Our project is an image tracking sentry paintball gun. In the beginning, we decided to divide the project into two main parts. The first part was the software/servo control, and the second part was the actual physical mount.


The software has a couple main components. The first thing it must do is bring in a live video feed from a camera, and then use our software to detect motion. Video is basically just a bunch of pictures stiched together, so we had to be able to process 30 pictures per second. After it finds where motion is occuring in a certain frame, it must send this data to a chip that controls servos. Once the chips receive data, they have to move the gun accordingly.

So our first task was getting the image processing working. To do this, we decided to use OpenCV, which is an open source computer vision library developed by Intel. It allowed us to develop in C++, a language we were all very familiar with. For a guide on how to install OpenCV, click here. While developing, we found a book called Learning OpenCV: Computer Vision with the OpenCV Library. It has a lot of good examples in it, and explains a lot of different algorithms in detail.

The method we used for motion tracking is actually very simple, but effective. In the code, it always keeps a copy of the previous image. It compares the current image with the previous image, and determines differences in each individual pixel. It creates a new image with these differences in it. This method is often referred to as edge detection, because it tends to pick up the edges of objects in motion. As you can see below, the first picture shows the current frame, the second picture shows the previous frame, and the last picture shows the difference between them.

One problem with this method is that a small change in lighting can cause motion to be detected where there isn't any. We tried to compensate for this by setting a threshold for motion. If the change in any individual pixel wasn't substantial enough, we marked that pixel as having no change. So this leaves us with an image where we have the edges of moving objects. From there, we loop through each pixel and wherever we see a change, we note that pixels position. Once we are done, we take an average of all of the positions of all of the pixels where we saw a change. Once we do this for both the x and y axes, we can pinpoint one individual pixel where we want our paintball gun to shoot. This pixel is highlighted in the above picture as the big green square. Here is our C++ code we used to do the image tracking.

In our video, our resolution is 320x240(height x width). So this above algorithm gives us an number for x between 0 and 319 and a number for y between 0 and 239. Now that we have this number, we must send it to the chips controlling the servos. We chose to use Arduino Unos to control our servos because we all had used them before. To send the data between the laptop and the Arduinos, we used openFrameWorks, which is a library that allows for many things, one of them being serial communication. Sending the data with that library was relatively straightforward, and required very little coding on our part. An install guide for openFrameworks can be found here.

Once the Arduino received the data, it had to decide how to move the servos. When we got to this part of the project, we ran into a little roadblock. We hadn't considered that a value of (x,y) means absolutely nothing to the Arduino. We had to write code for it to understand where that point was. The solution we came up with requires us to calibrate the gun/servos to the webcam each time it is started up. It was also complicated by the fact that gears were used with the servos, so we had to code everything backwards. Here's a brief overview of the calibration method, ignoring the fact that we had gears on our servos and that the gears decrease the maximum rotation of the servos by a factor of two.

We would attatch a laser pointer to the barrel of the paintball gun and take a couple shots by hand to make sure that the gun shot where the laser was pointing. Once the laser was calibrated to the gun, we would look at the video feed and move the gun until the laser was in the very top right corner of the screen. We did this using a couple of pushbuttons on a breadboard that were wired to the Arduinos. Then, another button let the Arduinos know we were at the top right corner. The Arduinos would then save the angle that the servos were at. We will just deal with the X Servo in this explanation. Let's say that at that top right corner, the angle on the X servo was 150 degrees(Maximum X Servo Angle). We then moved the laser to the bottom left, and did the same thing. Let's say for our explanation that the X servo was at 50 degrees(Minimum X Servo Angle) here. We then knew that the angle we were looking at was 100(150-50) degrees. We use this information to determine what angle to move the servo to with the formula:

X Servo Angle = (Minimum X Servo Angle) + [(X Coordinate/X Resolution)*(Maximum X Servo Angle-Minimum X Servo Angle)]

where the X Coordinate is a pixel value between 0 and 319 and X Resolution is the number of pixels wide our video is(320).

So, if the pixel sent was 160, the our equation would yield

(50) + {(160/320)*(150-50)]} = 100

Our servo would move to the 100 degrees position.

After doing all of this we discovered a new problem. When we tried to control two servos from one Arduino, the motion wasn't fluent and would have caused problems because of the weight of the gun. To get around this problem, we used two Arduinos, one for the x servo, and one for the y.

Our last issue with the software was that we needed it to be able to lead the target based on their speed. We did this by keeping track of the last 20 X positions, measuring the distance between each one, and then taking an average of those changes in X. This gave us a number that represented the average change in X per frame, and by multiplying that number by an experimentally determined factor, we could decide how much to lead a target by. A video of this part of the software in action is shown below.


As you can see, it's not perfect, and it doesn't work perfectly when it gets towards the edge. When the target is further away though, it usually does a good enough job.


Gun Mount:

The stand that holds the paintball gun is relatively simple even though it is able to support the relatively large weight of the paintball gun and the stresses of constantly changing direction as the system tracks the target. The base which supports the yoke that holds the paintball gun and the CO2 canister also provides a mounting point for the servo motor which moves the gun in the horizontal plain. The base is made of a sheet of plywood which sits on four feet made from small sections of 2x4. The servo is mounted in a hole in the plywood to allow the yoke to be closer to the base and therefore wobble less. Both of the servos we used to move the gun are Hitec brand HS-5485HB 180⁰ rotation servos with 89 oz/inches of torque and rotational speed of 0.17sec/60° at six volts. Both servos are geared up at a 2:1 ratio to increase the torque available from 89 oz/inches to 178 oz/inches.

However this means that the speed is decreased by half as is the angle the gun can turn. However we calculated that the speed would not be a problem as no one could run in front of the gun faster than the servos could turn even at half speed. We also found that because the camera had a maximum viewing angle of only 60⁰, 90⁰ was plenty of travel. The yoke is attached to the base with a bolt at its center. To allow the bolt to move smoothly we inserted a bronze bearing in the wood. Bronze bearings are impregnated with oil so there is no need to lubricate this part. We made the yoke out of aluminum angle and flat stock that was fastened by rivets. We decided on aluminum rather than wood to decrease the weight of the yoke and thereby the load on the servo.

To attach the gun to the yoke we used a piece of 2x4 which we bored a hole in the center of to accept the barrel of the gun. We cut the block in half and added additional holes to allow us to clamp the barrel securely.  We also bored a hole through the back perpendicular to the first one which would accept a threaded rod. During testing the threaded rod sliped in the holes due to the weight of the gun. To fix this we glued the rod to the clamp with epoxy. The rod extends through the holes on either side of the yoke and on one side we attached a gear to mesh with the gear on the servo.

To allow the gun to rotate smoothly in the yoke we put to vinyl bushings in the holes bored in the yoke.  Because of the weight of the gun and the recoil from when it fired we decided to add a second piece of aluminum at a right angle to one that acts as the base of the yoke in order to attach stabilizing feet. We also added these feet at the ends of the base of the yoke. To allow these feet to move smoothly we attached an aluminum sheet to the base underneath the yoke and put white lithium grease on it. It was important to use the right grease or the feet would disintegrate over time.

After some testing, we found that the weight of the gun caused the yoke to torque significantly thereby disengaging the gears which moved the gun up and down. To correct this problem we added a brace between the two arms of the yoke. We also added rubberbands to pull at the end of the rod to help the gears mesh even better.

We also built a mechanism to pull the guns trigger. Initially we tried to use solenoids however these were unreliable as they had to be perfectly positioned to provide maximum pull so we switched to a servo. To attach the servo to the gun we built a plate out of aluminum which we attached to existing screw holes in the gun. To pull the trigger we put a small cable around the trigger and used an attachment for the servo to attach the cable to the servo.  When the servo turned the cable would be alternating pulled tight and loosened thereby firing the gun. This method is effective but slower than we would prefer. 

In essence while our design was not perfect and could have benefited from a larger budget and more time it performed  well. As can be seen in our demo videos the gun mount allowed us to both track, fire at and hit a moving target. The video also demonstrates that the gun mount could handle both the weigh and the recoil of the gun showing that our design functioned as we expected.