SRR 2016

Video List. https://www.youtube.com/watch?v=x6Dsqha9JPM&list=PL2dAOzo3a1SqxkO_c7mVzUcQh5leLHjsX&index=1

Problem :

When the team is able to bring the robot into a park for testing they only have enough time to test things that makes sure the mechanical aspects of the robot works i.e. the gripper mechanism to collect samples, power consumption, robot acceleration / deceleration, doesn't burst into flames, etc...

I can't bug them to put colored rocks in front of the robot in different lighting conditions, near debris, near shadows, etc... It would take too much time.

Solution :

1.) Turn my spare rpi2 into a webcam and take the pictures of colored rocks myself.

2.) Configure it to match the rpi2 + v2 camera used by the robot.

3.) Create a python script that uses the PiCamera and OpenCv library that will x-forward the rpi2's camera stream to a window on my laptop. This python script also opens a gui that can begin/stop recording images and fine tune camera settings such as contrast, saturation, etc... The recorded images are named after a string that contains the current camera settings + dd/mm/yy + time before being saved to disk.

4.) Physically secure the rpi2 on top of the laptop with rubber bands. Height of camera on robot is approx 3'3 ft and a tilt of 25 degrees, therefore place laptop at waist level when recording.

5.) Go to park and begin collecting test images of colored rocks matching that of the robot's camera.

6.) After test images have been collected, run scp script to download rpi2 images on mm/dd/yy into Jetson Tk1 for image processing.

Problem :

Colored rocks are more difficult to find than initially expected.

While a car and a human face have many distinguishable features independent of their color that makes them ideal for machine learning frameworks. Unfortunately colored rocks have too few features for machine learning to be worth the effort.

The sheer number of images required to account for different lighting conditions, irregular patches of grass, dirt, leaves, and even squirrels, would be impractical.

Solution :

1.) Create algorithm that is specific to the task at hand instead of using existing machine vision frameworks. Implement robot vision in C++ with the OpenCv Library + CUDA for use on the Jetson Tk1.

The final product can be described as a vision system that looks for colored rocks over n images instead of per image.

Implementation :

NOTE : This is a very simplified tldr of the overall implementation.

1.  Image thresholding phase converts the current frame into Luv colorspace and sorts found blobs according to their channel. Next it checks each blob against the feature descriptor that checks for things such as correct size, color, and position for each blob. Found points are the center of mass for a given blob, if any are found they are sent to the second phase i.e. point tracker.

2. Create a point tracker that takes in x,y positions that represent the center of mass for each found blob, for each channel in the Luv colorspace.

Keep in mind that the image threshold check has no sense of context for the blobs it finds. The point tracker is meant to counter this problem.

3. Found points are compared to found points to previous points. After reaching a certain number of hits n times, it is considered a colored rock. Then it returns a non-zero point(x,y) to the main ROS node that controls the robot's behavior. Once the colored rock has been picked up the vision system is re-initialized to be used again at the discretion of the main ROS node.