My Undergraduate Senior Project Progress Part 1 – Circle Detection

This will be the first in what I hope will be a series of posts detailing the progress on my senior project at California State University Long Beach. In this post I will first give a quick overview of what we are supposed to accomplish in my senior project, my groups’ current progress, and my progress with OpenCV and circle detection using the hough circle transform.

In the theme of meeting the needs of the aerospace industry, the goal of my senior project is to program a robot arm to recognize bolt holes using computer vision, and then pick and place bolts into those bolt holes, a task which large aerospace corporations have slowly starting automating. Besides the code for a robotic arm that can pick-and-place bolts, my group, which consists of myself and two other students, is also responsible for coming up with a mechanical manipulator for the robot arm which can grasp bolts.

We are starting this project from where a previous group left off last year, but they didn’t make much progress, and as a result, my group is starting from a clean slate. Regarding the robot arm, we will likely be utilizing a Dorna 1 or 2, a relatively cheap educational level robotic arm. One of my group members, David Barno, will be handling the mechanical design side of things and designing the manipulator to pickup, place, and thread bolts into the bolt holes.

To detect the bolt holes visually with a camera, our current line of thinking is to use an Xbox Kinect. I decided to opt for the Kinect because of its native depth-sensing capabilities, but I recently learned it is possible to use a single camera to sense depth, so we may explore that option in the future. Below is a picture I sketched indicating our current plan, as presented in class in our Conceptual Design Review (CDR):

My initial thoughts on the software we would use to solve the pick-and-place problem for this project are as follows. First, we would use OpenCV to recognize the bolt-holes, and calculate their location in 3D space. We would then pass those coordinates from OpenCV to MoveIt, Moveit being Robot Operating System’s (ROS) motion planning framework for robot arms, which would handle the path planning and inverse kinematics. Beyond the big picture of using OpenCV and Moveit, the specifics of solving the smaller individual problems in this project were unknown to me. What computer vision method would I need to recognize a circle? How does MoveIt integrate with OpenCV? All these questions and many unknowns remained, all of which to solve would require a lot of research, work, and trial and error. It was time to get started!

I would start on the software side of this project by trying to figure out OpenCV and learn how bolt-hole detection was already implemented in the real world. I didn’t have any previous computer vision experience, so this area of the project was one where I was the most unfamiliar with the possibilities, but also the most excited about because there was a lot to learn and the idea of programming a camera to recognize bolt holes was really intriguing!

One of the first things I did was google something similar to “OpenCV detecting bolt holes”, which quickly led me to the conclusion that detecting bolt holes could be boiled down to the problem of detecting a circle, which is what a bolt hole looks like when viewed from head-on. From my research, I found detecting circles, and by extension, bolt holes, is already a solved problem in the computer vision world that doesn’t require machine learning. From here, I could have just gone the route of studying some code examples and learning only what was strictly necessary, but I decided I should start learning the basics of computer vision from scratch, and learn some of the fundamentals before diving into circle detection right away without any idea of what was going on in the background.

I started learning OpenCV in Python and the basics of computer vision with this tutorial series. While really long, this tutorial series served its purpose by slowly getting me acquainted with all the basics of computer vision I would need to know. It took me a couple months in my free time outside of school to slowly work through these videos and follow along, but as of right now I am about 30 videos through the series, and I think a lot of those videos weren’t very helpful to the problem I’m trying to solve, but at least I have some additional background knowledge about computer vision now to boot. Regardless, this series gave me the knowledge I needed to start attacking the problem of recognizing a circle using computer vision and a webcam.

From my initial google searches and research, I had come to the conclusion that the best way for our group to detect a circle (bolt hole) would be to use the hough circle transform. This is an algorithm already built into OpenCV that allows one to robustly detect circles in a simple fashion without having to wade into the machine learning side of things.

Unfortunately, hough transforms (there is both a circle and line hough transform in OpenCV) is where the tutorial series I mentioned earlier falls off in quality as far as explanations go, so I had to do some more digging around to understand hough transforms better.

After focusing specifically on OpenCV’s circle hough transform function, last night using OpenCV and Python I was able to recognize circles in a sample image!

What amazes me is how few lines of code it takes to achieve this result. What you see in the picture above is the only code needed to robustly recognize circles. OpenCV handles all the heavy lifting in the background! As an aside, this was a theme with OpenCV I noticed, and I think its great. Often times, to get a mind-blowing result, all one needed to do was directly copy the OpenCV documentation. How cool is that! It almost feels like cheating. The code above is also very similar to the documentation example on OpenCV’s website for the hough circle transform. Unfortunately for me, I guess it wasn’t strictly necessary to watch all the tutorial videos that I did, but at least now I have a basic understanding of the background knowledge for OpenCV, computer vision, and the hough line and circle transform.

Next, I took the hough circle transform code above, and tried to get circle detection working in a live webcam video feed, which, with the help of my fake potted plants, lead to the following results later that same night:

From a live webcam view, I got a lot of spurious results similar to the above, where the code detected large amounts of false positives, circles that OpenCV thought existed which really did not. I did very little tuning on the hough circle algorithm, so this was expected.

However, when shown an actually circular shape, the algorithm, with very little tuning, was fairly reliably able to pick up the outline of a circle, as shown above! Detecting circles of any kind was really exciting to see, and once again, as my picture illustrates the code is really simple. In the future, to achieve better results, filtering will have to be added to reduce false positives. Because we are only interested in recognizing circles that are known to be valid bolt hole sizes, any circle that is larger or smaller than the expected size can be discarded.

And that’s it for this senior project update! At this point, I feel like I have done enough research and experimenting with OpenCV that the fog has mostly lifted on the computer vision side of this project, and I see a clear enough path forwards that I feel confident either passing the OpenCV part of this project off to another group member who really wants to dial in the circle detection, or setting it aside for now while I clear up questions I still have regarding how we will implement ROS and MoveIt into this project. I learned the basics of ROS about a year ago, but the next step for me will be to learn and experiment with MoveIt, which should hopefully really help tie the computer vision side of this project with the physical robot side of the project.