2D Object Recognition

Ching Teo, Cornelia Fermuller, Yiannis Aloimonos

Contour-Based Object Recognition using Image Torque

A key component for recognizing the action performed are the objects and tools involved. In this project, we have defined a set of 7 tools: "Saw", "Screwdriver", "Hammer", "Borer", "Triangle Ruler", "Marker", "Nail / Screws" and for objects, "fake" plastic wooden planks of a variety of sizes. We extend a bottom technique of grouping 2D contours known as the image torque (1) to determine "object-like" regions, and match it using a modified version of shape context matching in a hierarchical fashion (increasing scales) with the target object shape. The target objects are defined by a set of "codons", or contour fragments which are either learned from data or hand-drawn. At test time, an initial set of potential torque fixation points are selected from which we begin to match supporting fragments with the target. We use a modified version of Shape Context distance, augmented with the torque fixation points to form a rotationally invariant feature for computing similarity. Each contour is then re-weighted based on their compatibility with neigboring contours in supporting the presence of the target. By modulating the weights of the torque appropriately, we tune-up fixation points that are more likely to contain the object's location. Results for detected objects and their torque weights are shown below:

Predicting Object Location via Hand Trajectory Information

A second component of the work is to predict the location of objects that are to be manipulated by the hands. This is important for recognition as we want to recognize the object before it is occluded by the hand. We draw inspiration from the work of Jeanarrod that predicts that an intentional movement of grasping follows a velocity curve that has a peak linearly correlated with the distance of the hand to the target object. We used simple second order derivatives of the trajectories to effectively predict the location that the hand is moving and if the path intersects an object, we perform recognition. An example is shown below:

Integrating Contour-Based Recognition with Object Tracking

The final component of this work is the integration of object tracking with the contour-based recognition approach. We exploit the hand trajectory in this case to localize a smaller region that contains the object that is now partially occluded. We then run a contour extraction algorithm (Pb Edges) at an extremely low threshold to obtain as much edge information as possible for matching. An example of the process is shown here for detecting and tracking "Hammer":