Object Recognition

Main contributors Marcello Mulas, Michael Pfeiffer, Daniel Neil

Participants Kayode Sanni, Cornelia Fermuller, Cheston Tan, Ching Teo, John Chiasson, Ryad Benjamin Benosman, Luis Camunas

Demo 1

This demo recognizes 6 objects recorded with an eDVS camera.

Classification

The classification algorithm is based on a 200-200 hidden units DBN trained on about 2500 images per object.
The DBN weights are used to initialize a NN with a standard error back-propagation algorithm.
The deep learning algorithm are based on the  DeepLearnToolbox for Matlab.

Training set generation

Each input image is binary and 28x28 pixels in size. Events are accumulated for 20 ms (regardless of the polarity).
The resulting frame image is low-pass filtered with a Gaussian convolution and under-sampled to 28x28 size.
As final step a binary image is generated based on a threshold.

Source code

The  source code of this demo is written in Matlab.

  • Utility functions
    • initialize_retina.m – opens serial communication and enables stream of retina events
    • finalize_retina.m – closes serial communication and disables stream of retina events
    • SerPortCloseAll?.m – closes all serial communication streams
    • get_retina_events.m – returns all available retina events
    • generate_binary_frame.m – generates a binary image from a list of retina events
  • Utility scripts:
    • record_events.m – stores in a binary file the events recorded with an eDVS camera
    • replay_events.m – replays the recording of the events
    • generate_images.m – generates binary images based on a previously recorded stream of events
    • train_network.m – trains a DBN and a ANN with back-propagation on a binary image dataset
  • Demo main script:
    • recognize_objects.m – recognizes an object recorded with an eDVS camera
  • Data:
    • net.mat - trained network with back-propagation

Current limitations and future works

The object recognition accuracy is currently sensitive to a number of factors:

  • object orientation
  • light conditions
  • noise level
  • presence of other objects

To improve the demo performance the training dataset should include images of objects with different orientations and light conditions.
A noise removal filter may be applied to the recording.
An object tracking algorithm would help to reduce the interference of other objects in the background.
It might be necessary adding more layers to the DBN in order to allow for the recognition of more objects.

Screenshots

Demo 2

This demo recognizes 5 objects recorded by a  Pushbot.
Training set generation and classification are performed similarly to demo 1.

Source code

In addition to the previous functions the  source code of demo 2 includes:

  • Utility scripts:
    • record_events_with_control.m - records eDVS events while manually driving the Pushbot
  • Demo main script:
    • drive_and_recognize_objects.m – recognizes objects while manually driving the Pushbot
    • classify_objects.m – recognizes object from a recorded Pushbot session
  • Data:
    • net.mat - DBN trained network with back-propagation

Current limitations and future works

The object recognition accuracy is currently sensitive to a number of factors:

  • object size
  • visual perspective
  • background noise

Objects were recorded with different scales and perspective but DBN performance remains low in manually driving conditions.

Screenshots

Attachments