2011/learn11/ProjectResults

Computational Cognitive Neuroscience Methods for Guided Reinforcement Learning Project Results

REINFORCEMENT LEARNING IN A VIRTUAL ENVIRONMENT

'David Noelle'

UN-NATURAL LANGUAGE PROCESSING

Results from UN-NATURAL LANGUAGE PROCESSING (UNLP) project are described on this web page.

John Harris

OBJECT AND SPEECH RECOGNITION IN A LEABRA MODEL ON THE NAO ROBOT

'Brian Mingus'

The goal of this project was to take the Leabra Vision model (LVis) of 3D invariant object recognition, train it in a virtual world and then test generalization performance in the real world. The full demonstration included using LVis for both visual and auditory object recognition. For the auditory component the model was trained on the speech spectrograms of twelve speakers on ten category labels that it was good at recognizing visually. For the visual component the model was trained on 1000 objects from 100 categories from the Google Sketchup database (known as the  CU3D). In order to demonstrate generalization from the virtual world to the real world the Nao robot was used. It was situated such that it had three fields of view relative to its head pointing forward; left, center and right. Participants whose voices the LVis model was not trained on would place one real object, each from a different category, in each of Nao's fields of view. They would then say the name of the object that Nao should find in one of it's fields of view. Nao's microphones would then relay this utterance to the LVis model which would perform speech recognition on that word. Nao then initiated a visual search for the object of interest. For each field of view he would first foveate on the center, and then move his head 3 degrees left, up, right and down relative to the center, performing object recognition at each foveation. For each field of view he would then vote across his recognition labels in order to determine the category of the object. Upon encountering the object that the participant had indicated via speech, the robot would state that it had found the object, otherwise he would indicate failure to find the object.