2014/bench14

Notes from the First Meeting

Concerns

  • Repeatability of sampling / stimulus
  • Trying to solve High-level complex problems
  • Non-generalizability of benchmarks

Possible Benchmarking

  • Accuracy + Error
  • Biological Realism
  • Decathlon (necessity for tuning)
  • Power consumption / PErformance
  • Latency / Speed
  • Noise robustness
  • Area/node/Resources
  • Learning speed
  • Adaptability
  • Real-world Problems
  • Multi-sensory problems
  • Usability

Databases

  • Card->DVS->spikes
  • Bernabe MNIST->DVS->spikes
  • Spinning Digits
  • DVS freeway
  • ATIS cars
  • MNIST->rate code
  • TI Digits Audio Spiking
  • [for all above, can add noise]

Other communities

  • Robot tasks (multidimensional)
  • Simulated mazes
  • Eye tracking
  • Spike sorting
  • Cocktail party data

Final notes

  • A common repository is needed; INE should host it - talk to Giacomo, Ralph
  • Send out the word on google+, comp.neuro
  • Try and get high quality, shareable data and share!
  • Frontiers dataset articles and loop back publications
  • Licenses - Danny

Licenses

A useful summary of many dataset licenses:  http://www.dcc.ac.uk/resources/how-guides/license-research-data

Short summary

We should use the Open Data Commons Attribution license (ODC-BY 1.0). Here's the human readable summary:

It has the following characteristics (taken from the summary):

  • You are free:
    • To Share: To copy, distribute and use the database.
    • To Create: To produce works from the database.
    • To Adapt: To modify, transform and build upon the database.
  • As long as you
    • Attribute the source.

Notes from the Second Meeting

  • Data Formats
    • Always use an ASCII header or a readme file that describes the format
    • Share a Matlab file with the dataset that has example read / write
  • Theory of Benchmarking
    • Google Docs
    • Email J. Tapson if you want a specific section
    • J. Tapson is responsible for skeleton
    • Should include biological relevance
    • STPR benchmarks - email J. Tapson if you want to contribute a:
      • Dataset
      • Software Network
      • Hardware Network
    • An annual benchmark should occur - everyone brings their sensors and record the data on a common task.
    • K-H Meier is also working on a relevant position paper

Neuromorphic Challenges

  • We should come up with a list of Neuromorphic Challenges that we think only neuromorphic engineering can solve
  • These challenges can raise awareness and drive the field, as the Hilbert Problems did a century ago
  • There should be an annual meeting to update and evaluate the latest progress
  • This list may be taken as a definition of Neuromorphic Engineering, so we should be careful how we construct it
  • Example challenges:
    • Face recognition within energy and time constraints
    • Speaker recognition within noise, energy, and time constraints
    • Adaptive Motor Control
    • Operant classical conditioning
    • Limited memory / Limited time response

Attachments