Measuring progress in robotics: Benchmarking and the ‘measure-target confusion’

In Fabio Bonsignorio, John Hallam, Elena Messina & Angel P. Del Pobil (eds.), Metrics of sensory motor coordination and integration in robots and animals. Berlin: Springer. pp. 169-179 (2019)
Download Edit this record How to cite View on PhilPapers
Abstract
While it is often said that robotics should aspire to reproducible and measurable results that allow benchmarking, I argue that a focus on benchmarking can be a hindrance for progress in robotics. The reason is what I call the ‘measure-target confusion’, the confusion between a measure of progress and the target of progress. Progress on a benchmark (the measure) is not identical to scientific or technological progress (the target). In the past, several academic disciplines have been led into pursuing only reproducible and measurable ‘scientific’ results – robotics should be careful to follow that line because results that can be benchmarked must be specific and context-dependent, but robotics targets whole complex systems for a broad variety of contexts. While it is extremely valuable to improve benchmarks to reduce the distance be- tween measure and target, the general problem to measure progress towards more intelligent machines (the target) will not be solved by benchmarks alone; we need a balanced approach with sophisticated benchmarks, plus real-life testing, plus qualitative judgment.
PhilPapers/Archive ID
MLLMPI
Upload history
First archival date: 2018-06-09
Latest version: 2 (2019-04-30)
View other versions
Added to PP index
2018-06-09

Total views
226 ( #19,831 of 51,411 )

Recent downloads (6 months)
54 ( #9,943 of 51,411 )

How can I increase my downloads?

Downloads since first upload
This graph includes both downloads from PhilArchive and clicks on external links on PhilPapers.