A Caltech Library Service

Evaluation Metrics for Object Detection for Autonomous Systems

Badithela, Apurva and Wongpiromsarn, Tichakorn and Murray, Richard M. (2022) Evaluation Metrics for Object Detection for Autonomous Systems. . (Unpublished)

[img] PDF - Submitted Version
Creative Commons Attribution Non-commercial Share Alike.


Use this Persistent URL to link to this item:


This paper studies the evaluation of learning-based object detection models in conjunction with model-checking of formal specifications defined on an abstract model of an autonomous system and its environment. In particular, we define two metrics -- \emph{proposition-labeled} and \emph{class-labeled} confusion matrices -- for evaluating object detection, and we incorporate these metrics to compute the satisfaction probability of system-level safety requirements. While confusion matrices have been effective for comparative evaluation of classification and object detection models, our framework fills two key gaps. First, we relate the performance of object detection to formal requirements defined over downstream high-level planning tasks. In particular, we provide empirical results that show that the choice of a good object detection algorithm, with respect to formal requirements on the overall system, significantly depends on the downstream planning and control design. Secondly, unlike the traditional confusion matrix, our metrics account for variations in performance with respect to the distance between the ego and the object being detected. We demonstrate this framework on a car-pedestrian example by computing the satisfaction probabilities for safety requirements formalized in Linear Temporal Logic (LTL).

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Wongpiromsarn, Tichakorn0000-0002-3977-122X
Murray, Richard M.0000-0002-5785-7481
Additional Information:Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). We acknowledge funding from AFOSR Test and Evaluation Program, grant FA9550-19-1-0302, and from NSF grant CNS-2141153.
Funding AgencyGrant Number
Air Force Office of Scientific Research (AFOSR)FA9550-19-1-0302
Record Number:CaltechAUTHORS:20221219-234021838
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:118456
Deposited By: George Porter
Deposited On:20 Dec 2022 03:41
Last Modified:02 Jun 2023 01:29

Repository Staff Only: item control page