CaltechAUTHORS
  A Caltech Library Service

Validation of Average Error Rate Over Classifiers

Bax, Eric (1997) Validation of Average Error Rate Over Classifiers. California Institute of Technology , Pasadena, CA. (Unpublished) http://resolver.caltech.edu/CaltechCSTR:1997.cs-tr-97-17

[img]
Preview
Postscript - Submitted Version
See Usage Policy.

293Kb
[img] PDF - Submitted Version
See Usage Policy.

148Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechCSTR:1997.cs-tr-97-17

Abstract

We examine methods to estimate the average and variance of test error rates over a set of classifiers. We begin with the process of drawing a classifier at random for each example. Given validation data, the average test error rate can be estimated as if validating a single classifier. Given the test example inputs, the variance can be computed exactly. Next, we consider the process of drawing a classifier at random and using it on all examples. Once again, the expected test error rate can be validated as if validating a single classifier. However, the variance must be estimated by validating all classifers, which yields loose or uncertain bounds.


Item Type:Report or Paper (Technical Report)
Additional Information:© 1997 California Institute of Technology. Thanks to Zehra Cataltepe and Joseph Sill for their instructive conversations and helpful pointers. Thanks to Dr. Yaser Abu-Mostafa for teaching - the results in this paper were inspired by his class on learning theory. Thanks to Dr. Joel Franklin for advice and guidance. Also, thanks to an anonymous referee for invaluable advice on the presentation of these results.
Group:Computer Science Technical Reports
Subject Keywords:machine learning, Vapnik-Chervonenkis, validation
DOI:10.7907/Z90P0X25
Record Number:CaltechCSTR:1997.cs-tr-97-17
Persistent URL:http://resolver.caltech.edu/CaltechCSTR:1997.cs-tr-97-17
Usage Policy:You are granted permission for individual, educational, research and non-commercial reproduction, distribution, display and performance of this work in any format.
ID Code:26825
Collection:CaltechCSTR
Deposited By: Imported from CaltechCSTR
Deposited On:30 Apr 2001
Last Modified:28 Mar 2017 15:49

Repository Staff Only: item control page