Seminar on Computational Learning and Adaptation

  Dealing with Unreliable Labelers and Implications for Labeling Strategies

Chuck Lam
Department of Electrical Engineering
Stanford University

Machine learning and pattern recognition researchers have long studied the effect of labeling noise. Most of those work have assumed that the labeling process is fixed. However, in many situations, such as with the Open Mind Initiative (www.openmind.org), the system architect can specify a labeling strategy. For example, she can have the labelers spend more of their effort cross-checking each others' work, thus trading off the size of the dataset for quality. This talk will examine such labeling strategy trade-offs from a couple perspectives. The first is in labeling test data for classifier evaluation. The second perspective is in the learning rate of an ideal domain with labeling noise. In both cases we found the classifier's achievable error rate to play a significant role in designing the labeling strategy.

Return to the seminar schedule