Evaluating our algorithms is a challenge, because ground truth is difficult or impossible to find. We decided to carry out a number of complementary evaluations. Since the system consists of two independent procedures, centerline refinement and lane clustering, we can determine the weaknesses and strengths of our overall system by examining the two components separately. The overall performance of the system is important for evaluating its commercial viability and the interaction of its components.
If our approach to learning lane models is viable as a cost-saving alternative to manually encoding lane structure, it must achieve ``acceptable'' performance without needing ``excessive'' training data. Although these terms depend on particular business assumptions regarding cost and profit, we can use learning curves to estimate how performance improves as more training data becomes available. Since our training data are fairly accurate and our algorithms are based on plausible geometric assumptions, we expect the learning curves to show that the system approaches its best performance on a segment after only processing a few traces that pass over the segment.
To test our algorithms and empirically investigate their behavior, we collected 44 position traces along a 15 kilometer section of Interstate Highway 280 between Redwood City and Palo Alto, California. The positions were calculated twice a second from a differential GPS system using a Novatel DGPS receiver and a CSI differential corrections unit obtaining corrections from the U. S. Coast Guard beacon network. The data were then matched to the commercially-available digital map to determine what segments each trace traversed. Since the traces did not follow the same path, different segments received different numbers of passes. Each of the 42 total segments of Interstate 280 in the target region received between 9 and 35 passes. We did not consider the difference in coverage to have a significant impact, so the results are averaged over all segments. All segments had four lanes, but all four lanes were not covered by any trace for a few segments. The traces generally stayed in one lane for the entire duration, and each point was tagged with the current lane, an integer from one to four.