Designing a Comprehensive Procedure for Flagging Archived Traffic Data: A Case Study

Archived data management systems (ADMS) are extensively used for storing historical traffic data (e.g., volume, speed, occupancy) collected from traffic sensors. Archived traffic data have important uses for engineering and planning applications such as ramp meter timing, work zone planning, and performance management. They are also an important data source for transportation research. Various flagging procedures have been implemented in ADMS to identify invalid or questionable archived traffic data, however, those flagging procedures may not be comprehensive enough to maintain adequate data quality. This study presents the findings of a literature search and a user survey to discuss the possible gap between the state-of-the-practice and the state-of-the-art validity tests, identifies complex yet effective validity tests which are favored by users, and recommends the procedure that prioritizes the implementation of validity tests in ADMS. To aid the implementation, different methods to establish quantitative rules and practical thresholds for candidate validity tests have been proposed. This study underscores the importance of keeping the basic validity tests required to maintain minimum data quality and adding more advanced tests to detect less obvious yet important data issues. The recommended tests along with the flagging procedure are demonstrated through a case study based on one detector station in Wisconsin. Results of the case study show that the guide is useful in the development of a comprehensive flagging procedure for better data quality.