Towards High-Quality Big Data: Lessons from FIT

Summary form only given, as follows. The complete presentation was not made available for publication as part of the conference proceedings. Data are being generated, collected, and analyzed today at an unprecedented scale, and data-driven decision making is sweeping through all aspects of society. As the use of big data has grown, so too have concerns that poor-quality data, prevalent in large data sets, can have serious adverse consequences on data-driven decision making. Responsible data science thus requires a recognition of the importance of veracity, the fourth "V" of big data. In this talk, we first present a vision of high-quality big data and highlight the substantial challenges that the first three V’s, volume, velocity, and variety, bring to dealing with veracity in big data. We then present the FIT Family of adaptive, data-driven statistical tools that we have designed, developed, and deployed at AT&T for continuous data quality monitoring of a large and diverse collection of continuously evolving data. These tools monitor data movement to discover missing, partial, duplicated, and delayed data; identify changes in the content of spatiotemporal streams; and pinpoint anomaly hotspots based on persistence, pervasiveness, and priority. We conclude with lessons from FIT relevant to big data quality that are cause for optimism.