Identifying Anomalous Data Entries in Repeated Surveys✩
Volume 22, Issue 3 (2024): Special issue: The Government Advances in Statistical Programming (GASP) 2023 conference, pp. 436–455
Pub. online: 8 August 2024
Type: Statistical Data Science
Open Access
Open Access
✩
The findings and conclusions in this article are those of the authors and should not be construed to represent any official USDA or US Government determination or policy. This research was supported in part by the intramural research program of the US Department of Agriculture, National Agriculture Statistics Service.
Received
30 November 2023
30 November 2023
Accepted
16 April 2024
16 April 2024
Published
8 August 2024
8 August 2024
Abstract
The presence of outliers in a dataset can substantially bias the results of statistical analyses. In general, micro edits are often performed manually on all records to correct for outliers. A set of constraints and decision rules is used to simplify the editing process. However, agricultural data collected through repeated surveys are characterized by complex relationships that make revision and vetting challenging. Therefore, maintaining high data-quality standards is not sustainable in short timeframes. The United States Department of Agriculture’s (USDA’s) National Agricultural Statistics Service (NASS) has partially automated its editing process to improve the accuracy of final estimates. NASS has investigated several methods to modernize its anomaly detection system because simple decision rules may not detect anomalies that break linear relationships. In this article, a computationally efficient method that identifies format-inconsistent, historical, tail, and relational anomalies at the data-entry level is introduced. Four separate scores (i.e., one for each anomaly type) are computed for all nonmissing values in a dataset. A distribution-free method motivated by the Bienaymé-Chebyshev’s inequality is used for scoring the data entries. Fuzzy logic is then considered for combining four individual scores into one final score to determine the outliers. The performance of the proposed approach is illustrated with an application to NASS survey data.
References
Agostinelli C, Leung A, Yohai VJ, Zamar RH (2015). Robust estimation of multivariate location and scatter in the presence of cellwise and casewise contamination. Test, 24(3): 441–461. https://doi.org/10.1007/s11749-015-0450-6
Alqallaf F, Van Aelst S, Yohai VJ, Zamar RH (2009). Propagation of outliers in multivariate data. The Annals of Statistics, 37(1): 311–331. https://doi.org/10.1214/07-AOS588
Chepulis MA, Shevlyakov G (2020). On outlier detection with the Chebyshev type inequalities. Journal of the Belarusian State University. Mathematics and Informatics, 3: 28–35. https://doi.org/10.33581/2520-6508-2020-3-28-35
Dagum L, Menon R (1998). OpenMP: An industry standard API for shared-memory programming. IEEE Computational Science and Engineering, 5(1): 46–55. https://doi.org/10.1109/99.660313
Daniel JW, Gragg WB, Kaufman L, Stewart GW (1976). Reorthogonalization and stable algorithms for updating the Gram-Schmidt $QR$ factorization. Mathematics of Computation, 30(136): 772–795. https://doi.org/10.1090/S0025-5718-1976-0431641-8
Filzmoser P, Gregorich M (2020). Multivariate outlier detection in applied data analysis: Global, local, compositional and cellwise outliers. Mathematical Geosciences, 52(8): 1049–1066. https://doi.org/10.1007/s11004-020-09861-6
Flynn M (1966). Very high-speed computing systems. Proceedings of the IEEE, 54(12): 1901–1909. https://doi.org/10.1109/PROC.1966.5273
Gupta MM, Qi J (1991). Theory of t-norms and fuzzy inference methods. Fuzzy Sets and Systems, 40(3): 431–450. https://doi.org/10.1016/0165-0114(91)90171-L
Hampel FR (1974). The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346): 383–393. https://doi.org/10.1080/01621459.1974.10482962
Heydarian M, Doyle TE, Samavi R (2022). MLCM: Multi-label confusion matrix. IEEE Access, 10: 19083–19095. https://doi.org/10.1109/ACCESS.2022.3151048
O’Gorman TJ (1994). The effect of cosmic rays on the soft error rate of a DRAM at ground level. IEEE Transactions on Electron Devices, 41(4): 553–557. https://doi.org/10.1109/16.278509
Raymaekers J, Rousseeuw PJ (2019). Handling cellwise outliers by sparse regression and robust covariance. arXiv preprint: https://arxiv.org/abs/1912.12446.
Rousseeuw PJ, Van den Bossche W (2018). Detecting deviating data cells. Technometrics, 60(2): 135–145. https://doi.org/10.1080/00401706.2017.1340909
Rubin DB (1976). Inference and missing data. Biometrika, 63(3): 581–592. https://doi.org/10.1093/biomet/63.3.581
Sedgewick R (1978). Implementing quicksort programs. Communications of the ACM, 21(10): 847–857. https://doi.org/10.1145/359619.359631
