JDS Journal of Data Science 1683-86021680-743X1680-743X School of Statistics, Renmin University of China JDS1136 10.6339/24-JDS1136 Statistical Data Science Identifying Anomalous Data Entries in Repeated Surveys✩ https://orcid.org/0000-0002-0446-1328 SartoreLucaluca.sartore@usda.govlsartore@niss.org12∗ https://orcid.org/0000-0003-3387-3484 ChenLu12 van WartJustin2 https://orcid.org/0009-0008-9482-5316 DauAndrew2 https://orcid.org/0000-0001-9828-968X BejleriValbona2 National Institute of Statistical Sciences, 1750 K Street NW Suite 1100, Washington DC, 20006, USA United States Department of Agriculture, National Agriculture Statistics Service, 1400 Independence Avenue SW, Washington DC, 20250, USA

The findings and conclusions in this article are those of the authors and should not be construed to represent any official USDA or US Government determination or policy. This research was supported in part by the intramural research program of the US Department of Agriculture, National Agriculture Statistics Service.

Corresponding author. Email: luca.sartore@usda.gov or lsartore@niss.org. 2024882024223436455 Supplementary Material 301120231642024 2024 The Author(s). Published by the School of Statistics and the Center for Applied Statistics, Renmin University of China.2024 Open access article under the CC BY license.

The presence of outliers in a dataset can substantially bias the results of statistical analyses. In general, micro edits are often performed manually on all records to correct for outliers. A set of constraints and decision rules is used to simplify the editing process. However, agricultural data collected through repeated surveys are characterized by complex relationships that make revision and vetting challenging. Therefore, maintaining high data-quality standards is not sustainable in short timeframes. The United States Department of Agriculture’s (USDA’s) National Agricultural Statistics Service (NASS) has partially automated its editing process to improve the accuracy of final estimates. NASS has investigated several methods to modernize its anomaly detection system because simple decision rules may not detect anomalies that break linear relationships. In this article, a computationally efficient method that identifies format-inconsistent, historical, tail, and relational anomalies at the data-entry level is introduced. Four separate scores (i.e., one for each anomaly type) are computed for all nonmissing values in a dataset. A distribution-free method motivated by the Bienaymé-Chebyshev’s inequality is used for scoring the data entries. Fuzzy logic is then considered for combining four individual scores into one final score to determine the outliers. The performance of the proposed approach is illustrated with an application to NASS survey data.

 agricultural data Bienaymé-Chebyshev’s inequality cellwise outliers fuzzy logic outlier detection statistical analysis This research was supported by the intramural research program of the US Department of Agriculture, National Agricultural Statistics Service (NASS). References Agostinelli C, Leung A, Yohai VJ, Zamar RH (2015). Robust estimation of multivariate location and scatter in the presence of cellwise and casewise contamination. Test, 24(3): 441–461. https://doi.org/10.1007/s11749-015-0450-6 Alqallaf F, Van Aelst S, Yohai VJ, Zamar RH (2009). Propagation of outliers in multivariate data. The Annals of Statistics, 37(1): 311–331. https://doi.org/10.1214/07-AOS588 Bienaymé IJ (1867). Considérations à l’appui de la découverte de Laplace sur la loi de probabilité dans la méthode des moindres carrés. Journal de Mathématiques Pures et Appliquées, 2(12): 158–176. Chepulis MA, Shevlyakov G (2020). On outlier detection with the Chebyshev type inequalities. Journal of the Belarusian State University. Mathematics and Informatics, 3: 28–35. https://doi.org/10.33581/2520-6508-2020-3-28-35 Dagum L, Menon R (1998). OpenMP: An industry standard API for shared-memory programming. IEEE Computational Science and Engineering, 5(1): 46–55. https://doi.org/10.1109/99.660313 Daniel JW, Gragg WB, Kaufman L, Stewart GW (1976). Reorthogonalization and stable algorithms for updating the Gram-Schmidt Q R factorization. Mathematics of Computation, 30(136): 772–795. https://doi.org/10.1090/S0025-5718-1976-0431641-8 De Waal T, Pannekoek J, Scholtus S (2011). Handbook of Statistical Data Editing and Imputation, volume 563. John Wiley & Sons. Filzmoser P, Gregorich M (2020). Multivariate outlier detection in applied data analysis: Global, local, compositional and cellwise outliers. Mathematical Geosciences, 52(8): 1049–1066. https://doi.org/10.1007/s11004-020-09861-6 Flynn M (1966). Very high-speed computing systems. Proceedings of the IEEE, 54(12): 1901–1909. https://doi.org/10.1109/PROC.1966.5273 Gupta MM, Qi J (1991). Theory of t-norms and fuzzy inference methods. Fuzzy Sets and Systems, 40(3): 431–450. https://doi.org/10.1016/0165-0114(91)90171-L Hampel FR (1974). The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346): 383–393. https://doi.org/10.1080/01621459.1974.10482962 Heydarian M, Doyle TE, Samavi R (2022). MLCM: Multi-label confusion matrix. IEEE Access, 10: 19083–19095. https://doi.org/10.1109/ACCESS.2022.3151048 Hidiroglou MA, Berthelot JM (1986). Statistical editing and imputation for periodic business surveys. Survey Methodology, 12(1): 73–83. Huber PJ, Ronchetti EM (1981). Robust Statistics. John Wiley & Sons, New York. Miller D, Robbins M, Habiger J (2010). Examining the challenges of missing data analysis in phase three of the agricultural resource management survey. JSM Proceedings. American Statistical Association Section on Survey Research Methods. O’Gorman TJ (1994). The effect of cosmic rays on the soft error rate of a DRAM at ground level. IEEE Transactions on Electron Devices, 41(4): 553–557. https://doi.org/10.1109/16.278509 Raymaekers J, Rousseeuw PJ (2019). Handling cellwise outliers by sparse regression and robust covariance. arXiv preprint: https://arxiv.org/abs/1912.12446. Raymaekers J, Rousseeuw PJ, Van den Bossche W, Hubert M (2023). cellWise: Analyzing data with cellwise outliers. CRAN, R package version 2.5.2. Rousseeuw PJ, Van den Bossche W (2018). Detecting deviating data cells. Technometrics, 60(2): 135–145. https://doi.org/10.1080/00401706.2017.1340909 Rubin DB (1976). Inference and missing data. Biometrika, 63(3): 581–592. https://doi.org/10.1093/biomet/63.3.581 Sandqvist AP (2016). Identifizierung von Ausreissern in eindimensionalen gewichteten Umfragedaten. KOF Analysen, 2016(2): 45–56. Sedgewick R (1978). Implementing quicksort programs. Communications of the ACM, 21(10): 847–857. https://doi.org/10.1145/359619.359631 Stigler SM (1973). The asymptotic distribution of the trimmed mean. The Annals of Statistics, 1(3): 472–477. Tchebichef P (1867). Des valeurs moyennes. Journal de Mathématiques Pures et Appliquées, 2(12): 177–184. Zwillinger D (2018). Standard Mathematical Tables and Formulas. CRC Press.