JDS Journal of Data Science 1683-86021680-743X1680-743X School of Statistics, Renmin University of China JDS1123 10.6339/24-JDS1123 Statistical Data Science Producing Fast and Convenient Machine Learning Benchmarks in R with the stressor Package HaycockSamhaycock.sam@outlook.com1∗ BeanBrennan1 BurchfieldEmily2 Department of Mathematics and Statistics, Utah State University, Logan, UT, USA Department of Environmental Sciences, Emory University, Atlanta, GA, USA Corresponding author. Email: haycock.sam@outlook.com. 2024462024222239258 Supplementary Material

The supplementary materials associated with this paper contain all the data and code necessary to reproduce the figures and tables shown in this paper. Dataset descriptions have been provided in the text, but additional information about the files can be found in the README file contained in the supplementary materials folder.

28720232122024 2024 The Author(s). Published by the School of Statistics and the Center for Applied Statistics, Renmin University of China.2024 Open access article under the CC BY license.

The programming overhead required to implement machine learning workflows creates a barrier for many discipline-specific researchers with limited programming experience. The stressor package provides an R interface to Python’s PyCaret package, which automatically tunes and trains 14-18 machine learning (ML) models for use in accuracy comparisons. In addition to providing an R interface to PyCaret, stressor also contains functions that facilitate synthetic data generation and variants of cross-validation that allow for easy benchmarking of the ability of machine-learning models to extrapolate or compete with simpler models on simpler data forms. We show the utility of stressor on two agricultural datasets, one using classification models to predict crop suitability and another using regression models to predict crop yields. Full ML benchmarking workflows can be completed in only a few lines of code with relatively small computational cost. The results, and more importantly the workflow, provide a template for how applied researchers can quickly generate accuracy comparisons of many machine learning models with very little programming.

 agriculture benchmarking cross-validation machine learning Python R References Aguilar J, Gramig GG, Hendrickson JR, Archer DW, Forcella F, Liebig MA (2015). Crop species diversity changes in the United States: 1978–2012. PLoS ONE. 10(8): 1–4. https://doi.org/10.1371/journal.pone.0136580. Ali M (2020). PyCaret: An open source, low-code machine learning library in Python. PyCaret version 1.0.0. https://www.pycaret.org (Accessed May 17, 2023). Bowles M (2015). Machine Learning in Python: Essential Techniques for Predictive Analysis. John Wiley & Sons, Hoboken, NJ, USA. Brenning A (2012). Spatial cross-validation and bootstrap for the assessment of prediction rules in remote sensing: The R package sperrorest. In: 2012 IEEE International Geoscience and Remote Sensing Symposium, 5372–5375. IEEE. https://doi.org/10.1109/IGARSS.2012.6352393 (Accessed Dec 29, 2023). Brenning A, Long S, Fieguth P (2012). Detecting rock glacier flow structures using Gabor filters and ikonos imagery. Remote Sensing of Environment, 125: 227–237. https://doi.org/10.1016/j.rse.2012.07.005. Burchfield EK (2022). Shifting cultivation geographies in the central and eastern US. Environmental Research Letters, 17(5): 1–11. https://doi.org/10.1088/1748-9326/ac6c3d. Burchfield EK, Nelson KS (2021). Agricultural yield geographies in the United States. Environmental Research Letters, 16(5): 1–12. https://doi.org/10.1088/1748-9326/abe88d. Chen T, He T, Benesty M, Khotilovich V, Tang Y, Cho H, et al. (2023). xgboost: Extreme Gradient Boosting. R package version 1.7.6.1. https://CRAN.R-project.org/package=xgboost. Crane-Droesch A (2018). Machine learning methods for crop yield prediction and climate change impact assessment in agriculture. Environmental Research Letters, 13(11): 114003. https://doi.org/10.1088/1748-9326/aae159. Culp M, Johnson K, Michailidis G (2016). ada: The R Package Ada for Stochastic Boosting. R package version 2.0-5. https://CRAN.R-project.org/package=ada (Accessed May 17, 2023). Friedman J, Hastie T, Tibshirani R (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1): 1–22. https://doi.org/10.18637/jss.v033.i01. Goslee SC (2020). Drivers of agricultural diversity in the contiguous United States. Frontiers in Sustainable Food Systems, 4: 75. https://doi.org/10.3389/fsufs.2020.00075. Gramacy RB (2007). tgp: An R package for Bayesian nonstationary, semiparametric nonlinear regression and design by treed Gaussian process models. Journal of Statistical Software, 19: 1–46. https://doi.org/10.18637/jss.v019.i09. Greenwell B, Boehmke B, Cunningham J, GBM Developers (2022). gbm: Generalized Boosted Regression Models. R package version 2.1.8.1. https://CRAN.R-project.org/package=gbm. Harrison D, Rubinfeld DL (1978). Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5(1): 81–102. https://doi.org/10.1016/0095-0696(78)90006-2. Hastie T, Efron B (2022). lars: Least Angle Regression, Lasso and Forward Stagewise. R package version 1.3. https://CRAN.R-project.org/package=lars. Hastie T, Tibshirani R, Friedman JH, Friedman JH (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2. Springer, New York, NY USA. Haycock S (2023). stressor: An R package for benchmarking machine learning models. Utah State University Digital Commons. 1-75. https://doi.org/10.26076/2am5-9f67. Hengl T, Miller MA, Križan J, Shepherd KD, Sila A, Kilibarda M, et al. (2021). African soil properties and nutrients mapped at 30 m spatial resolution using two-scale ensemble machine learning. Scientific Reports, 11(1): 1–18. https://doi.org/10.1038/s41598-021-85639-y. Hothorn T (2023). CRAN task view: Machine learning & statistical learning. Version 2023-07-20. https://CRAN.R-project.org/view=MachineLearning. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, et al. (2017). Lightgbm: A highly efficient gradient boosting decision tree. In: Advances in Neural Information Processing Systems. volume 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf (Accessed May 17, 2023). Krueger T, Braun M (2022). CVST: Fast Cross-Validation via Sequential Testing. R package version 0.2-3. https://CRAN.R-project.org/package=CVST. Kuhn M (2022). caret: Classification and Regression Training. R package version 6.0-93. https://CRAN.R-project.org/package=caret. Kuhn M, Wickham H (2020). Tidymodels: a collection of packages for modeling and machine learning using tidyverse principles. https://www.tidymodels.org. Lang M, Binder M, Richter J, Schratz P, Pfisterer F, Coors S, et al. (2019). mlr3: A modern object-oriented machine learning framework in R. Journal of Open Source Software. 1903. https://doi.org/10.21105/joss.01903. Le HM, Eriksson A, Do TT, Milford M (2019). A binary optimization approach for constrained k-means clustering. In: Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Revised Selected Papers, Part IV. Perth, Australia. December 2–6, 2018, 383–398. Springer. https://doi.org/10.1007/978-3-030-20870-7_24. Leisch F, Dimitriadou E (2021). mlbench: Machine Learning Benchmark Problems. R package version 2.1-3.1. https://cran.r-project.org/package=mlbench. Liang XZ, Wu Y, Chambers RG, Schmoldt DL, Gao W, Liu C, et al. (2017). Determining climate effects on US total agricultural productivity. Proceedings of the National Academy of Sciences, 114(12): E2285–E2292. https://doi.org/10.1073/pnas.1615922114. Lovelace R, Nowosad J, Muenchow J (2019). Geocomputation with R. CRC Press. https://r.geocompx.org/spatial-cv.html (Accessed: Dec 29, 2023). Lundell JF (2017). There has to be an easier way: A simple alternative for parameter tuning of supervised learning methods. In: JSM Proceedings, Statistical Computing Section, 3028–3036. American Statistical Association, Alexandria, VA. https://cran.r-project.org/package=EZtune. Majka M (2019). naivebayes: High Performance Implementation of the Naive Bayes Algorithm in R. R package version 0.9.7. https://CRAN.R-project.org/package=naivebayes. Meyer D, Dimitriadou E, Hornik K, Weingessel A, Leisch F (2022). e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. R package version 1.7-12. https://CRAN.R-project.org/package=e1071. Neunhoeffer M, Sternberg S (2019). How cross-validation can go wrong and what to do about it. Political Analysis, 27(1): 101–106. https://doi.org/10.1017/pan.2018.39. Papadakis M, Tsagris M, Fafalios S (2023). Rfast: A Collection of Efficient and Extremely Fast R Functions. R package version 2.1.0. https://CRAN.R-project.org/package=Rfast. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12: 2825–2830. https://www.jmlr.org/papers/volume12/pedregosa11a/pedregosa11a.pdf?ref=https:/ (Accessed Dec 29, 2023). Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. (2013). sklearn.model_selection.randomizedsearchcv. Journal of Machine Learning Research, 12: 2825–2830. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html (Accessed May 17, 2023). Ploton P, Mortier F, Réjou-Méchain M, Barbier N, Picard N, Rossi V, et al. (2020). Spatial validation reveals poor predictive performance of large-scale ecological mapping models. Nature Communications, 11(1): 1–11. https://doi.org/10.1038/s41467-020-18321-y. R Core Team (2022). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. Raschka S, Patterson J, Nolet C (2020). Machine learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence. Information, 11(4): 1–33. https://doi.org/10.3390/info11040193. Ray DK, Gerber JS, MacDonald GK, West PC (2015). Climate variation explains a third of global crop yield variability. Nature Communications, 6(1). https://doi.org/10.1038/ncomms6989. Roberts DR, Bahn V, Ciuti S, Boyce MS, Elith J, Guillera-Arroita G, et al. (2017a). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8): 913–929. https://doi.org/10.1111/ecog.02881. Roberts MJ, Braun NO, Sinclair TR, Lobell DB, Schlenker W (2017b). Comparing and combining process-based crop models and statistical models with some implications for climate change. Environmental Research Letters, 12(9): 095010. https://doi.org/10.1088/1748-9326/aa7f33. Roberts MJ, Schlenker W, Eyer J (2013). Agronomic weather measures in econometric models of crop yield with implications for climate change. American Journal of Agricultural Economics, 95(2): 236–243. https://doi.org/10.1093/ajae/aas047. Rong X (2022). deepnet: Deep Learning Toolkit in R. R package version 0.2.1. https://CRAN.R-project.org/package=deepnet. Rosenzweig C, Jones JW, Hatfield JL, Ruane AC, Boote KJ, Thorburn P, et al. (2013). The agricultural model intercomparison and improvement project (agmip): Protocols and pilot studies. Agricultural and Forest Meteorology, 170: 166–182. https://doi.org/10.1016/j.agrformet.2012.09.011. Schratz P, Becker M (2023). mlr3spatiotempcv: Spatiotemporal Resampling Methods for ‘mlr3’. https://mlr3spatiotempcv.mlr-org.com/. Schumacher BL, Burchfield EK, Bean B, Yost MA (2023). Leveraging important covariate groups for corn yield prediction. Agriculture, 13(3). https://doi.org/10.3390/agriculture13030618. Soltani A (2012). Modeling physiology of crop development, growth and yield. CABi. https://www.cabidigitallibrary.org/doi/book/10.1079/9781845939700.0000. Spangler K, Schumacher BL, Bean B, Burchfield EK (2022). Path dependencies in US agriculture: Regional factors of diversification. Agriculture, Ecosystems & Environment, 333: 107957. https://doi.org/10.1016/j.agee.2022.107957. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1): 1929–1958. https://dl.acm.org/doi/abs/10.5555/2627435.2670313. Therneau T (2018). deming: Deming, Theil-Sen, Passing-Bablock and Total Least Squares Regression. R package version 1.4. https://CRAN.R-project.org/package=deming. Therneau T, Atkinson B (2022). rpart: Recursive Partitioning and Regression Trees. R package version 4.1.19. https://CRAN.R-project.org/package=rpart. Urban DW, Sheffield J, Lobell DB (2015). The impacts of future climate and carbon dioxide changes on the average and variability of us maize yields under two emission scenarios. Environmental Research Letters, 10(4): 045003. https://doi.org/10.1088/1748-9326/10/4/045003. USDA (2019). 2017 Census of Agriculture. https://www.nass.usda.gov/AgCensus (Accessed Dec 29, 2023). Ushey K, Allaire J, Tang Y (2022). reticulate: Interface to Python. R package version 1.25. https://CRAN.R-project.org/package=reticulate. van Klompenburg T, Kassahun A, Catal C (2020). Crop yield prediction using machine learning: A systematic literature review. Computers and Electronics in Agriculture, 177: 105709. https://doi.org/10.1016/j.compag.2020.105709. Venables WN, Ripley BD (2002). Modern Applied Statistics with S. Springer, New York, fourth edition. ISBN 0-387-95457-0. Wadoux AMC, Heuvelink GB, De Bruin S, Brus DJ (2021). Spatial cross-validation is not the right way to evaluate map accuracy. Ecological Modelling, 457: 109692. https://doi.org/10.1016/j.ecolmodel.2021.109692. Wang XD, Chen RC, Yan F, Zeng ZQ, Hong CQ (2019). Fast adaptive k-means subspace clustering for high-dimensional data. IEEE Access, 7: 42639–42651. https://doi.org/10.1109/ACCESS.2019.2907043. Wright MN, Ziegler A (2017). Ranger: A fast implementation of random forests for high dimensional data in C++ and R. Journal of Statistical Software, 77(1): 1–17. https://doi.org/10.18637/jss.v077.i01.