JDS Journal of Data Science 1683-86021680-743X1680-743X School of Statistics, Renmin University of China JDS1071 10.6339/22-JDS1071 Statistical Data Science High-Dimensional Nonlinear Spatio-Temporal Filtering by Compressing Hierarchical Sparse Cholesky Factors ChakrabortyAnirban1 KatzfussMatthiaskatzfuss@gmail.com1∗ Department of Statistics, Texas A&M University, 3143 TAMU, College Station, TX 77843, USA Corresponding author. Email: katzfuss@gmail.com. 20223102022204461474 Supplementary Material

R code to reproduce our results and figures is available at https://github.com/katzfuss-group/CHVfilter.

1820222892022 2022 The Author(s). Published by the School of Statistics and the Center for Applied Statistics, Renmin University of China.2022 Open access article under the CC BY license.

Spatio-temporal filtering is a common and challenging task in many environmental applications, where the evolution is often nonlinear and the dimension of the spatial state may be very high. We propose a scalable filtering approach based on a hierarchical sparse Cholesky representation of the filtering covariance matrix. At each time point, we compress the sparse Cholesky factor into a dense matrix with a small number of columns. After applying the evolution to each of these columns, we decompress to obtain a hierarchical sparse Cholesky factor of the forecast covariance, which can then be updated based on newly available data. We illustrate the Cholesky evolution via an equivalent representation in terms of spatial basis functions. We also demonstrate the advantage of our method in numerical comparisons, including using a high-dimensional and nonlinear Lorenz model.

 basis functions data assimilation hierarchical Vecchia approximation Lorenz model unscented Kalman filter National Science FoundationDMS–1654083DMS–1953005National Aeronautics and Space Administration80NM0018F0527MK was partially supported by National Science Foundation (NSF) Grants DMS–1654083 and DMS–1953005, and by the National Aeronautics and Space Administration (80NM0018F0527). References Arasaratnam I, Haykin S (2009). Cubature Kalman filters. IEEE Transactions on Automatic Control, 54(6): 1254–1269. Castrillón-Candás JE, Genton MG, Yokota R (2016). Multi-level restricted maximum likelihood covariance estimation and kriging for large non-gridded spatial datasets. Spatial Statistics, 18: 105–124. Spatial Statistics Avignon: Emerging Patterns. Castrillón-Candás JE, Li J, Eijkhout V (2013). A discrete adapted hierarchical basis solver for radial basis function interpolation. BIT, 53(1): 57–86. Chandrasekar J, Kim IS, Bernstein DS, Ridley AJ (2008). Reduced-rank unscented Kalman filtering using Cholesky-based decomposition. In: 2008 American Control Conference, 1274–1279. Datta A, Banerjee S, Finley AO, Gelfand AE (2016). Hierarchical nearest-neighbor Gaussian process models for large geostatistical datasets. Journal of the American Statistical Association, 111(514): 800–812. Fang C, Liu J, Ye S, Zhang J (2020). The geometric unscented Kalman filter. arXiv preprint: https://arxiv.org/abs/2009.13079. Gneiting T, Katzfuss M (2014). Probabilistic forecasting. Annual Review of Statistics and Its Application, 1(1): 125–151. Gordon N, Salmond D, Smith A (1993). Novel approach to nonlinear/non-Gaussian Bayesian state estimation. IEE Proceedings. Part F. Radar and Signal Processing, 140(2): 107–113. Grewal MS, Andrews AP (1993). Kalman Filtering: Theory and Applications. Prentice Hall. Guinness J (2018). Permutation and grouping methods for sharpening Gaussian process approximations. Technometrics, 60(4): 415–429. Jurek M, Katzfuss M (2022). Hierarchical sparse Cholesky decomposition with applications to high-dimensional spatio-temporal filtering. Statistics and Computing, 32: 15. Kalman R (1960). A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1): 35–45. Kang M, Katzfuss M (2021). Correlation-based sparse inverse Cholesky factorization for fast Gaussian-process inference. arXiv preprint: https://arxiv.org/abs/2112.14591. Katzfuss M (2017). A multi-resolution approximation for massive spatial datasets. Journal of the American Statistical Association, 112(517): 201–214. Katzfuss M, Guinness J (2021). A general framework for Vecchia approximations of Gaussian processes. Statistical Science, 36(1): 124–141. Katzfuss M, Guinness J, Gong W, Zilber D (2020). Vecchia approximations of Gaussian-process predictions. Journal of Agricultural, Biological, and Environmental Statistics, 25(3): 383–414. Khazraj H, Faria da Silva F, Bak CL (2016). A performance comparison between extended Kalman Filter and unscented Kalman Filter in power system dynamic state estimation. In: 2016 51st International Universities Power Engineering Conference (UPEC), 1–6. Liu JS, Chen R (1998). Sequential Monte Carlo methods for dynamic systems. Journal of the American Statistical Association, 93(443): 1032–1044. Lorenz EN (2005). Designing chaotic models. Journal of the Atmospheric Sciences, 62(5): 1574–1587. Meng D, Miao L, Shao H, Shen J (2018). A seventh-degree cubature Kalman filter. Asian Journal of Control, 20(1): 250–262. Nychka DW, Anderson JL (2010). Data assimilation. In: Handbook of Spatial Statistics (AE Gelfand, PJ Diggle, M Fuentes, P Guttorp, eds.), 477–494. CRC Press. Chapter 27. Ott E, Hunt BR, Szunyogh I, Zimin AV, Kostelich EJ, Corazza M, et al. (2004). A local ensemble Kalman filter for atmospheric data assimilation. Tellus A, 56: 415–428. Pitt MK, Shephard N (1999). Filtering via simulation: Auxiliary particle filters. Journal of the American Statistical Association, 94(446): 590–599. Schäfer F, Katzfuss M, Owhadi H (2021). Sparse Cholesky factorization by Kullback-Leibler minimization. SIAM Journal on Scientific Computing, 43(3): A2019–A2046. Shumway RH, Stoffer DS (2000). Time Series Analysis and Its Applications. Springer. St-Pierre M, Gingras D (2004). Comparison between the unscented Kalman filter and the extended Kalman filter for the position estimation module of an integrated navigation information system. In: IEEE Intelligent Vehicles Symposium, 2004, 831–835. IEEE. Stein ML, Chi Z, Welty L (2004). Approximating likelihoods for large spatial data sets. Journal of the Royal Statistical Society, Series B, 66(2): 275–296. Vecchia A (1988). Estimation and model identification for continuous spatial processes. Journal of the Royal Statistical Society, Series B, 50(2): 297–312. Wan E, Van Der Merwe R (2000). The unscented Kalman filter for nonlinear estimation. In: Adaptive Systems for Signal Processing, Communications, and Control. Lake Louise, Canada, 153–158. Wang S, Feng J, Chi KT (2013). Spherical simplex-radial cubature Kalman filter. IEEE Signal Processing Letters, 21(1): 43–46. West M, Harrison J (1997). Bayesian Forecasting and Dynamic Models. Springer Series in Statistics. Springer-Verlag. Zilber D, Katzfuss M (2021). Vecchia-Laplace approximations of generalized Gaussian processes for big non-Gaussian spatial data. Computational Statistics & Data Analysis, 153: 107081.