A Statistician’s Selective Review of Neural Network Modeling: Algorithms and Applications
Pub. online: 20 January 2025
Type: Data Science Reviews
Open Access
Open Access
Received
25 October 2024
25 October 2024
Accepted
8 January 2025
8 January 2025
Published
20 January 2025
20 January 2025
Abstract
Deep neural networks have a wide range of applications in data science. This paper reviews neural network modeling algorithms and their applications in both supervised and unsupervised learning. Key examples include: (i) binary classification and (ii) nonparametric regression function estimation, both implemented with feedforward neural networks ($\mathrm{FNN}$); (iii) sequential data prediction using long short-term memory ($\mathrm{LSTM}$) networks; and (iv) image classification using convolutional neural networks ($\mathrm{CNN}$). All implementations are provided in $\mathrm{MATLAB}$, making these methods accessible to statisticians and data scientists to support learning and practical application.
Supplementary material
Supplementary MaterialThe MATLAB implementation, including a README file, is available at https://github.com/ChunmingZhangUW/Review-NNM_JDS. The supplementary file includes Appendix A for the proof of Proposition 1 and Appendix B for numerical illustrations of
LSTM models in Section 5.2.
References
Alzubaidi L, Zhang J, Humaidi AJ, et al. (2021). Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data, 8(1): 53. https://doi.org/10.1186/s40537-021-00444-8
Coleman TF, Li Y (1994). On the convergence of reflective Newton methods for large-scale nonlinear minimization subject to bounds. Mathematical Programming, 67(2): 189–224. https://doi.org/10.1007/BF01582221
Farrell MH, Liang T, Misra S (2021). Deep neural networks for estimation and inference. Econometrica, 89(1): 181–213. https://doi.org/10.3982/ECTA16901
Goodfellow I, Bengio Y, Courville A (2016). Deep Learning. MIT Press. http://www.deeplearningbook.org.
Higham CF, Higham DJ (2019). Deep learning: An introduction for applied mathematicians. SIAM Review, 61(4): 860–891. https://doi.org/10.1137/18M1165748
Hinton GE (2007). Learning multiple layers of representation. Trends in Cognitive Sciences, 11(10): 428–434. https://doi.org/10.1016/j.tics.2007.09.004
Hinton GE, Osindero S, Teh Y-W (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18(7): 1527–1554. https://doi.org/10.1162/neco.2006.18.7.1527
Jordan MI, Mitchell TM (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245): 255–260. https://doi.org/10.1126/science.aaa8415
Kingma DP, Ba J (2015). Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings (Y Bengio, Y LeCun, eds.). ArXiv, Ithaca, NY. https://hdl.handle.net/11245/1.505367.
Kramer MA (1991). Nonlinear principal component analysis using autoassociative neural networks. AIChE Journal, 37(2): 233–243. https://doi.org/10.1002/aic.690370209
Muir D (2024). Adam stochastic gradient descent optimization. https://github.com/DylanMuir/fmin_adam.
Vogl TP, Mangis J, Rigler A, Zink W, Alkon D (1988). Accelerating the convergence of the back-propagation method. Biological Cybernetics, 59: 257–263. https://doi.org/10.1007/BF00332914
Zhang C, Zhu L, Shen Y (2023). Robust estimation in regression and classification methods for large dimensional data. Machine Learning, 112(9): 3361–3411. https://doi.org/10.1007/s10994-023-06349-2
Zhong R, Zhang J, Zhang C (2024). Nonlinear functional principal component analysis using neural networks. arXiv preprint: https://arxiv.org/abs/2306.14388.
