JDS Journal of Data Science 1683-86021680-743X1680-743X School of Statistics, Renmin University of China JDS1024 10.6339/21-JDS1024 Statistical Data Science Hypothesis Testing for Hierarchical Structures in Cognitive Diagnosis Models MaChenchen1 XuGongjungongjun@umich.edu1∗ Department of Statistics, University of Michigan, West Hall, 1085 S University Ave, Ann Arbor, MI 48109, USA Corresponding author. Email: gongjun@umich.edu. 202214102021203279302 Supplementary Material

More comprehensive simulation results are presented in the supplementary material. Specifically, bootstrap results for DINA and GDINA models under both null hypothesis and alternative hypothesis with different sample sizes and noise levels are plotted there. We also include the codes for simulations and real data analysis.

2620211192021 2022 The Author(s). Published by the School of Statistics and the Center for Applied Statistics, Renmin University of China.2022 Open access article under the CC BY license.

Cognitive Diagnosis Models (CDMs) are a special family of discrete latent variable models widely used in educational, psychological and social sciences. In many applications of CDMs, certain hierarchical structures among the latent attributes are assumed by researchers to characterize their dependence structure. Specifically, a directed acyclic graph is used to specify hierarchical constraints on the allowable configurations of the discrete latent attributes. In this paper, we consider the important yet unaddressed problem of testing the existence of latent hierarchical structures in CDMs. We first introduce the concept of testability of hierarchical structures in CDMs and present sufficient conditions. Then we study the asymptotic behaviors of the likelihood ratio test (LRT) statistic, which is widely used for testing nested models. Due to the irregularity of the problem, the asymptotic distribution of LRT becomes nonstandard and tends to provide unsatisfactory finite sample performance under practical conditions. We provide statistical insights on such failures, and propose to use parametric bootstrap to perform the testing. We also demonstrate the effectiveness and superiority of parametric bootstrap for testing the latent hierarchies over non-parametric bootstrap and the naïve Chi-squared test through comprehensive simulations and an educational assessment dataset.

 bootstrapping latent hierarchical structure likelihood ratio test National Science FoundationCAREER SES-1846747Institute of Education SciencesR305D200015This research is partially supported by National Science Foundation CAREER SES-1846747 and Institute of Education Sciences R305D200015. References Chen J (2017). On finite mixture models. Statistical Theory and Related Fields, 1(1): 15–27. Chen Y, Moustaki I, Zhang H (2020). A note on likelihood ratio tests for models with latent variables. Psychometrika, 85: 1–17. Dahlgren MA, Hult H, Dahlgren LO, af Segerstad HH, Johansson K (2006). From senior student to novice worker: Learning trajectories in political science, psychology and mechanical engineering. Studies in Higher Education, 31(5): 569–586. de la Torre J (2011). The generalized DINA model framework. Psychometrika, 76(2): 179–199. de la Torre J, van der Ark LA, Rossi G (2018). Analysis of clinical data from a cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development, 51(4): 281–296. DiBello LV, Stout WF, Roussos LA (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In: Edited by Paul D. Nichols, Susan F. Chipman, Robert L. Brennan, Cognitively Diagnostic Assessment, 361–389. Routledge. Efron B (1979). Bootstrap methods: Another look at the Jackknife. The Annals of Statistics, 7(1): 1–26. George AC, Robitzsch A (2015). Cognitive diagnosis models in R: A didactic. The Quantitative Methods for Psychology, 11(3): 189–205. Gu Y, Xu G (2019). Learning attribute patterns in high-dimensional structured latent attribute models. Journal of Machine Learning Research, 20(2019): 1–58. Gu Y, Xu G (2020). Partial identifiability of restricted latent class models. Annals of Statistics, 48(4): 2082–2107. Gu Y, Xu G (2021). Identifiability of hierarchical latent attribute models. arXiv preprint arXiv:1906.07869. Haertel EH (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26(4): 301–321. Henson RA, Templin JL, Willse JT (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2): 191. Jimoyiannis A, Komis V (2001). Computer simulations in physics teaching and learning: A case study on students’ understanding of trajectory motion. Computers & Education, 36(2): 183–204. Junker BW, Sijtsma K (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3): 258–272. Leighton JP, Gierl MJ, Hunka SM (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41(3): 205–237. Nylund KL, Asparouhov T, Muthén BO (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling: A Multidisciplinary Journal, 14(4): 535–569. O’Brien KL, Baggett HC, Brooks WA, Feikin DR, Hammitt LL, Higdon MM, et al. (2019). Causes of severe pneumonia requiring hospital admission in children without HIV infection from Africa and Asia: The PERCH multi-country case-control study. The Lancet, 394(10200): 757–779. Reckase MD (2009). Multidimensional item response theory models. In: Multidimensional Item Response Theory, 79–112. Springer. Self SG, Liang KY (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82(398): 605–610. Simon MA, Tzur R (2004). Explicating the role of mathematical tasks in conceptual learning: An elaboration of the hypothetical learning trajectory. Mathematical Thinking and Learning, 6(2): 91–104. Tatsuoka KK (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20: 345–354. Tatsuoka KK (1990). Toward an integration of item-response theory and cognitive error diagnosis. In: Edited by Norman Frederiksen, Robert Glaser, Alan Lesgold, and Michael G. Shafto, Diagnostic Monitoring of Skill and Knowledge Acquisition, 453–488. Routledge. Templin J, Bradshaw L (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79(2): 317–339. Templin JL, Henson RA (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3): 287. von Davier M (2005). A general diagnostic model applied to language testing data. ETS Research Report Series, 2005(2): 1–35. Wang C, Gierl MJ (2011). Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills in critical reading. Journal of Educational Measurement, 48(2): 165–187. Wu Z, Deloria-Knoll M, Hammitt LL, Zeger SL (for Child Health Core Team PER) (2016a). Partially latent class models for case–control studies of childhood pneumonia etiology. Journal of the Royal Statistical Society: Series C (Applied Statistics), 65(1): 97–114. Wu Z, Deloria-Knoll M, Zeger SL (2016b). Nested partially latent class models for dependent binary data; estimating disease etiology. Biostatistics, 18(2): 200–213. Xu G (2017). Identifiability of restricted latent class models with binary responses. The Annals of Statistics, 45(2): 675–707. Xu G, Shang Z (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523): 1284–1295. Xu G, Zhang S (2016). Identifiability of diagnostic classification models. Psychometrika, 81(3): 625–649.