Hypothesis Testing for Hierarchical Structures in Cognitive Diagnosis Models

Ma, Chenchen; Xu, Gongjun

doi:10.6339/21-JDS1024

Journal of Data Science

Hypothesis Testing for Hierarchical Structures in Cognitive Diagnosis Models

Volume 20, Issue 3 (2022): Special Issue: Data Science Meets Social Sciences, pp. 279–302

Chenchen Ma Gongjun Xu

https://doi.org/10.6339/21-JDS1024

Pub. online: 14 October 2021 Type: Statistical Data Science

Open Access

Received
2 June 2021

Accepted
11 September 2021

Published
14 October 2021

Abstract

Cognitive Diagnosis Models (CDMs) are a special family of discrete latent variable models widely used in educational, psychological and social sciences. In many applications of CDMs, certain hierarchical structures among the latent attributes are assumed by researchers to characterize their dependence structure. Specifically, a directed acyclic graph is used to specify hierarchical constraints on the allowable configurations of the discrete latent attributes. In this paper, we consider the important yet unaddressed problem of testing the existence of latent hierarchical structures in CDMs. We first introduce the concept of testability of hierarchical structures in CDMs and present sufficient conditions. Then we study the asymptotic behaviors of the likelihood ratio test (LRT) statistic, which is widely used for testing nested models. Due to the irregularity of the problem, the asymptotic distribution of LRT becomes nonstandard and tends to provide unsatisfactory finite sample performance under practical conditions. We provide statistical insights on such failures, and propose to use parametric bootstrap to perform the testing. We also demonstrate the effectiveness and superiority of parametric bootstrap for testing the latent hierarchies over non-parametric bootstrap and the naïve Chi-squared test through comprehensive simulations and an educational assessment dataset.

Supplementary material

Supplementary Material

More comprehensive simulation results are presented in the supplementary material. Specifically, bootstrap results for DINA and GDINA models under both null hypothesis and alternative hypothesis with different sample sizes and noise levels are plotted there. We also include the codes for simulations and real data analysis.

References

Chen J (2017). On finite mixture models. Statistical Theory and Related Fields, 1(1): 15–27.

Chen Y, Moustaki I, Zhang H (2020). A note on likelihood ratio tests for models with latent variables. Psychometrika, 85: 1–17.

Dahlgren MA, Hult H, Dahlgren LO, af Segerstad HH, Johansson K (2006). From senior student to novice worker: Learning trajectories in political science, psychology and mechanical engineering. Studies in Higher Education, 31(5): 569–586.

de la Torre J (2011). The generalized DINA model framework. Psychometrika, 76(2): 179–199.

de la Torre J, van der Ark LA, Rossi G (2018). Analysis of clinical data from a cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development, 51(4): 281–296.

DiBello LV, Stout WF, Roussos LA (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In: Edited by Paul D. Nichols, Susan F. Chipman, Robert L. Brennan, Cognitively Diagnostic Assessment, 361–389. Routledge.

Efron B (1979). Bootstrap methods: Another look at the Jackknife. The Annals of Statistics, 7(1): 1–26.

George AC, Robitzsch A (2015). Cognitive diagnosis models in R: A didactic. The Quantitative Methods for Psychology, 11(3): 189–205.

Gu Y, Xu G (2019). Learning attribute patterns in high-dimensional structured latent attribute models. Journal of Machine Learning Research, 20(2019): 1–58.

Gu Y, Xu G (2020). Partial identifiability of restricted latent class models. Annals of Statistics, 48(4): 2082–2107.

Gu Y, Xu G (2021). Identifiability of hierarchical latent attribute models. arXiv preprint arXiv:1906.07869.

Haertel EH (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26(4): 301–321.

Henson RA, Templin JL, Willse JT (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2): 191.

Jimoyiannis A, Komis V (2001). Computer simulations in physics teaching and learning: A case study on students’ understanding of trajectory motion. Computers & Education, 36(2): 183–204.

Junker BW, Sijtsma K (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3): 258–272.

Leighton JP, Gierl MJ, Hunka SM (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41(3): 205–237.

Nylund KL, Asparouhov T, Muthén BO (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling: A Multidisciplinary Journal, 14(4): 535–569.

O’Brien KL, Baggett HC, Brooks WA, Feikin DR, Hammitt LL, Higdon MM, et al. (2019). Causes of severe pneumonia requiring hospital admission in children without HIV infection from Africa and Asia: The PERCH multi-country case-control study. The Lancet, 394(10200): 757–779.

Reckase MD (2009). Multidimensional item response theory models. In: Multidimensional Item Response Theory, 79–112. Springer.

Self SG, Liang KY (1987). Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions. Journal of the American Statistical Association, 82(398): 605–610.

Simon MA, Tzur R (2004). Explicating the role of mathematical tasks in conceptual learning: An elaboration of the hypothetical learning trajectory. Mathematical Thinking and Learning, 6(2): 91–104.

Tatsuoka KK (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20: 345–354.

Tatsuoka KK (1990). Toward an integration of item-response theory and cognitive error diagnosis. In: Edited by Norman Frederiksen, Robert Glaser, Alan Lesgold, and Michael G. Shafto, Diagnostic Monitoring of Skill and Knowledge Acquisition, 453–488. Routledge.

Templin J, Bradshaw L (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79(2): 317–339.

Templin JL, Henson RA (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3): 287.

von Davier M (2005). A general diagnostic model applied to language testing data. ETS Research Report Series, 2005(2): 1–35.

Wang C, Gierl MJ (2011). Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills in critical reading. Journal of Educational Measurement, 48(2): 165–187.

Wu Z, Deloria-Knoll M, Hammitt LL, Zeger SL (for Child Health Core Team PER) (2016a). Partially latent class models for case–control studies of childhood pneumonia etiology. Journal of the Royal Statistical Society: Series C (Applied Statistics), 65(1): 97–114.

Wu Z, Deloria-Knoll M, Zeger SL (2016b). Nested partially latent class models for dependent binary data; estimating disease etiology. Biostatistics, 18(2): 200–213.

Xu G (2017). Identifiability of restricted latent class models with binary responses. The Annals of Statistics, 45(2): 675–707.

Xu G, Shang Z (2018). Identifying latent structures in restricted latent class models. Journal of the American Statistical Association, 113(523): 1284–1295.

Xu G, Zhang S (2016). Identifiability of diagnostic classification models. Psychometrika, 81(3): 625–649.

2022 The Author(s). Published by the School of Statistics and the Center for Applied Statistics, Renmin University of China.

Open access article under the CC BY license.

Keywords

bootstrapping latent hierarchical structure likelihood ratio test

Funding

This research is partially supported by National Science Foundation CAREER SES-1846747 and Institute of Education Sciences R305D200015.

Metrics

since February 2021

1910 Article info views	0 Full article views
668 PDF downloads	320 XML downloads

RSS

Authors

Abstract

Supplementary material

References

Export citation

Copy and paste formatted citation

Download citation in file