Development of a Distractor-Based Cognitive Diagnostic Test (DB-CDT) on Thermodynamics Material to Detect High School Students' Data Literacy Skills
DOI:
10.29303/jppipa.v12i2.13899Published:
2026-02-25Downloads
Abstract
This study addresses the lack of specialized diagnostic instruments capable of pinpointing specific cognitive failures in physics data literacy, particularly within Thermodynamics. To bridge this gap, a Distractor-Based Cognitive Diagnostic Test (DB-CDT) was developed and validated for eleventh-grade high school students in Yogyakarta. Utilizing a Research and Development (R&D) approach with the ADDIE framework, the instrument was built upon the DINA (Deterministic Input, Noisy-AND Gate) cognitive diagnostic model. Expert validation by three specialists yielded an excellent average V-Aiken value of 1.00. Empirical testing on a sample of 540 students across high, medium, and low-tier schools confirmed high psychometric quality: an average INFIT MNSQ of 1.00 (within the 0.77–1.30 range), item difficulty between -2 and +2, and high reliability (0.92). DB-CDT distinguishes itself by using a Q-Matrix to link specific distractors to cognitive errors, such as failures in sign conversion or process interpretation. Analysis of student profiles revealed that the experimental group—utilizing targeted diagnostic feedback—achieved superior mastery in analyzing (96.6%) and interpreting data (90%), whereas the control group, receiving conventional assessment, struggled significantly with inference and synthesis. Theoretically, this research advances the application of Cognitive Diagnostic Models in physics education. Practically, it provides educators with a sensitive tool to profile individual student weaknesses, allowing for more focused instructional interventions in complex scientific topics.
Keywords:
Data literacy Distractor-based cognitive diagnostic test ThermodynamicsReferences
Branch, R. M. (2009). Instructional Design: The ADDIE Approach. Springer.
Bulut, O., & Beiting-Parrish, M. (2024). The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challenges. Chinese/English Journal of Educational Measurement and Evaluation, 5(3). https://doi.org/10.59863/miql7785
Cao, X., Lu, H., Wu, Q., & Hsu, Y. (2025). Systematic review and meta-analysis of the impact of STEM education on students learning outcomes. Frontiers in Psychology, 16(August), 1–15. https://doi.org/10.3389/fpsyg.2025.1579474
Chin, H., & Chew, C. M. (2023). Validation of cognitive models for subtraction of time involving years and centuries. International Journal of Assessment Tools in Education, 10(2), 175–196. https://doi.org/10.21449/ijate.1160120
Davison, M. L., Weiss, D. J., DeWeese, J. N., Ersan, O., Biancarosa, G., & Kennedy, P. C. (2023). A Diagnostic Tree Model for Adaptive Assessment of Complex Cognitive Processes Using Multidimensional Response Options. Journal of Educational and Behavioral Statistics, 48(6), 914–941. https://doi.org/10.3102/10769986231158301
Desstya, A., Sayekti, I. C., Abduh, M., & Sukartono, S. (2025). Development of a four-tier diagnostic test for misconceptions in natural science of primary school pupils. Journal of Turkish Science Education, 22(2), 338–353. https://doi.org/10.36681/tused.2025.017
Dogan, E. (2023). A Meta-Analysis Study on Data Literacy Education For School Administrators and Teachers. Kuramsal Eğitimbilim, 16(1), 199–217. https://doi.org/10.30831/akukeg.1134207
Downes, S. (2023). Three Frameworks for Data Literacy. 20th International Conference on Cognition and Exploratory Learning in Digital Age, 107–115. https://doi.org/10.33965/celda2023_202306l014
Guglietti, M. V. (2023). Redefining visual literacy in an era of visual overload: The use of reflective visual journals to expand students’ visual thinking. Journal of University Teaching and Learning Practice, 20(4). https://doi.org/10.53761/1.20.4.03
Hambleton, R. K., Swaminathan, H., & Rogers, D. J. (1992). Fundamentals of item response theory. Choice Reviews Online, 29(07), 29-4185-29–4185. https://doi.org/10.5860/CHOICE.29-4185
Kim, M., Duncan, C., Yip, S., Sankey, D., Kim, M., Duncan, C., Yip, S., & Dec, D. S. (2024). Beyond the theoretical and pedagogical constraints of cognitive load theory , and towards a new cognitive philosophy in education. Educational Philosophy and Theory, 1(1), 1–12. https://doi.org/10.1080/00131857.2024.2441389
Laliyo, L. A. R., La Kilo, A., Paputungan, M., Kunusa, W. R., Dama, L., & Panigoro, C. (2022). Rasch Modelling To Evaluate Reasoning Difficulties, Changes of Responses, and Item Misconception Pattern of Hydrolysis. Journal of Baltic Science Education, 21(5), 817–835. https://doi.org/10.33225/jbse/22.21.817
Laliyo, L. A. R., Tangio, J. S., Sumintono, B., Jahja, M., & Panigoro, C. (2020). Analytic approach of response pattern of diagnostic test items in evaluating students’ conceptual understanding of characteristics of particle of matter. Journal of Baltic Science Education, 19(5), 824–841. https://doi.org/10.33225/jbse/20.19.824
Lee, J., Alonzo, D., Beswick, K., Abril, J. M. V., Chew, A. W., & Oo, C. Z. (2024). Dimensions of teachers’ data literacy: A systematic review of literature from 1990 to 2021. In Educational Assessment, Evaluation and Accountability (Vol. 36, Issue 2, pp. 145–200). Springer Netherlands. https://doi.org/10.1007/s11092-024-09435-8
Li, Y. (2023). Impact of perceived recognition by physics instructors on women ’ s self-efficacy and interest. Physical Review Physics Education Research, 19(2), 20125. https://doi.org/10.1103/PhysRevPhysEducRes.19.020125
Lian, L. H., & Yew, W. (2023). Determining the psychometric properties of middle school statistical thinking testlet-based assessment tool. International Journal of Assessment Tools in Education, 10(4), 672–689. https://doi.org/10.21449/ijate.1255859
Mandinach, E. B., & Schildkamp, K. (2021). Misconceptions about data-based decision making in education: An exploration of the literature. Studies in Educational Evaluation, 69(September), 100842. https://doi.org/10.1016/j.stueduc.2020.100842
Mulyani, S., Haniza, N., Ramadhani, D. G., & Mahardiani, L. (2021). Rash Model Approach for Analysis of Misconception on Chemistry Learning with Distractor Analysis. JKPK (Jurnal Kimia Dan Pendidikan Kimia), 6(1), 98. https://doi.org/10.20961/jkpk.v6i1.50517
Munandar, H., Doyan, A., Susilawati, S., Hakim, S., Muliyadi, L., & Hamidi, H. (2024). Increasing Motivation to Study Physics Using PhET Media on Mechanical Energy Material. MANDALIKA : Journal of Social Science, 2(1), 1–5. https://doi.org/10.56566/mandalika.v2i1.70
Natalis, V., & Leyh, B. (2025). How Difficult is it to teach Thermodynamics in Tertiary Education? Teachers’ Opinions and Predominance of the Disorder Metaphor. International Journal of Research in Education and Science, 11(2), 190–215. https://doi.org/10.46328/ijres.3570
Net, W. W. W. P. (2023). Exploration of Diagnostic Testing Instruments: Validity, Reliability, and Item Characteristics. Pegem Journal of Education and Instruction, 13(3), 386–394. https://doi.org/10.47750/pegegog.13.03.39
Noh, M. F. M., Mohd Matore, M. E., Sulaiman, N. A., Azeman, M. T., Ishak, H., Othman, N., Mohd Rosli, N., & Sabtu, S. H. (2024). Cognitive Diagnostic Assessment in Educational Testing: A Score Strategy-Based Evaluation. E-Bangi Journal of Social Science and Humanities, 20(4). https://doi.org/10.17576/ebangi.2024.2104.21
OECD. (2023). PISA 2022 Results: The State of Learning and Equity in Education: Vol. I. Paris: OECD Publishing.
Panahi, A., & Mohebbi, H. (2022). Cognitive Diagnostic Assessment of IELTS Listening: Providing Feedback from its Internal Structure. Language Teaching Research Quarterly, 29, 147–160. https://doi.org/10.32038/ltrq.2022.29.10
Prasanth, P., Reshma, P., & Udayanandan, K. M. (2023). Thermodynamics in Microcanonical Ensemble. European Journal of Physics Education, 14(1), 14–22. Retrieved from https://files.eric.ed.gov/fulltext/EJ1413982.pdf
Pusporini, W., Rosana, D., & Setyawarno, D. (2026). Innovative Online Diagnostic Assessment for Evaluating Science Learning Readiness in the Merdeka Curriculum Framework. Journal of Education and Teaching (JET), 7(1), 1–19. https://doi.org/10.51454/jet.v7i1.546
Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic Measurement (Theory, Methods, and Applications). New York: The Guilford Press.
Scaria, N., John, S., Kennedy, J., Agarwal, K., Seth, D., & Subramani, D. (2025). Dissecting Physics Reasoning in Small Language Models : A Multi-Dimensional Analysis from an Educational Perspective. ArXiv. https://doi.org/2505.20707v2 [cs.CL] null
Soeharto, S. (2021). Development of A Diagnostic Assessment Test to Evaluate Science Misconceptions in Terms of School Grades: A Rasch Measurement Approach. Journal of Turkish Science Education, 18(3), 351–370. https://doi.org/10.36681/tused.2021.78
Suryadi, Mahardika, I. K., Supeno, & Sudarti. (2021). Data literacy of high school students on physics learning. Journal of Physics: Conference Series, 1839(1), 012025. https://doi.org/10.1088/1742-6596/1839/1/012025
Wladis, C., Verkuilen, J., Mccluskey, S., Offenholley, K., Licwinko, S., Lee, J. K., Wladis, C., Verkuilen, J., Mccluskey, S., Offenholley, K., Dawes, D., Wladis, C., Verkuilen, J., Mccluskey, S., Offenholley, K., & Dawes, D. (2020). Relationships between procedural fluency and conceptual understanding in algebra for postsecondary students To cite this version : HAL Id : hal-02416499 Relationships between procedural fluency and conceptual understanding in algebra for postsecondary stu. In Eleventh Congress of the European Society for Research in Mathematics Education. Retrieved from https://hal.science/hal-02416499/
Yi, Y. S. (2023). Another Reason for CDA in Language Assessment: A Critical Synthesis from a Perspective of Validating Test Constructs. Language Teaching Research Quarterly, 37, 248–265. https://doi.org/10.32038/ltrq.2023.37.14
License
Copyright (c) 2026 Supahar, Ayu Mar’ati Barokatun Ni’mah, Reny Afifah, Putri Edryna Larasati
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with Jurnal Penelitian Pendidikan IPA, agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution 4.0 International License (CC-BY License). This license allows authors to use all articles, data sets, graphics, and appendices in data mining applications, search engines, web sites, blogs, and other platforms by providing an appropriate reference. The journal allows the author(s) to hold the copyright without restrictions and will retain publishing rights without restrictions.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in Jurnal Penelitian Pendidikan IPA.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
