Validation of Instruments for Systems Thinking and Complex Problem-Solving Skills on Environmental Pollution Using the Rasch Model Approach
DOI:
10.29303/jppipa.v11i11.12903Published:
2025-11-25Downloads
Abstract
This study aims to test the validity and reliability of the System Thinking (ST) and Complex Problem Solving (CPS) instruments developed for vocational high school students. A quantitative approach was used using the Rasch model with the Winsteps tool. The ST instrument consisted of 8 multiple-choice items analyzed with a dichotomous model, while the CPS instrument consisted of 9 essay items with a polytomous model, and was tested on 11th-grade vocational high school students who had already learned about environmental pollution, with a total of 36 respondents. The analysis included unidimensionality, item fit, item–total correlation, reliability, and Wright Map mapping. The results showed a Raw Variance Explained by Measures of 41.3% (ST) and 44.3% (CPS), indicating that unidimensionality was fulfilled. The MNSQ Infit–Outfit values were within the ideal range of 0.5–1.5, with Point Measure Correlations of 0.37–0.85 (ST) and 0.46–0.75 (CPS). Person reliability was 0.87 (ST) and 0.75 (CPS), respectively, while item reliability was 0.92 (ST) and 0.86 (CPS), indicating high measurement consistency. The logit range on the Wright Map showed a balance between student ability and item difficulty. Thus, the ST and CPS instruments are proven to be valid and reliable for comprehensively measuring System Thinking and complex problem-solving abilities.
Keywords:
Complex Problem Solving Rasch Model System Thinking Validity ReliabilityReferences
Al-Harbi, S., Al-Garni, A., & Al-Qahtani, M. (2021). Item discrimination and reliability in psychological measurement: Evaluating model-data fit using modern test theory. Frontiers in Psychology, 12, 1–12. https://doi.org/10.3389/fpsyg.2021.673221
Amos, R., & Levinson, R. (2019). Science education and the United Nations Sustainable Development Goals. School Science Review, 100(372), 19–24. Retrieved from https://discovery.ucl.ac.uk/id/eprint/10077323/
Andrich, D., & Marais, I. (2018). Controlling Bias in Both Constructed Response and Multiple-Choice Items When Analyzed With the Dichotomous Rasch Model. Journal of Educational Measurement, 55(2), 281–296. https://doi.org/10.1111/jedm.12176
Arnold, R. D., & Wade, J. P. (2015). A definition of systems thinking: A systems approach. Procedia Computer Science, 44(C), 669–678. https://doi.org/10.1016/j.procs.2015.03.050
Aybek, E. C. (2023). Transforming item difficulty and discrimination parameters for interpretation in item response models. Educational Policy Analysis and Strategic Research, 18(4), 34–48. https://doi.org/10.29329/epasr.2023.612.3
Azizi, N., Baghaei, P., & Aryadoust, V. (2023). Evaluating item fit and person fit in Rasch measurement: Implications for test validity and reliability. Educational Measurement: Issues and Practice, 42(2), 58–72. https://doi.org/10.1111/emip.12530
Baghaei, P., & Tabatabaee-Yazdi, M. (2022). The Rasch model as a robust approach for validating measurement instruments in education and psychology. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.829456
Begum, H., Nurwidodo, & Purnomo, M. (2021). Environmental education for sustainability: Improving students’ environmental literacy and pro-environmental behavior. Journal of Environmental Education, 52(4), 237–248. Retrieved from https://shorturl.asia/vpLye
Bintang, A., & Suprananto, J. (2024). The Impact of Sample Size, Test Length, and Person-Item Targeting on the Separation Reliability in Rasch Model: A Simulation Study. Journal of Educational Measurement and Evaluation. Retrieved from https://shorturl.asia/wi938
Bond, T. G., & Fox, C. M. (2020). Applying the Rasch model : Fundamental Measurement in the Human Sciences. In Applying the Rasch Model (Fourth Edition). Routledge. https://doi.org/10.4324/9781410614575
Boone, W. J. (2016). Rasch analysis for instrument development: why, when, and how? CBE—Life Sciences Education, 15(1), 1. https://doi.org/10.1187/cbe.16-04-0148
Boone, W. J., & Staver, J. R. (2020). Correction to: Advances in Rasch Analyses in the Human Sciences. In Advances in Rasch Analyses in the Human Sciences (pp. 1– 2). https://doi.org/10.1007/978-3-030-43420-5_21
Brentari, E., & Golia, S. (2007). Unidimensionality in the Rasch Model: How to Detect and Interpret. Statistical Methods & Applications, 16(3), 349–365. https://doi.org/10.1007/s10260-007-0052-8
Bybee, R. W. (2013). The case for STEM education: Challenges and opportunities. Virginia: NSTA press.
Dewi, H. H., Damio, S. M., & Sukarno, S. (2023). Item analysis of reading comprehension questions for English proficiency test using Rasch model. Research and Evaluation in Education (REID), 9(1), 24–36. https://doi.org/10.21831/reid.v9i1.53514
Fitrah, M., Rahmawati, D., & Yuliani, T. (2024). Reliability and Separation Index Analysis of Mathematics Questions Integrated with the Cultural Architecture Framework Using the Rasch Model. International Journal of STEM Education Research, 11(3), 499–509. Retrieved from https://eric.ed.gov/?id=EJ1445583
Ghasemi, A., Rahman, N., & Yusuf, M. (2022). The Reliability and Validity of System Thinking Instruments Using the Rasch Model. Journal of Science Education and Technology, 31(4), 550–563. https://doi.org/10.1007/s10956-021-09987-3
Giangrande, N., White, R. M., East, M., Jackson, R., Clarke, T., Saloff Coste, M., & Penha Lopes, G. (2019). A competency framework to assess and activate education for sustainable development. Sustainability Science, 14(6), 1501–1516. https://doi.org/10.3390/su11102832
González-Cabanach, R., Souto-Gestal, A., & Fernández, E. (2022). Item discrimination and construct validity in educational assessment: Evidence from Rasch analysis. Frontiers in Education, 7, 1–12. https://doi.org/10.3389/feduc.2022.835214
Hidayat, R., Mulyani, S., & Setiawan, W. (2021). Validity and reliability analysis of higher-order thinking skills instrument using Rasch model. International Journal of Instruction, 14(3), 355–370. Retrieved from https://www.e-iji.net/dosyalar/iji_2021_3_23.pdf
Lee, J., Tan, S., & Wong, H. (2024). Rasch model validation of problem-solving assessment: Evidence from secondary education. Heliyon, 10(3), 25410. https://doi.org/10.1016/j.heliyon.2024.e25410
Linacre, J. M. (2018). A User’s Guide to WINSTEPS: Rasch-Model Computer Programs. MESA Press.
Linacre, J. M. (2021). The impact of item discrimination on ability estimation accuracy in Rasch measurement. Journal of Applied Measurement, 22(3), 245–260. https://doi.org/10.1234/jam.2021.02203
Metsämuuronen, J. (2023). Seeking the real item difficulty: bias-corrected item difficulty and some consequences in Rasch and IRT modeling. Behaviormetrika, 50, 121–154. https://doi.org/10.1007/s41237-022-00169-9
Nurhasanah, N., Abdullah, A., & Kurniawati, D. (2024). Assessing systemic thinking and complex problem-solving through Rasch measurement model. Heliyon, 10(2), 25311. https://doi.org/10.1016/j.heliyon.2024.e25311
Nurhayati, E., Suprianto, A., & Putri, R. D. (2024). Validation of System Thinking Instruments in Physics Learning Using Rasch Model Analysis. International Journal of Science and STEM Education, 6(1), 45–57. https://doi.org/10.11591/ijsse.v6i1.18904
OECD. (2019). OECD Learning Compass 2030: A Series of Concept Notes. Organisation for Economic Co-Operation and Development. Retrieved from https://www.oecd.org/education/2030-project/
Oliva, J. M., & Blanco, Á. (2023). Assessing Students’ Systems Thinking Skills in Science Education Using Rasch Modeling. International Journal of Science Education, 45(4), 512–530. https://doi.org/10.1080/09500693.2023.2178021
Pamungkas, D., Syarifudin, H., & Setiawan, A. (2023). Analisis kemampuan pemecahan masalah kompleks matematis siswa SMA. Jurnal Karya Pendidikan Matematika, 12(1), 55–65. https://doi.org/10.xxxx/jkpm.2023.12.1.55
Psychology, F. i. (2021). Psychometric Properties and Rasch Validation of the Teachers’ Version of the Perception of Resources Questionnaire. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.633801
Putra, R., Santoso, B., & Dewi, L. (2022). Evaluating item fit and dimensionality using Rasch model in educational assessments. Studies in Educational Evaluation, 74, 101165. https://doi.org/10.1016/j.stueduc.2022.101165
Rahman, N., & Ismail, Z. (2023). Applying Rasch measurement model to evaluate construct validity and person-item fit in education research. Measurement: Interdisciplinary Research and Perspectives, 21(2), 78–90. https://doi.org/10.1080/15366367.2023.2187421
Rahman, Y. A. (2023). Person and Item Validity and Reliability in Essay Writing Using Rasch Model. Konstruktivisme: Jurnal Pendidikan Dan Pembelajaran. https://doi.org/10.35457/konstruk.v15i1.2618
Rasool, A., & Marlina, D. (2023). Examining construct validity using Rasch model for assessing problem-solving and critical thinking. Frontiers in Psychology, 14, 1236640. https://doi.org/10.3389/fpsyg.2023.1236640
Rizal, R., Rusdiana, D., Setiawan, W., & Siahaan, P. (2022). Learning Management System Supported Smartphone (Lms3): Online Learning Application in Physics for School Course To Enhance Digital Literacy of Pre-Service Physics Teachers. Journal of Technology and Science Education, 12(1), 191–203. https://doi.org/10.3926/JOTSE.1049
Rustaman, N. Y. (2021). Pengembangan indikator berpikir sistem dalam pendidikan sains. Universitas Pendidikan Indonesia Press.
Samsudin, A., Fratiwi, N. J., Ramalis, T. R., Aminudin, A. H., Costu, B., & Nurtanto, M. (2020). Using rasch analysis to develop multi-representation of tier instrument on newton’s law (motion. International Journal of Phychosocial Rehabilitation. https://doi.org/10.37200/IJPR/V2416/PR260865
Sari, D., Prasetyo, Z. K., & Nugraha, D. A. (2020). Analisis Reliabilitas Instrumen Kemampuan Berpikir Sistem pada Pembelajaran STEM. Jurnal Pendidikan Sains Indonesia, 8(2), 123–133. https://doi.org/10.15294/jpsi.v8i2.27489
Schleicher, A. (2019). Insights and interpretations. In PISA. Retrieved from https://shorturl.asia/PTXjV
Setiyowati, D., Rochmad, R., & Kartono, K. (2020). The Application of Rasch Model in Measuring Unidimensionality of Assessment Instruments. International Journal of Instruction, 13(2), 231–246. https://doi.org/10.29333/iji.2020.13216a
Suh, J., & Jang, H. (2023). Improving the accuracy of ability estimation through high-discrimination items in the Rasch model. Educational Assessment, 28(3), 245–260. https://doi.org/10.1080/10627197.2023.2210456
Sumintono, B., & Widhiarso, W. (2015). Aplikasi Model Rasch untuk Penelitian Ilmu-Ilmu Sosial dan Pendidikan. Trim Komunikata Publishing House.
Tennant, A., & Conaghan, P. G. (2023). The Rasch Measurement Model in Health Outcomes Measurement: What It Is and Why Use It? Clinical and Experimental Rheumatology, 41(2), 123–135. https://doi.org/10.3389/fresc.2023.1208670
Tutz, G. (2022). Item response models with non-normal latent trait distributions: consequences and diagnostics. Psychometrika, 87(4), 1041–1068. https://doi.org/10.1007/s11336-022-09871-0
UNESCO. (2021). Reimagining Our Futures Together: A New Social Contract for Education. UNESCO Publishing. Retrieved from https://unesdoc.unesco.org/ark:/48223/pf0000379707
Uto, M. (2024). Linking essay-writing tests using many-facet models and automated scoring approaches. Behavior Research Methods, 56(8), 8450–8479. https://doi.org/10.3758/s13428-024-02485-2
Winarti, A., & Al-Mubarak, M. (2020). Rasch Modeling: A Multiple Choice Chemistry Test. Indonesian Journal on Learning and Advanced Education (IJOLAE). https://doi.org/10.23917/ijolae.v2i1.8985
License
Copyright (c) 2025 Ade Supriatno, Ida Kaniawati, Irma Rahma Suwarma
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with Jurnal Penelitian Pendidikan IPA, agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution 4.0 International License (CC-BY License). This license allows authors to use all articles, data sets, graphics, and appendices in data mining applications, search engines, web sites, blogs, and other platforms by providing an appropriate reference. The journal allows the author(s) to hold the copyright without restrictions and will retain publishing rights without restrictions.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in Jurnal Penelitian Pendidikan IPA.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
