Development of PISA-based Chemistry Test Instrument for Measuring Madrasah Aliyah Students’ Scientific Literacy
DOI:
10.29303/jppipa.v11i11.12808Published:
2025-11-25Downloads
Abstract
This study aims to produce a valid and reliable PISA-based chemistry test instrument on the acid–base concept to measure Madrasah Aliyah students’ scientific literacy. The test instrument was developed using the ADDIE model (analyze, design, develop, implement, and evaluate). A total of 25 multiple-choice questions were tested on 60 Grade XII students majoring in natural sciences at Madrasah Aliyah in Subang. From the Rasch analysis, a Cronbach's alpha value of 0.71 indicates that the reliability of the test instrument is acceptable. In addition, the person reliability value of 0.67 indicates that the respondents' answers were quite consistent, and the item reliability value of 0.83 indicates that the reliability of the questions was good. All questions were declared valid because they met the outfit MNSQ, outfit ZSTD, and Pt-Measure Correlation criteria. The level of difficulty of the items was distributed into five very easy items, seven easy items, nine difficult items, and four very difficult items. The discriminating power of the items was classified into 24 items in the good category and one item in the sufficient category. In addition, 21 items had well-functioning distractors, while four items had distractors that did not function well.
Keywords:
PISA-based test instrument scientific literacy ADDIE Rasch modelReferences
Afriani, E., Susilaningsih, E., Haryani, S., & Prasetya, A. T. (2023). Analisis Kompetensi Minimum Siswa pada Materi Hidrolisis Garam melalui Pengembangan Instrumen Tes Bermuatan AKM dengan Konteks Saintifik Daily Life. Chemistry in Education, 12(2), 162-170. https://doi.org/10.15294/chemined.v12i2.69259
Anggraeni, C., Permanasari, A., & Heliawati, L. (2022). Students' Scientific Literacy in Chemistry Learning through Collaborative Techniques as a Pillar of 21st-Century Skills. Journal of Innovation in Educational and Cultural Research, 3(3), 457-462. https://doi.org/10.46843/jiecr.v3i3.162
Anggraini, K. E., & Setianingsih, R. (2022). Analisis Kemampuan Numerasi Siswa SMA dalam Menyelesaikan Soal Asesmen Kompetensi Minimum (AKM). Jurnal Mathedunesa, 11(3), 837-849. https://doi.org/10.26740/mathedunesa.v11n3.p837-849
Ardianti, R., Surahman, E., & Sujarwanto, E. (2022). Pengembangan Instrumen Penilaian Literasi Sains pada Bahasan Usaha dan Energi di Madrasah Aliyah. Journal for Physics Education and Applied Physics, 4(1), 9-14. https://doi.org/10.37058/diffraction.v4i1.5359
Arikunto, S. (2006). Prosedur Penelitian Suatu Pendekatan Praktik. Jakarta: PT Rineka Cipta.
Arikunto, S. (2009). Dasar-dasar Evaluasi Pendidikan Edisi. Jakarta: PT Bumi Aksara.
Baghaei, P. (2008). The Rasch Model as a Construct Validation Tool. Rasch measurement transactions, 22(1), 1145-1162.
Bagiyono. (2017). Analisis Tingkat Kesukaran dan Daya Pembeda Soal Ujian Pelatihan Radiografi Tingkat 1. Widyanuklida, 16(1), 1–12. http://jurnal.batan.go.id/index.php/widyanuklida/article/view/4068.
Branch, R. M. (2009). Instructional Design: The ADDIE Approach. London: Springer.
Broietti, F. C. D., Nora, P. S., & Costa, S. L. R. (2019). Dimensions of Science Learning: A Study on PISA Test Questions Involving Chemistry Content. Acta Scientiae, 21(1), 95-115. https://doi.org/10.17648/acta.scientiae.v21iss1id4947
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Sciences (3rd ed.). New York: Routledge.
Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch Analysis in the Human Sciences. London: Springer.
Boone, W. J., & Staver, J. R. (2020). Advances in Rasch Analyses in the Human Sciences. London: Springer International Publishing.
Chamisijatin, L., Pantiwati, Y., & Zaenab, S. (2022). Pendampingan Peningkatan Mutu Satuan Pendidikan melalui Penyusunan Tiga Instrumen Utama di SMP Muhammadiyah 02 Kota Batu. Jurnal Abdimas (Journal of Community Service), 4(2), 249–260. https://doi.org/10.36312/sasambo.v4i2.673
Chang, R. (2008). General Chemistry: The Essential Concept. (5th ed.). New York: Mc Graw Hill Company.
Chong, J., Mokshein, S. E., & Mustapha, R. (2023). Instrument Validation based on the Six Aspects of Messick Validity Framework using the Rasch Rating Scale Model. Publication at: https://www.researchgate.net/publication/366956609
Daud, K.A.M., Khidzir, N.Z., Ismail, A.R, & Abdullah, F.A. (2018). Validity and Reliability of Instrument to Measure Social Media Skills among Small and Medium Entrepreneurs at Pengkalan Datu River. International Journal of Development and Sustainability, 7(3), 1026-1037.
Davis, L. L. (1992). Instrument Review: Getting the Most from a Panel of Experts. Applied Nursing Research, 5(4), 194–197. https://doi.org/10.1016/S0897-1897(05)80008-4
Djaen, N., Rahayu, S., Yahmin, & Muntholib. (2021). Chemical Literacy of First-Year Students on Carbon Chemistry. Jurnal Pembelajaran Kimia, 6(1), 41-62. https://doi.org/10.17977/um026v6i12021p041
Firman, H. (2000). Penilaian Hasil Belajar dalam Pengajaran Kimia. Bandung: Jurusan Pendidikan Kimia FPMIPA UPI.
Gignac, G.E. (2009). Psychometrics and the Measurement of Emotional Intelligence. In: Parker, J., Saklofske, D., & Stough, C. (Eds.), Assessing Emotional Intelligence (pp. 9-40). The Springer Series on Human Exceptionality. Boston: Springer. https://doi.org/10.1007/978-0-387-88370-0_2
Gliem, J.A., & Gliem, R.R. (2003). Calculating. Interpreting, and Reporting Cronbach's Alpha Reliability Coefficient for Likert-Type Scales (Eds). Midwest Research to Practice Conference in Adult, Continuing, and Community Education (pp. 82-88). Ohio : Ohio State University.
Gurel, D. K., Eryilmaz, A., & McDermott, L. C. (2015). A Review and Comparison of Diagnostic Instruments to Identify Students’ Misconceptions in Science. Eurasia Journal of Mathematics, Science and Technology Education, 11(5), 989–1008. https://doi.org/10.12973/eurasia.2015.1369a
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education, 15(3), 309–333. https://doi.org/10.1207/S15324818AME1503_5
Hailaya, W., Alagumalai, S., & Ben, F. (2014). Examining the Utility of Assessment Literacy Inventory and its Portability to Education Systems in the Asia Pacific Region. Australian Journal of Education, 58(3), 297–317. https://doi.org/10.1177/0004944114542984
Kalkbrenner, M. T. (2021). A Practical Guide to Instrument Development and Score Validation in the Social Sciences: The Measure Approach. Practical Assessment, Research, and Evaluation, 26(1) , 1-18. https://doi.org/10.7275/svg4-e671
Kemenag. (2022). Framework Asesmen Kompetensi Madrasah Indonesia (AKMI) 2022. Jakarta: Direktorat KSKK Madrasah, Ditjen Pendis Kemenag RI.
Kemendikbud. (2020). AKM dan Implikasinya pada Pembelajaran. Jakarta: Pusat Asesmen dan Pembelajaran, Badan Penelitian, Pengembangan dan Perbukuan, Kementrian Pendidikan dan Kebudayaan.
Kemendikbudristek. (2023). Capaian Pembelajaran untuk SMA/MA/Program Paket C pada Kurikulum Merdeka. Jakarta: Badan Standar, Kurikulum, dan Asesmen Pendidikan, Pusat Kurikulum dan Pembelajaran.
Lawshe, C.H. (1975). A Quantitative Approach to Content Validity. Personnel Psychology, 28(4), 563-575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
Maharani, S. D., Suganda, V. A., Laihat, M., Harini, B., Pulungan, M., & Safitri, M. L. O. (2022). Asesmen Pembelajaran di Sekolah Dasar. Palembang: Bening Media Publishing.
Mufidah, N., & Ardhana, I. A. (2024). Science Literacy Profile of Eleventh-Grade High School Students in Acid-Base. Jurnal Pembelajaran Kimia, 9(1), 40-45. http://dx.doi.org/10.17977/um026v9i12024p%25p
Muntholib, M., Khusmawardani, E., Utomo, Y., Muchson, M., & Yahmin, Y. (2020). Development and Implementation of Multiple-choice Chemical Literacy Survey in Acid-Base Chemistry. AIP Conference Proceedings of The 3rd International Conference on Mathematics and Science Education (ICoMSE). https://doi.org/10.1063/5.0000547
Muntholib, M., Ibnu, S., Rahayu, S., Fajaroh, F., Kusairi, S., & Kuswandi, B. (2020). Chemical Literacy: Performance of First Year Chemistry Students on Chemical Kinetics. Indonesian Journal of Chemistry, 20(2), 468-482. https://doi.org/10.22146/ijc.43651
Nahadi, N., Siswaningsih, W., Purnawarman, P., Lestari, T., Febriani, A. E., & Rohmawati T. (2022). Development of Minimum Competency Assessment (AKM) on Chemical Materials. Moroccan Journal of Chemistry, 10(3), 452-463. https://doi.org/10.48317/IMIST.PRSM/morjchem-v10i3.33067
Nurdini, N., Suhandi, A., Ramalis, T. R., A. Samsudin, A., Fratiwi, N. J., & Costu, B. (2020). Developing Multitier Instrument of Fluids Concepts (MIFO) to Measure Student’s Conception: A Rasch Analysis Approach. Journal of Advanced Research in Dynamical and Control Systems, 12(6), 3069–3083. https://doi.org/10.5373/jardcs/v12i6/s20201273
OECD. (2019). PISA 2018 Assessment and Analytical Framework. Paris: OECD Publishing.
OECD. (2023). PISA 2025 Science Framework (Draft). Paris: OECD Publishing.
Ovira, E. D. (2018). Pengembangan dan Validasi Tes Kimia dengan Framework PISA pada Materi Kelas XI Semester 1. MENARA Ilmu, 12(1), 33-41. https://doi.org/10.33559/mi.v12i80.641
Petrucci, R. H., Herring, F. G., Madura, J. D., & Bissonnette, C. (2017). General Chemistry : Principles and Modern Applications. (11th ed.). Kanada: Pearson.
Purniasari, L., Masykuri, M., & Ariani, S. R. D. (2021). Analisis Butir Soal Ujian Sekolah Mata Pelajaran Kimia SMAN 1 Kutowinangun Tahun Pelajaran 2019/2020 Menggunakan Model Iteman dan Rasch. Jurnal Pendidikan Kimia, 10(2), 205–214. https://doi.org/10.20961/jpkim.v10i2.48244
Putri, A. A. A., Hussain, H., & Ramdhani. (2022). Pengembangan Instrumen Tes Literasi Sains pada Dimensi Pengetahuan Materi Asam Basa. Jurnal Inovasi Pendidikan Matematika dan IPA, 2(4), 536-547. https://doi.org/10.51878/SCIENCE.V2I4.1797
Polit, D. F., & Beck, C. T. (2006). The Content Validity Index: Are You Sure You Know What's Being Reported? Critique and Recommendations. Research in Nursing & Health, 29(5), 489–497. https://doi.org/10.1002/nur.20147
Ridwan et al. (2013). Development of a Student Scientific Literacy Test Instrument on the Topic of Diversity of Living Creatures. Journal of Biology Education and Learning, 4(1), 71–78.
Rusiyah., Eraku, S. S., & Supadmi, S. (2020). Analisis Soal Ujian Akhir Semester Mata Pelajaran Geografi Dengan Menggunakan Pemodelan Rasch. Jurnal Geografi Dan Pembelajaran Geografi, 5(1), 11. https://doi.org/10.31851/swarnabhumi.v5i1.4136
Saputri, E. N., Wigati, I., & Laksono, P. J. (2022). Kemampuan Literasi Kimia pada Aspek Kompetensi Sains pada Materi Asam Basa. Prosiding Seminar Nasional Pendidikan Kimia, 223–231.
Shofiyah, N., & Sartika, S. B. (2018). Buku Ajar Mata Kuliah Asesmen Pembelajaran. Sidoarjo: Umsida Press.
Sudirman, Rusilowati, A., & Susilaningsih, E. (2024). Development of Multiple-Choice Test Instruments to Improve Scientific Literacy in Madrasah Aliyah (MA). International Journal of Scientific Research and Management, 12(6), 3465-3475. https://doi.org/10.18535/ijsrm/v12i06.el04
Sumintono, B., & Widhiarso, W. (2015). Aplikasi Pemodelan Rasch pada Assessment Pendidikan. Bandung: Trim Komunikata.
Suryadi, A. (2024). Pemanfaatan Hasil AKMI untuk Pembelajaran Berkelanjutan di Madrasah. Jurnal Pelita Manajemen Pendidikan, 1(1), 1-8. https://doi.org/10.65226/jpmp.v1i1.1
Susanti, L. D., Pahrudin, A., & Yetri, Y. (2021). Analisis Pelaksanaan Asesmen Kompetensi Madrasah Indonesia (AKMI). Journal of Interdisciplinary Science and Education, 1(2), 17-24. https://doi.org/10.70371/jise.v1i2.23
Susanto, E., Susanta, A., & Rusdi, R. (2022). Pelatihan Penyusunan Instrumen Tes Matematika Online Berbasis PISA bagi Guru Matematika SMP Bengkulu. Jurnal Pengabdian Masyrakat, 2(3), 114−120. https://doi.org/10.47065/jpm.v2i3.330
Thornhill-Miller, B., Camarda, A., Mercier, M., Burkhardt, J.-M., Morisseau, T., Bourgeois-Bougrine, S., Vinchon, F., El Hayek, S., Augereau-Landais, M., Mourey, F., Feybesse, C., Sundquist, D., & Lubart, T. (2023). Creativity, Critical Thinking, Communication, and Collaboration: Assessment, Certification, and Promotion of 21st Century Skills for the Future of Work and Education. Journal of Intelligence, 11(54), 1-32. https://doi.org/10.3390/jintelligence11030054.
Ultay, N., & Calik, M. (2016). A comparison of different teaching designs of 'acids and bases' subject. Eurasia Journal of Mathematics, Science & Technology Education, 12(1), 57-86. https://doi.org/10.12973/eurasia.2016.1422a
Vera, R., Kaniawati, I., & Utama, J. A. (2023). Using the Rasch Model to Develop a Measure of Students’ Problem Solving Ability in Optical Instruments. Jurnal Pendidikan MIPA, 24(2), 419-431. http://dx.doi.org/10.23960/jpmipa/v24i2.pp419-431
Wilson, F. R., Pan, W., & Schumsky, D. A. (2012). Recalculation of the Critical Values for Lawshe’s Content Validity Ratio. Measurement and Evaluation in Counseling and Development, 45(3), 197-210. https://doi.org/10.1177/0748175612440286
Yuliani, G., Dianhar, H., & Suhendar, A. (2022). Kimia untuk SMA/MA Kelas XII. Jakarta: Kementerian Pendidikan, Kebudayaan, Riset, dan Teknologi.
Zhang, L., et al. (2023). Development and Validation of an Instrument for Assessing Scientific Literacy from Junior to Senior High School. Disciplinary and Interdisciplinary Science Education Research, 5(21), 1-15. https://doi.org/10.1186/s43031-023-00093-2
License
Copyright (c) 2025 Winda Ariani, Nahadi, Heli Siti Halimatul Munawaroh
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with Jurnal Penelitian Pendidikan IPA, agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution 4.0 International License (CC-BY License). This license allows authors to use all articles, data sets, graphics, and appendices in data mining applications, search engines, web sites, blogs, and other platforms by providing an appropriate reference. The journal allows the author(s) to hold the copyright without restrictions and will retain publishing rights without restrictions.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in Jurnal Penelitian Pendidikan IPA.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
