Evaluating Artificial Intelligence in Solving Physics Problems: A Comprehensive Analysis of Essay and Multiple-Choice Questions on Electrical Circuits
DOI:
10.29303/jppipa.v12i3.14494Published:
2026-03-25Downloads
Abstract
This study aims to examine the performance of artificial intelligence, specifically ChatGPT, in solving physics problems on electrical circuits by comparing essay and multiple-choice formats. The study employed a research and development approach combined with descriptive qualitative analysis to develop and validate equivalent test items with identical content and cognitive demands. The validated items were administered to ChatGPT, and its responses were analyzed based on accuracy and problem-solving processes. The results show that ChatGPT achieved higher accuracy in multiple-choice questions (100%) compared to essay questions (60%). Errors in essay responses were primarily conceptual and occurred in image-based and structurally complex circuit problems, particularly during the initial interpretation stage. These findings indicate that question format significantly influences AI performance, with multiple-choice questions promoting more structured reasoning while potentially overestimating AI capability. In conclusion, combining essay and multiple-choice formats provides a more comprehensive evaluation of AI problem-solving performance in physics education.
Keywords:
Artificial intelligence ChatGPT Electrical circuits Essay question Multiple-choice questionReferences
Agyare, B., Asare, J., Kraishan, A., Nkrumah, I., & Adjekum, D. K. (2025). A cross-national assessment of artificial intelligence (AI) Chatbot user perceptions in collegiate physics education. Computers and Education: Artificial Intelligence, 8, 100365. https://doi.org/10.1016/j.caeai.2025.100365 DOI: https://doi.org/10.1016/j.caeai.2025.100365
Aji, D. P., Akhsan, H., & Marlina, L. (2025). Development of Augmented Reality-Based and Artificial Intelligence-Assisted E-Modules on Global Warming Materials to Improve Critical Thinking Skills of High School Learners. JPPIPA: Jurnal Penelitian Pendidikan IPA, 11(2), 696–707. https://doi.org/10.29303/jppipa.v11i2.9993 DOI: https://doi.org/10.29303/jppipa.v11i2.9993
Ayuni, R. T., Jaedun, A., Zafrullah, Z., & Ramadhani, A. M. (2024). Trends in the Use of Artificial Intelligence in Science Education: Bibliometric & Biblioshiny Analysis (1975-2024). JPPIPA: Jurnal Penelitian Pendidikan IPA, 10(10), 740–756. https://doi.org/10.29303/jppipa.v10i10.7846 DOI: https://doi.org/10.29303/jppipa.v10i10.7846
Azap, S. (2025). The role of artificial intelligence in language teacher education: EFL learners’ views on the concept of “ChatGPT” through metaphorical analysis. Social Sciences & Humanities Open, 11, 101394. https://doi.org/10.1016/j.ssaho.2025.101394 DOI: https://doi.org/10.1016/j.ssaho.2025.101394
Bates, T., Cobo, C., Mariño, O., & Wheeler, S. (2020). Can artificial intelligence transform higher education? International Journal of Educational Technology in Higher Education, 17(42), 1–12. https://doi.org/10.1186/s41239-020-00218-x DOI: https://doi.org/10.1186/s41239-020-00218-x
Berliana, J. I., Winantoro, R. C., & Damariswara, R. (2024). Analisis Penggunaan AI dalam Mengerjakan Tugas pada Mahasiswa Universitas Nusantara PGRI Kediri Prodi PGSD Kelas 1C. Jurnal Review Pendidikan Dan Pengajaran (JRPP), 7(3), 9724–9731. https://doi.org/10.31004/jrpp.v7i3.30459
Cahyaningrum, I. Y., Fuady, A., & Sunismi, S. (2023). Analisis Butir Soal Sumatif Akhir Semester Ganjil Mata Pelajaran Matematika Kelas VII dengan Berbantuan Aplikasi Software Anates. Mathema Journal, 5(2), 67–81. https://doi.org/10.33365/jm.v5i2.2601 DOI: https://doi.org/10.33365/jm.v5i2.2601
Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20(22), 1–22. https://doi.org/10.1186/s41239-023-00392-8 DOI: https://doi.org/10.1186/s41239-023-00392-8
Firman, F., & Firdaus, F. (2023). Efektivitas Artificial Intelligence (AI) dalam Menjawab Soal Higher Order Thinking Skill (HOTS). Indonesian Journal of Educational Science (IJES), 6(1), 15–30. https://doi.org/10.31605/ijes.v6i1.2899
Hariadi, A. F., Degeng, I. N. S., Setyosari, P., Ulfa, S., & Alhakim, F. F. (2025). The Effect of Hybrid Learning Method Assisted by Artificial Intelligence Camera Controller and Student Engagement on Learning Outcomes. JPPIPA: Jurnal Penelitian Pendidikan IPA, 11(9), 777–786. https://doi.org/10.29303/jppipa.v11i9.12034 DOI: https://doi.org/10.29303/jppipa.v11i9.12034
Lin, C., Huang, A. Y. Q., & Lu, O. H. T. (2023). Artificial intelligence in intelligent tutoring systems toward sustainable education: A systematic review. Smart Learning Environments, 10(41), 1–22. https://doi.org/10.1186/s40561-023-00260-y DOI: https://doi.org/10.1186/s40561-023-00260-y
Mutakin, D. (2023). Pemanfaatan Software SPSS untuk Analisis Instrumen Penelitian dalam Penelitian Tindakan Kelas. Jurnal Inovasi Dan Manajemen Pendidikan, 3(1), 33–49. https://doi.org/10.12928/jimp.v3i1.7727 DOI: https://doi.org/10.12928/jimp.v3i1.7727
Nam, B. H., & Bai, Q. (2023). ChatGPT and its ethical implications for STEM research and higher education: A media discourse analysis. International Journal of STEM Education, 10(66), 1–24. https://doi.org/10.1186/s40594-023-00452-5 DOI: https://doi.org/10.1186/s40594-023-00452-5
Niu, Z., Tan, Z.-M., Yu, H., Gu, J.-F., Chen, G., & Huang, W. (2025). Typhoon Science Meets Artificial Intelligence: A Roundtable on Bridging Physics Based and Data-Driven Paradigms. Tropical Cyclone Research and Review, 14(3), 311–316. https://doi.org/10.1016/j.tcrr.2025.08.006 DOI: https://doi.org/10.1016/j.tcrr.2025.08.006
Nurhuda, D., Kumala, S. A., & Widiyatun, F. (2023). Analisis Kecerdasan Buatan ChatGPT dalam Penyelesaian Soal Fisika Bergambar pada Materi Resistor. Jurnal Luminous: Riset Ilmiah Pendidikan Fisika, 4(2), 62–70. https://doi.org/10.31851/luminous.v4i2.12232 DOI: https://doi.org/10.31851/luminous.v4i2.12232
Özçelik, N. P., & Ekşi, G. Y. (2024). Cultivating writing skills: The role of ChatGPT as a learning assistant—A case study. Smart Learning Environments, 11(10), 1–18. https://doi.org/10.1186/s40561-024-00296-8 DOI: https://doi.org/10.1186/s40561-024-00296-8
Pairunan, A. S., Nurjannah, N., Haeruddin, H., Wahyono, U., & Pasaribu, M. (2025). Development of Artificial Intelligence Based E-Learning Modules to Improve Students’ Critical Thinking and Communication Skills. JPPIPA: Jurnal Penelitian Pendidikan IPA, 11(9), 993–1005. https://doi.org/10.29303/jppipa.v11i9.12050 DOI: https://doi.org/10.29303/jppipa.v11i9.12050
Prananta, A. W., Susanto, N., Purwantoro, A., & Fuadah, N. (2023). ChatGPT Artificial Intelligence Integration in Science Learning Media: Systematic Literature Review. JPPIPA: Jurnal Penelitian Pendidikan IPA, 9(7), 315–321. https://doi.org/10.29303/jppipa.v9i7.4386 DOI: https://doi.org/10.29303/jppipa.v9i7.4386
Prasetya, F., Fortuna, A., Samala, A. D., Latifa, D. K., Andriani, W., Gusti, U. A., … García, J. L. C. (2025). Harnessing artificial intelligence to revolutionize vocational education: Emerging trends, challenges, and contributions to SDGs 2030. Social Sciences & Humanities Open, 11, 101401. https://doi.org/10.1016/j.ssaho.2025.101401 DOI: https://doi.org/10.1016/j.ssaho.2025.101401
Purba, S., Waluyo, B. D., Pangaribuan, W., Zulkarnain, S. A. B., & Astrid, E. (2025). Development and Implementation of AI-Driven Learning Media for Low-Power Inverter Mastery Using the ADDIE Model. JPPIPA: Jurnal Penelitian Pendidikan IPA, 11(8), 1098–1109. https://doi.org/10.29303/jppipa.v11i8.11573 DOI: https://doi.org/10.29303/jppipa.v11i8.11573
Purnama, I., Wijaya, R. F., Harahap, A., & Edi, F. (2025). Artificial Intelligence as a Science Teacher Assistant: An Analysis of Machine Learning Utilization in Diagnosing Student Misconceptions: A Review. JPPIPA: Jurnal Penelitian Pendidikan IPA, 11(12), 1–7. https://doi.org/10.29303/jppipa.v11i12.13089 DOI: https://doi.org/10.29303/jppipa.v11i12.13089
Puspitasari, W. D., & Febrinita, F. (2021). Pengujian Validasi Isi (Content Validity) Angket Persepsi Mahasiswa terhadap Pembelajaran Daring Matakuliah Matematika Komputasi. Journal Focus Action of Research Mathematic (Factor M), 4(1), 77–90. https://doi.org/10.30762/factor_m.v4i1.3254 DOI: https://doi.org/10.30762/factor_m.v4i1.3254
Salas‑Pilco, S. Z., & Yang, Y. (2022). Artificial intelligence applications in Latin American higher education: A systematic review. International Journal of Educational Technology in Higher Education, 19(21), 1–20. https://doi.org/10.1186/s41239‑022‑00326‑w DOI: https://doi.org/10.1186/s41239-022-00326-w
Schmidt, D. A., Alboloushi, B., Thomas, A., & Magalhaes, R. (2025). Integrating artificial intelligence in higher education: Perceptions, challenges, and strategies for academic innovation. Computers and Education Open, 9, 100274. https://doi.org/10.1016/j.caeo.2025.100274 DOI: https://doi.org/10.1016/j.caeo.2025.100274
Shahzad, M. F., Xu, S., & Javed, I. (2024). ChatGPT awareness, acceptance, and adoption in higher education: The role of trust as a cornerstone. International Journal of Educational Technology in Higher Education, 21(46), 1–23. https://doi.org/10.1186/s41239-024-00478-x DOI: https://doi.org/10.1186/s41239-024-00478-x
Smith-Mutegi, D., Mamo, Y., Kim, J., Crompton, H., & McConnell, M. (2025). Perceptions of STEM education and artificial intelligence: A Twitter (X) sentiment analysis. International Journal of STEM Education, 12(9), 1–18. https://doi.org/10.1186/s40594‑025‑00527‑5 DOI: https://doi.org/10.1186/s40594-025-00527-5
Sperling, K., Stenberg, C.-J., McGrath, C., Åkerfeldt, A., Heintz, F., & Stenliden, L. (2024). In search of artificial intelligence (AI) literacy in teacher education: A scoping review. Computers and Education Open, 6, 100169. https://doi.org/10.1016/j.caeo.2024.100169 DOI: https://doi.org/10.1016/j.caeo.2024.100169
Sugiyono, S. (2021). Metode Penelitian Kualitatif. Bandung: Alfabeta.
Wang, S., Wang, F., Zhu, Z., Wang, J., Tran, T., & Du, Z. (2024). Artificial intelligence in education: A systematic literature review. Expert Systems With Applications, 252, 124167. https://doi.org/10.1016/j.eswa.2024.124167 DOI: https://doi.org/10.1016/j.eswa.2024.124167
Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21(40), 1–22. https://doi.org/10.1186/s41239-024-00468-z DOI: https://doi.org/10.1186/s41239-024-00468-z
Yusof, M. S. B. (2019). ABC of Content Validation and Content Validity Index Calculation. Education in Medicine Journal, 11(2), 49–54. https://doi.org/10.21315/eimj2019.11.2.6 DOI: https://doi.org/10.21315/eimj2019.11.2.6
Zawacki-Richter, O., Bai, J. Y. H., Lee, K., Tryon, P. J. S. van, & Prinsloo, P. (2024). New advances in artificial intelligence applications in higher education? International Journal of Educational Technology in Higher Education, 21(32), 1–4. https://doi.org/10.1186/s41239-024-00464-3 DOI: https://doi.org/10.1186/s41239-024-00464-3
Zulpan, Z., Sahrul, S., Yusron, A., Hanida, R. S., & Marfu’ah, S. (2024). Valitidas Tes Objektif Bentuk Pilihan Ganda. Jurnal Pendidikan Matematika, 1(3), 1–10. https://doi.org/10.47134/ppm.v1i3.812 DOI: https://doi.org/10.47134/ppm.v1i3.812
License
Copyright (c) 2026 Aji Saputra, Muliadi, Hutri Handayani Isra
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with Jurnal Penelitian Pendidikan IPA, agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution 4.0 International License (CC-BY License). This license allows authors to use all articles, data sets, graphics, and appendices in data mining applications, search engines, web sites, blogs, and other platforms by providing an appropriate reference. The journal allows the author(s) to hold the copyright without restrictions and will retain publishing rights without restrictions.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in Jurnal Penelitian Pendidikan IPA.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
