Evaluating AI Detection Performance Across Writing Modalities in Engineering Interaction
DOI:
10.29303/jppipa.v12i4.9637Published:
2026-04-30Downloads
Abstract
This study proposes a conceptual framework for AI-based detection of handwritten and technical English writing in electrical engineering students, integrating multiple input modalities, including handwritten, typed, hybrid, and transcribed texts. A neural network–based detection system was evaluated using classification metrics and statistical analysis. The results demonstrate that detection accuracy varies significantly across modalities, ranging from 0.72 to 0.89, with typed and transcribed texts achieving higher consistency. Mean detection scores increased progressively from handwritten (0.41-0.50) to typed (0.52-0.58), hybrid (0.60-0.66), and transcribed texts (0.68-0.76), indicating the influence of text standardization. Variability was highest in handwritten and hybrid categories, with standard deviation reaching 0.10–0.11 and wider confidence intervals (e.g., 0.36-0.56), reflecting classification uncertainty. Heatmap analysis further shows that precision and recall in transcribed texts can exceed 0.85-0.89, while handwritten texts drop below 0.45 in some cases. The findings reveal that L2 writing characteristics and technical language significantly affect detection outcomes, leading to potential misclassification. This study highlights the need for context-aware AI detection systems that incorporate modality-specific features, ensuring fairness and reliability in engineering education.
Keywords:
AI detection Education Engineering L2 writing Writing modalityReferences
Adi, S. P., Simanjuntak, A. V. H., Supendi, P., Daryono, D., Prakoso, T. A., & Muksin, U. (2024). Different Faulting of the 2023 (Mw 5.7 and 5.9) South-Central Java Earthquakes in the Backthrust Fault System. Geotechnical and Geological Engineering, 42, 3123–3135. https://doi.org/10.1007/s10706-023-02720-1 DOI: https://doi.org/10.1007/s10706-023-02720-1
Anggriani, T., Ginting, C. N., Chiuman, L., & Rosari, A. (2024). Potential Test of Jojoba Oil-Based Sunscreen in Lotion Preparations. International Journal of Health and Pharmaceutical (IJHP), 4(2), 235–241. https://doi.org/10.51601/ijhp.v4i2.150 DOI: https://doi.org/10.51601/ijhp.v4i2.249
Ansari, K., Walo, J., Simanjuntak, A. V., & Wezka, K. (2024). Evaluation of Recent Tectonic Movement in Northeast Japan Using Long-Term GNSS and Tide Gauge Measurements. Journal of Structural Geology, 188, 105258. https://doi.org/10.1016/j.jsg.2024.105258 DOI: https://doi.org/10.1016/j.jsg.2024.105258
Asmiddin, A. M., Nurhuda, P., Megahati S, R. R. P., & Susanto, N. (2023). Advantages and Disadvantages of ChatGPT in Science Learning: A Systematic Literature Review. Jurnal Penelitian Pendidikan IPA, 9(12), 1335–1341. https://doi.org/10.29303/jppipa.v9i12.6576 DOI: https://doi.org/10.29303/jppipa.v9i12.6576
Asnawi, Y., Simanjuntak, A., Muksin, U., Okubo, M., Putri, S. I., Rizal, S., & Syukri, M. (2022). Soil Classification in a Seismically Active Environment Based on Shear Wave Velocity and HVSR Data. Global Journal of Environmental Science and Management, 8(3), 297–314. https://doi.org/10.22034/gjesm.2022.03.01
Banjarnahor, S., Simatupang, L. L., Anggriani, T., Sinaga, R. M., Harahap, S. Y., & Handoko, E. (2025). Effectiveness of a Mobile Application Empowerment Educational Program on Nurses’ Attitudes and Competencies in Patient Safety. Gema Lingkungan Kesehatan, 23(4), 566–573. https://doi.org/10.36568/gelinkes.v23i4.273 DOI: https://doi.org/10.36568/gelinkes.v23i4.273
Biber, D., & Gray, B. (2016). Grammatical Complexity in Academic English: Linguistic Change in Writing. Cambridge University Press. https://doi.org/10.1017/CBO9780511920776 DOI: https://doi.org/10.1017/CBO9780511920776
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021). All That’s “Human” Is Not Gold: Evaluating Human Evaluation of Generated Text. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 7282–7296. https://doi.org/10.18653/v1/2021.acl-long.565 DOI: https://doi.org/10.18653/v1/2021.acl-long.565
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and Cheating: Ensuring Academic Integrity in the Era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148 DOI: https://doi.org/10.1080/14703297.2023.2190148
Crossley, S. A., & McNamara, D. S. (2012). Predicting Second Language Writing Proficiency: The Roles of Cohesion and Linguistic Sophistication. Journal of Research in Reading, 35(2), 115–135. https://doi.org/10.1111/j.1467-9817.2010.01449.x DOI: https://doi.org/10.1111/j.1467-9817.2010.01449.x
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M., Alalwan, A. A., Al-Emran, M., Al-Hawari, M. A., Al-Qaysi, N., & Wamba, S. F. (2023). So What If ChatGPT Wrote It? Multidisciplinary Perspectives on Opportunities, Challenges and Implications of Generative Conversational AI for Research, Practice and Policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642 DOI: https://doi.org/10.1016/j.ijinfomgt.2023.102642
Fütterer, T., Fischer, C., Alekseeva, A., Chen, X., Tate, T., Warschauer, M., & Gerjets, P. (2023). ChatGPT in Education: Global Reactions to AI Innovations. Scientific Reports, 13, 15310. https://doi.org/10.1038/s41598-023-42227-6 DOI: https://doi.org/10.1038/s41598-023-42227-6
Gehrmann, S., Strobelt, H., & Rush, A. M. (2019). GLTR: Statistical Detection and Visualization of Generated Text. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 111–116. https://doi.org/10.18653/v1/P19-3019 DOI: https://doi.org/10.18653/v1/P19-3019
Ippolito, D., Duckworth, D., Callison-Burch, C., & Eck, D. (2020). Automatic Detection of Generated Text is Easiest When Humans Are Fooled. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1, 1808–1822. https://doi.org/10.18653/v1/2020.acl-main.164 DOI: https://doi.org/10.18653/v1/2020.acl-main.164
Kadir, A., Rampengan, D. D., Simanjuntak, A. V. H., Anggriani, T., Nauval, M. D., Adnan, M. I., Lastri, S., & Iqhrammullah, M. (2026). Economic Burden of Work-Related Low Back Pain in Indonesia Before and During the COVID-19 Era, 2019–2021: Analysis of Global Burden of Disease Estimates. ClinicoEconomics and Outcomes Research, 1–16. https://doi.org/10.2147/CEOR.S575081 DOI: https://doi.org/10.2147/CEOR.S575081
Karim, B. Q., Haryanto, H., & Susanti, E. Y. (2025). AI in Education: Transforming Student Engagement for the Digital Age. Jurnal Penelitian Pendidikan IPA, 11(2), 1127–1136. https://doi.org/10.29303/jppipa.v11i2.10469 DOI: https://doi.org/10.29303/jppipa.v11i2.10469
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., Kuhn, J., & Kasneci, G. (2023). ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274 DOI: https://doi.org/10.1016/j.lindif.2023.102274
Kyle, K., & Crossley, S. A. (2018). Measuring Syntactic Complexity in L2 Writing Using Fine-Grained Clausal and Phrasal Indices. Journal of Second Language Writing, 40, 1–12. https://doi.org/10.1016/j.jslw.2018.02.002 DOI: https://doi.org/10.1111/modl.12468
Lamonge, A. S. L., Polii, G. B., Laka, A. A. M. L., Simanjuntak, A. V. H., Tangkulung, G. G., Timbuleng, J. V. D., Anggriani, T., & Ristiyono, L. (2024). The Needs of Public Health Mitigation as the Impact of Earthquake Disasters in North Sulawesi Region, Indonesia. Journal of Geoscience, Engineering, Environment, and Technology, 9(4), 453–460. https://doi.org/10.25299/jgeet.2024.9.04.19452 DOI: https://doi.org/10.25299/jgeet.2024.9.04.19452
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT Detectors are Biased Against Non-Native English Writers. Patterns, 4(7), 100779. https://doi.org/10.1016/j.patter.2023.100779 DOI: https://doi.org/10.1016/j.patter.2023.100779
Liu, Z., Yao, Z., Li, F., & Luo, B. (2024). Check Me If You Can: Detecting ChatGPT-Generated Academic Writing Using CheckGPT. Applied Sciences, 14(2), 625. https://doi.org/10.3390/app14020625 DOI: https://doi.org/10.3390/app14020625
Mitchell, E., Lee, Y. K., Khazatsky, A., Manning, C. D., & Finn, C. (2023). DetectGPT: Zero-Shot Machine-Generated Text Detection Using Probability Curvature. Proceedings of the 40th International Conference on Machine Learning. https://doi.org/10.48550/arXiv.2301.11305
Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J. M., & López-Meneses, E. (2023). Impact of the Implementation of ChatGPT in Education: A Systematic Review. Computers, 12(8), 153. https://doi.org/10.3390/computers12080153 DOI: https://doi.org/10.3390/computers12080153
Muksin, U., Riana, E., Rudyanto, A., Bauer, K., Simanjuntak, A. V. H., & Weber, M. (2023). Neural Network-Based Classification of Rock Properties and Seismic Vulnerability. Global Journal of Environmental Science and Management, 9(1), 15–30. https://doi.org/10.22034/gjesm.2023.01.02
Pasari, S., Simanjuntak, A. V. H., Mehta, A., Neha, & Sharma, Y. (2021a). A Synoptic View of the Natural Time Distribution and Contemporary Earthquake Hazards in Sumatra, Indonesia. Natural Hazards, 108, 309–321. https://doi.org/10.1007/s11069-021-04682-0 DOI: https://doi.org/10.1007/s11069-021-04682-0
Pasari, S., Simanjuntak, A. V. H., Neha, n., Sharma, Y., & Mehta, A. (2021c). Nowcasting Earthquakes in Sulawesi Island, Indonesia. Geoscience Letters, 8, 27. https://doi.org/10.1186/s40562-021-00197-5 DOI: https://doi.org/10.1186/s40562-021-00197-5
Pasari, S., Simanjuntak, A. V., Mehta, A., & Sharma, Y. (2021b). The Current State of Earthquake Potential on Java Island, Indonesia. Pure and Applied Geophysics, 178, 2789–2806. https://doi.org/10.1007/s00024-021-02781-4 DOI: https://doi.org/10.1007/s00024-021-02781-4
Prananta, A. W., Megahati S, R. R. P., & Susanto, N. (2023a). Transforming Education and Learning Through ChatGPT: A Systematic Literature Review. Jurnal Penelitian Pendidikan IPA, 9(11), 1031–1037. https://doi.org/10.29303/jppipa.v9i11.5468 DOI: https://doi.org/10.29303/jppipa.v9i11.5468
Prananta, A. W., Susanto, N., Purwantoro, A., & Fuadah, N. (2023b). ChatGPT Artificial Intelligence Integration in Science Learning Media: Systematic Literature Review. Jurnal Penelitian Pendidikan IPA, 9(7), 315–321. https://doi.org/10.29303/jppipa.v9i7.4386 DOI: https://doi.org/10.29303/jppipa.v9i7.4386
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit Spewer or the End of Traditional Assessments in Higher Education? Journal of Applied Learning & Teaching, 6(1), 342–363. https://doi.org/10.37074/jalt.2023.6.1.9 DOI: https://doi.org/10.37074/jalt.2023.6.1.9
Rusdiawan, A., Hasriyani, E., Zeswita, A. L., Megahati S, R. R. P., & Susanto, N. (2024). Can ChatGPT Be Integrated into Blended Learning in Science: A Systematic Literature Review. Jurnal Penelitian Pendidikan IPA, 10(2), 38–44. https://doi.org/10.29303/jppipa.v10i2.7001 DOI: https://doi.org/10.29303/jppipa.v10i2.7001
Saragih, Y., Nurhuda, P., Sudirman, S., Indriati, G., & Susanto, N. (2023). ChatGPT for the Future of Science Learning: A Systematic Literature Review. Jurnal Penelitian Pendidikan IPA, 9(SpecialIssue), 143–149. https://doi.org/10.29303/jppipa.v9iSpecialIssue.6232 DOI: https://doi.org/10.29303/jppipa.v9iSpecialIssue.6232
Setiadi, T. A. P., Sirait, A. M. M., Halauwet, Y., Simanjuntak, A. V. H., & Susilo, A. (2025). Unveiling the Dynamics of Deep-Focus Earthquakes: Insights from the 2020 Jepara Event Mw 6.7 and Subduction Processes Beneath Java. Rudarsko-Geološko-Naftni Zbornik, 40(5), 57–66. https://doi.org/10.17794/rgn.2025.5.5 DOI: https://doi.org/10.17794/rgn.2025.5.5
Sidabutar, A. D., Pratama, I. H., Fadillah, Q., Angwyn, W., Rimadeni, Y., & Anggriani, T. (2025). Maternal Age at Marriage and Risk of Childhood Stunting: Evidence from a Case-Control Study in North Sumatra, Indonesia. Gema Lingkungan Kesehatan, 23(4), 549–555. https://doi.org/10.36568/gelinkes.v23i4.403 DOI: https://doi.org/10.36568/gelinkes.v23i4.403
Simanjuntak, A. V. H., Ansari, K., Mase, L. Z., Setiadi, T. A. P., & Muksin U. (2025c). Spatio-Temporal Clustering and Mechanism Analysis of the 2018 Palu Earthquake Sequence. Geotechnical and Geological Engineering, 43, 414. https://doi.org/10.1007/s10706-025-03350-5 DOI: https://doi.org/10.1007/s10706-025-03350-5
Simanjuntak, A. V. H., Palgunadi, K. H., Supendi, P., Muksin, U., Gunawan, E., Widiyantoro, S., Rawlinson, N., Daryono, M. R., Daryono, D., Karnawati, D., Hanifa, N. R., Pratama, C., & Ida, R. (2024). The Western Extension of the Balantak Fault Revealed by the 2021 Earthquake Cascade in Sulawesi, Indonesia. Geoscience Letters, 11(35). https://doi.org/10.1186/s40562-024-00353-7 DOI: https://doi.org/10.1186/s40562-024-00353-7
Simanjuntak, A. V. H., Palgunadi, K. H., Supendi, P., Daryono, D., Prakoso, T. A., & Muksin, U. (2023). New Insight on the Active Fault System in the Halmahera Volcanic Arc, Indonesia, Derived from the 2022 Tobelo Earthquakes. Seismological Research Letters, 94(6), 2586–2594. https://doi.org/10.1785/0220230006 DOI: https://doi.org/10.1785/0220230006
Simanjuntak, A. V., Ansari, K., Kagda, M., Mase, L. Z., Susilo, A., Sirait, A. M. M., Asnawi, Y., Lubis, M. Z., & Andinisari, R. (2025b). Seismicity Clustering and Statistical Modeling of the Intermediate-Depth Earthquakes in the Weber Deep, Banda Arc. Transportation Infrastructure Geotechnology, 12(214). https://doi.org/10.1007/s40515-025-00663-z DOI: https://doi.org/10.1007/s40515-025-00663-z
Simanjuntak, A. V., Palgunadi, K. H., Syaifuddin, F., Muksin, U., Riama, N. F., Sirait, A. M. M., Setiadi, T. A. P., Kusumawati, D., Sahara, D. P., Anshori, M., Triahadini, A., Suryanto, W., Sulandari, B., Saleh, H. M., Iskandar, Y., Ahmadiyah, A. S., & Warnana, D. D. (2025a). Evidence of a New Backthrust Fault System from the 2022 Southeast Java, Indonesia Earthquake Sequence: Insights from Seismological Analysis. Journal of Asian Earth Sciences, 106845. https://doi.org/10.1016/j.jseaes.2025.106845 DOI: https://doi.org/10.1016/j.jseaes.2025.106845
Smith, R. (2007). An Overview of the Tesseract OCR Engine. Ninth International Conference on Document Analysis and Recognition, 2, 629–633. https://doi.org/10.1109/ICDAR.2007.4376991 DOI: https://doi.org/10.1109/ICDAR.2007.4376991
Sokolova, M., & Lapalme, G. (2009). A Systematic Analysis of Performance Measures for Classification Tasks. Information Processing & Management, 45(4), 427–437. https://doi.org/10.1016/j.ipm.2009.03.002 DOI: https://doi.org/10.1016/j.ipm.2009.03.002
Wairisal, P. L., Elijonnahdi, E., Susanto, N., & Megahati S, R. R. P. (2023). Freedom to Learn and Freedom to Teach in Science Learning Through ChatGPT: Systematic Literature Review. Jurnal Penelitian Pendidikan IPA, 9(10), 784–790. https://doi.org/10.29303/jppipa.v9i10.5089 DOI: https://doi.org/10.29303/jppipa.v9i10.5089
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of Detection Tools for AI-Generated Text. International Journal for Educational Integrity, 19, 26. https://doi.org/10.1007/s40979-023-00146-z DOI: https://doi.org/10.1007/s40979-023-00146-z
Winskel, H. (2014). The Development of Writing Skills in Engineering Students. Journal of Engineering Education, 13(3), 447–470. https://doi.org/10.1002/jee.20036 DOI: https://doi.org/10.1002/jee.20036
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019). Defending Against Neural Fake News. Advances in Neural Information Processing Systems, 32, 9054–9065. https://doi.org/10.48550/arXiv.1905.12616
Zumhari, Z., Hutajulu, E., Sibarani, B., Sirait, R., Anggriani, T., & Matondang, A. A. (2025). Advancing Environmental and Health Pollution Monitoring in Medan, Indonesia: A Mechatronics-Based Meta-Analysis. Journal of Geoscience, Engineering, Environment, and Technology, 10(3), 402–407. https://doi.org/10.25299/jgeet.2025.10.3.24694 DOI: https://doi.org/10.25299/jgeet.2025.10.3.24694
License
Copyright (c) 2026 Meidi W. Lestari, Agustina Ginting, Harris Aminudin, C. Cholish, Ulfa Hasnita
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with Jurnal Penelitian Pendidikan IPA, agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution 4.0 International License (CC-BY License). This license allows authors to use all articles, data sets, graphics, and appendices in data mining applications, search engines, web sites, blogs, and other platforms by providing an appropriate reference. The journal allows the author(s) to hold the copyright without restrictions and will retain publishing rights without restrictions.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in Jurnal Penelitian Pendidikan IPA.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
