Transformers in Machine Learning: Literature Review
DOI:
10.29303/jppipa.v9i9.5040Published:
2023-09-25Issue:
Vol. 9 No. 9 (2023): SeptemberKeywords:
Accuracy, Machine learning, TransformerReview
Downloads
How to Cite
Downloads
Metrics
Abstract
In this study, the researcher presents an approach regarding methods in Transformer Machine Learning. Initially, transformers are neural network architectures that are considered as inputs. Transformers are widely used in various studies with various objects. The transformer is one of the deep learning architectures that can be modified. Transformers are also mechanisms that study contextual relationships between words. Transformers are used for text compression in readings. Transformers are used to recognize chemical images with an accuracy rate of 96%. Transformers are used to detect a person's emotions. Transformer to detect emotions in social media conversations, for example, on Facebook with happy, sad, and angry categories. Figure 1 illustrates the encoder and decoder process through the input process and produces output. the purpose of this study is to only review literature from various journals that discuss transformers. This explanation is also done by presenting the subject or dataset, data analysis method, year, and accuracy achieved. By using the methods presented, researchers can conclude results in search of the highest accuracy and opportunities for further research.
References
Abed, M., Imteaz, M. A., Ahmed, A. N., & Huang, Y. F. (2023). A novel application of transformer neural network (TNN) for estimating pan evaporation rate. Applied Water Science, 13(2), 31. https://doi.org/10.1007/s13201-022-01834-w
Acheampong, F. A., Wenyu, C., & Nunooâ€Mensah, H. (2020). Textâ€based emotion detection: Advances, challenges, and opportunities. Engineering Reports, 2(7). https://doi.org/10.1002/eng2.12189
Alqudsi, A., & El-Hag, A. (2019). Application of machine learning in transformer health index prediction. Energies, 12(14), 2694. https://doi.org/10.3390/en12142694
Arshed, M. A., Mumtaz, S., Ibrahim, M., Ahmed, S., Tahir, M., & Shafi, M. (2023). Multi-Class Skin Cancer Classification Using Vision Transformer Networks and Convolutional Neural Network-Based Pre-Trained Models. Information, 14(7), 415. https://doi.org/10.3390/info14070415
Durairaj, A. K., & Chinnalagu, A. (2021). Transformer based Contextual Model for Sentiment Analysis of Customer Reviews: A Fine-tuned BERT. International Journal of Advanced Computer Science and Applications, 12(11). Retrieved from https://thesai.org/Downloads/Volume12No11/Paper_53-Transformer_based_Contextual_Model_for_Sentiment_Analysis.pdf
Bacco, L., Cimino, A., Dell’Orletta, F., & Merone, M. (2021). Explainable Sentiment Analysis: A Hierarchical Transformer-Based Extractive Summarization Approach. Electronics, 10(18), 2195. https://doi.org/10.3390/electronics10182195
Bohatyrewicz, P., PÅ‚owucha, J., & Subocz, J. (2019). Condition Assessment of Power Transformers Based on Health Index Value. Applied Sciences, 9(22), 4877. https://doi.org/10.3390/app9224877
Caucheteux, C., & King, J.-R. (2022). Brains and algorithms partially converge in natural language processing. Communications Biology, 5(1), 134. https://doi.org/10.1038/s42003-022-03036-1
Putri, C. A. (2020). Analisis sentimen review film berbahasa Inggris dengan pendekatan bidirectional encoder representations from transformers. JATISI (Jurnal Teknik Informatika dan Sistem Informasi), 6(2), 181-193. https://doi.org/10.35957/jatisi.v6i2.206
Pasaribu, D. J. M., Kusrini, K., & Sudarmawan, S. (2020). Peningkatan Akurasi Klasifikasi Sentimen Ulasan Makanan Amazon dengan Bidirectional LSTM dan Bert Embedding. Inspiration: Jurnal Teknologi Informasi dan Komunikasi, 10(1), 9-20. http://dx.doi.org/10.35585/inspir.v10i1.2568
Yang, D., Myronenko, A., Wang, X., Xu, Z., Roth, H. R., & Xu, D. (2021). T-AutoML: Automated machine learning for lesion segmentation using transformers in 3d medical imaging. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3962-3974). Retrieved from https://openaccess.thecvf.com/content/ICCV2021/papers/Yang_T-AutoML_Automated_Machine_Learning_for_Lesion_Segmentation_Using_Transformers_in_ICCV_2021_paper.pdf
Ekstedt, E., & Skantze, G. (2020). Turngpt: a transformer-based language model for predicting turn-taking in spoken dialog. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.268
Ghosh, S., Ekbal, A., & Bhattacharyya, P. (2023). VAD-assisted multitask transformer framework for emotion recognition and intensity prediction on suicide notes. Information Processing & Management, 60(2), 103234. https://doi.org/10.1016/j.ipm.2022.103234
Ghojogh, B., & Ghodsi, A. (2020). Attention mechanism, transformers, BERT, and GPT: tutorial and survey. https://doi.org/10.31219/osf.io/m6gcn
Graterol, W., Diaz-Amado, J., Cardinale, Y., Dongo, I., Lopes-Silva, E., & Santos-Libarino, C. (2021). Emotion Detection for Social Robots Based on NLP Transformers and an Emotion Ontology. Sensors, 21(4), 1322. https://doi.org/10.3390/s21041322
Grechishnikova, D. (2021). Transformer neural network for protein-specific de novo drug generation as a machine translation problem. Scientific Reports, 11(1), 321. https://doi.org/10.1038/s41598-020-79682-4
He, K., Gan, C., Li, Z., Rekik, I., Yin, Z., Ji, W., Gao, Y., Wang, Q., Zhang, J., & Shen, D. (2023). Transformers in medical image analysis. Intelligent Medicine, 3(1), 59–78. https://doi.org/10.1016/j.imed.2022.07.002
Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023). Natural language processing: State of the art, current trends and challenges. Multimedia Tools and Applications, 82(3), 3713–3744. https://doi.org/10.1007/s11042-022-13428-4
Li, Q., Wu, C., Wang, Z., & Zheng, K. (2020). Hierarchical Transformer Network for Utterance-Level Emotion Recognition. Applied Sciences, 10(13), 4447. https://doi.org/10.3390/app10134447
Li, Z., Zhang, Z., Zhao, H., Wang, R., Chen, K., Utiyama, M., & Sumita, E. (2021). Text compression-aided transformer encoding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7), 3840-3857. Retrieved from https://ieeexplore.ieee.org/abstract/document/9354025
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022). A survey of transformers. AI Open, 3, 111–132. https://doi.org/10.1016/j.aiopen.2022.10.001
Liu, H.-I., & Chen, W.-L. (2022). X-Transformer: A Machine Translation Model Enhanced by the Self-Attention Mechanism. Applied Sciences, 12(9), 4502. https://doi.org/10.3390/app12094502
Luitse, D., & Denkena, W. (2021). The great transformer: Examining the role of large language models in the political economy of AI. Big Data & Society, 8(2), 20539517211047734. https://doi.org/10.1177/20539517211047734
Moutik, O., Sekkat, H., Tigani, S., Chehri, A., Saadane, R., Tchakoucht, T. A., & Paul, A. (2023). Convolutional Neural Networks or Vision Transformers: Who Will Win the Race for Action Recognitions in Visual Data? Sensors, 23(2), 734. https://doi.org/10.3390/s23020734
Öztürk, O., & Özcan A. (2022). Ideology Detection Using Transformer-Based Machine Learning Models. Retrieved from https://rb.gy/28t72
Paaß, G., & Giesselbach, S. (2023). Improving Pre-trained Language Models. In G. Paaß & S. Giesselbach, Foundation Models for Natural Language Processing (pp. 79–159). Springer International Publishing. https://doi.org/10.1007/978-3-031-23190-2_3
Park, J., Choi, K., Jeon, S., Kim, D., & Park, J. (2019). A Bi-Directional Transformer for Musical Chord. 20th International Society for Music Information Retrieval Conferences Netherland. https://doi.org/10.13140/RG.2.2.12303.51362
Piccialli, F., Di Cola, V. S., Giampaolo, F., & Cuomo, S. (2021). The Role of Artificial Intelligence in Fighting the COVID-19 Pandemic. Information Systems Frontiers, 23(6), 1467–1497. https://doi.org/10.1007/s10796-021-10131-x
Prottasha, N. J., Sami, A. A., Kowsher, M., Murad, S. A., Bairagi, A. K., Masud, M., & Baz, M. (2022). Transfer Learning for Sentiment Analysis Using BERT Based Supervised Fine-Tuning. Sensors, 22(11), 4157. https://doi.org/10.3390/s22114157
Putra, M. F. D. A., & Hidayatullah, A. F. (2021). Tinjauan Literatur: Named Entity Recognition pada Ulasan Wisata. AUTOMATA, 2(1). Retrieved from https://journal.uii.ac.id/AUTOMATA/article/view/17391
Rajan, K., Zielesny, A., & Steinbeck, C. (2021). DECIMER 1.0: deep learning for chemical image recognition using transformers. Journal of Cheminformatics, 13(1), 1-16. https://doi.org/10.1186/s13321-021-00538-8
Ramos-Pérez, E., Alonso-González, P. J., & Núñez-Velázquez, J. J. (2021). Multi-Transformer: A New Neural Network-Based Architecture for Forecasting S&P Volatility. Mathematics, 9(15), 1794. https://doi.org/10.3390/math9151794
Reinauer, R., Caorsi, M., & Berkouk, N. (2021). Persformer: A transformer architecture for topological machine learning. https://doi.org/10.48550/arXiv.2112.15210
Rendragraha, A. D., Bijaksana, M. A., & Romadhony, A. (2021). Pendekatan metode Transformers untuk deteksi bahasa kasar dalam komentar berita online Indonesia. eProceedings of Engineering, 8(2). Retrieved from https://repository.telkomuniversity.ac.id/pustaka/files/167651/jurnal_eproc/pendekatan-metode-transformers-untuk-deteksi-bahasa-kasar-dalam-komentar-berita-online-indonesia.pdf
Röder, L. L. (2023). Neural Network Architectures for Absorption Spectroscopy. https://doi.org/10.21203/rs.3.rs-2832856/v1
Sanmarchi, F., Bucci, A., Nuzzolese, A. G., Carullo, G., Toscano, F., Nante, N., & Golinelli, D. (2023). A step-by-step researcher’s guide to the use of an AI-based transformer in epidemiology: An exploratory analysis of ChatGPT using the STROBE checklist for observational studies. Journal of Public Health. https://doi.org/10.1007/s10389-023-01936-y
Santangelo, O. E., Gentile, V., Pizzo, S., Giordano, D., & Cedrone, F. (2023). Machine Learning and Prediction of Infectious Diseases: A Systematic Review. Machine Learning and Knowledge Extraction, 5(1), 175–198. https://doi.org/10.3390/make5010013
Singla, S., & Ramachandra, N. (2020). Comparative analysis of transformer based pre-trained NLP Models. Int. J. Comput. Sci. Eng, 8, 40-44. https://doi.org/10.26438/ijcse/v8i11.4044
Tabinda Kokab, S., Asghar, S., & Naz, S. (2022). Transformer-based deep learning models for the sentiment analysis of social media data. Array, 14, 100157. https://doi.org/10.1016/j.array.2022.100157
Taye, M. M. (2023). Understanding of Machine Learning with Deep Learning: Architectures, Workflow, Applications and Future Directions. Computers, 12(5), 91. https://doi.org/10.3390/computers12050091
Thoyyibah, T., Abdurachman, E., Heryadi, Y., & Zahra, A. (2022). Transformer Model in Music Mood Classification. International Journal of Applied Engineering and Technology (London), 4(1), 40–44. Retrieved from https://romanpub.com/resources/ijaet v4-1-2022-08.pdf
Tran, Q. T., Davies, K., & Roose, L. (2020). Machine learning for assessing the service transformer health using an energy monitor device. IOSR Journal of Electrical and Electronics Engineering, 15(6), 1-6. Retrieved from https://www.iosrjournals.org/iosr-jeee/Papers/Vol15-Issue6/Series-1/A1506010106.pdf
Valencia, F., Arcos, H., & Quilumba, F. (2021). Comparison of Machine Learning Algorithms for the Prediction of Mechanical Stress in Three-Phase Power Transformer Winding Conductors. Journal of Electrical and Computer Engineering, 2021, 1-9. https://doi.org/10.1155/2021/4657696. 2021
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., & Chao, L. S. (2019). Learning deep transformer models for machine translation. https://doi.org/10.48550/arXiv.1906.01787
Wu, N., Green, B., Ben, X., & O'Banion, S. (2020). Deep transformer models for time series forecasting: The influenza prevalence case. https://doi.org/10.48550/arXiv.2001.08317
Wu, B., Wang, L., & Zeng, Y.-R. (2023). Interpretable tourism demand forecasting with temporal fusion transformers amid COVID-19. Applied Intelligence, 53(11), 14493–14514. https://doi.org/10.1007/s10489-022-04254-0
Xiao, Y., Jin, Y., Cheng, R., & Hao, K. (2022). Hybrid attention-based transformer block model for distant supervision relation extraction. Neurocomputing, 470, 29-39. https://doi.org/10.1016/j.neucom.2021.10.037
Yang, G. R., & Wang, X.-J. (2020). Artificial Neural Networks for Neuroscientists: A Primer. Neuron, 107(6), 1048–1070. https://doi.org/10.1016/j.neuron.2020.09.005
Zhong, P., Wang, D., & Miao, C. (2019). Knowledge-enriched transformer for emotion detection in textual conversations. https://doi.org/10.48550/arXiv.1909.10681
Author Biographies
Thoyyibah T, Universitas Pamulang
Wasis Haryono, Universitas Pamulang
Achmad Udin Zailani, Universitas Pamulang
Yan Mitha Djaksana, Universitas Pamulang
Neny Rosmawarni, Universitas Pembangunan Nasional Veteran Jakarta
Nunik Destria Arianti, Nusa Putra University
License
Copyright (c) 2023 Thoyyibah T, Wasis Haryono, Achmad Udin Zailani, Yan Mitha Djaksana, Neny Rosmawarni, Nunik Destria Arianti
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with Jurnal Penelitian Pendidikan IPA, agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution 4.0 International License (CC-BY License). This license allows authors to use all articles, data sets, graphics, and appendices in data mining applications, search engines, web sites, blogs, and other platforms by providing an appropriate reference. The journal allows the author(s) to hold the copyright without restrictions and will retain publishing rights without restrictions.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in Jurnal Penelitian Pendidikan IPA.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
