The Framework and Types of Chemical Literacy Tests: A Systematic Review

: The development of chemical literacy measurement instruments is influenced by changes and progress in frameworks, methods, and educational technologies. The study aimed to investigate the framework and test types that are utilized to measure students’ chemical literacy levels. With adherence to the PRISMA 20202, thirty-five empirical studies published between 2014 and 2023 are analyzed to find publication characteristics, chemical literacy frameworks, and test types. The finding showed that most publications were recorded in 2019


Introduction
In line with the rapid development of science and technology, the sustainability of human lives depends on the prudence of each individual to take advantage of these two things.It corresponds with the main goal of developing these two aspects, to improve human prosperity and to tackle problems facing society.Community wisdom can be built through the mastery of scientific literacy, which emphasizes scientific ways of understanding, and thinking critically and creatively about the natural world (Maienschein, 1998).Scientific literacy represents the ability to use evidence and data, to evaluate the quality of information and arguments presented by scientists and, in the mass media (Dragoş & Mih, 2015).Achieving a high level of scientific literacy means that a person becomes more confident and qualified in dealing with issues that arise in everyday life related to science, besides that it also allows them to have better job opportunities (Ploj Virtič, 2022).
The term "scientific literacy" was introduced in the late 1950s, which was used to express a broad knowledge of science and particularly the purpose of science education (Bybee, 2015).The meaning of scientific literacy is always developing, and therefore no single accepted definition of the conception (Al Sultan, Henson, & Lickteig, 2021).The definition widely used in current research is that proposed by PISA, as the conception is used for the basis testing in 90 countries.PISA reveals scientific literacy as an individual's ability to engage with science-related issues, and with the ideas of science, related to their role as reflective citizens (Osborne, 2023).
In chemistry, scientific literacy is the starting point for developing the term chemical literacy (CL).It denotes the activation of knowledge, skills, acquirements, and other elements retaining solidarity with the eligible educational goals (Mozeika & Bilbokaite, 2010;Thummathong & Thathong, 2018).CL aims to realize informed citizens who can make responsible decisions and take deliberate actions based on chemical thinking (Talanquer & Sevian, 2014).Moreover, CL is interpreted as the ability to understand and critically evaluate ideas, information, and arguments circulating in society and related to chemical content, enabling someone to deal with situations faced by members of society in scientific and technological contexts in everyday life.Hence, CL is important for educational programs in the secondary and tertiary levels.Three aspects underlie the need for accommodating chemical literacy: economic and political involvement, practical personal reasons, and cultural reasons related to ideals, values, and norms (Kohen, Herscovitz, & Dori, 2020).CL supports students to understand the role of chemistry in life and society and acquire the skills to actively participate in debates regarding relevant socio-political and economic issues.
Diagnostics of CL levels are reviewed through students' ability to use and handle the information provided related to chemical problems.It also measures the ability of students to use knowledge and chemistry skills when understanding information of daily problems (Ceyhan Cigdemoglu & Geban, 2015;Witte & Beers, 2003).The skills mentioned are the capacity to understand the information provided, select the information needed, change the information provided to other forms, and assess information from an acceptability or reasonable aspect.
CL development can be pursued in two components of education, the learning process and assessment.The assessment must pay attention to three aspects; the cognition model, the types of observations that will provide evidence of their competence, and the interpretation process to understand the evidence (Stowe & Cooper, 2019).The cognition model refers to evidence-based theories about how students develop, organize, and use knowledge in the knowledge domain.Observation is an attempt to find beliefs related to student understanding.The interpretation of the evidence from the assessment depends on inferences that can be supported by cognitive theory.It is following the paradigm shift of chemistry learning from opinionbased theories to science-informed best practices (Hartman, Nelson, & Kirschner, 2022).
One of the techniques that is often used to assess the CL level is the test.The construction of the CL test is often influenced by changes and progress in frameworks, methods, and technologies that are used in science education, particularly with regard to scientific literacy.Many studies have been conducted to construct and implement the CL test (Ad'hiya & Laksono, 2018;Ad'Hiya & Laksono, 2018;Alwathoni, Saputro, Ashadi, & Masykuri, 2020;Arabbani, Mulyani, Mahardiani, & Ariani, 2019;C. Cigdemoglu, Arslan, & Cam, 2017;Muchtar, Nahadi, & Hernani, 2020;Wiyarsi, 2020).However, literature reviews on a specific topic of the studies are still limited.The systematic literature review has been published just investigating the PISA framework as a foothold to developing chemical literacy instrument (Suwahyu & Rahayu, 2023)

Methods
The current research, a systematic review, was conducted according to the PRISMA 2020 criteria (Page et al., 2021).PRISMA provides guidelines for reporting systematic reviews that are more transparent, complete, and accurate.This study consists of three main stages: identification, screening, and inclusion as shown in Figure 1.
In the identification stage, articles are searched through the Scopus database.Scopus has global and regional coverage of scientific journals, conference proceedings, and books with high-quality assurance through accurate content selection and re-evaluation (Baas, Schotten, Plume, Côté, & Karimi, 2020).Keywords used in the searching process are chemical literacy, test, and assessment.The search strings in the database are TITLE-ABS-KEY("chemical literacy") AND TITLE-ABS-KEY(test) and TITLE-ABS-KEY("chemical literacy") AND TITLE-ABS-KEY(assessment). Both strings identified 69 articles.The duplicate articles were eliminated after identifying the documents.In addition, an automatics tool was used to eliminate the articles published before 2014.An article without insufficient information about the author was excluded.As the outcomes, 53 articles were eligible for the consecutive stages.

Figure 1. Flowchart selection of articles
The screening was done by evaluating titles and abstracts against the inclusion criteria presented.A set of inclusion criteria is presented in Table 1.The results of the screening process were seven articles excluded because it was irrelevant to the study.Furthermore, 46 documents were sought for retrieval and one of them failed to be retrieved.The failed document cannot be accessed in full paper version.Many 45 articles were further processed to assess the eligibility.The process depicted that there were nine articles excluded because frameworks of CL were not described (n = 5), the types of tests were not explained (n = 4), and the article was not written in English (n = 1).
There were 35 articles included in this study which were examined and reviewed.Microsoft Excel was cultivated for structuring under the category.Then, all selected articles were analyzed interpretively according to the research questions.

Results and Discussion
The Characteristics of Published Articles of The CL Test The majority of the articles were written by Indonesian authors (86%).The characteristics of selected papers have been analyzed based on publication year, the participants, and the research design.Figure 2 presents the number of publications in the period.A remarkable portion of the articles was published between 2019 -2021, with nine articles published in 2020, followed by seven papers in 2021 and six papers in In 2022, the number of papers decrease significantly.However, recently the number of publications has increased slightly.However, recently the number of publications has increased slightly.There is still a large chance of increasing the number because the recent papers analyzed were published in the early period of the year.

Figure 2. Analyzed paper by year
As seen in Figure 3, most of the research participants in the field were high school students.Many 74% of the research aims to develop or implement CL instruments in secondary education.The rest of the studies targeted the university students.The selection of participants was related to the forerunner of CL which is scientific literacy in the PISA version.Scientific literacy defined by PISA emphasizes educating future citizens with an age target of 15 years old or those who have completed a compulsory education program (Bybee, R., & McCrae, 2011).Moreover, chemistry in high school should emphasize the inquiry, scrutiny, and information-sharing that is fundamental to scientific literacy (American Chemical Society, 2012).
The published paper on the CL test utilized a broad research design.The major portion of the included studies implemented descriptive and experimental research, 16 publications employed descriptive design and 11 publications used experimental design.The research methods of the papers are shown in Table 2.

Figure 3. Participants involved in the studies
Quantitative descriptive is a popular choice for authors as the purpose is to describe individuals, events, or conditions by studying them as they are in nature which is concentrating on the quantity of responses.It focused on implementing constructed CL-test items or exploring students' CL level, only one study using the design aimed to construct CL-test items.While experimental research was more widely used to test a learning model or method to increase the students' CL level.

Frameworks of CL Test
In response to the second question, the framework of CL that was used to develop test items was identified and the findings were tabulated in Table 3.Two popular frameworks used as a reference for the development of CL tests were PISA and Shwart, et a.l (Shwartz, Ben-Zvi, & Hofstein, 2006).CL framework by Shwartz, et. al   Chemical literacy is one of the components of scientific literacy (Mozeika & Bilbokaite, 2010), therefore domain scientific literacy of PISA is relevant to benchmark CL development.Thus, domains of the scientific literacy framework defined by PISA and the CL framework defined by Shwartz are almost similar to one another.In table 4 the comparison of both frameworks is presented.Each framework has particular advantages, PISA frameworks are simple and explicit (Muntholib et al., 2020), whilst Shwartz et al. are more detailed and specific to the chemistry field.Cigdemoglu (2020) constructed the CL framework by merging the PISA and Shwartz frameworks and integrating argument immersion.CL dimensions proposed by Cigdemoglu included content knowledge, higher-order thinking, and interest.

Affective
Having views of chemistry and its applications.Furthermore, an individual expresses interest in chemical issues and topics, especially in nonformal scope.
There are six publications incorporating PISA or Shwartz frameworks with the framework of the other.The study was done by Thummatong et al ( 2016) integrated four frameworks which are PISA, Shwartz et al., Chang & Ciu, and Mateapinikul. Studies Chang & Ciu, and. Chang & Ciu (2005) arranged scientific literacy in six components; scientific cognition, process skills, application of science, habits of mind, nature of science, and attitude towards science.Moreover, three studies (Ad'hiya & Laksono, 2018;Ad'Hiya & Laksono, 2018;Prastiwi & Laksono, 2018) integrated framework of analytical thinking proposed by Ramirez & Ganaden (2008) which have three indicators; differentiating, organizing, and attributing.

The Test Types Employed to Assess Students' CL Level
In terms of third research questions, all terms related to test type were identified.The finding shows that three test types developed by the researchers are open-ended questions, multiple choice, and fill-theblank questions.The results of data analysis show more than half (54%) of selected papers employed open-ended questions; multiple choice was selected by 26% research group.Six publications combine open-ended questions and multiple choice, and only one study used fill-theblank questions (Eny & Wiyarsi, 2019) None type of question is considered ideal on a test, and all these types of questions vary in strengths and weaknesses.Multiple choice has strength in reliability, cost-effectiveness, and time-saving, but it is just appropriate to measure surface information for a particular skill (Polat, 2020).On the other hand, openended questions can be utilized to measure high-order thinking skills, such as critical and synthesis thinking, however, has time-consuming issues in scoring and are less reliable.The authors use more open-ended questions in the CL test because CL domains are highlevel thinking and acting skills, besides that it can explore deeper into information mastered by students.

Conclusion
This systematic review has critically reviewed papers related to the CL test to identify publication characters, frameworks, and test types.A number of 35 articles were selected with adherence to the guidelines for reporting systematic review proposed by PRISMA 2020.The finding showed that the most publications are recorded in 2019 and the respondents who were involved in the research in the largest number are high school students.Descriptive and experimental designs are popular choices for researchers to construct and explore CL tests.Related to literacy framework, the vast majority of studies develop CL tests using PISA and Shwartz et al frameworks.Moreover, open-ended questions are the most widely used to measure CL.Nevertheless, this study has limitations, considering the comprehensiveness of the database contents, this systematic review only explores Scopus.This systematic contributes significantly contribution to providing insight into the framework and types of tests in the development of CL measurement.Moreover, further research ought to inquire into the implementation of the CL test.Besides that, it is important to integrate innovative technology testing to create more authentic tests.


Records identification from scopus  Search term: Chemical literacy test (n = 46) Chemistry literacy assessment (n = 23) Records remove before screening  Duplicate records removed (n = 14)  Records marked as ineligible by automation tools (n = 1)  Records removed as no author's name (Framework is not described (n = 5)  Test types is not described (n = 4)  Not written in english (n was employed by 25 studies, while the scientific literacy framework by PISA is referred to 18 studies.A total of 23 publications referred single framework (PISA, Cigdemoglu), and the rest of the publications merged two or more frameworks.
. Therefore, it is important to understand more deeply about the test techniques of CL.This research aims to investigate the framework and test types which are utilized to explore the CL of students.Three research questions were proposed: (1) what are the characteristics of published articles of the CL test?(2) What are the frameworks of the CL test?(3) what are the test types employed to assess students' CL levels?

Table 1 .
Inclusion and exclusion criteria for article screening

Table 2 .
The research methods used in the studies

Table 3 .
Framework referred the studies

Table 4 .
Differences framework CL proposed by PISA and Shwartz