Consistent or Inconsistent? Expert-based Cognitive Model vs Student-based Response Cognitive Model of Cognitive Diagnostic Assessment in Factorisation of Algebraic Fractions

Expert-based Pg. Abstract This study aims to discuss the process of developing a Cognitive Diagnostic Assessment combination with cognitive models for assessing students' mastery of algebraic skills. It also highlights the factors considered in the development of the CDA. The integrated assessment triangle's theoretical framework and evidence-centred design is used to guide the development and validation of the hierarchical attribute method (AHM), expert-based cognitive model, student-response cognitive model, and the use of the simulation of the Hierarchical Consistency Index (HCI) to assess the consistency of the two models. It began with the critical phase of producing a high-quality CDA, where experts would define the related attributes and items based on the DSKP, textbook and related documents and a pilot test of open-ended questions distributed to students. The details of CDA's development and the construction of its components, such as cognitive models and psychometric analyses, are also discussed.


Introduction
Cognitive diagnostic assessment (CDA) is commonly used to evaluate students' achievements such as mastery and non-mastery, strengths and weaknesses, and mistake analysis in skills or subskills in certain topics. Recently, (Toprak & Cakir, 2020), (Sia et al., 2019), (Shanley, 2016), (L. Li et al., 2020) and (Ralston et al., 2018) conducted an extensive study to assess students' comprehension of certain topics using CDA. As a result, Cognitive Diagnostic Assessment (CDA) is one of the finest tools to use in assessment to obtain a clear understanding of students' competency in each skill. CDA is a learning theory and psychrometric model used to analyse students' performance in depth for each ability evaluated and may be presented statistically for references. For example, the study of (Chin et al., 2021) on the design of CDA for time multiplication in primary schools in Malaysia. The study focused on developing and validating an online CDA that used ordered multiple-choice (OMC) questions to assess students' grasp of 'Multiplication of Time' attributes. This study identified pupils' mastery in each attribute of multiplication related to time. However, in secondary schools, students in Malaysia have difficulty conceptual understanding algebra, leading to low performance in algebraic skills, particularly at the higher levels of schooling (Mustaffa, 2017). This is in line with the findings of the TIMSS 2019 study. For example, items 2 and 3 in the TIMSS 2019 examination were classified as highest-level items since the question was produced using the cognitive domain and the content domain is algebra. Unfortunately, Malaysian students who correctly answered the test were in the bottom three, placing them in the lowest three countries. Students who lack a solid understanding of algebra are too accountable for this. They cannot relate mathematical operations and algebraic formulas to the idea of variables (Jupri & Drijvers, 2016). Hence, to overcome the worse become worst, it is clear why it is important to use CDA to access students' mastery in algebraic skills and yield the significant influence on their progress in learning mathematics at the higher level.

Cognitive Diagnostic Assessment
CDA is a components for KPM classroom-based assessment (CBA) (KPM, 2016b). However, there are reports that teachers'' application of CBA in the classroom is inconsistent with the vision and objective of CBA. CBA allows teachers to assess pupils' ability to process information in various psychomotor, cognitive, and emotional methods. Evaluation occurs continually throughout the years (KPM, 2016a).However, some teachers place a greater emphasis on the rubrics system, where students are categorised based performance indicators or tahap penguasaan (TP), than observing and evaluating students' improvement. As a result, parents tend to doubt teachers' capacity to evaluate the students, undermining the professionalism and their careers ( Wan Omar, 2019). Hence, teachers' evaluation alone is not sufficient for CBA. Thus, introducing CDA to teachers and educators may assist them in evaluating students' cognitive skill development and justifying the level of skill mastery backed with strong evidence (statistical data) and improve their teaching techniques (Sia et al., 2019). In the meantime, traditional diagnostic examination provide teachers and students with information about what students have previously learned in terms of content and expectations. Numerous studies have been conducted in Malaysia to investigate various difficulties associated with learning algebra and have resulted in recommendations for improved teaching methods Abdullah et al (2016) Ying et al., 2020;Zila et al (2020) yet, little attention has been paid to assessing students' mastery of the algebraic skill, which encompasses conceptual knowledge and procedural skill phases. For many educators, the issue of assessing pupils' algebraic skills in terms of conceptual knowledge and operational skill is still new and challenging to resolve. As a result, teachers need a strategy that provides comprehensive information on each student's mastery status, including their conceptual knowledge and procedural skill structure. This information should focus on the students' needs while also offering a relevant assessment for them. Assessment tools must be established or structured in such a way that they provide teachers with information about each student's cognitive profile and emphasise the specific cognitive features that need to be improved. CDAs may provide information on students' cognitive and learning processes, allowing for the possibility of influencing meaningful student learning.

Issues on Students' Achievement in Algebra
Algebra forms the foundation for studying mathematics at the tertiary level. It is also the core for critical disciplines in the science, technology, engineering, and mathematics (STEM). These disciplines are critical for students' future preparedness and to survive Industry Revolution 4.0. (IR 4.0). Algebra covers a vast range of subtopics and is taught in the mathematics subject. Subsequently, algebra has become a significant component in mathematics learning from the elementary level until university level. It is inherently intertwined with the science and mathematics education (Daud & Ayub, 2019). Using certain symbols or letters to presents concepts or ideas of simplification and problem solving (Wu, 2001). Algebra is crucial in mathematics because it is connected to other areas of study such as geometry, calculus, basic matrices, trigonometry, statistics, and vectors (Jupri & Drijvers, 2016;Makonye & Stepwell, 2016). For instance, algebraic problems may be solved geometrically by first visualising functions or equations, or they can be solved algebraically. Consequently, students must possess a strong understanding of algebraic conceptual knowledge to quickly follow the learning in each field of mathematics related to that knowledge and solve problems using the appropriate procedural skill. In other words, the fact that algebra is necessary for success in mathematics, particularly at the higher-level stage requiring higher order thinking abilities, cannot be disputed anymore (Alexendra, 2020).
Algebra, specifically algebraic skills involving fractions, has been introduced to pupils in Malaysian primary school. In this regard, primary level algebra expose pupils to the principles of the unknown ideas and applying to simple arithmetic calculations to building a strong foundation in algebra (Somasundram et al., 2017). The fundamentals of algebra are taught in lower secondary schools as a continuation of the primary mathematics while algebraic computations such as factoring algebraic fractions are taught in upper secondary schools (KPM, 2016b). However, pupils' ability in acquiring mathematical abilities has been found unsatisfactory (Sugiarti and Retnawati, 2019), eroding their interest in the subject during secondary school (Mat et al., 2020). It has been scientifically shown that pupils' engagement in algebra instruction is unpredictable and students' attention is often inconsistently throughout math lessons (Skilling et al., 2016). Additionally, issues in mastering algebraic skills, such as factorisation of algebraic fractions, resulted in students encountering difficulty with word problems, impairing their ability to fully grasp algebraic skills (Sugiarti and Retnawati, 2019). Because of these issues, students' scores on the Malaysian Certificate of Education (MCE) or known as Sijil Pelajaran Malaysia (SPM), show a strong link with their performance on higherlevel exams (colleges and universities). Furthermore, students who perform well in mathematics in SPM are likelier to do well in mathematics in college (Mahlan et al., 2017). Unfortunately, based on a comparison of SPM students' achievement in 2017, 2018, and 2019 decreased, respectively. Hence, from the standpoint of content algebra, it is reasonable to presume that higher graders already possess solid mathematical foundations, having completed the whole algebra curriculum in lower secondary school (Abdullah et al., 2016). Because of this, secondary school students must have a solid grasp of algebraic concepts and skills, which will benefit them in university and beyond. Therefore, assessing the knowledge of algebraic skills among students of lower secondary schools is an appropriate course of action. According to the situation in Malaysia, CDA must be used to diagnose students' problems early on to avert a worsening scenario and, most importantly, to ensure that all students have an equal opportunity to receive better and more quality instruction strategised by the teacher after their problems with algebra are identified.

The Development of Cognitive Diagnostic Assessment
Cognitive Diagnostic Assessment (CDA) is a type of educational assessment in which students' general knowledge/conceptual/understanding and cognitive abilities are evaluated to discover cognitive strengths and deficiencies . CDA has contributed significantly to identifying students' answers at higher diagnostic, informative, and finegrained data levels. Numerous statistical and psychometric approaches, including Cognitive Diagnostic Models (CDMs), have been developed for extracting this kind of information from diagnostic and non-diagnostic data. These frameworks supply two types of knowledge to testing participants: information on test takers' mastery/non-mastery patterns of qualities and information on the diagnostic capabilities of test items and yielding the success of quality CDAs. Therefore, in this study, the cognitive model will be constructed as a pillar of the development of CDA. The cognitive model mainly focuses on hierarchical orders to the attributes related to solving the items involved. Based on Attribute hierarchical method (AHM) (Leighton et al., 2004), the experts will determine the related attributes and arrange them in hierarchical order depending on the attributes' complexity. The process of determination of the attributes generally is the process of constructing the expert-based cognitive models. Meanwhile, the student's answers in each item reflect their attributes achievement. Hence, the students-response cognitive model is constructed based on genuine students' responses during CDA implementation. According to these two cognitive models as a pillar of the development of CDA, this study's objective specifically focuses on the consistency between two cognitive models by using the measurement of the Hierarchical Consistency Index (HCI).

Expert-Based Cognitive Model
The mathematics curriculum documents (KSSM syllabus and DSKP for form one and two), mathematics textbook (form 1 and form 2) and past year PT3 mathematics exam questions (code number fifty-five) will be used to identify a list of skills related to factorisations of algebraic fraction. This collection of algebraic skills, referred to as attributes, will be combined to form an expert-based cognitive model. Due to AHM's properties, these attributes will arranged in increasing order of complexity and organise using Boolean Matrix (Tatsuoka, 2009). A cognitive model will be developed using Q-matrixes to anticipate the students' responses. As the model will be developed based on the experiences of experts, the arrangement of attributes will be referred to as an expert-based cognitive model. A pool of CDAs is generated based on cognitive attributes and Q-matrix development to determine the conceptual and procedural stage of algebraic competence necessary to understand the subject factorisation and algebraic fraction. Experts then assessed the proposed CDA in terms of its structure and arrangement of attributes. This validation aims to determine if the proposed model, also known as an expert-based model, fits the cognitive model of students' responses in CDA. The expert-based model will be enhanced if the pilot test indicates a discrepancy between it and the cognitive model of students' responses. The expert-based model will be delivered to form two students in secondary schools as part of a pilot study. The answers will be analysed to determine whether the aim was met. If there is a discrepancy between these two models, the cycle can be improved to rectify the attributes/items that cannot accomplish the aim. This improvement cycle will be repeated until all attributes/items achieve consistency between the two models.

Student-Respond Cognitive Model
After the preparation and finalisation stages, CDA will be distributed to form two students in chosen schools. Students will complete all CDA items in one hour and thirty minutes, corresponding to the time of the national test SPM for paper one. The data, specifically the students' responses to each item will then be analysed to determine students' mastery of each attribute based. The consistency of the attribute's arrangement of algebraic skill based on the student's response will be determined using the hierarchical consistency index (HCI) and fit analysis (item discrimination index, item difficulties, and attribute reliability) to determine the response's consistency, validity and reliability of the CDA, and the quality of items based on AHM. (Cui & Leighton, 2009) devised fit studies to measure attributes' consistency and compliance with AHM criteria. Numerous academics Cui (2007); Wang & Gierl (2007) have discussed and supported the use of HCI. In this light, the HCI expects students to successfully answer simple items measuring attributes if and only if they had previously correctly answered complicated items on an educational test meant to measure a set of hierarchically ordered attributes. The purpose of the person-fit statistic is to determine whether the students' actual item response patterns match the expected response patterns based on the hierarchical connection between test item attributes. In other words, HCI determines the consistency of the observed and expected response patterns. The purpose of this exercise is to determine the validity of the attribute hierarchy using the students' responses.

The Procedures of CDA Development Theoretical Framework
The Assessment Triangle introduced by (Pellegrino et al., 2001) and Evidence-Centered Design (ECD) by (Mislevy et al., 2003) are used in the theoretical framework in this study on the development of CDA. The evaluation triangle comprises three vertexes symbolising the process's components: cognition, observation, and interpretation. The assessment triangle is useful because it offers a basic and overarching framework for organising the components of CDA development. ECD is a framework that "involves the creation, building, and organisation of specialised information pieces" (Mislevy et al., 2003). This section highlights the key components of creating a valid CDA utilising both the assessment triangle and ECD. It should be noted once again that the evaluation design process is not always linear (DiBello et al., 2007). Although attribute definition often comes before task development, the practicality of building tasks that assess a certain set of attributes could determine how attributes are specified. Figure 1 depicts an outline for the framework of ECD merged with the assessment triangle. Cognition vertex collects and interprests facts about the topic of interest while focusing on the core of knowledge development (Mislevy, 2011;Mislevy et al., 2006). Consequently, the CDA development phase begins with a thorough analysis of the area of concern (Tjoe & Torre, 2013b). A study by Tjoe & Torre (2013a) exhaustedly documented their work on a proportional reasoning evaluation for middle school students, allowing for a study of student thinking.
The following sections present how the process works. The hierarchical attributes based on the complexity and Table Specification test will be constructed at this stage. Experts will review students' responses in terms of their level of competency and error analysis in algebraic skills by doing the preliminary test. After that, the experts will arrange the related attributes to be tested based on the DSKP and mathematics textbook. The format of the items is based on PT3 and SPM format for paper 1. All the processes are the basis in the development of the expert-based cognitive model where prediction that students' response is consistent with the arrangement of attributes. Thus, students are expected score higher for easy items, and vice versa based on the arrangement of the attributes. The objective of the study is achieved if students' answers align with the expectation.
The observation vertex (Mislevy, 2011) explains the importance of information in detail. Domain modelling necessitates detailed statements to explain the type of evidence needed to logically deduce students' knowledge in the domain. A solid interpretation of the data-task interaction will provide input on how the tasks should be organised to elicit the required information. Initial CDA activities based on desired data presume that the tasks, both singly and collectively, will compute the attributes as predicted. Consequently, the domainmodelling layer contains an intermediary way of gathering valid data that validates attribute use, eventually leading to the assessment's judgement of the interpretation or use argument. As a result, the typical validation technique seeks scientific evidence that verifies a hypothesis. This section will use the diagnostic task to obtain students' responses to the preliminary test.
The preliminary (pencil and paper) tests consists of items that will test students' performance in algebraic skills based on DSKP content. For this step, students' errors and the cognitive process can be analysed to guide the experts in constructing quality and related attributes based on the study's objective.
In the interpretive vertex, the Conceptual Assessment Framework (CAF) acts as a system for designing assessments. It is an assessment model built using domain knowledge and other data. The CAF is primarily concerned with a formal overview of the study field that outlines how data will be acquired. It may be constructed or modified to employ the principles of observational reasoning. It is a means of producing evidence about the stated set of skills from the observation vertex of the assessment task. Various mathematical approaches may be employed to understand the assessment results. A mathematical model that categorises learners' skills into different levels in large-scale evaluations is employed, while an intuitive approach simplifies understanding these traits. A task model, multiple-choice, open-ended, and teacher-created assessments, as well as a test style example, are included in the toolkit. It also includes many explanations of how students should behave throughout the lecture and the results of those exams. The evaluation may be used to assist students in understanding the essence of the characteristics being evaluated and how to improve classroom instruction. The evidence paradigm bridges the gap between the student and work models. It is divided into two parts: an assessment component and a calculating component. This is maybe the most closely connected component of ECD to the representation vertexes of the evaluation triangle. The assessment section provides the technique for evaluating assignments and explains how observations might be statistically represented. The measuring aspect is the mathematical model used to analyse a response trend. It is the procedure that allows for the estimate of the degree of a latent variable (ability). The test comprises multiple-choice questions with only one right answers. The answers are marked as correct or incorrect. The nature and purpose of these products call for rubrics for scoring, which might entail the use of a mathematical model. The calculation models, large amounts of data may be measured efficiently, realistically, and precisely. Hence, in this section, the student-response cognitive model is constructed based on the students' response to the CDA. A psychometric statistical analysis known as HCI will be implemented to analyse the students' response by gauging the consistency between the expert-based cognitive model and student-response cognitive model

Conclusion
The Assessment Triangle (Pellegrino et al., 2001) supports the foundation for the CDA. Based on the framework, the three core parts, namely (i) cognition, (ii) observation, and (iii) interpretation, must be integrated to assure the provision of more accurate and systematic measurements of students' knowledge and skill development (Shute et al., 2016) between both models. It is then combined with ECD, which is a process involving the creation, building, and organisation of specialised knowledge components, or assessment design items, to inform requirements that comprise the substantive statement underlying an assessment (Mislevy et al., 2003). DiBello et al (2007) introduced ECD as a framework for the assessment triangle, providing the necessary accuracy to each vertex corner. As a result, the related elements of ECD under the needed vertex of the assessment triangle will be highlighted to give a realistic starting point for the assessment design phase since it describes actionable measures. Hence, the key components of developing a correct CDA utilising both the assessment triangle and ECD. Integration of Assessment Triangle and the ECD mainly aims to develop the expert-based model in a systematic, valuable, and reliable way. In this sense, the cognitive models must be supported by adequate empirical data to assure their validity for guiding item development and score interpretation. The linear hierarchical structure of the cognitive models in this research will be supported by correlations across the attributes in the two cognitive models and a decrease in the mean mastery probability as attribute complexity increases. This linear hierarchical structure of the cognitive models will also be reflected in the psychometric features of the CDA with OMC items, where the mean item difficulty index decreases as the complexity of the attributes increases. This finally leads to the study's modeldata fit (Roberts et al., 2014), indicating the consistency of student-response cognitive models with expert-based cognitive models. The attributes showed by students in executing Factorisation Algebraic Fractions align with the expectations of mathematics education experts, with a good model-data fit. Thus, the construct cognitive model will be obtained from both cognitive models will be valid for guiding OMC item development and meaningful score interpretation in the CDA (Briggs et al., 2006). Finally, the big why for developing the CDA is to assist teachers to assess students' understanding of attributes linked to Factorisation of Algebraic Fractions by comparing the consistency between the expert-based cognitive models and student-response cognitive model. This completed CDA model might serve as a reference for future researchers and test developers in constructing a CDA for other topics in mathematics and other subjects. It is importance for psychometric perspective to portray the good and valid results. Thus, the CDA developed is a valid and reliable tool that may assist instructors in ensuring students' mastery of attributes linked to Factorisation of Algebraic Fractions. Based on this diagnostic information, teachers might organise remedial courses or strategies to assist students in overcoming cognitive limitations and improve their' ability in Mathematics.