Fuzzy Delphi Method: A Step-by-Step Guide to Obtain Expert Consensus on MUETBot Functionalities

This study presents a guide to using the Fuzzy Delphi Method (FDM) to obtain expert consensus on the functionalities of MUETBot, a technological tool designed to support Malaysian University English Test (MUET) reading skills. The methodology involves a step-by-step process of questionnaire development, expert panel selection, data collection and analysis, and iterative feedback and revision. The findings demonstrate a high level of consensus among experts regarding the importance of specific functionalities, such as providing sample MUET reading papers and instant feedback on answers. The implications of this study highlight the importance of expert input in designing instructional technological tools, emphasizing the need for tools like MUETBot to align with experts' perceptions of what is essential. Future studies could explore the effectiveness of MUETBot in improving students' performance in the MUET exam.


Introduction
The Malaysian University English Test (MUET) is a crucial examination that assesses the English language proficiency of Malaysian students, particularly those aspiring to pursue higher education (Harun et al., 2021).Proficiency in English is essential for success in academia and the professional world, making the preparation for MUET a significant concern for students and educators alike.To support students in mastering the requisite skills, the development of educational tools such as MUETBot has gained traction, aiming to enhance students' learning experience and performance in the MUET examination.
MUETBot is an instructional technological tool designed to assist students in improving their reading skills, a key component of the MUET examination (Yin & Hanif, 2024).The development of MUETBot necessitates a comprehensive understanding of the functionalities and features that will best support students' learning needs.However, determining these functionalities requires expert input to ensure that MUETBot aligns with the expectations and Vol 14, Issue 4, (2024) E-ISSN: 2222-6990 requirements of both educators and students.Hence, the involvement of a panel of experts to provide consensus on MUETBot's functionalities is imperative to its effectiveness and relevance in the educational context (Ridhuan & Nurulrabihah, 2020).
The primary objective of this study is to assess and validate the functionalities of MUETBot based on the consensus of a group of experts using the Fuzzy Delphi Method (FDM).By leveraging the expertise and insights of the expert panel, this study aims to identify the key functionalities that MUETBot should possess to effectively support students in enhancing their MUET reading skills.Additionally, this research seeks to provide a step-by-step guide for obtaining expert consensus on educational technological tools, offering a valuable methodology for future research in this area.
This study holds significant implications for the field of educational technology and language learning.By elucidating the process of obtaining expert consensus on MUETBot functionalities, this research contributes to the advancement of instructional design principles tailored to the specific needs of MUET students.Furthermore, the findings of this study can inform the development of similar educational tools for other language proficiency examinations, thereby benefiting a broader spectrum of students seeking to enhance their language skills for academic and professional pursuits.

Literature Review
The Fuzzy Delphi Method (FDM) is a variation of the traditional Delphi method that incorporates fuzzy logic to handle uncertainties and ambiguities in expert judgment (Hierro et al., 2021).FDM has gained popularity in various fields, including education, due to its ability to obtain more reliable and consistent results by accommodating diverse and sometimes conflicting expert opinions.In educational research, the use of FDM has been particularly valuable in obtaining expert consensus on complex issues related to curriculum development, assessment, and educational technology.
The Fuzzy Delphi Method (FDM) involves a series of procedures to obtain consensus from a group of experts on a particular topic (see in Figure 2).According to Mohd Ridhuan & Nurulrabihah (2020), the first step, the expert determination process, focuses on selecting the right experts for the study.Jones & Twiss (1978) suggest that the optimal number of experts should be between 10 to 50, while Adler and Ziglio (1996) propose 10-15 experts when there is high consistency among them.Ocampo et al. (2018) argue that the quality of results is not strongly related to the number of experts, indicating that a study does not necessarily require many experts.
The second step, the linguistic selection process, involves using a Fuzzy Scale, which is a 7-point scale derived from the Likert Scale (Table 1), to measure the level of agreement among experts.This scale allows for a more nuanced analysis of expert opinions compared to a traditional Likert Scale.The Fuzzy Scale assigns fuzzy values to each Likert scale value, ranging from 0 to 1, to quantify the degree of agreement or disagreement.

Level of Agreement
Likert Scale Fuzzy Scale Strongly Agree 7 0.9 1 1 Very Agree 6 0.7 0.9 1 Agree 5 0.5 0.7 0.9 Moderately Agree 4 0.3 0.5 0.7 Disagree 3 0.1 0.3 0.5 Strongly Disagree 2 0 0.1 0.3 Very Strongly Disagree 1 0 0 0.1 In the third step, the researcher keys in the Likert Scale values into the item space for each item, converting them into fuzzy scale values.This process allows for a standardized method of inputting expert opinions into the FDM framework.The fourth step involves obtaining fuzzy average values (m1, m2, m3) from the fuzzy scale, which represent the consensus among experts for each item.A triangular fuzzy number consists of three values: m1, m2, and m3, which correspond to the minimum value, the most plausible value, and the maximum value, respectively.Figure 1 illustrates these three values in the triangular fuzzy number.The values in a triangular fuzzy number fall within the range of 0 to 1, similar to other fuzzy numbers.
Figure 1.Values in Triangular Fuzzy Number (Ridhuan & Nurtulrabihah, 2020) The fifth step is the process of determining the threshold value 'd,' which is used to determine whether an item is accepted or rejected based on the level of agreement among experts.The sixth step involves assigning the number of experts to the red boxes according to the total number of experts in the study, which helps in visualizing the level of agreement among experts.
The seventh step is the process of determining the percentage of agreement for each item and overall items, which provides a quantitative measure of expert consensus.The eighth step is the fuzzy evaluation process, which involves determining scores (ranking/priority) for each item based on the level of agreement among experts.Items that do not meet the conditions set by the FDM are rejected, as indicated by values and cells appearing in red in the table.
Finally, the ninth step involves presenting the results of data analysis in a table, with displayed values and cells appearing in red indicating non-compliance with the conditions set by the Fuzzy Delphi Method.This suggests that expert agreement rejects those elements, providing a clear summary of the consensus reached by the expert panel.In the context of educational technology or language learning, FDM has been used in similar studies to obtain expert consensus on various aspects such as curriculum design, assessment methods, and the development of educational tools (Liu et al., 2023;Vasodavan et al., 2021;Yeh et al., 2024).Overall, the Fuzzy Delphi Method has proven to be a valuable tool in obtaining expert consensus in complex educational research projects.Its ability to handle uncertainties and diverse opinions makes it a suitable method for obtaining reliable and consistent results in the field of education.

Methodology
The methodology of the study involved several key steps to ensure the development of MUETBot was informed by expert consensus and met the needs of MUET students.The study utilized the Fuzzy Delphi Method (FDM) to obtain expert consensus on the design and functionality of MUETBot, focusing on enhancing students' reading skills and comprehension.The methodology included the development of a questionnaire, distribution to an expert panel, data collection and analysis, fuzzy aggregation techniques, and an iterative process of feedback and revisions until consensus was achieved.
The development of the questionnaire for the Fuzzy Delphi Method (FDM) was based on a thorough review of the literature related to MUET (Malaysian University English Test) reading skills, comprehension practices, and assessment methods.The constructs of "Notes of MUET Reading Skills," "MUET Reading Comprehension Practices," and "Assessments" were identified as key areas for expert assessment and feedback.Each construct was designed to encompass specific aspects of MUET preparation and performance, ensuring a comprehensive evaluation framework.
The questionnaire was structured to include items framed using fuzzy logic terms and a 7-point Likert scale.This design allowed the expert panel to express their opinions and expectations with a nuanced degree of importance, enabling a more nuanced understanding of their perspectives (Ridhuan & Nurtulrabihah, 2020).The use of fuzzy logic terms also facilitated the incorporation of expert judgment and qualitative insights into the analysis process, enhancing the depth and relevance of the findings.
The expert panel consisted of 13 distinguished individuals recognized for their mastery of MUET, expertise in language teaching, and specialized knowledge in instructional technology.The selection of panel members was based on their professional qualifications, experience, and contributions to the field, ensuring a high level of expertise and credibility in assessing the functionalities of MUETBot.
The distribution of the questionnaire to the expert panel was conducted in a manner that ensured anonymity, integrity, and impartiality.Experts were encouraged to provide wellconsidered and informed opinions based on their expertise, without any external influence or bias.This approach facilitated honest and unbiased feedback, enhancing the reliability and validity of the data collected.
The process of collecting and analyzing the data followed a structured approach based on the FDM methodology.The data were analyzed using fuzzy aggregation techniques, which involved calculating mean scores and standard deviations for each item to assess the degree of consensus among the expert panel.Items with high consensus were identified as areas of agreement, while items with low consensus were marked for further discussion and refinement in subsequent rounds.
The iterative process of feedback, discussion, and questionnaire revisions continued until a satisfactory level of consensus was achieved among the expert panel.This iterative approach allowed for the incorporation of expert opinions and insights, ensuring that the design and development of MUETBot were grounded in the valuable feedback and consensus among experts.The final results of the FDM process provided a detailed account of the methodology used, the aggregation techniques employed, and the implications of the expert panel's views and expectations for the design and development of MUETBot.
By employing the Fuzzy Delphi Method, the design and development of MUETBot were grounded in the valuable insights and consensus among experts, ensuring that the resulting instructional technological tool met the needs and expectations of its intended users.

Findings and Discussions
Table 2 summarizes the findings of the Fuzzy Delphi Method (FDM) analysis for the functionalities of MUETBot based on expert consensus.The threshold value (d) and the percentage of agreement among the expert group are presented for each item, along with the fuzzy score (A) indicating the level of consensus.The analysis of expert opinions using fuzzy logic terms revealed a strong consensus regarding the key features and functionalities of MUETBot, a tool designed to support students in preparing for the Malaysian University English Test (MUET).The expert panel unanimously agreed on the importance of incorporating the general MUET format into the tool, providing sample MUET reading papers, and offering explanations for reading skills.These aspects received high fuzzy scores of 0.913, 0.944, and 0.908, respectively, indicating a high level of importance attributed to them by the experts.
Similarly, there was unanimous agreement on the importance of providing instant feedback on answers, explanations for correct answers, and practice of different parts of the exam separately.These functionalities received fuzzy scores of 0.951 each, demonstrating a strong consensus among the experts.Additionally, the panel unanimously agreed on the importance of providing supplementary learning resources and explanations for unfamiliar vocabulary, with fuzzy scores of 0.877 and 0.951, respectively.
While the percentage of agreement was slightly lower (84.62%), the panel still agreed on the importance of providing choices for difficulty levels, with a fuzzy score of 0.795.Furthermore, the panel unanimously agreed on the importance of dividing assessments into parts, tracking progress and performance, and having a leaderboard for performance comparison, with fuzzy scores of 0.936, 0.905, and 0.921, respectively.
The findings of the Fuzzy Delphi Method (FDM) analysis align closely with the research objectives, which aimed to assess the functionalities of MUETBot based on expert consensus.The high level of agreement among the expert panel indicates a clear consensus on the importance of incorporating specific features into MUETBot to enhance students' preparation for the MUET exam.By providing functionalities such as sample MUET reading papers, explanations for reading skills, and instant feedback on answers, MUETBot can effectively support students in improving their reading skills, which are key objectives of the study.
The expert panel's views and expectations have significant implications for the design and development of MUETBot.The consensus among experts indicates that these functionalities are considered essential for MUETBot to be effective in supporting students.Therefore, it is crucial for the developers of MUETBot to prioritize the implementation of these features to ensure that the tool meets the needs and expectations of its intended users.Additionally, the findings suggest that MUETBot should be designed to be user-friendly and accessible, as these aspects are likely to enhance its effectiveness and usability.

Conclusion
This study's key findings through the application of the Fuzzy Delphi Method (FDM) clearly indicate that experts concur on the crucial functionalities for MUETBot, a support tool for students preparing for the Malaysian University English Test (MUET).Specifically, the consensus among the expert panel affirmed the necessity of including distinct features within MUETBot, such as providing sample MUET reading papers, detailed explanations for reading skills, and immediate feedback on answers, which are instrumental in enhancing the MUET preparatory process.These findings demonstrate the pivotal role of expert agreement in determining the essential features of educational technology tools and underscore the need for such features to be prioritized in the design and development of MUETBot.
Acknowledging the limitations of this study, particularly the homogeneity of the expert panel, it is suggested that future research should engage a more diverse group of experts to attain a wider spectrum of insights.Furthermore, it would be beneficial to extend the study beyond the scope of MUETBot's functionalities to encompass additional factors that may influence its effectiveness, such as technological limitations and user preferences.
In light of these findings, it is recommended that subsequent research should assess the actual impact of MUETBot on students' performance in the MUET exam and explore its usability across a broader educational context.There is also a need for continuous enhancement of MUETBot, incorporating the feedback from both students and educators, to ensure that it remains a relevant and effective learning aid.Thus, this study lays the groundwork for future endeavours in advancing instructional technology to support language proficiency and academic success.

Table 2
Functionalities of MUETBot based on Fuzzy Delphi Analysis (FDM)