The Validity and Reliability of The Teaching Supervision Instruments in Culinary Field: An Analysis of The Rasch Measurement Model

The implementation of teaching supervision is mandatory for all teachers. Feedback in teaching supervision is a very important aspect. Not only that, supervisors need to have sufficient skills, updated knowledge on the changes in subject syllabus, and reasonable recommendations to improve the teachers they supervised. There is a notion that an accurate teaching supervision model using specific instruments needs to be created so that the teaching and learning supervision can be carried out more effectively. Along with the issues and focus, this study aimed to test the validity and reliability of teaching supervision instruments in the culinary field. The design of the study involves the development of instruments based on the data gathered from interviews, where this study utilised a Modified Delphi Methodology approach. The researcher conducted a test on 100 respondents who were among administrators tasked to carry out teaching supervision in classrooms. The Rasch Measurement Model with Winsteps 3.73.0 software was used to obtain the validity and reliability of each item in the study of teaching supervision in culinary field. In the instrument evaluation phase, the data analysis went through five stages to fulfil the criteria and conditions for instrument usability. The results of the analysis show that of the 45 items in the teaching supervision in culinary field study, two items were dropped and 43 remained. With this, the research instrument can be used as a reference for school administrators in the actual teaching supervision.


Introduction
The supervision of teaching and learning is vital to ensure the effectiveness of any teaching method. According to Glickman, Gordon, and Ross-Gordon (2014), the teaching supervision is akin to the "glue of a successful school", where it becomes an essential factor in guaranteeing school's efficacy. In short, when teachers and students have the same objective in class, the teaching and learning process is considered complete. This aspect is also in the same vein with teachers' supervision by the school administration, where if done consistently, drives the attainment of school's objective. To this end, a teaching supervisor must have the knowledge, interpersonal skills and technical skills to accomplish the teaching observation successfully (Anizam & Farzeeha, 2014).
At school level, teaching supervision is compulsory in all classrooms. This exercise has been mandated by the Ministry of Education Malaysia through the Professional Circular No. 3/1987: The Supervision of Teaching and Learning in Classrooms by School Principal/Headmaster. The circular explains that headmasters and principals in schools are required to monitor and supervise the teaching and learning that happens in classrooms. The circular also includes a clear guideline to implement the supervision process.
To make sure that the teaching and learning supervision runs smoothly, headmasters and principals must select the teaching supervisor among the school administrator body, such as the Senior Assistant of Administration and Curriculum, Senior Assistant for Students Affairs, Senior Assistance for Co-Curriculum, Afternoon Session Supervisor, Heads of Subjects, and Head of Subjects Committee. However, the headmaster or principal assumes the final responsibility in the supervision of teaching and learning.
Jemaah Nazir Sekolah (JNS), a professional body authorised in supervising teaching and learning at schools, reported in the Malaysian Education Blueprint (2013 -2025) the different understanding that schools and Jemaah Nazir had regarding the meaning of quality teaching and learning. The outcomes from the School Self-Evaluation Report done by schools' administrators in Malaysia showed 63% schools determined they achieved 'Good' and 'Excellent' ratings for their teaching and learning process. However, the evaluation done by JNS revealed only 13% schools in all of Malaysia actually reached 'Good' and 'Excellent' level. This demonstrated the dissimilar understanding mentioned earlier, even though both parties utilised the same evaluation instruments. This raises the questions as to how schools and JNS define quality teaching and learning, when both were given the same tools to determine the ratings. To present the perspective of how the current supervision of teaching and learning is done, Abdul Rahman (2011) summarised the situation by saying how a physics teacher who graduated in 2000 needs the support and advice from a principal who obtained a degree in Malay Literature in 1980. Ong'ondo and Borg (2011) confirmed this observation by detailing how supervisors' feedback mostly focused on the common pedagogical aspects of teaching that had little to no relation with the subject taught. It is clear through this revelation that the teaching supervision in schools concentrates on the methodology side of teaching but leaves the teachers unclear on how the methodology can help better deliver the subjects involved.
The same problem was also detected by Hamidah, Jamal, Sharifah, and Syed (2016) who found teachers to provide unfavourable responses when inquired about the feedback of the supervision they received. This is worrying, as Weber, Chandler, and Finley (2016) highlighted the importance of feedback to guarantee a continuous improvement, where the readiness and acceptance of comments on the teacher's performance must be present. Further, Hamdan and Nurlia (2015) also agreed on the significance of supervision feedbacks. They opined that supervisors must have sufficient pedagogical knowledge, updated knowledge of subject syllabus they are assessing, and the skills to identify areas of improvements.
teachers' pedagogical skills elucidated a lack of differentiation between general practice of teaching from those relevant for teaching culinary subjects (Feng, Su, & Yang, 2011). The same issue was also raised by Ambarwangi and Rachman (2015) and Paulsen and Martin (2013), where they put forth the need to develop an accurate model for teaching supervision for culinary subjects to warrant an effective teaching and learning supervision.
As such, it stands to reason that the instrument to supervise teaching and learning must be suitable with a particular field, which in this case, the culinary field. Any instruments developed should be put to the test to determine their efficacy to be used in all culinary schools in the country. In line with the issue and focus of this study, this research aims to test the validity and reliability of teaching supervision instruments in the culinary field.

Research Methodology
The methodology used in this study involved the instruments developed through interviews, where the Modified Delphi Methodology was utilised. Through this method, the researcher obtained data from interviews with culinary experts, literature reviews and documents analysis. Next, the researcher developed an instrument based on the data gathered. This instrument was put through the validity and reliability test by administering a survey on culinary teachers in the instrument evaluation phase.
The instrument evaluation phase began by developing the questionnaire on teaching supervision for culinary field from the JAK analysis in the second and third cycle of Modified Delphi Methodology. This was followed by two pilot tests. An assessment was conducted to determine the suitability of the instrument with the identified constructs. The validity of the constructs obtained from the actual study proved that the constructs in the instrument were appropriate. The researcher then carried out an actual test on 100 respondents, which consisted of administrators who conducted teaching supervision in secondary schools.
The Rasch Measurement Model with Winsteps 3.73.0 software was used to determine the validity and reliability of each item in the teaching supervision in the culinary field study. The Rasch analysis was utilised to measure the difficulty level for each item in the questionnaire, which aided in the instrument refining process. According to Wright and Mok (2004), the Rasch Model is useful for the measurement process in social sciences because it meets the following five criteria or assumptions: (a) produces a consistent size with the same time interval, (b) provides accurate estimation process, (c) detects misfits or outliers, (d) overcomes missing data and (e) produces nonrepetitive measurements (independent of the parameters examined). The validity of the instrument can be identified by referring to key analysis such as item polarity, item-individual map, itemindividual mismatch, item-segregation, unidimensional, item-matching and rating scale.

Data Analysis
In the instrument evaluation phase, the data analysis must go through several stages by fulfilling the conditions and requirement for instrument usability. They are: STAGE 1 Testing the reliability and items isolation/respondent index. Table 1 shows the summary of the reliability and items segregation index for the constructs in the Teaching Supervision for Culinary Field research. The items reliability index shows a high value of 0.97. Further, the items segregation index indicates the respondent's ability to isolate the difficulty of the items to several levels in the measured construct. Index for the isolation of the items in the Teaching Supervision for Culinary Field study shows a good value of 5.66. Table 1 Reliability and Index of Item Isolation for Teaching Supervision Instruments in the Culinary Field: Actual Study  Table 2 shows the summary of the reliability and isolation index of the respondents measuring the constructs in the Teaching Supervision for Culinary Field study. The reliability index of the respondents recorded a high value of 0.81. The items segregation index indicates that the quality of the items is capable of indicating respondents to several abilities. Further, the isolation index of respondents showed a good value of 2.06.    Table 3 shows the summary for the items polarity that measured identical constructs. The value for items polarity, The Point Measure Correlation (PTMEA), shows positive value with 0.3 logits. Items that complied with the set PTMEA values confirmed that they were measured in only the same direction. Items A10 and B11 were dropped as their minimum PTMEA values were less than 0.3 logits.  Table 4 shows the summary for the mismatched constructs in the Teaching Supervision in the Culinary Field study. Mean-square (MNSQ) and z-standardised (ZSTD) was used to detect data that contradicted the Rasch measurement model. The MNSQ value for Likert scale items is between 0.60 to 1.40 logits. Items outside the MNSQ value rate need to be dropped or repaired. The ZSTD values indicate whether the data matches the Rasch measurement model perfectly. It also represents the probability of the data being significant if the data corresponds to the Rasch measurement model. The ZSTD values range from -2.00 to +2.00. However, the researcher decided to ignore the value for ZSTD if the MNSQ value was accepted with reference to Linacre (2011). A total of 15 misfit items were identified out of 45 from the Teaching Supervision in Culinary Field study based on the value of the MNSQ infit/outfit index. They were three items that were taken from the Planning and Preparation constructs, four items from the Execution and Presentation construct, two items from the Evaluation Method construct, four items from the Objectives Attainment and Reflection construct, and two from Teacher's Professional Responsibility construct.

STAGE 4 Determining and Detecting Items Measuring Single or Unidimensional Constructs (Standard Residual Variance)
Dimensional uniformity ensures that the instrument can be firmly measured and achieved based on the Residual Principal Component Analysis technique. This technique is able to detect the ability of an instrument to measure in a uniform dimension with an acceptable level of interference. Table 5 shows a summary of the Principal Component Analysis (PCA) based on the variance explained by measuring the competency constructs. The PCA value is accepted because it exceeds 20%, which is 31.2%. For unexplained variance by 1st contrast (size), the degree of interference of the item in one contrast is accepted if it is worth less than 15%. The value of unexplained variance by 1st contrast (size) for the constructs in the Teaching Supervision in Culinary Field research is 8.6%, which is within the acceptable value.  Looking at Table 6, six items have residual correlation values that are more than 0.7 logits. This situation was caused by respondents being confused by items that shared overlapping qualities. These items need to be repaired or removed by looking at their MNSQ and ZSTD values. The item with MNSQ infit value approaching 1.00 and ZSTD value approaching 0.00 should be retained, and the other should be removed. Table 4.32 shows the items that were dropped after going through the screening process by looking at the MNSQ value approaching the value of 1.00 and the ZSTD approaching the value of 0.00. After going through the screening process, no items were dropped.
Determining and confirming the level of difficulty of items and the abilities of respondents. Table 7 shows that Teaching Supervision in Culinary Field research had respondents who were able to answer items with the highest level of difficulty. The measurement value of the 11th and 46th respondents is +5.63 logits, above the value of item C3, which is +3.41. Figure 1 shows the Wright Map item distribution and respondents' abilities for the items in the Teaching Supervision in Culinary Field study. There is also an easy item that can measure weak respondents, which is B11 at -2.45.    The final findings of the study at the validation stage for the five constructs of food preparation and serving instruments showed item reliability index of 0.97 and respondent reliability of 0.81 then this study instrument is acceptable, high and excellent in parallel as suggested by Bond and Fox (2015). This indicates that in general the Food Preparation and Presentation instruments are consistent and stable when administered on other samples that have similar and almost identical characteristics.

Conclusion
In conclusion, the Food Preparation and Presentation instruments are of good quality based on the reliability index of items and respondents as well as the isolation index of items and respondents is achieving standards based on Rasch measurement model. The research instrument can be used as a reference for school administrators and to be adapted in the actual teaching supervision. The development of this teaching supervision instrument can also assist school administrators who do not have the background in culinary arts to still carry out effective teaching supervision.
Researcher suggest several further studies that can be done so that further studies can provide information in the preparation of instruments in the field of food preparation and serving. The suggestions are (i) build instruments related to theoretical teaching supervision in the field of food preparation and serving, (ii) conduct research on the construction of other vocational field supervision instruments, and (iii) Add construct and item instruments in the field of food preparation and serving to further strengthen the instrument.