Evaluation of Automotive Technology Diploma Program at Malaysian Vocational College: Validity and Reliability

Program evaluation is a systematic approach to gathering, analysing, and applying data to address issues with initiatives, policies, and programs, especially with regards to their efficacy and efficiency. Therefore, this pilot study aims to test the reliability of four dimensions in this study, which is the context dimension which has two sub-dimensions, namely curriculum goals and curriculum objectives. The respondents for the pilot study to be 25 people because the number of the population for the entire respondent is limited. The study findings show the Cronbach's alpha value for each pilot study variable, the context variable has a value of 0.902, the input variable has a value of 0.930, the process variable has a value of 0.937, the product variable has a value of 0.909 and the entire research instrument has a value of 0.979. The research instrument has high reliability and therefore the next step can be continued. Therefore, this study can be continued to the next level and carried out on 51 vocational colleges which will involve 132 to 136 respondents.


Introduction
The branding of vocational secondary schools to Vocational Colleges (KV) began in 2012 with the implementation of Vocational Education Transformation.KV is a post-secondary vocational education and training institution that implements the Vocational College Standard Curriculum (KSKV) to form human capital with character, knowledge, and high skills.Vocational High Schools (SMV) which only produced Malaysian Certificate of Education (SPM) graduates at one time, are now recognized for producing graduates with Malaysian Vocational Diploma (DVM).SMV's name has been changed to Vocational College since 2012.The impact is that KV is a university-level higher education institution and its first graduate graduated in 2015.As of June 2015, the Fourth Cohort of the Automotive Technology Diploma (DTA) Program has successfully completed their studies at KV and received their degrees on in September 2015 at Universiti Sains Islam Malaysia, Nilai, Negeri Sembilan.In fact, it aims to give value to KV's mission in producing skilled and professional workforce.

Problem Statement
Program evaluation is important to see the efficiency, effectiveness, and weaknesses of implementation either during the implementation of the program or after five months of its implementation (Ghazali et al., 2017).Educational institutions can be guaranteed their quality when program evaluation becomes a routine institutional practice to see the rate of return on investment made.The most critical aspect is the measurement of course learning outcomes.It is very difficult to design and implement when measuring the impact resulting from any determined curriculum implementation (Sharifah, 2017).In the context of learning at KV, aspects of teaching qualifications, lecturer personality and lecturer competence have a direct relationship with the quality of teaching in the lecture room (Fah & Osman, 2011).The lecturer's performance in delivering lessons includes the ability to plan, implement, and evaluate the teaching and learning process (Mundarti, 2007).The course evaluation study can see the level of the lecturer's teaching in the lecture room at KV. Aspects of lecture room facilities and workshops are also found to have an impact on the quality of student learning.Aspects of educational facilities such as classrooms for learning and teaching and learning resources have implications for teaching and learning.The lack of equipment will affect the quality of teacher training, past studies in education evaluation found that aspects of learning facilities are not adequately provided.

Research Objective
This pilot study aims to test the reliability of four dimensions in this study, which is the context dimension which has two sub-dimensions, namely curriculum goals and curriculum objectives.The second dimension is the input dimension which has four sub-dimensions namely student content, pedagogical skills, evaluation, and assessment as well as selection and use of teaching and learning (P&P) resources.The third dimension is the process dimension and has four sub-dimensions which are implementation of P&P based on lesson content, implementation of pedagogical skills, implementation of evaluation and assessment and implementation of selection and use of P&P resources.The last dimension is the product dimension which has one subdimension which is the effectiveness of DTA curriculum implementation.

Significance of the Study
The importance of quality, effective and excellent novice teachers to ensure that learning among students is also of quality cannot be denied producing best practices for professional development.Researchers believe it can bridge the gap between the missions to be achieved, including any problems in the best practices of beginning teacher professionalism.This study will also be beneficial in building best practice instruments of beginning teacher professional development and, in turn, assisting policymakers and strengthening teacher professional development programs.Therefore, a conceptual framework must be carefully planned to ensure that this study can run smoothly and effectively.

Content Validity
Content validity refers to the degree to which an assessment instrument is appropriate for representing the target constructs it is designed to measure.The researcher conducted content validity by appointing eight experts.The experts are appointed based on their expertise in the field of educational leadership, educational policy, educational administration, educational management as well as psychometrics and educational evaluation.To carry out content validity, at least two experts are required, experts are asked to examine each item, match it with the respective domain, and evaluate the relevance of each item to the content (Thorn & Deitz, 1989).To conduct content validity for this study, the researcher has appointed four experts with one for psychometric validity.
The instruments of this study were taken from past researchers and in connection with that, experts in the field of leadership and education were consulted to get their feedback.
Experts were asked to evaluate this research instrument from the aspects of appropriateness, importance, content, clarity of meaning and psychometrics for lecturers who teach DVM.All suggestions and comments from experts are considered and used in the process of refining the instrument to ensure that the meaning, language, and content of the instrument used in this study are appropriate.

CONTENTS OF THE LESSON
1.I was able to convey the contents of the Automotive lesson that will be taught.2. I can translate the contents of the Automotive lesson that will be taught.2. I can use various approaches (questioning, discussion, quiz etc) to assess student understanding.3. I was able to provide Automotive assignments to students.4. I was able to administer Automotive tests and exams to assess student understanding. 5.I am able to continuously assess students' understanding of Automotive.6.I can check student assignments to assess student understanding.7. I was able to check the students' Automotive assignments to find out the students' weaknesses and strengths.D 4

SELECTION AND USE OF R&D RESOURCES
1.I was able to surf the internet to access Automotive P&P resources.2. I was able to use library materials (print media) to obtain Automotive P&P resources.3. I was able to use other electronic media to get Automotive P&P resources.4. I can find alternatives if Automotive reference sources are not sufficient.5.I was able to effectively use a variety of Automotive P&P resources.E1 DTA CURRICULUM IMPLEMENTATION EFFECTIVENESS 1.I strive to produce automotive technologists who appreciate FPK.
2. I am able to check students' assignments to assess their understanding.
3. I am able to review students' assignments to assess their weaknesses.4. I try to assess the ability of students to apply Automotive knowledge.5.I try to evaluate the effectiveness of improving students' Automotive knowledge.I am able to evaluate the effectiveness of improving Automotive skills to students 7. I am able to administer tests and exams to assess students' understanding of Automotive 8.I am able to collect student achievement information by using an appropriate assessment approach.9.I am able to analyze student achievement to see their understanding by using various appropriate assessment approaches 10.I am able to evaluate the effectiveness of teaching based on student learning problems.11.I am able to use the evaluation results to plan the improvement of Automotive teaching in the future 12.I am able to use peer feedback to evaluate Automotive teaching 13.I am able to modify the teaching approach based on student evaluation feedback.

Pilot Study
A pilot study is the first step of the entire study protocol and is often a small-sized study that helps in the planning and modification of the main study (In, 2017).A good rule of thumb for conducting pilot studies is on at least 12 to 50 people before conducting a full-scale study (Sheatsley, 1983;Sudman, 1983).Respondents that exceed 50 people during the pilot study will provide the researcher with more information and problems that may arise, but the benefits are decreasing, because the financial and time costs for the pilot study are higher than the benefits of discovering small issues and inconsistencies in the instrument (Ruel et al., 2016).Therefore, in this study, the researcher has set the respondents for the pilot study to be 30 people because the number of the population for the entire respondent is limited.However, only 25 respondents answered the distributed questionnaire.

Reliability of Research Instruments
The data obtained from the pilot study was analysed using SPSS with the internal consistency method and the method used was through the calculation of the Cronbach's alpha coefficient index of the research instrument.Reliability is a concept related to the consistency and stability of an instrument (Creswell, 2013).Table 2 shows the Cronbach's alpha value for each pilot study variable, the context variable has a value of 0.902, the input variable has a value of 0.930, the process variable has a value of 0.937, the product variable has a value of 0.909 and the entire research instrument has a value of 0.979.The research instrument has high reliability and therefore the next step can be continued.(Hassan et al.,2019) and carried out on 51 vocational colleges which will involve 132 to 136 respondents.This also shows that the adapted instrument is suitable for vocational college lecturers and it is hoped that the research conducted will provide the best results.
understand the requirements of DTA Education in Malaysia.2. I understand the goals of the National Education Philosophy. 3. I understand the Objectives of the Malaysian Vocational College 4. I understand the learning outcomes of the DTA curriculum 5.I understand the Objectives of the DTA curriculum 6.I understand the Mission and Vision of DTA B 2 OBJECTIVE 1.I am ready to impart knowledge related to DTA subject components.2. I am willing to translate scientific skills in the subject component of DTA.3.I am willing to master knowledge in the field of Automotive. 4. I am willing to master all skills in the Automotive field. 5.I am ready to achieve the learning outcomes of the DTA curriculum with the course objectives.6.I am willing to master automotive skills based on generic skills.7. I am ready to achieve the learning outcomes of the DTA curriculum with the students' apply the contents of the Automotive lesson that will be taught.2. I translate the contents of the Automotive lesson that will be taught.3. I translated the content of the Automotive lesson based on the syllabus (pro-forma) 4. I deliver the content of Automotive lessons based on the latest reference sources. 5.I organized the content of Automotive lessons from concrete to abstract.C 2PEDAGOGICAL SKILLS1.I apply various Automotive teaching activities to improve student understanding.2. I diversify Automotive teaching methods to enable students to use thinking skills.3. I vary the Automotive teaching approach to teach the same topic.
4. I create a variety of Automotive learning experiences for students.5.I modified the Automotive teaching strategy based on student feedback.C 3 EVALUATION AND ASSESSMENT 1.I plan assessment techniques based on student ability.2. I use various approaches (questioning, discussion, quiz etc) to assess students' automotive understanding.3. I provide Automotive assignments to students.4. I administer Automotive tests and exams to assess student understanding. 5.I continuously evaluate students' understanding of Automotive.6.I check the students' Automotive assignments to find out the students' weaknesses and 7. I made an evaluation of the effectiveness of Automotive teaching based on student learning problems.C 4 SELECTION AND USE OF R&D RESOURCES 1.I surfed the internet to access Automotive P&P resources.2. I use library materials (print media) to get Automotive P&P resources.3. I use other electronic media to get Automotive P&P resources.4. I Access alternative sources if Automotive reference sources are insufficient.5.I use a variety of Automotive P&P resources effectively.D 1 3. I was able to implement the content of the Automotive lesson based on the syllabus (pro-forma).4. I can plan the content of Automotive lessons based on the latest reference sources. 5.I can organize the content of Automotive lessons from concrete to abstract.D 2 PEDAGOGICAL SKILLS 1.I apply various Automotive teaching activities to improve student understanding.2. I use various Automotive teaching methods to enable students to use thinking skills.3. I use a variety of Automotive teaching approaches to teach the same topic.4. I provide students with a variety of Automotive learning experiences.5.I modified the Automotive teaching strategy based on student feedback.D 3 EVALUATION AND ASSESSMENT 1.I can plan assessment techniques based on students' abilities.

Table 1
Items of Instrument

Table 2
Cronbach's Alpha Value of Each DimensionConclusionOverall, this study focuses on conducting a pilot study starting from adapting this instrument.This research instrument was taken from past studies and adapted to lecturers at vocational colleges in Malaysia.Reliability of this instrument, it was found that Cronbach's alpha value has a value between 0.902 to 0.937, and according toCohen et al., (2018)Cronbach's alpha value >0.90 has a very reliable interpretation.Therefore, this study can be continued to the next level Abu