Construct Validity and Reliability in Content Knowledge of Design and Technology Subject: A Rasch Measurement Model Approaches for Pilot Study

Validity and reliability are the main important issues when developing a new instrument. This pilot study aims to test the validity and reliability of instruments in content knowledge for the Design and Technology subject in Malaysia. The content knowledge of RBT subject instruments that had been prepared consists of 94 dichotomous items and is distributed to 83 teachers. The purpose of the establishment of this instrument is to measure the five main constructs of this study, which is; (1) Introduction to Design and Technology; (2) Introduction to Design Project Management; (3) Product Making; (4) Introduction to Apply Technology; and (5) Introduction to Design and Technology in entrepreneurship. The approach used to examine the validity and reliability of the items and respondents in this study is emanate from the Rasch Measurement Model Approaches which is much more valid and well-grounded compared to just focus on the output produced by Cronbach’s Alpha. The Winsteps software Version 3.73 has been used to check on the functionality of the items in the aspects such as the item reliability and the separation of item-respondent, polarity item, the suitability of the item to measure the construct, the item difficulty level, and the respondent’s ability. It also allows the removal of items based on the statistics of polarity item and the suitability of the item. At the end of the analysis, it is found that there is a total of 10 items that were discarded because they did not meet the inspection criteria specified in accordance to the Rasch Model. The final instrument recorded a total of 84 items that can only be used to measure the five constructs of the study. Since this study was established as a pilot study, then the distribution made to the actual respondents can be carried out to measure the five main constructs of this study. This study shows that the Rasch Model can help researchers build a good instrument as the items constructed offset psychometric standards.


Introduction
The Design and Technology Curriculum also emphasizes the skills required by RBT teachers in handling this subject. Teachers need to communicate knowledge in an integrated manner with activities to be carried out to students. In contrast to the delivery of teaching and learning (PdP) theoretical subjects, RBT requires teachers to improve their skills in real situations (Alwi, Kamis and Ismail, 2018). Therefore, according to Harun (2014) has stated that to ensure that students are able to master RBT subjects, teachers need to have teaching skills, subject knowledge, mastering materials and technology and using the Tools / Learning Aid Materials (ABBM) according to the needs of the subject. Design and Technology (RBT) are the subjects offered at all primary and secondary levels in Malaysia. RBT aims to provide students with a sense of value, aesthetics, practical skills and technology with creative, critical, innovative, inventive and entrepreneurial thinking for them to develop communication skills and generate ideas for creating new products and systems that meet human needs and improve the quality of life (KPM, 2015).
KSSM RBT focuses on four domains such as Appreciation of Design, Technology Application, Product Creation and Product Design Evaluation. Pupils will apply knowledge and skills through project design and production activities. It is supported by Moore, Johnson, Peters-Burton, and Guzey (2016) in details on the STEM core which has links in line with Design and Technology (RBT) subjects which challenge students' potential by using Engineering Design approaches, to develop critical and creative thinking through relevant technology-assisted design activities. Students can learn from failure in designing solutions in Engineering designs by improving existing designs.
Malaysia's Achievement in the Current Aggregated World Ranking Design report for 2010-2017, Malaysia ranked 51 out of 97 participating countries with 30 award-winning groups and total number 101 points compared to the first position by the United States with 584 and 2168 (World Bank Institute, 2017) points out that efforts need to be intensified to improve this poor performance. This transformation demonstrates that teachers as a policy implementer should have the knowledge, skills and values in implementing the planned policies (Appanna, Tajularipin & Wulandari, 2015). Shulman (1986) says Content Content is an understanding of a subject as a discipline. According to him, content knowledge is a knowledge structure that includes the theory, concept and principles of a discipline of learning or subjects. Examples of Design and Technology (RBT) subjects offered at primary schools are different with their RBT offered at secondary schools. Therefore, to teach a subject, teachers need to have a good and up-to-date content knowledge (Koehler, Greenhalgh, Rosenberg & Keenan, 2017, Cherner & Smith, 2016Nordin, 2014). The findings show that teachers teaching in schools need to have knowledge of the theories, concepts and principles contained in the learning standard.

Analysis Using Rasch Model
Rasch model with the application of WINSTEPS version 3.72 was used to analyze the data as well as to test the validity and reliability of the instrument. The Rasch model incorporates a method for ordering person according to their ability and ordering items according to their difficulty Fox, 2015). According to Bond & Fox (2015), the criteria in Table 1.0 below used as benchmarks for determining the validity of the instrument. Knowledge items are dichotomous. Item compatibility starts with Mean Square (MNSQ) value not exceeding Mean Infit (MNSQ) with + / (-) S.D). Misfit and Outlier can be detected by seeing ZSTD values larger or out of limit t +/-2logit (Azrilah et al., 2015). The MNSQ range should be at the range of 0.70 logits up to 1.30 logits (Bond & Fox 2015) for dichotomic or multiple-choice test items (knowledge items). The ZSTD value is the accepted value between -2.0 to 2.0 (Bond & Fox, 2015) and according to Fox and Jones (2005) the ZSTD value can be ignored if MNSQ has been accepted. The Rasch model approach is used to look at the validity and reliability of the instrument more deeply through some diagnosis. Only four diagnoses were performed: checking to functional items in terms of (i) item reliability and separation; (ii) detect the polarity of items that measure the constructs based on the value of PTMEA CORR; (iii) the fit of constructing items; and (iv) determine the dependent items based on the standard residual correlation value; (v) determine the item difficulty level and the ability of the respondents. These four diagnoses complement the requirements required to verify the validity and reliability of the instrument due to the objective of the study only to produce a measurement model and a structural model. This is because if the researcher wants to produce a questionnaire instrument then all the diagnosis of Rasch's model should be followed as described earlier.
The pilot study was conducted to obtain construct validity and item reliability using Rasch model which produced 94 knowledge items of a tests instrument that had been developed by the researcher through the quantitative data collection.

Objective
This study aims to test the validity and reliability in content knowledge of Design and technology instruments for teacher at the secondary school using Rasch analysis. The objectives of this study are to: (1) Testing the reliability and separation index of the item and respondent; (2) Identify the polarity item that measures the constructs; (3) Examine the suitability of item (item fit) of the instrument; (4) Detect unidimensionality of construct; and (5) Determine the item difficulty level and the ability of the respondents.

Methodology
This pilot study was carried out by using a quantitative approach by distributing the test instrument regarding the content knowledge of design and technology subject to the selected respondents. The sample for this pilot study involved a total of 83 teachers which specifically teaching in design and technology subject for secondary school. According to , the number of respondents which is suitable and considered as adequate for the pilot study is between 25 and 100 people. The findings generated from this pilot study will then be analyzed using the Winsteps software Version 3.73 alongside the Rasch Measurement Model Approach. The content knowledge of design and technology subject instrument that had been constructed consists of 94 items which comprises the five main constructs, namely (1) Introduction to Design and Technology; (2) Introduction to Design Project Management; (3) Product Making; (4) Introduction to Apply Technology; and (5) Introduction to Design and Technology in entrepreneurship.

Results and Findings
In accordance with the Rasch Measurement Model Approach, the researcher had conducted a test on the functionality of the item in terms of (1) the reliability and separation index of the item and respondent; (2) the polarity item that measures the constructs of the study based on the value of PTMEA CORR; (3) the suitability of item (item fit) of the instrument; (4) the unidimensionality of construct; and (5) ) the map of item difficulty level and the ability of the respondents. The description and explanation for each item tested on the functionality are as follows.

Reliability and Item Separation
Based on the Rasch Measurement Model Approach, the value of Cronbach's Alpha (α) that its reliability can be accepted is between 0.71 -0.99 where this value is at its best (71% -99%) as described in Table 1.2 (Bond & Fox, 2007). Very good and effective with a high level of consistency Good and is acceptable Acceptable The item needs refinement The item needs to be discarded Source: (Bond & Fox, 2007) To determine the reliability of the respective items, the statistical analysis by using the Rasch Measurement Model Approach was used with reference to the reliability value and the value of the item separation. The result of the analysis found that the reliability value obtained based on Cronbach's Alpha (α) value was 0.97 as shown in Table 1.3 below. The value obtained clearly shows that the instruments used are in very good condition and is acceptable, thus it can be used in real research. The analysis for the entire instrument was also performed by looking at the reliability value and the separation values of the items and respondents. Based on Table 1.4 as shown below, the reliability value of the item is 0.83, which indicates that it is in good condition and is acceptable (Bond & Fox, 2007). Meanwhile, the value of item separation is 2.20, this value can be used because the items are new items. As suggested by Fox and Jones (2005), the value that shows a good index separation is a value that is greater or more than the value of 2.0.

Table 1.4: Reliability and Item Separation Value for the Entire Construct Instruments: Pilot Study
Whilst, based on Table 1.5 below, the reliability value of the respondents is 0.94 and the respondent's separation value is 3.91. This shows that the reliability of the respondents is very good and effective with a high level of consistency. The value of the item and respondent separation which is more than 2.0 is considered as good (Fox & Jones, 2005;Bond & Fox, 2007).

Table 1.5: Reliability and Respondent Separation Value for the Entire Construct Instruments: Pilot Study
Polarity Item by PTMEA CORR Value The Point Measure Correlation or PTMEA CORR value examination was made to identify the polarity items in the study, which is intended to test the extent of which the established constructs can achieve its goals. If the value found in the PTMEA CORR section is positive (+), then it indicates the respective item can achieve its goals of measuring the construct that needs to be measured (Bond & Fox, 2007).
In contrast, if the value is negative (-), then the established item does not measure the construct that needs to be measured. So, the item needs to be revised or discarded because it does not address the question, or it is too difficult for the respondents to answer it. Based on Table 1.6 below, there are no items show negatif value and near to value 0. Based on the findings, it shows that the items are positively moving in one direction to measure the constructs. And it does not contradict with the constructs that need to be measured.

Item Fit in Measuring the Constructs
Item fit measured the constructs through the infit and outfit Mean Square (MNSQ). According to Bond and Fox (2015), the outfit and infit MNSQ should be in the range of 0.70 to 1.33 to ensure that the items are suitable for measuring the constructs. But the outfit index MNSQ noteworthy in advance compared infit MNSQ for determining congruity of items that measure a construct or latent variable (Sumintono, 2017). If the infit or outfit MNSQ value more than 1.33 logit, then it gives the meaning of confusing item. If the MNSQ value is less than 0.70 logit, it shows that the item is too easily anticipated by the respondents (Linacre, 2007). Beside that the outfit and infit ZSTD value should also be within -2.00 to +2.00 (Bond & Fox, 2015). But if the outfit and infit MNSQ be accepted, then the ZSTD index can be ignored (Linacre, 2007).
Therefore, if this condition is not met, then the item should be either removed or refined. The Table 1.7 below shows the misfit order featuring items having the largest MNSQ and the smallest MNSQ analysis statistics: misfit order. Based on Table 1.7 below, found that 14 items from introduction to design and technology, 7 items from introduction to design project management, 9 items from product making, 20 items from apply technology and 1 item from introduction to design and technology in entrepreneurship that are not in the specified range and it should be revised or refined.  , 2015). Researcher refers to two criteria intesting the unidimensionality of an instrument namely the value of Principal Component Analysis of Residual (PCA) and (ii) the level of distortion of items or unexplained variance 1 st contrast (Azrilah et al. 2015). According to Runnels (2012) the good PCA value is at least 20% and more than 40% and the unexplained variance 1 st contrast is 15% maximum (Azrilah et al., 2015). Local independance is a value referring to the individual abilities of an item is not related to another item in the same construct. Values that meet local independance requirements are less than 0.7 (Linacre, 2007). Table 1.8 below represents the findings of Principal Component Analysis (PCA) based on variance explained by measure for content knowledge. The PCA value for content knowledge 30.9% is accepted as it exceeds 20%. The value of unxplained variance by 1 st contrast (size) to be in the desired specification is the content knowledge is 5.8%. shows an item having a residual value correlation that exceeds 0.7 logits, P4B33, P4B34, P4B25, P4B39, P3B16, P3B17, P1B15 and P1B16 items. All these items go through the filter process by looking at the value of seeing MNSQ values approaching the value of 1.00 and ZSTD approaching the value of 0.00. After the filtering process, P3B16 and P3B17 items have been dropped.  Figure 1.10 below represents item difficulty locations and distribution of examinees along the logit scale. Item difficulty measures from +1.21 to -1.43 logit. Meanwhile, the respondents' ability estimates from +1.53 to 0.16, which is slightly higher than the item difficulty measurement. The mean for both measurements is approximately around the same location, thus indicating that the items for this sample are well targeted. The map has greatly assisted the researcher in locating the area where most items are located particularly to see whether this is parallel with the spread of the respondents. .10 shows the number of respondent's ability and item difficulty on the logit scale. All the items are scattered and point towards the ability of respondents' diversity. Respondents that have high satisfaction located at the above on the scale, while the respondents that have low satisfaction are located below on the scale. The most difficult items are introduction to design and technology: P1B15 (1.82 logit) which located on the near upper scale. While the easiest item is introduction to design and technology: P1B1 (-2.23 logit). This shows that the difficult items can be answered by the highly capable respondents, while the easier item can be easily answered by the respondents of high ability and low ability (Linacre, 2007).

Discussions and Conclusion
After data analysis, each item is being revised following the standard index and the conditions that must be followed to achieve the standards of validity and reliability of the instrument based on the Rasch measurement model. The item removal refines, and purification were conducted by referring and considering the views and expert evaluation. Based on the results obtained, there are 11 items that do not meet the requirements analysis and should be discarded. Whereas 40 items are appropriately refined in accordance with the context and significance of the study. 43 items were retaining from 94 items. Overall summary of the related items in the questionnaire is shown in Table 1.11 below. Based on this research, it can be concluded that the validity and reliability of an instrument are a very important aspect to consider in developing a new instrument for a study. Overall from this analysis, it is found that a total of 11 items that were dropped are questionable items on validity and reliability. Thus, based on the validity and reliability test made on this instrument, it indicates that this instrument is fits to be used by school or other researchers for future study. The implications of this analysis help researchers in developing a good instrument for the school subject.