Validity and Reliability of Competency Analysis Instrument for Cooperative Board Members using Rasch Measurement Model Approach

Reliability


Introduction
Until now, there is a limited number of literatures in the topic of Co-operative Board Members (CBM) competency in Malaysia. A study by Din et al (2012) that analysed school CBM competency only reviewed 100 grade A schools' cooperatives and did not comprehensively review all CBM job scopes. Another work by Ali et al (2020) studied the influence of interpersonal traits competency, involvement effectiveness competency, employeecommunity relationship competency, and financial and planning competency of CBM on cooperative's performance. This study found that CBM failed to perform the competency elements related to their cooperative job scopes.
In general, competency means eligibility, ability, willingness, and efficiency in performing a task (Abbas, 2018). Competency refers to factors required by a person to perform his/her job well (Ahmad & Khalid, 2009). Various researchers have found that competency is the combination of three main elements namely knowledge, skills, and attitude, which are required to perform a responsibility or task (Abdullah, 2017;Fuad, 2016;Yusoff & Liew, 2002;Triwani et al., 2019). Competency can also be defined as the ability of a person to perform a job (Shariff, 2015). It can be linked to an individual's work performance Erman (2017) and may influence an organisation's performance (Ahmad & Khalid, 2009a). Among the criteria required for a cooperative to be listed as one of the 100 Best Cooperatives in Malaysia include high revenue and good governance (Kassim, 2021) that can be realised via the appointment of competent Board Members who are efficient in conducting their tasks and responsibilities (Abd Rahman & Zakaria, 2018).
Previous studies have found that it is crucial to review a person's specific job scopes in order to analyse the person's competency (Ahmad & Khalid, 2009b;Baharuddin & Hashim, 2004). In this study, researchers developed an instrument determination table to come out with a questionnaire to study and analyse current CBM competency level as well as their competency gaps in the case of Malaysia. Researchers adapted 5 CBM competency elements by Din et al (2012) namely (i) management and administration, (ii) financial management, (iii) strategic management, (iv) governance management, and (v) business management, taking into account the basic competency elements (knowledge, skills, and attitude) and other source materials including Co-operative Societies Act 1993, Co-operative Regulations 2010, by-laws on roles and responsibilities of CBM in managing cooperatives, and governance practices guidelines. The instrument development process also involved two experts, which consisted of competency elements experts and Rasch Measurement Model experts.
Instruments are crucial for gathering data or information to collect data or responses that are in line with their study goals. However, due to differences in geographic locations, culture, belief systems, and individual characteristics of the respondents, the results of a similar instrument administered in several situations may vary. Thus, measurement tools employed in research must be legitimate and dependable in order to ensure that they are beneficial. The instrument developed was tested using a pilot study. The pilot study was used to measure the consistency of each instrument item (Nur Yunus et al., 2017) and can be utilised to minimise errors in the actual study by verifying the instrument's validity and reliability. According to Wainer and Braun (1988), consistency is when the same item is tested several times on the same subject at different time intervals, the score results or the answers given are approximately the same. The pilot study was analysed using several data analysis procedures to confirm the validity and reliability of the instrument developed. It is crucial to determine the validity and reliability of an instrument to verify that it is accurate and free from defects. For this research, the researchers conducted the validity and reliability tests using Rasch Measurement Model to ensure that the instrument's validity and reliability can be proven empirically.

Literature Review
Rasch Measurement Model is a measurement model that was developed considering each respondent's ability to answer a questionnaire, test or instrument and the level of difficulty for each item or test (Rasch, 1980). Normally, researchers refer to Cronbach's Alpha reliability coefficient to measure item's reliability level in an instrument, where, in normal measurement, the reliability is often referred to based on the True Score Test Theory (TSTT) or known as the classical model.
Validity and reliability of items included in a research instrument can be verified using Rasch Measurement Model (Rasch, 1960). Based on results of the pilot study conducted, this model was able to produce several types of diagnoses in analysing the validity and reliability of instrument's constructs as provided in Table 1. This model is becoming more popular among researchers who require an accurate measurement tool (Jalil & Siew, 2022;Jumiatmoko et al., 2021) and used to determine the validity and reliability of a research instrument (Alias et al., 2019;Ariffin et al., 2010;Kamis et al., 2012;Katip et al., 2017;Salleh et al., 2015;Said et al., 2018).

Methodology
The study was conducted using Rasch Measurement Model, and the instrument implemented involves a quantitative method. Validity and reliability were measured using Winsteps version 5.2.2.0 software. Questionnaire for the study was distributed via Google Forms to 30 respondents who held the positions of CBM in cooperatives in the state of Johor. The participants involved in the pilot study will not be included in the actual study. According to Johanson & Brooks (2010), 30 is the minimum and sufficient number of respondents for analysing validity and reliability of a pilot study.
This study analysed five main constructs, three basic concepts and contained 43 items. The five main constructs are management and administration, financial management, strategic management, governance management and business management; while the three main concepts are knowledge, skills and attitude. In this study, four-point Likert scale was used for the scoring.
The prepared questionnaire set was segregated into two sections namely section A (demography) and section B (current competency). Breakdown of the questionnaire set is given in Table 2.  Table 3, results of this study's unidimensionality analysis showed that the gross variance value explained by the measure was 51.1%, while raw variance value explained by the measures was at least 40% (Linacre, 2005). Meanwhile, the biggest secondary dimension in the first contrast was 7.5%, which was not more than 15%. According to Abdul Aziz et al (2014), unexplained variance in the first contrast should not be more than 15%.

Local Independence
Local independence is used to assess the extent to which a respondent's response for an item is not related to other items. Table 4 shows the result of local independence analysis. There are 2 items with values of more than 0.7 namely ES2 and EA1 items. A standard value of more than 0.70 shows that the range did not meet the local independence condition (Linacre, 2012), or in other words, the respondents felt that those items were similar and confusing. The study may improve the two items by improving the sentence structure and comprehension or by dropping the items. A researcher needs to ensure that the content validity is still intact if they plan to drop the said items.

Table 4
Analysis of Local Independence

Item Polarity via Point Measure Correlation Value
An analysis of item polarity via reviewing the Point Measure Correlation (PMC) value aims to test the extent to which the development of a construct serves its intended purpose. Positive PMC shows the extent of which the item measures the constructs to be measured (Bond & Fox, 2015). According to Linacre (2005), a high value of item polarity means the item is able to differentiate respondent's ability and measure the construct being measured. On the other hand, negative polarity value means the developed item does not measure the construct to be measured. Table 5 shows that all 43 items obtained positive PMC values, and no negative PMC value was recorded. The positive PMC values between 0.37 and 0.86 showed the items developed were able to measure the construct to be measured, and each item moved parallel with other items.

Item Fit
This analysis aimed to review fitness of each item developed using the construct represented using infit and outfit MNSQ values. According to Bond & Fox (2015), the infit and outfit MNSQ values should be in the range of 0.5 to 1.5 to ensure the items are suitable for measuring the constructs. If the MNSQ value is less than 0.5, this means the item is too easily anticipated by the respondents. However, if the MNSQ is more than 1.5, this means the item confuses the respondents (Linacre, 2012). Based on Table 6, the minimum infit MNSQ value is 0.39 and maximum infit MNSQ value is 2.31, while the minimum outfit MNSQ value is 0.36 and maximum outfit MNSQ value is 2.25. 0.62 0.60 30 1.13 1.10 * Less than 0.5, ** More than 1.5 Item fit analysis found that 37 items had infit MNSQ values and 36 items had outfit MNSQ value between 0.5 and 1.5, while other items recorded values of outside the desired range. Meanwhile, the values for infit MNSQ and outfit MNSQ were 2, and two items recorded values of less than 0.5. This showed that those items can be easily predicted and anticipated by the respondents. Aside from that, both infit MNSQ and outfit MNSQ values had 4 items that obtained MNSQ values more than 1.5, revealing that those items were confusing for respondents to answer. However, for items that did not achieve the desired MNSQ value between 0.5 and 1.5, the said items can still be considered, refined, or dropped based on researcher's requirements and expert's advice. The final instrument showed that 43 items were suitable to measure constructs of CBM competency analysis. However, improvements were made by splitting one item into two, resulting in a total of 44 items for the actual study.

Determination of Instrument Reliability Level Instrument Reliability Level
This study's reliability level was verified using Cronbach's Alpha score that had the range between 0.00 and 1.00. Based on Rasch Measurement Model, a good and acceptable Cronbach's Alpha score is between 0.70 and 1.00, where a score of 0.80-1.00 shows very good and effective at a high level of consistency. The full table for Cronbach's Alpha score is given in Table 7.  Table 7 Cronbach's Alpha Score (Bond & Fox, 2015) Cronbach's Alpha Score Reliability Level 0.8 to 1.0 Very good, effective at a high level of consistency 0.7 to 0.8 Good and acceptable 0.6 to 0.7 Acceptable < 0.6 Items need to be refined < 0.5 Items need to be dropped Instrument reliability index for this study was between 0.96 and 0.98 as per Table 8, while overall reliability index was at 0.97. This showed that the study's instrument was very reliable, effective, and highly consistent. Thus, it can be used in a real study (Bond & Fox, 2015).  Table 9 shows the respondent reliability value of 0.96, which means that the respondents were in a very good and effective condition with a high level of consistency. Meanwhile, respondent separation index was 5.17, which corresponded to a good condition. A good value for respondent separation index is at least 2.0 (Linacre, 2005). Table 9 Analysis of Respondent Reliability Index and Respondent Separation Index

Item Reliability Level and Separation Index
Next, Table 10 shows the item reliability value of 0.85, which means that the items were acceptable to measure what it was intended to measure (Abdul Aziz et al., 2014). Meanwhile, item separation index, which classifies respondents or items into categories, recorded a good condition or value of 2.38. A good value for item separation index is at least 2.0 (Linacre, 2012).

Table 10
Analysis of Item Reliability Index and Item Separation Index

Conclusion
Overall, the results from the Winsteps analysis recorded high item reliability index and high respondent reliability index. Aside from that, the items can also be organised based on a hierarchy of difficulties that showed that the CBM competency analysis instrument studied was both reliable and valid. Next, the researchers used Rasch Measurement Model to conduct relevant analyses including unidimensionality, local independence, item polarity, item fit, respondent reliability and separation values, and item reliability and separation values to determine the validity and reliability of the instrument. Based on the analyses, the items that did not meet the required conditions or standard indices were refined, and one item was further broken down into two, following the experts' advice (the items had two different contents). The validity and reliability results showed that the research instrument developed met the quality required to be used to measure the level of competency of CBM in Malaysia.