Content Validity and Reliability of Questionnaire on Students’ Perception Towards Online Learning During Covid-19 Pandemic

This study aims to develop instrument to facilitate undergraduates in online learning by investigating their perceptions towards the use of it. A quantitative approach has been undertaken which include review of past literature review and integration of models related to assess students’ perception towards online learning during the pandemic. The students’ perceptions are assessed based on readability and satisfaction in terms of motivation, technology access, opinion, self-directed learning and online teaching approach. The instrument was validated by two experts and their rating were calculated using Cohen Kappa value to examine the level of agreement. Cronbach Alpha reliability index was also computed to investigate the reliability level of the instrument. The finding shows that the instrument had substantial agreement among raters and strong internal consistency.


Introduction
When Covid-19 pandemic hits the world, many countries imposed various types of lockdown to curb the spread of this virus. This pandemic has forced the government to announce physical closure of business, outdoor activities, and all levels of education institutions; school, university, kindergarten. With this sudden decision, universities have to embrace the technological solutions and opted for online learning to ensure the continuation of learning of the students. The mode of learning from face-to-face classroom interaction has completely shifted to a full online learning method, either conducted in a synchronous or asynchronous method. Online learning is an alternative method of learning that is dependents on the use of internet and technologies with no physical interaction like in a normal classroom between the students and lecturers (Tamrat & Teferra, 2020). It is currently regarded as a mainstream education (Coates, Wen & Shi, 2020) and a subcategory of distance education (Bates, 2005).
Online learning offers convenience of a learning experiences which can be tailored to students' need and schedule. Advances in technology allows students to study at the comfort of their own homes, while still able to interact with their friends, listening to lectures, watch recorded lectures at flexible times and even participating in online discussions. Due to the pandemic, Malaysian Higher Education Ministry has given the approval for all educational institutions to implement online learning or e-learning, taken into consideration that students and teachers both have access and infrastructures for online learning and teaching to take place (Malay Mail, 2020). Even though, online learning is an established platform of learning at universities, and has been used for many years, some challenges are inevitable. Among the challenges are limited functionality of the online platform, the platform was seldom used by lecturers and students, and there are no courses that are conducted fully online before this (Chung, Noor & Mathew, 2020).
Furthermore, the COVID-19 outbreak has caused university students and lecturers to deviate from their academic norms, in terms of daily routine, teaching and learning activities, objectives, nature of completing assignments, and social relationships (Chen et al., 2020, Sahu, 2020. This led to high levels of anxiety and worries among the students particularly about their academic endeavours. A year has passed since the first lockdown, and now Malaysia is still battling against COVID-19, and recently the government has announced the Movement Control Order for the third time to curb the spreading of this pandemic. Hence, there is a need to assess and revise the students' perceptions towards online learning in terms of students' readiness and satisfaction to ensure the effective of the online learning implemented. Readiness or preparedness of students is the students' ability to continue and adapt to online learning. Students' readiness can be assessed in terms of their motivation, knowledge and skills using technological tools, availability of suitable devices and software to access online learning and the ability to participate in self-directed learning. According to Nganga,Wararu and Nakweya (2020), the online learning preparedness among students varies from one institutions to the others. This influenced by how much exposures in terms of training and knowledge of online learning that the students received. Are they adequately trained on how to actively function in online learning? Chung, Subramaniam and Dass (2020) reported that Malaysian students' online learning readiness were at slightly moderate level and showed that some of the students were still not ready for online learning due to the lack of learners' control, self-directed learning and online communication efficacy.
The implementation of online learning needs to be revised from time to time in order to improve its effectiveness. Hence, it is important to investigate the students' satisfaction towards online learning in order to measure goals and objectives of lessons conducted via online learning. Sharma et al. (2020) asserted that learners, technological and instructors' dimensions in line with course design and coordination significantly regulate the learners' satisfaction towards online learning. This study aimed at investigating content validity and reliability of questionnaire to investigate students' perception towards online learning during COVID-19 pandemic. The students' perception is assessed in terms of readiness and satisfaction based on five identified domains namely, motivation, access and use of technology, opinion, self-directed learning and online teaching approach.

Domains Identifications
Based on previous research, five main domains were identified that assess perception towards online learning based on satisfaction and readiness. These domains are briefly summarized in Table 1.0.

Motivation
Motivation refers to the combination of desire and effort made to achieve a goal; it links the individual's rationale for any activity such as language learning with the range of behaviours and degree of effort employed in achieving goals (Gardner, 1988). Motivation can also be defined as one's direction to behaviour or what causes a person to want to repeat a behaviour and vice versa (Alizadeh, 2016). Motivation significantly assist students' efforts, actions, desires and need to achieve their learning goals (Hung et al., 2010). Understanding students' motivation and preferences toward learning is vital for improving the planning, production and implementation of educational resources (Federico, 2000). In this study, students' motivation for online learning looks at students' openness to participate in online learning, their feeling and interest which will influence their readiness and satisfaction towards process and outcome of online learning itself.

Access and use of technology
Online learning requires students to have hardware (i.e. computer, tablet, mobile phone), software (i.e. application), and connection (i.e. internet) in order to get access to information, lesson, assignment given on online platform. Hence, it is important for students to be equipped with the right devices, knowledge and skills in dealing with technology. This study investigate access and use of technology among the students by looking at their technology related knowledge, access to technology -enhanced devices and skills in utilising these technologies to achieve educational goal objective in higher education (Chung, Noor & Mathew, 2020;Hung et al., 2010).

Opinions on online learning
Sudden shift from traditional face to face learning to online learning may contribute to students' diverse opinions regarding online learning. This matters as their opinions will influence their action, effort in terms of readiness, and overall satisfaction towards online learning. This study looks at students' opinion on the benefits or disadvantages of online learning as well as their preference in comparison with traditional method of learning. According to Doculan (2016), the usefulness of e-learning (i.e. online learning) can only be experienced if the institution is deemed prepared. For example, institution needs to firstly access their readiness and capability of online learning before implementation in order to avoid technical obstacles faced by provider (institution) and users (students, lecturers). The usefulness of online learning can also be evaluated based on process and outcome of learning itself. Olayemi, Adamu & Olayemi (2021) stated that digital creativity, academic improvement, ease in using online learning, reductions in financial cost and flexibilities are some of the usefulness of online learning.

Self-directed learning during online learning
Since learning at tertiary level mostly done independently and the students are geared to be autonomous learners, it is important to investigate how self-directed learning is conducted in context of online learning. Self-directed learning involves process in which individual take responsibilities to understand their learning needs, identifying learning goals, identifying resources for learning, choosing suitable learning strategies and evaluate learning outcomes (Knowles, 1975). In this study, self-directed learning looks at students' initiative to take control of their own learning in order to accomplish their goals.
Online teaching approach Derived from Sharma et al.'s (2020) study on assessment of students' satisfaction on online learning, online teaching approach or instructors' dimension measure teachers' characteristics, frequency of interaction, feedback and content delivery. This study further elaborates the teachers' characteristics by looking at their competencies in terms of content, knowledge of ICT, how they conduct the lesson as well as how they support and encourage the students in online learning.

Methodology
Based on previous study, five domains (refer to Table 1) related to students' perceptions towards online learning have been identified. The purpose of this study is to measure validity and reliability of questionnaires developed to access students' perception towards on online learning during COVID-19 pandemic. Instrumentation Set of questionnaire which consisted of 56 items in 5 domains was adapted from past studies. The items are on five-point Likert scale (1=strongly disagree, 2= disagree, 3=neutral, 4= agree, 5=strongly agree). The first domain which is on motivation consisted of 9 items, the second domain is on students' ability to access and use the technology consisted of 14 items, the third domain is on opinion on online learning consisted of 11 items, the fourth domain is on self-directed learning consisted of 8 items, and the fifth domain is on online teaching approach consisted of 14 items.

Respondents
Two experts in the field of education were appointed to conduct content validity and face validity. To verify the reliability of the instrument, the questionnaires were distributed to 43 undergraduate students from various academic programmes at one of the local universities in Malaysia.

Validity of Instrument
Content validity refers to "judgments on the content and logical structure of an instrument as it is to be used in a particular study" (Fraenkel, Wallen, & Hyun, 2012:162). In assessing the content validity, two experts in the field of education were selected. The selection of the experts was based on two criteria; they are lecturer serving at public or private universities in Malaysia, who, at the time the research was carried out, were involve in online teaching and learning during the pandemic.
The experts were approached through emails and in this email the information sheet of the study and consent form were also made available for them. Once they agreed, the documents containing brief explanation of the domain and checklist for the questionnaire. The experts were required to rate each item with (0 is disagree and 1 is agree). The items were categorised into readiness and satisfaction. One item can be in both categories. Experts may agree and disagree to the categorization of each item to assess students' perception towards online learning. Comments for each disagree item was also included for suitable amendments to be made.
To negate any biasness that may affect their judgement that an individual might have, an inter-rater reliability test was conducted to exclude the differences and to make the assessment more reliable (Creswell 2014). The reliability index was calculated using Cohen's Kappa analysis measurement. It calculates the percentage of approval among the experts. Calculation formula of Fleiss's Kappa used to determine the confidence index (κ).

Figure 1. Cohen's Kappa Calculation
The researcher gathered the number of agreed and disagreed items to be calculated using the Fleiss's Kappa calculation to obtain the Kappa's value. According to Bryman (2004), the closer the kappa value is to 1, the higher agreement among the experts. This study refers to Cohen Kappa Agreement Level Indicator taken from Bernard and Ryan (2010) ( Table 2.0).

Reliability of instrument
Ary, Jacobs, and Sorensen (2010) define reliability of a measuring instrument as the "degree of consistency with which it measures whatever it is measuring" (p. 236). Questionnaire developed in this study was adapted from several sources, hence, a pilot study was necessary to ensure the reliability of the questionnaire. Furthermore, it is to ensure that the items were not vague, confusing and the responses given are able to answer the research objectives. The internal reliability of the items in the questionnaire was determined by computing the Cronbach Alpha reliability index for each domain as well as overall items. According to Jackson Κappa value = P_Rated-P_Expected N-P_Expected P-Rated = the number of agreed items P_Expected =50% of the items expected to be agreed upon N = total number of items measured for agreement (2006), a scale has a good internal consistency if the Cronbach Alpha coefficient is 0.7 and above.

Content validity
The results analysed were based on five domains which consisted of 56 items to investigate the students' perception towards online learning. Content validity was conducted by two experts and their rating of the items in five domains were analysed using Cohen Kappa analysis. The two experts were labelled as Rater 1 and Rater 2.  Table 3.0 show the mean kappa value obtained for the items in the questionnaire were 0.71 and 0.89 respectively which indicated substantial agreement as described in Table 2.0. The mean Kappa value of the two experts was 0.80 which also indicated substantial agreement. The Cohen's Kappa agreement indicator reflects of the experts' agreement reviews that helped in refining and validating the items. Table 4.0 below denoted the mean kappa value of each domain which reflect different level of agreements. For domain on opinion on online learning and online teaching approach, mean kappa value represent almost perfect agreement. This means that both raters agreed on all the items in these two domains. As for motivation domain which comprises of 9 items has mean kappa value of 0.67 indicating substantial agreement. In addition, the mean kappa values for access and use of technology domain and self-domain domain, both also shows substantial agreement between raters. For each disagreed item, the raters gave justifications and suggestions to further improve the content validity of the questionnaire. For example, the Rater 2 commented to change the word 'system' in B6 item under motivation domain (I am confident to use online learning system) to application to suits the study better. Each items evaluated, commented and corrected by the raters, was addressed. However, if items that did not fulfil the requirement of the study after being modified, the items will be excluded from the questionnaire. From the comments obtained, the two raters gave opinions on different items. For example, Rater 1 mostly commented to exclude or include the items in either satisfaction or readiness category. Meanwhile, Rater 2 comments mostly on the words and terms used in the items.

Reliability
A pilot study was conducted to understand the target group and to ensure the reliability of the research instruments. The data from the pilot study were computed using Cronbach Alpha reliability index for each item. Table 6.0 shows the Cronbach Alpha coefficient value for each domain. .973 The Cronbach Alpha coefficient for overall items is 0.973, which according to Jackson (2006) is considered strong reliability level. In addition, the coefficient values for item according to its domains also indicate good to string reliability level.

Conclusion
The present study was conducted to validate the content and reliability of the instrument used to investigate students' perception towards online learning. Based on content validity, the data showed that it achieved a substantial agreement among the raters which reflect good content validity. The items in the questionnaire also consisted of good to strong internal consistency based on Cronbach Alpha coefficient value.