Exploring On-Campus Student Satisfaction on Learning Via Mooc

Satisfaction is the key to determine students’ intention to continue using a Massive Open Online Course (MOOC) and examining these students’ MOOC learning experience can provide insights into their satisfaction. Thus, this study aims to explore the MOOC learning experience of on-campus students who took up a MOOC on ICT Competency to identify aspects that are satisfying as well as dissatisfying to them. This study employed a qualitative approach in which eight students who had completed the ICT Competency MOOC were purposively chosen. The critical incident technique (CIT) was employed to collect and analyze information about significant experiences or critical occurrences of these participants during their MOOC learning. The meanings to these critical occurrences were collected to determine whether each experience inferred satisfaction towards the MOOC learning experience or otherwise. Participants were interviewed individually and the interview was guided by the eight aspects of Badrul Khan’s e-learning framework. The study reveals six satisfaction factors and another six dissatisfaction factors that point to six important lessons in designing and implementing a MOOC. The six lessons learned include the importance of providing flexibility in learning; providing a user-friendly interface and appealing as well as comprehendible learning materials; providing manageable, relevant assessments with clear assessment instructions; providing adequate instructor engagement; providing essential infrastructure and stable technological affordances; and incorporating anti-plagiarism strategies.


Introduction
Massive Open Online Courses or MOOCs are online courses that focus on big-scale interactive participation and open to anyone at any time via the web while paying vastly low, if any, fees (Liyanagunawardena, 2015;Liyanagunawardena, Adams, & Williams, 2013). MOOCs are classified as a sustaining innovation that creates new opportunities for learners to pursue knowledge and skills which would otherwise unavailable (Al-Imarah & Shields, 2019). Haavind and Sistek-Chandler (2015) have categorized MOOCs pedagogy into five types: cMOOC, xMOOC, pMOOC, hMOOC, and mMOOC. A cMOOC, which is based on connectivism, focuses on collaboration, sharing and construction of meaning among the participants (Terras & Ramsay, 2015); a xMOOC is based on behaviourism (Guardia, Maina, & Sangra, 2013) and aims to deliver learning contents to massive number of learners (Terras & Ramsay, 2015); a pMOOC emphasizes collaboration on a specific project or problem (Kim & Chung, 2015); a hMOOCs or hybrid MOOC refers to a MOOC that is blended with face-to-face class (Haavind & Sistek-Chandler, 2015); whereas a mMOOC or mini-MOOC refers to a MOOC with less than 500 participants (Haavind & Sistek-Chandler, 2015).
Public universities in Malaysia have started implementing MOOC on the OpenLearning platform since 2014 (Rajaendram, 2016). The development of MOOC in Malaysia is in line with a few national plans, which include the National Economic Model, Economic Transformation Programme and the Malaysian Education Blueprint for Higher Education (2015-2025) (Ghazali & Nordin, 2017). The study by Sari, Bonk, and Zhu (2020) has reported that out of 46 MOOCs in Malaysia and Indonesia that they surveyed, 50% was delivered in a blended mode, 21.7% was led by online instructors with assistant or tutor support (21.7%), 10.9% was led instructors but without tutor support, 4.4% learnerdriven and 8.7% self-paced. This study focuses on a university course that is offered to all undergraduate students of the university. The course, known as ICT Competency, and is fully delivered via a MOOC which is led by instructors but without tutor support. As the number of student enrolments is more than 500 and the course is designed in a rather behaviourist manner, this MOOC is considered as a x-MOOC.
High dropout rate is acknowledged as a challenge to MOOCs (Chen & Zhang, 2017). However, there are also studies such as Anutariya and Thongsuntia (2019); Korableva, Durand, Kalimullina, and Stepanova (2019); Shukor and Abdullah (2019); Xing (2019), found that course features and design can affect student performance, engagement, satisfaction and/or dropout rate. The study by Lu, Wang, and Lu (2019) reveals that satisfaction is the key to determine students' intention to continue using a MOOC and examining these students' MOOC learning experience can provide insights into their satisfaction. Thus, this study aims to explore the MOOC learning experience of students who took the ICT Competency MOOC to identify aspects that are satisfying as well as dissatisfying to them. Insights on these satisfying and dissatisfying aspects will inspire necessary strategies to improve the current implementation.

Research Design
This study employed the qualitative approach. Specifically, the critical incident technique (CIT), which was proposed by Flanagan (1954) was used. The CIT involves a step-by-step approach to collect information about human experiences and observations on incidents that are regarded as significant to the people involved and fulfilled the defined criteria of a study (Hughes, Williamson, & Lloyd, 2007;Lotfi & Yarahmadi, 2014). This technique is used to obtain rich, qualitative information from ICT Competency MOOC students about their significant incidents during the MOOC learning process. The meanings to these critical occurrences are collected to determine whether an experience infers satisfaction towards a student's MOOC learning experience or otherwise. Participants were interviewed individually and the interview was guided by the eight aspects of elearning framework proposed by Khan (2005). This e-Learning framework points out factors that would be needed for creating a successful online learning experience for diverse learners (Colorado & Eberle, 2012) and guides the planning and designing of e-Learning systems for higher educational institutions (Queirós & Leal, 2013). Hence, this framework was employed in this study.

Sample
Using the purposive sampling technique, eight participants, four males and four females, were chosen among the undergraduate students of the university to participate in this study. These informants had undergone the ICT Competency course fully via MOOC throughout an academic semester. Their age ranged from 18 to 23 years old and had given their consent to participate in this study.

Instrument
An interview guide that focused on discovering critical incidents that participants experienced when utilizing the ICT Competency MOOC was produced. This guide was designed based on the eight dimensions of e-learning framework as proposed by Khan (2005), which include the pedagogical, technological, interface, evaluation, management, resource support, ethical and institutional dimensions. This guide intends to trigger participants to recall critical incidents that are related to each of these important dimensions of e-Learning.

Critical Incident Technique (CIT) Five-Step-Process
CIT is a good qualitative research approach that gives a practical step-by-step method to collect and analyse information about significant experiences or memorable elements of events to the people involved. According to Flanagan (1954), CIT technique comprises five preliminary steps, which include (i) establishing general aims; (ii) establishing plans and specifications; (iii) collecting data; (iv) analyzing data, and (v) interpreting and reporting.
Firstly, the vital step of CIT is to define the task or activity to be studied and establish its purposes. In this study, the aim of CIT is to discover students' satisfaction as well as dissatisfaction towards a learning delivery that was conducted fully via MOOC. The second step is to establish plans and specifications. This includes developing a detailed and defensible plan of arguments and perspectives from many facets for data collection. It is a crucial step as it involves identification of critical incidents as well as recording critical behaviors. In this study, the e-Learning framework by Khan (2005) was employed to trigger the recalling of critical incidents related to all eight important dimensions of e-Learning. The pedagogical dimension focused on the course content, learners' expectations and learning outcomes; the technological dimension addressed issues related to software and hardware used in the course; the evaluation aspect concerned on the course assessment; the management dimension emphasized the continuation, upgrade and maintenance of the learning environment; the resource support dimension looked into technical and human resources support throughout their enrolment to the MOOC; the ethical dimension addressed issues that are related to social affect, bias, diversity, information accessibility as well as legal issues such as privacy, plagiarism and copyright; and finally, the institutional dimension focused on administrative affairs, academic affairs and student services.
Data collection is the third step. The researcher noted all critical incidents. Participants were reminded to focus on incidents that they had experienced while learning via the ICT Competency MOOC. The researcher had also gained permission from participants to audio record their interview sessions. Participants were requested to access the ICT Competency MOOC during the interview session as to trigger their memory on the critical incidents that they had experienced. In addition, they were given ample time during the interview to recall their respective critical incidents. For each participant, the researcher moved from one e-Learning dimension to the next dimension only when the participant was ready. The fourth step is data analysis in which critical incidents were identified and related themes on the critical behaviours were identified. Finally, in the interpreting and reporting step, the researcher explained the interpretation of the critical behaviors identified and presented the findings as thematic narrative, recommendations and tabulations.

Data collection and analysis
This study employed interview as the main data collection technique. Every participant was given a consent letter before an interview session was started. After the participant had signed the consent letter, which included the permission for audio recording, the interview session was conducted for about 30 minutes.
The researcher produced a transcription for each interview and all critical incidents were thoroughly categorized to any of the eight aspects of the e-learning framework which are pedagogical, technological, interface, evaluation, management, resource support, ethical and institutional. The themes that were discovered for each e-Learning aspect were then identified. Technological -anytime, anywhere access From the technological dimension, almost all participants were satisfied with the anytime and anywhere accessibility of the MOOC. Participant 2, 3, 4, 6, 7 and 8 mentioned that they did not need to attend conventional physical class. Participant 2 and 7 felt contented as they could easily access the learning system via their smartphones or laptops. Participant 1, 2, 3, 7 and 8 also felt comfortable as they could participate in the learning process anywhere. This aspect tremendously affects students' satisfaction as it can save their time as well as energy, particularly for those who stay outside the campus or residential colleges that are far from their respective faculties. Indeed, anytime, anywhere access is regarded as one of the most attractive features of online learning (Du, Zhang, Shelton, & Hung, 2019). It affords flexibility in learning, which emphasizes student choice to enhance education quality and satisfy students with diverse needs (Li & Wong, 2018).

Interface -user-friendly
Referring to Table 1, 50% of the participants stated that the interface of the online system influenced their satisfaction towards the ICT Competency MOOC. Participant 4 mentioned the interface design was interesting and user-friendly even when using it for the first time. Participant 6 indicated the icons employed were easy to understand and the text written was clear and easy to follow. Generally, participants were satisfied as they did not face any difficulty and confusion while using the online learning system, which indicates a good match between the course designer's expectations and that the participant's intention in using the MOOC. As emphasised by Sethi (2017), the importance of a good interface design in MOOC that reduces the mismatch between the communication between the course designer and learner is crucial in order to reduce MOOC dropout rates. Studies to identify factors that more favorable by MOOC users such as the one by Korableva, Durand, Kalimullina and Stepanova (2019), further ascertain the importance of a user-friendly interface design.

Evaluation -Manageable Assessments
Participant 2, 5, 7 and 8 recalled their learning assessment incidents and were satisfied with how they were assessed. Three types of assessments were involved, namely individual assignment, group assignment and final examination. Participant 4 reported the individual assignment was easy as they were only required to comment on given videos and do simple tasks such as drawing an ICT technology. Participant 4 also mentioned the task for the group assignment was simple and straightforward as they only need to create a short video about ICT Competency. According to Participant 8, the freedom to choose his own group members eased the process to accomplish the group assignment too. As for the final examination, all questions were based on given lecture notes. Thus, these participants were pleased as they did not need to find any external resources in preparing for the examination. In short, these participants were satisfied as they found the assessments were manageable.

Pedagogical -Attractive and Comprehendible Learning Materials
As shown in Table 1, five out of eight informants were satisfied with the pedagogy employed in this MOOC. Participant 3, 4, 5, 7 and 8 complimented the lecture notes as interesting and easy to comprehend. Participant 3, 4 and 5 viewed the videos were significant in assisting them to understand the lecture notes. Participant 4 reported the length of each video which was in the range of 5 to 10 minutes was very helpful in maintaining his attention. It has been long recognized that audio-visual materials are useful in stimulating and facilitating learning (Cakir, 2006). The finding on the video duration is consistent with the finding by Kim, Guo, Seaton, Mitros, Gajos, and Miller (2014) who examined 862 videos in four Massive Open Online Courses (MOOCs) on edX as well as Guo, Kim and Rubin (2014) who examined 6.9 million video watching sessions on the edX MOOC platform. Both studies found that shorter videos are much more engaging.
Participant 5 highlighted the benefit of MOOC learning from the instructor's perspective where in conventional classes, instructors will often repeat their instruction to ensure the information is delivered to students and this process can be tiring. Generally, all these five participants were satisfied with the online learning experience brought by the MOOC.

Engagement with lecturer-instructor engagement
Another satisfying pedagogical aspect that was derived from the interview data is the engagement of the instructors with their students in the MOOC. Participant 3 and 5 felt the presence of the instructor while learning via MOOC as the lecturer kept frequent interaction with students via online chat, provided frequent reminders on students' tasks and responded to the participants' comments. The presence of the online instructor in an online learning environment plays an important role to engage online learners (Lipscomb, 2019). The instructor's online guidance and assistance has also shown significant positive impact on the completion of students' learning tasks (Ma, Han, Yang, & Cheng, 2015) and contributes to online teaching quality (Welch, Orso, Doolittle, & Areepattamannil, 2015).

Resource support -adapting to students' preferences
Participant 3 reported an interesting satisfaction factor which is related to the resource support dimension. Students were allowed to enrol to the MOOC using their Google email accounts instead of restricting them to the use of official email accounts provided the university. Participant 3 highlighted this flexibility as convenient and helpful as notifications of the MOOC updates and interactions were sent to her Google email account that she frequently checked. This factor points to the benefit of adapting to students' preferences to facilitate their learning process.   Table 1, four participants reported incidents that were classified as dissatisfying. Participant 1 did not see the relevancy of the weekly online activities with the course contents and syllabus. Specific examples on the company logo creation and drawing of any ICT tool activities were given. Vai and Sosulski (2015) emphasize the importance of employing the course syllabus to guide the design of online learning materials, activities and assessments. Moreover, as stressed by Gagne (1985), course goals and objectives need to be clearly communicated with students in any course design. Jaggars and Xu (2016) further assert the importance to explicitly explain the relevance of the course objectives and course assessments as it contributes to student satisfaction. Hence, there is a possibility that the course goals and objectives were not clearly communicated and caused these participants to be unable to relate the course contents with those activities.

Dissatisfaction factors towards ICT Competency MOOC
Participant 5 highlighted the upload of new weekly resources by her instructor was not done consistently at a specific time and date. Consequently, the participant had to frequently check for course updates and this was regarded as burdensome. Employing short message notification (Flores, 2019;Stroud, Peacock, & Curry, 2020) is a possible method to alleviate this problem. Students will be notified when the course is updated. In the earlier section, Participant 3 mentioned on the notifications that was sent to her Google email that she frequently checked has helped to facilitate her learning. This provides evidence on the benefit of providing such notification.
An interesting aspect that was overlooked by the course instructor is the fact that students did not know each other and thus, causing difficulties for them to look for their group members. This incident was brought up by Participant 4. Allowing students to pick their own group members may lead to healthy, preferred collaboration as students (Hilliard, 2006) are likely to choose to work with those who share common interests or have compatible study schedules (Lieberman, 2018). However, it can be challenging when students themselves are not meeting in person and they do not understand each other's personalities and behaviour. Although assigning groups can be similarly challenging, particularly if the instructor and students are not meeting in person, the information on student general characteristics can provide some guides for the instructor to form more appropriate groups for collaborative work (Lieberman, 2018). Lieberman (2018) as also highlights other strategies to foster good online collaboration such as establishing tasks for individual group member, assigning one person in each group to serve as a liaison between the group and the instructor for providing updates on the group's progress, obtaining individual student's reflection on his or her own performance as well as peer assessment, and providing sufficient amount of instructor-student online interaction.

Technological -poor internet connection
Good internet access is compulsory for MOOC learning. Participants 1 and 3 reported their poor internet access at their residential colleges as critical incidents that affected their learning process. Participant 1 had to use her own mobile data to proceed with her MOOC learning and Participant 3 was unpleased as she failed to submit one of her weekly tasks.

Evaluation -unclear assessment instruction
Three out of eight informants reported incidents that implied their dissatisfaction towards to their course assessments. Participant 1, 2 and 3 highlighted that the description of assignment tasks were not made available in the MOOC. As this MOOC was offered to on-campus students, the course instructor had opted to convey the task description to a class representative who then transferred the message to the whole class. Participant 1 stressed that such approach was dissatisfying as the class representative was unable to effectively convey the message. Such dissatisfaction is alignment with the survey study by King (2014) that revealed that the instructor's feedback on assignments or assessments was rated by online students as the most important factor to engage their learning. Students were frustrated when expectations for assignments were not adequately communicated (Hara and Kling, 1999).

Engagement with lecturer-lack of instructor engagement
Although a couple of participants (Participant 3 and 5) were satisfied with their instructor's engagement during their online learning, another five participants reported the contradictory experiences. Participant 2 and 6 did not feel their instructor's presence. Participant 4 was still carried away by the conventional experience in which he felt the need to physically meet his instructor in the physical office. Participant 7 and 8 found the late responses provided by their instructor was discouraging. The importance of frequent and effective instructor-student interaction in an online learning environment is highlighted in many studies such as Jaggars and Xu, 2016;Lieberman, 2018;andYoung, 2006. Martin, Wang andSadaf (2018) examined 12 online facilitation strategies and found instructors' timely response to questions and assignments/projects was rated the highest in all four constructs (instructor presence, instructor connection, engagement and learning by students). This once again reaffirms the need to have adequate instructor engagement in an online learning environment.

Ethical -plagiarism
An ethical concern on plagiarism was raised by 75% of the participants, which points to the severity of this aspect. According to Participant 1, 2, 3, 4, 5 and 6, students could easily copy and paste answers posted by their virtual peers and reposted them as theirs or copied their peers' ideas. This show the needs for some redesigning of the way questions are posted to eliminate plagiarism. For example, a student will only able to see their peers' posts after submitting his or hers. Plagiarism is an on-going problem in higher education that exists in both online and face-to-face modalities (Greenberger, Holbeck, Steele, & Dyer, 2016). McCord (2008) suggests strategies such as using plagiarism detection tools, implementing academic integrity policies and education programs as well as improving the design of student assignments to alleviate this problem.

Management -unstable MOOC system
At the time of the course was implemented, some technical hiccups on the MOOC platform were reported as critical incidents by five participants. The cause for these hiccups was unknown although it may be potentially caused by internet connectivity. Participant 3 faced trouble as she could not submit her works. Participant 4 experienced the inability to log into the platform. Participant 3, 4, 6, 7 and 8 reported problem with the progress indicator that did not display the accurate progress despite their completion of the required tasks. Participants 4 regarded his one week experience of disrupted access to the MOOC platform was demotivating.

Conclusion
This study reveals six satisfaction factors and another six dissatisfaction factors that point to some important lessons in designing and implementing ICT Competency MOOC. The first lesson learned is about students' preference for flexibility in their learning. Students were satisfied with the flexibility of this online learning mode as they can learn anytime and anywhere. Providing flexible technological options that allow the online learning mechanism to be adapted to students' preferences is also found to be satisfying. Allowing the use of Google mail for enrolling to a MOOC as explained in the earlier section is a sample of such technological flexibility. The second lesson learned is about the importance of providing a user-friendly interface as well as appealing and comprehendible learning contents that are chunked into the right amount to facilitate learning as navigation and learning via MOOC is mostly self-directed. The third lesson learned revolves around course assessments. Providing manageable assessments, a clear communication of the assessment expectations and explaining the relevance of the course assessments to the course goals are some aspects that need to be taken into account when providing assessments via MOOC. Despite the unavailability of a physical instructor in a MOOC, the virtual presence of the instructor with active interaction with students is essential to increase students' online learning engagement. This point to the fourth lesson learned. The fifth lesson learned emphasizes on the importance of providing the essential infrastructure and stable technological affordances to avoid MOOC dropout as well as unfairness to those experiencing MOOC access or related technological hiccups. The final lesson learned points to the need to incorporate various strategies to alleviate plagiarism among MOOC students. This study also provides solid evidence on the applicability of CIT to discover students' satisfaction and dissatisfaction towards a MOOC that eventually provides insights on how to further improve the MOOC. The six learned lessons can also be used to guide the future design and development of any other MOOCs. Future work may include examining the satisfaction and dissatisfaction among MOOC students with different characteristics and background. A big scale survey based on the derived satisfaction and dissatisfaction factors among the student population may also be conducted to provide insights into the criticality of each factor.