Developing a Measuring Tool for Learners’ Satisfaction in Synchronous Learning Environments

The challenges faced by the language learners of Universiti Teknologi MARA (UiTM) in synchronous learning platform are much discussed and researched by the academics. In particular, this study investigates a research instrument or tool that is used to describe the online learning engagement issues faced by the students. This Student Learning and Satisfaction in Online Learning Environments (SLS-OLE) survey was distributed to 150 students of UiTM to analyse and measure the relationships among the variables using exploratory factor analysis (Structural Equation Model approach). The variables are course organization/structure, learner interaction, student engagement, instructor presence, student satisfaction, and perceived learning. The respondents involved were undergraduates who went through Online Distance Learning (ODL) during the Movement Control Order due to COVID 19 pandemic. The findings of this study discovered that certain items and sections of the survey like “I learned less in the course than I anticipated” under Perceived Learning may not be relevant to UiTM students. Therefore, these findings allow the researchers to verify and improve the survey to gather more accurate data that represents the issues of online learning engagement in ODL synchronous learning of UiTM.


Introduction
On 18 March 2020, the Malaysian government implemented cordon sanitaire as a preventive measure against the spread of a pandemic, COVID 19 (https://www.pmo.gov.my/2020/03/movement-control-order/). It was a move that was deemed necessary to curb the spread of the disease among the people in Malaysia. Malaysia made the decision swiftly and effectively as the rest of the world was still hesitant to implement such a strict move. Though the action took us by surprise, Malaysians generally comply with this order. It affected all walks of life, all industries, and all sectors. The education system faced this unprecedented predicament with much challenge and struggle. It was literally imposed on tertiary education, to increase Online Distance Learning (ODL) from 30% to 100%. This was done almost overnight, and it was an immense pressure on both the students and educators. Some educators were still grappling with the idea of working from home, but the demand of the curriculum was still real and something they needed to fulfil.
Soon after the directive from the Ministry of Education to implement ODL, came the tsunami or shockwave of a multitude of suggestions of classroom applications like Google Classroom, WebEx and Kahoot. The unfamiliar became familiar to many. Educators and learners were both excited and worried (Lim, 2020). For some it was a phenomenon that they embrace as endless opportunities to reach and teach students. A case of Universiti Malaysia Sarawak (UNIMAS) Malaysia reported positivity among the students and instructors during the transition of face-to-face to ODL . However, for some educators, they shun away amidst the confusion and chaos. They were overwhelmed due to hours of online material preparations and the chaotic environment working from home. In a recent study on Malaysian university students, the students revealed that if given a choice they would prefer face-to-face classroom amidst the challenges that they face during this movement control order (Chung et al., 2020). Commonly known as the 2020 Movement Control Order (MCO or PKP), has recently been lifted in 2022. After all this chaos, however, we finally realize that ODL can be done with appropriate methods of delivery. As synergy between the instructors and students must be developed and designed according to the issues and findings.

Problem Statement
Stauffer (2020) mentioned that one of the elements of successful online learning is interaction between the teachers and students. A study by Allam et al (2020) found that during the pandemic the computer literacy among undergraduate students was at a high level however self-directed learning and motivation were at low level. A review of literature shows that the ODL issues during the pandemic can be categorized into six: technological readiness, technological knowledge, financial, facility, emotional, and domestic situations (Ahmadon et. al., 2020). With the challenges that both teacher and students are experiencing, various modifications and improvisations need to be carried out to accommodate learning and teaching that include aspects of classroom management, delivery and assessment in order to ensure motivation and engagement are always at the forefront.
There should be a balanced integration between asynchronous and synchronous learning to make ODL effective (Wathall, 2020;Yunita & Maisarah, 2020). However, a prevalent issue with synchronous ODL is the inadequate technology support that mainly hinders the transition to ODL. The engagement in ODL comes from the active interaction between teacher and student (Zainol, 2021;Sugino, 2021). Fitria (2021) investigated students' perception towards synchronous Learning during COVID-19 pandemic in English Language Teaching (ELT). It was concluded that the quality of learning outcomes is still determined by how the teacher delivers the learning material. In fact, there is no relationship between the quality of learning outcomes and the online synchronous platform used. Another study of Malaysian undergraduates indicates that students perceive online learning using ICT tools as beneficial. Interestingly, however, the students also believe that online synchronous learning is not as effective as traditional classes (Shukri et al., 2021). Thus, the perception of students while being positive, they still believe in the effectiveness of an instructor to make them motivated, interested and engaged. As COVID 19 has shifted from pandemic to endemic in Malaysia, there are lessons to be learnt from the experiences of the students and lecturers in facing ODL. Hence, 100% ODL is possible but with improvement. One of the tools to measure student engagement is Student Learning and Satisfaction in Online Learning Environments Instrument (SLS-OLE) developed by (Gray and DiLoreta, 2016). This paper investigates the research instrument or tool that is used to describe the learning engagement issues faced by the students. Using exploratory factor analysis (EFA), which is a statistical technique, the observed variables in the survey can be verified for further research.

Objective of the Study
To verify the variables (course organization/structure, learner interaction, student engagement, instructor presence, student satisfaction, and perceived learning) of the Student Learning and Satisfaction in Online Learning Environments Instrument (SLS-OLE) survey.

Literature Review The On-going Definition of ODL
Online learning or often referred to as "e-learning" poses a different kind of environment which sets it apart from face-to-face traditional classroom (Newby 2000). Among them are learners who are isolated by distance or geographical barriers, learners usually comprise adults or homebound individuals and experts or teachers being linked at remote locations via the Internet. From the constructivist point of view, learning is a process of negotiating meaning from experience (Piaget, 1975). Piaget (1975) believes that a learner actively constructs knowledge based on the information he gathers from the environment. Based on this information the learner, modifie(s) and elaborate(s) the existing mental processes. For this process to operate effectively there must be a close relationship between the existing mental structures and the structure of the new experiences. That is, experience per se is not necessarily beneficial (Brown & Desforges, 1979, p. 21) In this point of view of learning, the environment that a learner is in plays an important role in shaping his course of learning. From the 'prompts' that the learner receives from the learning environment, he adapts his existing experiences with new ones. However, the new experiences must be meaningful in order for the learner to learn. In teaching, constructivism entails that information can be conveyed but understanding is totally dependent on the experience of the learners. Learners do not simply mimic or repeat information that is given to them, but they look for meaning and constantly adjust the information to their existing knowledge (Tam, 2000). Merely placing learners into a learning environment does not necessarily ensure learning. Therefore, the online learners in an online learning environment need to get the right 'prompts' or experiences in order for them to fully maximize their own learning potentials.
The term online learning or e-learning is often being equated to distance learning since the main element is the physical separation of the teacher and learner during instruction. Interaction is done via the use of technology to facilitate teaching and learning. However, Stauffer (2020) defines online learning and distance learning based on three key elements which are location, interaction and intention:

Location
With online learning or e-learning, students can be together in the classroom with an instructor while working through their digital lessons and assessments. Nonetheless, for distance learning, students work online at home while the teacher assigns work and checks in digitally.

Interaction
Online learning will involve in-person interaction between the teacher and students on a regular basis. This is because online learning is used as a blended learning technique along with other teaching strategies. Distance learning includes no in-person or face-to-face interaction between teachers and students. However, teachers rely on digital forms of communication such as messaging apps, video calls, discussion boards, and even learning management system (LMS).

Intention
Online learning is designed to be used in combination with a variety of other in-person teaching methods. It is a supplemental way of combining teaching strategies in the classroom to provide a variety of learning opportunities for your students. Distance learning is a method for delivering instruction solely online, not as a variation in your teaching style.
However, during MCO, the shift has been made so drastically that the question of when to use technology in the classroom is no longer relevant. The moment has arrived and about to stay for a while. The sudden shift that has caused the chaos. Therefore, Online Distance Learning has to be redefined since blended language learning cannot be equated to ODL. Due to this, Wathall (2020) has taken the initiative to redefine blended learning as follows, "Blended learning incorporates both face to face synchronous and asynchronous learning opportunities to engage and motivate learners. The face-to-face synchronous element of blended learning could be facilitated in a brick-and-mortar physical building or the online environment through a video conferencing tool. The main purpose of blended learning is to maintain the connection between learners while promoting collaboration, critical and conceptual thinking. A blended learning approach provides learners with an integrated, connected and coherent pathway to learn, that utilizes the affordances of the eLearning ecology and is more effective than utilizing synchronous or asynchronous alone." (Wathall, 2020) Nevertheless, in many cases of blended success stories have been reported due to careful planning and the proportionate ratio considered based on various factors like learner preferences, facilities, and support.

Learner Autonomy, Self-directedness, and Instructors' Roles in ODL
The most frequently cited definition of learner autonomy in language learning, however, is by (Holec, 1981). Holec (1981, p. 3) defines autonomy as "the ability to take charge of one's own learning". He highlighted that the learner has a pivotal role in ascertaining the process of learning. More specifically, he describes that a learner who is autonomous should be able to take control of his own learning in making decisions in the following situations (Holec 1981, p. 3), 1. Determining the objectives 2. Defining the contents and progressions 3. Selecting methods and techniques to be used 4. Monitoring the procedure of acquisition (rhythm, time, place, etc)

Evaluating what has been acquired
According to Holec (1981), self-directed means the degree of learning on one's own, while autonomy is the conscious capability to control learning. Therefore, the role or the presence of the instructor in the learning process is not the yardstick for the learner's selfdirectedness, it is the learner who determines his selfdirectedness (Holec, 1981). Based on this definition of learner autonomy, later researchers added more dimensions to being autonomous. To illustrate, Merriam and Caffarella (2001: 309) advocate that "Autonomy, however, is not necessarily context free; there is a relationship between the personal and situational variables that must come into play for a person to be autonomous in certain learning situations". Knowles (1990) states that instructors ought to create a self-directed learning environment that is informal, mutually respectful, consensual, collaborative and supportive to these adult learners.
Dang (2010) wrote a paper on advantages and disadvantages of learner autonomy promotion in Vietnamese EFL education and provided suggestions for its implementation through the lenses of socio-cultural theory. Data was generated from personal reflections to illustrate the conflict where learner autonomy can be either enhanced or hindered in various community constraints. This study explored possible situational constraints in the community that resulted in conflicts with the conditions for learner autonomy. In the context of education in Vietnam, learners are usually passive in class and familiar with rote learning. It is recommended that the instructors should determine their learners' level of autonomy and design an appropriate course to facilitate their engagement. Most significantly, the instructors need to consider the local socio-cultural characteristics of the learners to foster autonomy. In theory, learner autonomy is desirable due to the idea of independence of determining one's own learning direction. Nonetheless, researchers (August, 2006;GodwinJones, 2011;Jarvis, 2012;Reinders & White, 2011) have discovered that learner autonomy can be detrimental in learning if not supervised or structured by instructors.

Students' Engagement in ODL
Akbari et al (2016) looked into Astin's Theory of Student engagement. In their study, they investigated the learning environment to increase students' engagement. By a detailed comparison of a control group using face to face education and an experimental group using the social network Facebook, this study found significant differences between the two groups in terms of learning, engagement, and motivation. The Facebook group yielded a higher outcome in the TOEFL post-test scores than the face-to-face group with no differences in the pre-test scores. The Facebook group reports significantly higher levels of engagement and motivation than the face-to-face group.
Also, the output of the Kruskal Wallis Test indicated that there was a significant difference in students' achievement across students' engagement levels. Mulia (2020) The first finding was reported that in the four engagement aspects including behavioural, emotional, participation, and cognitive engagement, students were engaged positively in online language learning platforms. This study revealed that these students' engagement were the fundamental aspects that affect students to think cognitively. If the students showed high results in each aspect of engagement, they would get better learning outcomes.
Chiu (2021) It used a thematic analysis to analyse interview data from 36 students and 18 teachers. The findings highlight that (i) online learning environments that supported more autonomy were more likely to engage students cognitively in developing two important lifelong skills of digital literacy and self-regulated learning; and (ii) those environments that lacked emotional attachment, equipment and resources, coupled with perceived digital incompetence and ineffective learning experience of the students suppressed cognitive and emotional engagement. Hence, this study suggests how to satisfy the need for competence and relatedness to prepare and implement online learning. Eom et al (2006) examined the determinants of students' satisfaction and their perceived learning outcomes in the context of university online courses. In this study, structural equation modelling is applied to investigate the independent variables which were course structure, instructor feedback, selfmotivation, learning style, interaction, and instructor facilitation. Students who have completed at least one online course at a university participated in the study (397 responses). The results showed that all the predicted variables significantly affect students' satisfaction. However, only instructor feedback and learning style are significant. It is also highlighted that user satisfaction is a significant predictor of learning outcomes. The findings suggest online learning can benefit the students if it is targeted to learners with specific learning styles (visual and read/write learning styles) and meaningful instructor feedback. Below is the research model suggested by the study.  (Eom, Wen & Ashill, 2006) Kang and Im (2013) examined the factors in learner-instructor interaction that can predict the learner's outcomes in the online learning environment. Data from 654 respondents showed that factors related to instructional interaction predicted perceived learning achievement and satisfaction are significantly better than factors related to social interaction. However, it was revealed that social interaction such as social intimacy could adversely affect perceived learning achievement and satisfaction. This study is consistent with Richardson's et al (2016) study on the role of instructor presence in a successful online learning environment. The study acknowledges that instructor presence is influenced by many factors. Gray and DiLoreta (2016) expanded the earlier model by Eom, Wen and Ashill (2006). The survey is intended to investigate the relationships among course structure/organization, learner interaction, student engagement, and instructor presence on student satisfaction and perceived learning. Hence, the results of this study can be used to improve the quality of online teaching and learning. The Student Learning and Satisfaction in Online Learning Environments Instrument (SLS-OLE) was created by these researchers by reviewing an existing instrument and study by Eom, Wen and Ashill (2006), as well as numerous studies about online learning environments, student engagement, satisfaction, and learning, instructor presence, and learner interaction. It is hypothesized that there are relationships among these variables: course structure/organization, learner interaction, student engagement, and instructor presence on student satisfaction and improved learning. Thus, SLS-OLE was developed to gauge students' engagement that leads to effective learning in an online environment.  (Gray & DiLoreta, 2016) In 2018 Eom and Ashill argued that a significant reduction in dependent and independent variables and their measures is necessary for building an e-learning success model, and such a model should incorporate the interdependent (not independent) process nature of e-learning success. A total of 372 responses from students at a university in the Midwestern United States were used to examine the structural model. Results indicated that the e-learning success model satisfactorily explains and predicts the interdependency of six e-learning systems (course design quality, instructor, motivation, student-student dialog, student-instructor dialog, and self-regulated learning) and perceived learning outcomes.

Student Learning and Satisfaction in Online Learning Environments Instrument (SLS-OLE)
A recent study by Lagat and Concepcion (2022) adapted SLE-OLE to explore the degree of social interaction and collaborative learning in an online learning environment. However, this descriptive-correlational research only determined the relationship of 3 variables from the 6 variables: the level of students' social interaction, collaborative learning, and perceived learning in the online learning environment. The data were gathered from 288 teacher education students from a state university in the Philippines. Findings revealed a high level of social interaction, collaborative learning, and perceived learning in an online learning environment, and a significant relationship among the variables were also highlighted.

Method Research Design
This study employed a quantitative research design.

Research Procedure
The survey was distributed to students using Google form. The data were then analysed and interpreted. There were 150 students who responded to the survey. Studies show that with normally distributed indicator variables and no missing data, a reasonable sample size is about N = 150 (Muthén and Muthén, 2002). Thus, the study has sufficient data to be interpreted.

Research Analysis
The SLE-OLE survey has 6 constructs to show the relationship of these constructs of student perceived learning and satisfaction. The variables are Course Structure/Organization (5 items), Learner Interaction (7 items), Student Engagement (5 items), Instructor Presence (5 items), Student Satisfaction (6 items) and Perceived Learning (6 items) (Refer to Appendix).
Therefore, for this study, Exploratory Factor Analysis (EFA) was used as a statistical test to verify a set of observed or selected variables. EFA is essential to determine underlying constructs for a set of measured variables. In EFA, the Kaiser-Meyer-Olkin Measure of Sampling Adequacy was applied with value above 0.5 and a significance level for the Bartlett's test lower than 0.05 would indicate a substantial correlation in the data (Kaiser, 1974). Variable collinearity (usually above 0.4) suggests how strongly a single variable is correlated with other variables. Descriptive analysis was also employed in this study to gauge the reliability of the items in each of the constructs (Kumar, 2002). Cronbach's Alfa above 0.6 is considered to be good to excellent.

Respondents
A total of 150 undergraduates responded to the SLE-OLE survey. 72% (108 students) of the respondents are female and the rest are male (42 male students). The majority of the respondents are in semester 2 (42.33%) followed by semester 3 students (29.33%). The remaining respondents are semester 1 (19 students), semester 4 (16 students), semester 5 (2 students) and semester 6 (1 student).
The following section presents the results of Exploratory Factor Analysis (EFA) for each construct or variable.

Figure 3: EFA results for CSO
This construct reported a reliability of 0.33 ( Figure 3A). The Kaiser-Meyer-Olkin value exceeded the minimum value of 0.5 as shown in Figure 3B. Bartlett's test of sphericity of this construct also demonstrated that it was significant (p<0.001) as recommended by (Bahkia et al., 2019). The items in the construct also explained 73.2% of variance which surpassed the recommended point of 60% (Awang, 2015). Figure 3D reported that two components were obtained. However, two items (CS02 and CS03) experienced cross-loading. Therefore, both items will be removed from this construct. The remaining items, CS01, CS04, CS05, were maintained since the factor loading for these three items was higher than 0.50. These three items will stand as one component based on the results reported in Figure 3D. This construct reported a good reliability of 0.75 ( Figure 4A). The Kaiser-Meyer-Olkin value exceeded the minimum value of 0.5 as underlined in Figure 4B. Bartlett's test of sphericity results also demonstrated that this construct is significant (p<0.001) as suggested by (Bahkia et al., 2019). The items in the construct also explain 65.9% of variance which surpasses the recommended point of 60% (Awang, 2015). Figure 4D reported that no items were dropped as all items achieved the minimum factor loading of 0.5. Figure 4D also highlights that two components are obtained. The first component includes items LII, LI3, LI4, LI5, LI6 and LI7 while the second component includes only one item which is LI1.

Construct 3: Student Engagement (SE)
C D This construct reported an acceptable reliability of 0.61 ( Figure 5A). The Kaiser-Meyer-Olkin value exceeded the minimum value of 0.5 as underlined in Figure 5B. Bartlett's test of sphericity results also demonstrated that this construct is significant (p<0.001) as suggested by (Bahkia et al., 2019). The items in the construct also explain 59.2% of variance which did not surpass the recommended point of 60% (Awang, 2015). Figure 5D reported that no items were dropped as all items achieved the minimum factor loading of 0.5. Figure 5D also highlights that two components are attained. The first component includes items SE1, SE2, SE3, S4, and SE6. The second component in this component includes only one item which is SE5.

Figure 6: EFA results for Instructor Presence
This construct displays an excellent reliability of 0.7 as presented in Figure 6A. The Kaiser-Meyer-Olkin value exceeded the minimum value of 0.5 as underlined in Figure 6B. Bartlett's test of sphericity results also demonstrated that this construct is significant (p<0.001) as suggested by (Bahkia et al., 2019). The items in the construct also explain 54.8% of variance which did not surpass the recommended point of 60% (Awang, 2015). Figure 16 reported that one item was dropped which was Item IP2. The remaining items which include Items IP1, IP3, IP4, IP5, and IP6 stand as one component.

Figure 7: EFA results for Student Satisfaction
This construct reported an acceptable reliability of 0.59 ( Figure 7A). The Kaiser-Meyer-Olkin value exceeded the minimum value of 0.5 as underlined in Figure 7B. Bartlett's test of sphericity results also demonstrated that this construct is significant (p<0.001) as suggested by (Bahkia et al., 2019). The items in the construct also explain 64.6% of variance which fulfilled the recommended point of 60% (Awang, 2015). Figure 7D presents that no items are dropped as all items achieved the minimum factor loading of 0.5. The items are also shown to stand as one component as evidenced in Figure 7D.

Figure 8: EFA results for Perceived Learning
This construct reported an excellent reliability of 0.69 ( Figure 8A). The Kaiser-Meyer-Olkin value exceeded the minimum value of 0.5 as underlined in Figure 8B. Bartlett's test of sphericity results also demonstrated that this construct is significant (p<0.001) as suggested by (Bahkia et al., 2019). The items in the construct also explain 63.7% of variance which The learning tasks enhanced my understanding of the content. PL3 I learned less in the course than I anticipated. PL4 I learned skills that will help me in the future. PL5 The learning activities promoted the achievement of student learning outcomes. PL6 The course contributed to my professional development.