Usability Evaluation of a Virtual Reality Interactive Music Appreciation Module (E-Marz) for Secondary School

Vol. Abstract Advancement in Virtual Reality (VR) technology can be benefited for educational purposes. This article reports on a study undertaken to evaluate the usability of a Virtual Reality (VR) Interactive Music Appreciation Module (E-Marz) for the Lower Secondary School. The study has employed the Design and Development Research (DDR) which comprised of three different phases as its methodology. E-Marz was developed systematically based on the m-ADDIE Model which include the Evaluation Phase. The objective of the evaluation phase was to measure the level of usability of the module by using the System Usability Scale Questionnaire given to the field experts and the Post-Study System Usability Questionnaire given to students as the end users. The findings of the evaluation phase showed that E-Marz had obtained a high level of usability based on the expert views and the surveys on students as end users. The outcome of this study has contributed in terms of the development of an educational media which incorparated elements of VR for the teaching and learning of music appreciation in secondary schools. The module has the potential to assist teachers and students in facilitating for a more active and fun learning environment in the classroom through the application of the VR technology. The E-Marz module also served as a medium of early exposure for students to be familiar with the VR technology which will be widely used in the era of the 4.0 Industrial Revolution.


Introduction
The daily live of today's generation of students is very synonymous with the use of technological devices such as smartphones, tablets and other gadgets. These devices especially smartphones should not be limited for the purpose of social communication but also be benefited for educational purposes. Gadgets like the smartphones are devices that are cost effectively be employed to incorporate virtual reality (VR) technology to improve and make learning more interesting. Past studies have proven that virtual reality technology in education is effective to improve students skills in various different aspects as well as facilitating for a more active learning in the classroom (Stewart et al., 2010).
This article reports on a study aimed at developing a virtual reality software named as E-Marz, a digital learning module for the teaching and learning of music appreciation using smartphones. The digital module was targeted at the lower secondary school students in Malaysia as the end user. Specifically, the focus of this article is to discuss the evaluation phase of the study in which the final prototype of the module was evaluated in terms of its usability level including testing on the end users selected from Form 1 and Form 2 students at two secondary schools. The following were the two main questions that guided the evaluation on the usability of the E-Marz module prototype: (i) What is the overall usability level of the E-Marz module based on the expert views? (ii) What are the level of usability and satisfaction of the students towards the E-Marz module?

Statement of Problem
Empirical evidence suggests that technology can help in the development of one's musical knowledge (Macrides & Angeli, 2021;Savage, 2018;Wise et al., 2011). Education based on the latest technologies in the teaching and learning of music appreciation lessons can provide solutions to the issue of conventional teaching which tend to bore students (Rithaudin, 2016). Active approach in a more practical form should be employed in the teaching of music especially topics related to music from various cultures as it is more effective than using the conventional methods such as simply reading from the textbooks (Yie & Ying, 2017;Juriani & Ghaziah, 2013). However, the reality is that teacher-centered and unplanned learning can be observed occuring in many music teaching in the schools (Chan et al., 2012). The lack of knowledge, resources of teaching and learning aids as well as teachers incompetencies were among the factors identified to have contributed to the lack of emphasis on the teaching of topics on music appreciation in the classroom (Wong et al., 2015). In addition, it was identified that the inability of the students to remain focus on listening skills due to the age, interest and attitude factors had also caused the less effective music appreciation lessons (Angeli, 2018;Todd and Mishra, 2013). Therefore, improvement in the technological skills and creativity among music teachers are necessary in order to attract students to learn (Juriani & Ghaziah, 2013).
The lack of technology based in the teaching aids resources is also believed to be one of the main factors for music teachers not applying information technology elements in the teaching of music appreciation lessons which consequently could also reduce students interest in learning (Rithaudin, 2014). Innovation in pedagogical techniques and teaching aids are necessary in line with the current changes of the digital age that we are in today. The existence of new media such as VR can be benefited for educational purposes in order to attract the new generation of students to learn in a more enjoyable way.

Background of the Study
VR is a virtual environment technology that uses multimedia through computer generated simulations where users can interact and be in a created virtual world (Velev & Zlateva, 2017;Sherman & Craig, 2003). VR has features such as immersive, interaction, imagination, sensory response, virtual world, and combination of the use of three-dimensional graphics with connecting devices to produce immersive effects to the users in a virtual environment and interactive nature (Shen et al., 2017;Pan et al. al., 2006;Burdea & Coiffet, 2003).
VR technology is predicted to have a great potential for usage in the employment sector during the 4.0 industry era (Damiani et al., 2018;Laudante, 2017). Thus, technologies such as VR applied in education can indirectly provide early exposure and increase students literacy towards the new media technologies. However, there are many aspects that are still unknown about the effectiveness of this medium in the context of education especially in the field of music education (Rupp et al., 2019;Kavanagh et al., 2017). In this study, the VR technology was used for developing an interactive module for teaching and learning music; specifically on topics related to music appreciation at the lower secondary schools in Malaysia.
Music appreciation refers to the enhancement and exposure to music aimed at increasing a sense of appreciation for a particular genre of music (Woody, 2001) through listening skills (Hund, 2014). Cox (2001), defined music appreciation as a knowledge on rhythm, melody, harmony and musical literature through listening to quality standard works. Listening skills in music appreciation have various interpretations such as active and passive listening, audiation (auditory stimuli or feelings from sounds that are not visible to the naked eye), oral skills and dictation (Fautley, 2010).
In the context of learning in schools, the field of music appreciation focuses on the activity of active listening to music from various cultures (KPM, 2015). A common approach used to teach music appreciation involves an understanding of the basic elements of music and the historical background of a music. In this approach, students are introduced to the terminology for basic elements of music such as rhythm, melody, pitch, dynamics, tone color, musical instruments, tone, texture, form, style and also learn to identify the use of each of these basic elements of music through listening (Kamien, 2010;Woody, 2001). The music appreciation component became the researchers choice to be tested out as the content of E-Marz module on the conceptual appropriateness factor. The concept to the development of the E-Marz module was to create an active and meaningful learning as the users were placed in a VR environment where they could view various types of music performances by real musicians.
The level of usability of the E-Marz module prototype was evaluated based on the views of field experts and end users in terms of meeting its aim for the teaching and learning of music appreciation. The E-Marz module prototype that was evaluated comprised of E-Marz teaching software for teachers as teaching guide. Meanwhile for the students, E-Marz smartphone application and usage guide module were provided for the learning purposes.

What is E-Marz?
E-Marz is an interactive Virtual Reality software which was developed for the purpose of teaching and learning Music Appreciation for Form 1 and 2 secondary school students in Malaysia. A total of four sub-modules were developed namely Gamelan, Caklempong, Blues and Jazz. The E-Marz software was developed using the WebXR platform which could be accessed online using various types of devices such as smartphones, tablets, computers or standalone VR headseats such as Oculus Quest, and Htc Vive. The E-Marz software can be used either immersively or non-immersively. The details of the E-Marz software interface are described below:

Introduction Screen Interface
The introduction screen contains pictures and texts of the types of music for users to learn as shown in Figure 1. The users can tap on the selected music picture if using non-immersive mode on a smart device or computer. If using immersive mode, the users can use motion gaze techniques on the Mobile VR, or VR controllers for standalone VR devices.

Sub Module Interface
Each sub-module has a display according to its own theme. After selecting the type of music, the user will be taken to the main display of the sub-module. There are interactive icons which contains information on the selected type of music such as notes, exercises, 3-Dimensional models of musical instruments and 360° presentation videos in multimedia form accessible to the users. Figure 2 shows some examples of the screen interface displays for Gamelan and Caklempong representing the Traditional sub-modules. Meanwhile Figure 3 shows the Blues and Jazz sub-modules representing the popular music sub-modules. Figure  4 shows examples of the 3-Dimensional Models of Traditional and Popular Musical Instruments. Figure 5 shows examples of the Interactive Learning Notes. Similarly, Figure 6 shows examples of the the Interactive Exercises. Lastly, Figure 7 shows examples of the 360° Music Performance Video.

Methodology
The development of E-Marz software was based on the ADDIE instructional development model (Gustafon & Branch, 2002). The ADDIE model comprised of 5 phases namely Analyse, Design, Develop, Implement and Evaluate. A total of four sub-modules were developed namely Gamelan, Caklempong, Jazz and Blues music based on the textbook and also the Curriculum and Assessment Standard Document for Form 1 and 2 Secondary School Music Syllabus. Figure 1 shows the ADDIE model for developing the E-Marz module. During the evaluation phase, a usability test was conducted to ensure the system on the E-Marz software could be used smoothly by the end users. Usability is defined as the extent to which a product can be used by a particular consumer to achieve a purpose or objective determined through the effectiveness, efficiency and satisfaction in a particular usage context specification (ISO, 2018;Van der Weegen et al., 2014). The usability assessment can be seen from the aspect of ease of use, ease of understanding and acceptability of an evaluated product (Birnie et al., 2018).

Method of Collecting Data
The method used to evaluate the level of usability of E-MARZ module was by using usability questionnaires. The participants and respondents of the study were field experts and secondary school students. The questionnaires used for this purpose were the System Usability Scale (SUS) (Brooke, 1996) and the Post Study System Usability Questionnaire (PSSUQ) (Lewis, 2002). SUS is a standard questionnaire for testing the usability of a product developed by John Brooke in 1986 (Lewis & Sauro, 2018). The SUS and PSSUQ questionnaire instruments refers to standard questionnaires that is used comprehensively in a research to obtain the feedback on the usability of the user satisfaction (Zamzuri, 2018). The selected experts fill out the SUS questionnaire form after testing the module to see the usability in terms of the application system of the E-Marz module. Improvements to the module were made after the evaluation by the panel of experts before proceeding to test on the students as end users in the next stage. Meanwhile the PSSUQ questionnaire was used for the evaluation by the students as end users on the usability of the application system as well as their satisfaction after using the E -Marz application on the smartphone.

Sample
Nielsen (2000) explained that product usability assessment through four to five users could detect up to 80% of the usability problems. Therefore, to test the usability of the E -Marz module prototype system, a total of five field experts were selected as the participants of the study. All five field experts were music teachers with more than 5 years experience in teaching music at secondary school level as suggested by Berliner's (2004). Purposive sampling technique were adopted in selecting teachers as the panel of experts for the study.
The next phase was to evaluate the usability aspect on the lower secondary students as end users. A total of 82 students were selected as the respondents of the study. All the selected students were given four weeks to use the module during the music lessons with the presence of their music teacher. The selected students had used the E-Marz module system immersively and non-immersively on their smartphones. The evaluation on the module by the students were conducted using the Post Study System Usability Questionnaire (PSSUQ).

Validity and Reliability
The contents of the questionnaire as instrument of the study was evaluated for validity by four experts in language, information technology and curriculum. Meanwhile, the validity of the content of SUS and PSSUQ questionnaires were evaluated using the Content Validity Cotent Index calculation method based on the following formula: Total item scores received (Scale 3 or 4) CVI = Number of experts The findings of the content validity of the SUS and PSSUQ instruments obtained an average value of I-CVI = 1. Lynn (1986) and Polit et al (2006) stated that the content validation can be done with a minimum of three experts with the average of CVI score for each item is CVI = 1.
Meanwhile the reliability of the instrument explains the extent to which the scores in each items of the quationnaire are consistent as well as stable when tested several times (Ghazali & Sufean, 2018). Hair et al (2003) has set a total of 5 to 30 respondents to perform the reliability test of the instrument. Although the PSSUQ has a high reliability value and was widely used in past studies (Salvendy, 2012), pilot testing was still conducted for this study in order to test the suitability of the instrument in terms of reliability. The pilot study was conducted on 30 respondents who were not involved in the actual study in order to determine the value of Cronbach's Alpha coefficient (α) on the PSSUQ instrument. The value of the coefficient (α) obtained was 0.89. Taber (2018) considered the value of coefficient (α) ≥ 0.7 as a high reliability value in the field of educational research. Therefore with a coefficient (α) of 0.89, indicated that the PSSUQ instrument has a high overall reliability value. The reliability was not performed on the SUS instrument as it only involved five respondents and had used percentile while PSSUQ was used on the students to assess the usability of the E-Marz descriptive statistics.

Data Analysis
The responds obtained from the panel of experts on the usability of the module was analysed using the percentile value calculation formula as follows: SUS = 2.5(20 + SUM(x01,x03,x05,x07,x09) -SUM(y02,y04,y06,y08,y10)) 5 (Experts) (Lewis, 2018) The method of data analysis for the responds given by students towards the usability of the module had involved descriptive statistics which are mean score, standard deviation and percentage. The interpretation of the findings for the mean score had three levels of indicators including low, medium and high as follows:

Findings and Discussion
The result of the usability test showed that the E-Marz software had obtained a high level of usability. Detail result of the analysis of the responds given by the panel of experts for each sub module are as follows: Gamelan (89), Caklempong (85.5), Jazz (82) and Blues (88.5). Figure 2 displays the overall analysis of the SUS usability test for each sub module. The higher the percentile value, the better the usability level based on the SUS score indicator as shown in Table 1.

Figure 2:
The analysis of the overall SUS score findings Table 1: SUS Score Indicator Lewis & Sauro (2018). Item benchmarks for the system usability scale. Journal of Usability Studies,13(3).
The overall usability test of the E-Marz and end user satisfaction were conducted through a survey using the PSSUQ questionnaire. For this purpose, a total of 82 Form 1 and 2 secondary school students were selected and later used the E-Marz software during the music lesson had responded to the survey. Lewis, (2002) suggested that there are three aspects of usability to be assessed using the PSSUQ questionnaire which are the System Usefulness, Information Quality and Interface Quality. Descriptive analysis was employed by looking at the mean score value, standard deviation (Sd) and percentage for each aspect of usability statement. The findings as outlined in Table 2 showed that the E-Marz module obtained a high level of satisfaction among the students for each aspect of usability. The overall analysis as outlined in Figure 3 displays the average value of high mean scores for each aspect of usability which include 4.14 (System Usability), 4.04 (Information Quality) and 4.1 (Interface Quality). Table 3 shows the mean score which has been translated in percentage form for     Overall, the E-MARZ Module has a high level of usability based on the views of experts and end users. The E-MARZ module was evaluated using the interview method in order to obtain a more in depth responses from participants supported by statistical questionnaire data. The evaluation process conducted through triangulation has provided substantial information for the researchers to assess the usability of the E-MARZ software, application and teaching guide Usability of e-MARZ Application (PSSUQ ) Average Mean Score module. The findings showed that the E-MARZ module was able to facilitate an active and fun learning environment among users. In addition, the analysis also showed that the advantages of the E-MARZ software was in increasing the interest and involvement of students during the teaching and learning process. The interactive 360° VR technology integrated with the music appreciation lessons has shown several advantages which include aspects such as visuals and learning control in terms of student pace. Based on the experts view, these advantages could contribute to E-MARZ module as an option for teachers to employ in their music appreciation lessons as compared to conventional methods such as the use of textbooks or video viewing.

Conclusion
Advancement in Virtual Reality (VR) technology can be benefited for educational purposes.
Results of this study has strenghtened findings of previous studies on the advantages of using Virtual Reality in facilitating the teaching and learning process in the classroom. In this study, an evaluation was conducted on the E-Marz interactive module which was developed using the VR technology for the purpose of teaching and learning music appreciation. The results of this study showed that the E-Marz module has acquired a high level of usability and was ready to be used in the teaching and learning of music appreciation lessons in the secondary schools. The findings of this study has also strengthened findings of previous studies on the potential of smartphone applications which integrate virtual reality technology in creating a more active, immersive and fun learning environment among students. Teaching aids like the E-Marz module would be a good initial exposure for students to engage with the the virtual reality technology which is very much synonymous with the Industrial Revolution 4.0 era. This study was focused on the usability aspects of the module. Therefore, it is recommended that future studies could consider to investigate on the effectiveness of using the E-Marz module towards students cognitive achievement.