Systematic Review of Action Research in Management Information Systems Field

This paper provides a bird's eye view of action research methodology and its usage in the management information system field. In the review of literature, we pool paper from 8 prominent journals in the information systems domain. We started our literature review through a keyword search on various academic database, such as Science Direct, ProQuest, ACM Digital Library and Business Source Premier, the date range 2001-2019. We finalized a total of 42 empirical articles to review after we remove a number of theoretical papers. Through a designed meta-analysis, we found that there is no common framework to conduct research due to the variety of the research phenomena. Also, the change and reflection is the most dominant AR type in MIS papers. Further, we also found that majority of studies not follow the AR guidelines and, more importantly, fail to apply any validation to improve the rigor of the research.


Introduction
Action research (AR) has been accepted as approach for conducting qualitative research in management information systems (MIS) since 1990s (Baskerville, 1999;Baskerville and Myers, 2004). West et al. (1995) argued that AR is a suitable approach to information systems research because this method simultaneously links business practices and theoretical development. The purpose of AR is to "contribute both to the practical concerns of people in an immediate problematic situation and to goals of social science by joint collaboration within mutually acceptable ethical framework" (Rapoport, 1970, p. 499). The key assumption of AR is that the factors in social setting are hard understand to gain meaningful knowledge as whole entities (Baskerville, 1999). So, action researchers act in a 'helping role' to solve practice problem for practices in industry, but it allows for the evaluation and conceptualization of phenomena based on based on collected industry data along with theoretical knowledge (Baskerville and Wood-Harper, 1996;Baskerville and Myers, 2004). AR method has been applied in the MIS field over the decades (Lau, 1997;Baskerville & Wood-Harper, 1998). The application and purpose of the method has varied greatly despite a common goal of uncovering both practical and theoretical knowledge. AR has contributed to many different areas of research within the MIS discipline. In the organizational application of information technology topic space, action research has been used to study group support systems (GSS) in a specific organizational context leading to knowledge on GSS adoption and use (Davison & Vogel, 2000;Davison, 2001). In the process of systems development area, action research has been used to show risk analysis as an important component in the prototyping for IS (Baskerville & Stage, 1996). In MIS topic space, action research has been used to improve Software practices in four organizations, and it has led to knowledge on software management (Mathiassen, 2002;Mathiassen et al., 2002). In the organizational valuation of IS topic space, action research has been used to improve the use of competency systems in a number of organizations resulting in principles for quality design (Lindgren et al., 2003). In the societal impact of IS topic space, action research has been used to explore how IS can be used to reduce crime (Wastell et al., 2004). There have been frequent calls for MIS researchers to make their research more relevant to practice (Zmud, 1998). Yet, it seems MIS researchers continue to struggle with generating excellent research that is practically relevant (Baskerville & Myers, 2002). Action research seems a potential avenue to improve the practical relevance of MIS research. It aims to solve current practical problems all the while expanding scientific knowledge. Unlike other research methods where the researcher seeks to observe an organizational phenomena, the action researcher is concerned with creating organizational change and simultaneously studying the process (Babüroglu and Ravn, 1992). Kock (2004) concluded uncontrollability, contingency, and subjectivity are three threats of AR in the IS field. Uncontrollability threat refers to the extent of capabilities of academia to control the research environment because "rarely will an organization cede ultimate authority for organizational action to an external researcher" (Avison et al., 2001, pg. 30). Contingency threat refers to the research findings of AR that are difficult to generalize to other phenomenon. Subjectivity threat refers to the degree of bias of researchers because they get involved in the research with practitioners. As such, IS researchers always face certain level of challenges to report findings from AR (Mathiassen et al., 2012). In particular, there is no prior review scoped the extent of action research used in MIS discipline. To fill this gap, this paper aims to • provide a comprehensive overview of action research in the MIS field, including the tendency of topic, research model, secondary methodology, types of AR, and AR guideline and validation. The paper is organized as follows. In section 2, we briefly review the methodology procedure or the analyses. In Section 3, we provide our results and discussion from literature analysis and coding. In this section, the trends of research topics, research model, research methodology, AR types and AR guideline and validation are presented. Finally the conclusion is discussed in section 4.

Methodology
We use the methodology employed in Palvia et al. (2015). Figure 1 summarizes the methodology employed. For phase one, we decided to prepare a pool of papers from IS scholars' basket of eight journals. These include: European Journal of Information Systems, Information Systems Journal, Information Systems Research, Journal of AIS, Journal of Information Technology, Journal of MIS, Journal of Strategic Information Systems, and MIS Quarterly. In the next few subsections, we discuss our classification scheme. We present our synthesis and evaluation in the results and conclusion sections.

Phase 1: Journal collection
The papers were pooled from eight senior scholars' basket of journals since 2001. The journals selected are considered some of the top ranking publications in the IS field (Lowry et al., 2004;Peffers & Ya, 2003). As this study is focused on action research, we used a keyword search in various academic databases, such as Science Direct, ProQuest, ACM Digital Library and Business Source Premier. A summary of journals selected and a count of papers pooled in recent 20 years is presented in Table 1.

Journal Numbers
European Journal of Information Systems 15 Information Systems Journal 12  Since some of collected articles were conceptual and opinion papers on the of action research, we decided to remove those articles. Hence in the end, we ended up 42 empirical studies using action research methodology.

Phase 2: Classification
A five-dimensional framework comprised of topics, research models, research methodologies, types of action research and action research guidelines for classifying research articles was based on Palvia et al. (2015), and we use three of these dimensions and extended another two dimensions in this paper. In addition to this three dimensional framework from Palvia et al. (2015), we also adopt their classification for these three dimensions. This further discussed in subsequent sections. The purpose of proposed framework is to yield richer insights. We believe this multi-dimensional framework allows for the identification of trends in a multi-criterion scope.

Information Systems Topics
Many of these topics were originally from Palvia et al. (2004). These were derived by building on work done by Alavi and Carlson (1992) and Barki et al. (1993). In Palvia et al. (2004), topics were developed using the top three levels of Barki et al. (1993). Later in Palvia et al. (2015), some changes were made, and several topics were added to those in Palvia et al. (2004). The addition of new topics in Palvia et al. (2015) was done in order to adjust for the emergence of new research topics and trends in IS research after a 10 year period. In Palvia et al. (2015), some of these new topics were added at the start, a few topics were discovered and added during the coding process, and a few topics were eliminated. We establish consistency and conformity to previous research by using existing topic classification from Palvia et al. (2015) for IS literature. A list of topics based on classification is displayed in Table 2.

Research Models
As researchers examine phenomena, research models depict the variables and constructs and the relationships between these variables and constructs. We use the categorization for research models from Palvia et al. (2004) used in Palvia et al. (2015). The initial categorization for research models was originally brought about by Vessey and V Ramesh (2002), and their categorization included listing of variables, influence diagram, mathematical model, and combination. Palvia et al. (2004) refined and extended this categorization, and it is show in Table 3.  Palvia et al. (2003) developed classifications for research methodologies. In Palvia et al. (2015), these classifications was slightly modified and adopted. We adopt this latest classification from Palvia et al. (2015). The research methodology classification can be found in Table 4.

Types of AR
We borrow the classification for action research from Avison et al. (1999). Change and reflection action research is focuses on imposing change change and reflecting on the outcome. Conflict resolution is action science trying to resolve conflicts between espoused and applied theories. Participatory action is research emphasizes participants collaboration in implementing change. Lastly, action learning focuses on programmed instruction and experiential learning.

Action Research Guidelines and Validation
We use Suman and Evered's model as the AR guidelines because many studies have considered it as a canonical framework for action research (Baskerville, 1999;McKay and Marshall, 2001;Kock, 2004;Davison et al., 2004). Suman and Evered's model (illustrated in Figure 2) consists five stages of AR: Diagnosing, Action Planning, Action Taking, Evaluation, and Specifying Learning (Susman & Evered, 1978).

Figure 2. The Cyclical Model for Action Research (Susman & Evered, 1978)
In the diagnosing stage, action researchers need to identify the domain, problem, and find opportunity to solve it. Action planning stage involves considering appropriate actions that could develop and solve the identified problem to generate knowledge. In the action-taking stage, action researchers select and implement best actions for solving research problems. The evaluating stage the analysis of the outcomes associated with the implementation of actions. Based on the evaluations of outcome, action researchers generate knowledge, and this attempt to build an understanding the actions in the context of the situation in the study.

Coding and Inter-rater Reliability
The articles were reviewed and coded by three doctoral students. They reviewed and coded the 50 papers from eight journals (JIT, EJIS, ISJ, ISR, JAIS, JMIS, JSIS, and MISQ) in recent 20 years. To identify the coding for the topic dimension, each paper was examined based on its title, keywords, and abstract; this method has been used other IS authors (Farhoomand & Drury, 1999;Grover et al., 1993;Ives et al., 1980). To identify the coding for the other four dimensions, Weber (1990) method was used to examine both the abstract and the paper's content.
The authors coding the papers followed recommendations described in Weber (1990) and Stemler (2001) to make valid inferences from the text and improve reliability. Some of the coders were trained on coding methods as a part of a doctoral seminar course on research methods, and this improved the consistency of coding, enhanced the uniformity of coding, and reduced the ambiguity of coding. Standard definitions for the research methodologies, research models, and research approaches were developed to prevent dissimilar results (Krippendorff, 2012). Evaluate the reliability of our coding, we used inter-coder reliability (Weber, 1990). Table 5 provided inter-coder reliability statistics for the first phase and second phase for collected articles. Initially, we calculated the inter-coder reliability in a two-phase process. In the first phase, the doctoral students coded 15 papers and compared them. The differences were reconciled, and an additional 35 papers were coded.

Findings and Discussion
We report the results of our analysis for five dimensions with graphs and highlight important findings during the 15 year period for topic trends, research models, research methodologies, types of action research, AR guidelines, and AR validation.

Topic Trend
Action research builds a bridge to fill the gap between practice and theory (Kock, 2004). So, IS scholars have applied AR for many of topics. In the study, we find published AR articles have studied many IS topics such as healthcare information systems, knowledge management, and IS design and development. Using a word map, we display our finding in Figure 3. A larger the word size indicates a greater frequency of using action research methodology for the topic.
Second Phase Coder 2 R* K** Coder 1 88% 88% * Reliability, ** kappa Figure 4 shows the tendency to use a research model in the AR articles. We find that there has been a noticeable number of action research papers did not use a model to illustrate their constructs and relationships.

Secondary Research Methodology
Action researchers apply secondary methodology sequentially or parallelly to generate better understanding of the phenomenon (Chiasson et al., 2008). In the study, we find that almost half of selected articles used a secondary research methodology combined with action research. Most of them used a framework and conceptual model. Unlike other methodologies, the use of secondary methodology in AR allows action researcher support and direct their action plans.

Secondary methodology Frequency
Frameworks and Conceptual Model 14 Case Study 5 Design Science 1

Types of Action Research
We use Avison et al.'s (1999) classification for types of AR. Based on analysis, we find that 76% of AR articles use change and reflection. Participatory action research was ranked second with 12% of the total AR articles.

Action Research Guidelines and Validation
Action research has always been criticized for weak rigor for knowledge generation (Davison et al., 2004). A common guideline could help researchers to solve this issue. However, we find that over half of AR articles did not follow any AR guidelines. In addition, 68% of AR articles did not apply any form of validation for action research. Based on our analysis, we find that most of action researchers do not provide how they collect, code, and analyze their data. The relationship between researchers and practitioners is also not stated in the article. Action research is not purely controlled by researchers, practitioners facilitate and shape action research (Avison et al., 2001). Most of action researchers do not follow guidelines as they use this methodology. Additionally, they also don't provide validation for the research. We also found that most AR articles are focusing on change and reflection. This study contributes to the literature by providing a much-needed overview of action research in the MIS domain. Our study benefits both new and seasoned MIS researchers. Researchers who are interested in using action research can become informed about the currently studied topics, models usage, AR types employed, etc. New researchers can use our study to design their AR studies, and experienced researchers can evaluate the trends that can be used to guide their study accordingly based on their interests. A limitation of this paper is that only limited number of journals that were reviewed. But, we present keen insights into issues plaguing AR methodology. The primary objective of this paper was to analyze published papers using action research as research methodology in the MIS field. Therefore, we believe that we have done a good job by selecting journals that important to senior scholar in MIS. Issue important to senior academics is important to the field, and reviewing other journals were not in the defined scope of this project. There might be some critiques about selection of the topic of analysis. To compensate for potential flaws for topics selection, we started our work with a large pool of topics from previous studies, and during the coding process, we narrowed and revised the topics based on the observations from the literature. Therefore, the current list of topics is good representation of all the MIS topics considered for AR methodology.
In meta-analysis papers, another concern is the coding process and reliability of the results. To ensure that the results are reliable and consistent, we followed the discussed procedures recommended by many scholars. The coders went through several training sessions, and they started coding similar sets of papers together. They shared their views and understanding on coding. This process continues to the point where the coding process resulted in inter-coder reliability.