Abstract
Over the last decades there have been increased in the application of information systems (IS) in the form of digital health technologies to enhance access to healthcare services, improve healthcare quality and efficiency of healthcare systems. Evidence on the efficiency of digital health technologies in clinical settings to enable the adoption and scaling up of such technologies is buried in ever increasing and complex extant literature including grey due to lack of the application of robust evidence-based research in IS. By and large, in the medical and health domain, decision-making is based on evidence-based research. In this study, we advocate for the increased use of evidence-based research using systematic literature review (SLR) as reference point in IS interventions especially in healthcare settings as digital health interventions increase. We believe the increased use/application of evidence-based research in the domain of IS will be key to identifying and scaling up relevant digital health technologies, and one step forward from “potential” as they are known for. We demonstrate this by liking IS research to medical research in various aspects. We also provide a step-by-step guide to conducting systematic review in IS including strategies on how to harness evidence through strategic use of the contentious grey literature.
Keywords: IS, Digital health technologies, Evidence-based research, Health/Medical research, Systematic Review
Introduction
The discipline of information systems (IS) has embraced a continuum of research approaches to enable it to continue to expand and acquire knowledge, contain its rapid innovation and to continue to provide understanding to new concepts as they emerge. Empirically, qualitative, quantitative, mixed research methods, case research strategy etc. have been extensively and successfully applied to the field of IS. IS reviews have generally been narrative and criticised for being singular descriptive accounts of the contributions of scholars in the discipline. However, literature reviews play critical role in establishing the foundation of academic inquiries. It collates and synthesises literature to establish roadmap for research – establish research agenda and identify gaps through the understanding of the length and breadth of existing body of knowledge (Webster & Watson, 2002). True and valuable gaps can only be identified after exhaustive search and synthesis of literature in targeted domains and IS is indifferent. As such, approaches to scientific inquiries and in IS for that matter need to be valid, reliable, and repeatable and literature review research is foundational to these (Xiao & Watson, 2019).
IS research has drawn its methodologies largely from behavioural theories and practices that explain human behaviour by examining the experiences and their impact. This is present in the individual’s environment and the learned associations he or she has acquired through previous experience to develop IS that seek to solve specific practical world problems. On the other hand, design-science paradigm seeks to extend the boundaries of human and organizational capabilities by creating new and innovative artifact (Chatterjee & Hevner, 2010) (Hevner, March, Park, & Ram, 2004). Okoli and Schabram (2010) assert that information systems’ research methods are different from those of the health sciences. Information Systems is a combination of social science, business, and computing science, whose research methods are different from those of the health sciences, from which the systematic literature review (SLR) or systematic reviewed (SR) methodology has largely been developed. For example, qualitative, quantitative, and mixed research methodologies have all been extensively applied in research in IS domain and in health as well. However, with the advent of digital health or health information systems and the increased in the innovation and application of information technologies to improve health care and health outcome, evidence-based research has become imperative in IS research. In fact, evidence-based research outcomes will serve as the basis for decisions on the development, adoption, and use of IS that solves problems, especially in patient care and health systems in general will be based on convincing evidence from rigorous research. Thus, the main aim of this article is to develop guidelines for conducting evidence-based research in IS given its increased application in healthcare in the context digital health using SLR as the gold standard for evidence-based research. We believe this will help advance from the “potential” to actual capabilities of digital health technologies, adoption and scaling up in relevant settings. To achieve this objective, we demonstrate with various components of research design on how IS research can be adapted to those of health and medical sciences and develop a step-by-step guide on how to conduct rigorous systematic literature review in this context and beyond.
Systematic investigation used in both research and operational investigations collects or analyses data to answer a specific question about a phenomenon. Thus, a systematic investigation designed for the purpose of expanding (i.e., for the purpose of developing or contributing to) the knowledge base of a scientific discipline constitutes research. In this context, this is about establishing the needed evidence regarding the application and outcomes of health technologies. This is done to ensuring the validity and integrity of trial data, customizing the plan for monitoring intervention fidelity (i.e., including all potentially variable aspects of intervention fidelity, including delivery, receipt, acceptance, and intention to use), and adopting strategies to keep pace with technology change (DeVito Dabbs et al., 2013). In other words, it is about the utilization of these technologies to create and test the equipoise and dynamism of information technology and health research theories to advance safety and adaptability in the application of information systems and technologies in healthcare (Riley et al., 2011). As such, the establishment of facts/evidence regarding the efficacy/reliability of such technologies relies heavily on data and methods of analysing such data.
For example, with clinical trials of health interventions, clinical evidence on the use of these technologies is still patchy due to lack of evidence-based research. In fact, despite the proliferation of digital health interventions, descriptions of the unique considerations for conducting randomized trials of health IT interventions and intended consequences for use on patients are lacking (DeVito Dabbs et al., 2013). It is perceived that when comparing the pros and cons of digital health technologies, the pros outweigh the cons. However, an unintended consequence or errors from digital health systems such as computerised physician order entry (CPOE) could be fatal in addition to its known association with increased time for completion of selected physician workflow and financial challenges. For example, in the past, studies such as Berger and Kichak (2004) discovered a convincing level of association between the implementation and use of CPOE systems and expanded medical errors due to inadequately designed system interfaces as with lack of system end-user training and support.
On the one hand, IS research is defined as the inquiry into the impact from the adoption and use of IS on people’s behaviour, performance of their duties, groups, and organizations (Galliers, Markus, & Newell, 2007). On the other hand, management information systems (MIS) research is defined as “the systematic investigation of the development, operations, use and/or impact of an information (sub) system in an organizational environment” (Ives, Hamilton, and Davis 1980, p.11). Both definitions emphasis on the impact of information on organizations. Research in health and medical sciences systematically investigates any impact an intervention may have on identified problem or case.
As opposed to ontology and epistemology, critical realism has no preference for methods as this largely depends upon the research question and its accompanying characteristics such as causal mechanisms. On this, Mingers, Mutch, and Willcocks (2013) contend that critical realism offers potential opportunities to shift attention away from a focus on the data and methods of analysis towards real problems and their causes. However, real problems do not occur or exist in isolation. The data provide source to the problem including identifying their causes when it is analysed using appropriate method. Grover and Lyytinen (2015) proposed that one way to overcome the challenges of agonizing over the dearth of original and bold theorizing over IT is to “generate richer data and theory and engage in faster knowledge production cycles—cycles that feed back into stronger mid-range theorizing” (P.288). When integrating theory with empirics by creating disproportionate mid-range models that are difficult to consolidate, it is critical we examine and debate the negative impacts of the field’s dominant epistemic scripts and relax them by permitting IS scholarship that more fluidly accommodates alternative forms of knowledge production. One of such alternatives is literature review research that leads to conceptualisation of concepts and testing them empirically. Again, there is the need to move beyond statistical generalizability as one can observe that different researchers and philosophers have used the term generalizability to mean different things - Whether research is conducted quantitatively or qualitatively, there is only one scientifically acceptable way to establish a theory’s generalizability to a new setting: thus, it is for the theory to survive an empirical test in that setting (Lee & Baskerville, 2003).
However, with all these, as foundational, methodological review of past literature as a crucial endeavour for any academic research of which IS no exemption (King & Rodriguez, 1978; Levy & Ellis, 2006; Snyder, 2019) is lacking and in IS discipline. Literature review is taken for granted resulting in scanty evidence for the formation of insight into the rest of the research processes. In fact, there is a real dearth of scientific literature involving the systematic evaluation of information systems (King & Rodriguez, 1978). This has not improved since Webster and Watson (2002), the initial senior editors for MISQ Review identified that methods of structuring and presenting literature reviews remain a challenge. Nonetheless, IS continues to evolve, resulting in the proliferation and advancement of IS technologies. Empirical studies also continue without evidence-based foundations informing such advancements and future studies. Surely, that there have been some attempts to providing guidelines for conducting literature review in the IS discipline. For example, systematic literature review (Okoli, 2015b; Okoli & Schabram, 2010), standalone literature review (Okoli, 2015a), literature review (Bandara, Furtmueller, Gorbacheva, Miskon, & Beekhuyzen, 2015), to mention but just a few. We will summarise the key outputs of these works and more a little later. Worst yet, leading information systems journals such as MISQ, Informs, etc. do not appear to have interest in calling for such studies.
SLR provides an evidence-based research outcomes and serves as foundation for the advancement of IS development and application. For this study, we will stick to SLR. This research will contend that systematic literature review as a rigorous research method could serve as a theoretical foundation for empirical research and will be useful in linking IS research to policy and practice. Additionally, whilst systematic literature reviews provide exhaustive literature search and synthesis, there appears not any guide for building conceptual frameworks/models, which serve as foundational bedrock for testing, refining, building, and validating expanded existing and new theories in the domain of IS. We are not making the case for the literature review of every study be systematic given its very tedious nature. In fact, by far systematic literature review papers are some of the most frequently cited papers across disciplines. What we are encouraging is an increased in rigorous SLR studies that can provide reliable insight for future studies that researchers can reference to. As with a few other researchers who made attempt to address literature review shortcomings in literature review we concur with Brocke et al. (2009, p.2) in contending “that the process of searching the literature must be comprehensibly described. Only then can readers assess the exhaustiveness of the review and other scholars in the field can more confidently (re)use the results in their own research”.
But first, we provide an update on the trend of methodologically focused research in IS published in MISQ as a leading journal in the discipline. As a result, we retrieved 21 MISQ IS methodology-related studies and 1 from Informs for detail review to provide a trajectory for future related studies as the field continue to evolve. We did not find any systematic literature review study in the history of MISQ. In Table 1 below, we summarised characteristics of retrieved publications for detail analysis.
With its origins from the social sciences, mixed methods have now been embraced by other disciplines including but not limited to medical and health sciences, nursing, social and mental health, etc. (Wisdom & Creswell, 2013) and in IS as a social science discipline. Mixed research methodology provides the needed environment for researchers to achieve a better understanding of a phenomenon being researched by exposing relevant “connections” in the context of complementarity, completeness, developmental, expansion, corroboration/confirmation, compensation, and diversity (Venkatesh, Brown, & Bala, 2013). Better still, mixed research methods also help in uncovering “contradictions” in the context of gaps, unexplained findings and inconclusions that exist between qualitative and quantitative data (Shorten & Smith, 2017). The opportunity to integrate data allows for triangulation, laying the foundation for rigorous analysis and interpretations of phenomenon from wide and far, resulting in the development of novel or grounded theory that can be tested. The debate on when mixed methods research is most appropriate is on-going. However, one understanding is clear – the ability of mixed methods research to offer rich insight into various phenomena and develop novel theoretical perspectives. As suggested Venkatesh et al. (2013), if little attention can be paid to paradigmatic or cultural incommensurability when the research question, purpose, and context requirements have all been met, the questions that need to address this broad proposition in the context of IS are: what research questions can appropriately be answered by mixed methods research? What research purpose can be addressed using mixed research methods? And of course, the question of in what context in IS can mixed research methods be found to be appropriately useful or applicable?
Whilst advocating for case research strategy, Benbasat, Goldstein, and Mead (1987) reiterated that the goals of the researcher and the nature of the research topic influence the selection of a strategy (Benbasat, 1984). Following the recommendations by Yin (2003) on when to appropriately apply case study approach, Benbasat et al. (1987) emphasised on three situations/or reasons to validate the application of case study research in IS. They are: IS in practice or use in its natural settings, e.g., in organizations to help answering the “how” and “why” questions to gain insight into the nature and complexity of IS in use; and to research an under-research area. The relevance of case study is its potential to link research to practice. Case study research in IS is likened to systems requirements elicitation for systems development. Over time, a holistic/comprehensive and exhaustive secondary research on case study research will produce valuable outcomes that will enable academics in partnership with practitioners to offer insight into the future of IS as with guidelines describing how organizations could adopt, digitally transform, and manage the transition process sustainably given the technological change and innovative nature of IS.
As pioneers of case study research approach, Stake (1995) and Yin (2003) premised their philosophy on case study on constructivist paradigm. Constructivists claim that truth is relative and that it is dependent on one’s perspective. Although constructivists are inclined on saying that truth is subjective and dependent on the experience of the individuals, there is a limit to the degree of subjectivity, which does not allow a sweeping shift in the meaning of the context of a phenomenon being studied (Baxter & Jack, 2008).
IS discipline is known for its dynamism and enduring unabating change and innovation. There is acknowledgement for the need to cope with such dynamism as with keeping up with change and innovation. By and large IS research has been based on quantitative and positivist experimental models to determine relationships among variables (factors) and their impacts and the overarching impact of this relations on outcome variables or a concept of interest (Kaplan & Duchon, 1988). These relationships have generally been among technology, processes, people, and organizations. The interpretations and conclusions from these relationships provide insight for IS improvement, development, and advancement. As such, instruments used in establishing, determining, and measuring these relationships need to be credible – would have gone through validation process with high level of fidelity. 11 years after Straub (1989) called for new efforts to validate the positivist, quantitative instruments that IS researchers were using, it is unclear whether the 17% that reported reliability, 13% that validated their constructs or 19% that either utilizing pre-test or a pilot test has improved both in terms of percentages and rigorousness. Given the dynamic and innovative nature of IS, relying on archaic research methodologies and other older research resources, then we might be losing track of the real progress and advancement made in the discipline and those yet to be made. To determine the progress made in instrument validation, there need an exhaustive or comprehensive review of literature over a period determined to hold relevant milestones in the history the discipline to establish a new starting point for the future of the discipline.
Generalizability has the tendency to be used inappropriately. However, in a profound critique on this matter, Lee and Baskerville (2003) concluded that be it in case study environment or statistical premise, theory may never be generalized to a setting where it was never tested empirically tested and confirmed as with no sample size or the number of sites in a multisite case study would be an indicator of greater generalizability of a theory to new settings. Indeed, depending on the of individuals and the peculiarity of such experiences, unknown biases in collected data in both scenarios limited opportunities for subjects to freely say what they feel about a research subject. Do people choose answers that best relate to their understanding or that is there could have been a more suitable words/phrases as pertaining to individuals. And if these two are not sufficient to validate generalizability why would researchers not stop generalizing their research findings? What are the best research approaches those researchers can confidently use to generalize findings?
Behavioural and design sciences are the hallmark of IS research (Hevner et al., 2004). Hevner and colleagues expound that whilst behavioural science paradigm sought understanding and establishment of theories of human and/or organizational behaviour, design science sought to advance the boundaries of both human and organizational capabilities by creating new innovative products. As briefly touched on early, the relationships between humans, technology and organization epitomises the discipline of IS. Given their very abstract nature, the scopes of the concepts of human and organization encompass the understanding of creative nature of human and vision of organization and the sustainability of such creativity and the realization such vision. The ability of IS to create value lies in designing and developing products that support such creativities and vision and adapting their changing nature. Hevner et al. (2004) note that Benbasat and Zmud (1999) “argue that the relevance of IS research is directly related to its applicability in design, stating that the implications of empirical IS research should be implementable, … synthesize an existing body of research, [or] stimulate critical thinking among IS practitioners. For this to be achieved, there need an exhaustive synthesis of existing knowledge on relevant empirical IS research to gain insight into the trend of existing technologies to establish evidence sufficient to warrant implementation of outcomes of empirical studies. This exhaustive synthesis would be multi-facet/level structured but adaptable in its approach to establish facts that encourage critical thinking among IS practitioners.
No one approach to information systems research can provide the richness that information systems, as a discipline needs for further advancement. Recently, top journals including MISQ have focused intensely on theory expansions and development and less on research methods. If this study as with many others is anything to go by, we can attest to the aged of existing research that renews energy for research methods in IS.
Literature review as a research methodology in IS discipline has not received sufficient attention. At the same time, extant literature from empirical studies is growing at an exponential rate with rich knowledge that need to be collated and synthesised for purposes not limited to identifying needs for further research, develop theories, provide answers to formulated questions, and gain insight into research trend. Some common examples of literature review are the narrative review, SLR, the semi-SLR, and the integrative review.
Narrative literature reviews are “are comprehensive narrative syntheses of previously published information” (Green, Johnson, & Adams, 2006, p.103). Other types of narrative literature review are commentary and editorial (Carey, 1998; Gray & Chambers, 1997). The commentary type of narrative review that are expression of an opinion and are usually shorter than full length review articles. Authors usually synthesis extant literature that aligns or justifies the opinion of the author. The very “opinion” nature of commentaries make them bias given that they tend to focus on the perception or philosophy of the commentator. Editorials, as the name suggests, are usually written by editors of journals in which they synthesis existing literature on current issues. They are short and narrowly focused. Depending on how they are written, editorials are not always narrative reviews (Green et al., 2006, p.103).
Integrative review is usually a literature review that is conducted for the purpose of developing/discovering theoretical framework (use of this theory). Theoretical frameworks provide the required structure within which researchers can situate their research paradigms in the context of philosophy, epistemology, methodology and critical analysis (Adom, Hussein, & Agyem, 2018; Grant) in synthesising relevant literature to derive concepts.
SLR, which we are advocating for in IS research is the gold standard for evidence-based review developed within the medical discipline given its convergent nature when gathering facts for evidence-based practice. Information systems reinforces evidence-based practice given its ability to capture, process, and store data as evidence for decision-making. “If the raw data is valid, then the processed data, or “information”, can be considered as equivalent to evidence” (Rodrigues, 2000, p.1344). Information systems used in clinical settings are used as interventions for improved healthcare delivery.
We retrieved 10 MISQ IS research methodology-related studies for detail review to provide a trajectory for future related studies as the field continue to evolve. In the table below, we summarised the characteristics of retrieved publications.