3.6 Step 6: Appraise the quality of studies to include.
For this section, we define quality as the extent to which various parts
of studies coherently fit the whole. Thus, the match among various parts
of research design – research question and method, selection of
research subjects, sample size data collection process, measurements,
analysis, and reporting. In this section, particular attention needs to
be paid to the quality of each section of the entire research design.
For example, to prioritize papers according to their quality and to
exclude certain papers deemed not useful due to inferior methodological
quality (Okoli, 2015a). Okoli (2015a) posits that “perhaps the most
significant distinction between classes of quality appraisal methods is
whether the primary studies are quantitative (i.e., they obtain
knowledge by measuring numbers) or qualitative (i.e., use text or other
non-numeric data with discussion and argumentation to understand the
phenomenon); hence” (p.896).
When appraising the quality of qualitative studies, Okoli highlights
that study reviewers should endeavour to distinguish the lines of
arguments in the context of inference, assertion, or supposition drawing
on logics of how arguments were developed. For theoretical papers,
particularly those that aimed at deriving or discovering theories such
as grounded theory, the need to identify whether authors premised their
arguments along deductive or inductive reasoning is of great importance.
Research quality in IS when focused on digital health has become a
matter of life or death given the dearth of knowledge on the unintended
consequences and side effects of the application of digital health
technologies. As the primary objective of this study, to we posit that,
given the increased application of IS in the healthcare environment,
there is a growing evidence of identical interventions in the
application of IS in the healthcare domain. This warrants stricter
methodological quality of the highest standard. The implication of this
new evidence unlike Fink’s, is that the advocacy of Petticrew and
Roberts (2008) for “leniency of hierarchy of evidence” when conducting
SLR in social sciences no longer really hold much on which to reckon,
particularly for IS.
On incorporating non-peer reviewed/unpublished papers, we suggested
above a scoping review on grey literature for insight or broader idea on
the topic being researched as a starting point for systematic literature
review. Additionally, high-quality reports with quality/well-cited
references could be of great use. Again, as opposed to Fink (2005),
Okoli broke ranks with these differences to assert that both qualitative
and quantitative lend themselves to rigorous empirical structured
methodology of SR to achieve explicit, comprehensive exhaustive, and
reproducibility (Okoli, 2015a). To achieve the same level of quality
among multiple authors, a standard form developed by all authors would
be relevant. See an example developed by Fink.