COS are valuable, but methodological evidence can improve
robustness
Beune IM*, Ganzevoort W†, Gordijn SJ*
*Department of Obstetrics and Gynecology, University Medical Center
Groningen, University of Groningen, Groningen, The Netherlands.
†Department of Obstetrics and Gynecology, Amsterdam University Medical
Centers, University of Amsterdam, Amsterdam, The Netherlands.
Correspondence: Irene M. Beune,
irenebeune@hotmail.com
A core outcome set (COS) is the agreed minimum set of outcomes
to be measured in studies regarding a specific topic. A COS is
considered to encompass the most relevant outcomes and does not restrict
researchers. One should realize that outcomes not included in the COS
may actually be important for specific research questions and different
study-designs.
The COMET handbook (Core Outcome Measures in Effectiveness Trials), used
in the current study (Duffy et al. BJOG 2020 xxxx), describes consensus
methodology for COS development. (Williamson et al. Trials 2017,
18(Suppl 3):280). In a nutshell, it is advised to start with a
systematic review to identify all possible outcomes; then use the Delphi
strategy to converge opinions to consensus; and finally, the prioritized
list of outcomes is discussed in a face-to-face consensus meeting in
which the final COS is conducted. Published underlying evidence for this
approach is limited and alternatives may be considered.
1. It remains unknown whether a systematic review, is preferable over a
more liberal ‘scoping review’. A scoping review may cover all
important outcomes and chances are low that an outcome that requires asystematic review to identify it, is considered to be
fundamental for all research in the field.
2. COMET advises a response rate of 80% for each stakeholder group, but
there is no frame of reference to establish what attrition rate is
acceptable to avoid losing the strength of the panel. Did the drop-out
of 37% of the total group in the Delphi rounds in this study have a
significant effect on the final COS?
3. The crucial contribution of lay-experts is recognized by COMET, but
there is no advise for the number or percentage of lay-experts in a
panel. In previous COS procedures, lay-experts contribution varied from
4-50% (Williamson et al. Trials 2017, 18(Suppl 3):280).
4. A consensus-meeting facilitates acceleration of the consensus
building procedure because the panel members are in direct contact and
clarifications are readily available. The consensus meeting has inherent
pitfalls.
- In contrast to the online Delphi procedure, ‘strong voice’ may affect
voting-behavior, particularly when patients or lay experts are
impressed with knowledgeable professional experts.
- A small portion (4% in this study) of representatives in the
consensus-meeting make the final selection. In this study, 47 outcomes
were presented in the consensus-meeting, the ultimate COS consisted of
22 outcomes and contained 4 newly introduced outcomes. Electronic
meetings (international and COVID-19 proof) may reduce such selection
bias.
- A consensus-meeting held at the end of the Delphi procedure has a
major impact and no confirmation is sought if the original panel
agrees with the final COS. A consensus-meeting held at the beginning
or in between the Delphi rounds may have a different impact.
Delphi and COMET methodology are valid and valuable tool for consensus
building, particularly because a COS is never (only) a gold standard.
Since there is also no gold standard of the methodology, it remains
pivotal to appreciate the strengths and vulnerabilities of the
methodology by doing studies that strengthen the COMET and Delphi
methodology.
No disclosures: Completed disclosure of interest forms are
available to view online as supporting information.