Scope of ACCORD

ACCORD is a meta-research project to develop a reporting guideline for consensus methods used in health-related activities or research (Table 2) [29]. The guideline was designed to be applicable to simple and less structured methods (such as consensus meetings), more systematic methods (such as nominal group technique or Delphi), or any combination of methods utilized to achieve consensus. Therefore, the ACCORD checklist should be applicable to work involving any consensus methods. In addition, although ACCORD has been structured to help reporting in a scientific manuscript (with the traditional article sections such as introduction, methods, results, and discussion), the checklist items can assist authors in writing other types of text describing consensus activities.

ACCORD is a reporting guideline that provides a checklist of items that we recommend are included in any scientific publication in healthcare reporting the results of a consensus exercise. However, it is not a methodological guideline. It is not intended to provide guidance on how researchers and specialists should design their consensus activities, and it makes no judgment on which method is most appropriate in a particular context. Furthermore, ACCORD is not intended to be used for reporting research in fields outside health, such as social sciences, economics, or marketing.

Study design, setting, and ethics

The ACCORD project was registered prospectively on January 20, 2022 on the Open Science Framework [30] and the EQUATOR Network website [31], and received ethics approval from the Central University Research Ethics Committee at the University of Oxford (reference number: R81767/RE001). The ACCORD protocol has been previously published [29] and followed the EQUATOR Network recommendations for developing a reporting guideline [32,33], starting with a systematic review of the literature [19], followed by a modified Delphi process. In a planned change to the Delphi method as originally formulated, the preliminary list for voting was based on the findings of this systematic review rather than initial ideas or statements from the ACCORD Delphi panel, although the panel could suggest items during the first round of voting. In addition, the ACCORD Steering Committee made final decisions on item inclusion and refined the checklist wording as described below.

ACCORD Steering Committee

WTG and NH founded the ACCORD project, seeking endorsement from the International Society of Medical Publication Professionals (ISMPP) in April 2021. ISMPP provided practical support and guidance on the overall process at project outset but was not involved in checklist development. The ACCORD Steering Committee, established over the following months, was multidisciplinary in nature and comprised researchers from different countries and settings. Steering Committee recruitment was iterative, with new members invited as needs were identified by the founders and existing committee, to ensure inclusion of the desired range of expertise or experience. Potential members were identified via ISMPP, literature research, professional connections, and network recommendations. When the protocol was submitted for publication, the Steering Committee had 11 members (WTG, PL, EJvZ, AP, CCW, DT, KG, APH, NH, and Robert Matheis [RM] from ISMPP). Bernd Arents joined the Steering Committee in July 2021 but left in December of that year, as did RM in August 2022, both citing an excess of commitments as their reason for stepping down. Patient partners were invited as Delphi panelists. Paul Blazey joined the Steering Committee in September 2022 as a methodologist to support the execution of the ACCORD Delphi process and provide additional expertise on consensus methods.

The final Steering Committee responsible for the Delphi process and development of the checklist had members working in 4 different countries: Canada, United Kingdom, United States of America, and the Netherlands. A wide range of professional roles was represented by the Steering Committee with several members bringing experience from more than one area including clinician practitioners (medical doctor, physical therapist), methodologists (consensus methodologist, research methodologist, expert in evidence synthesis), medical publication professionals (including those working in the pharmaceutical industry), journal editors, a representative of the EQUATOR Network, and a representative of the public (S1 Text).

Protocol development

The ACCORD protocol was developed by the Steering Committee before the literature searches or Delphi rounds were commenced and has been published previously [29]. An overview of the methods used, together with some amendments made to the protocol during the development of ACCORD in response to new insights, is provided below.

Systematic review and development of preliminary checklist

A subgroup of the Steering Committee conducted a systematic review with the dual purpose of identifying existing evidence on the quality of reporting of consensus methods and generating the preliminary draft checklist of items that should be reported [19]. The systematic review has been published [19] and identified 18 studies that addressed the quality of reporting of consensus methods, with 14 studies focused on Delphi only and 4 studies including Delphi and other methods [19]. A list of deficiencies in consensus reporting was compiled based on the findings of the systematic review. Items in the preliminary checklist were subsequently derived from the systematic review both from the data extraction list (n = 30) [19] and from other information that was relevant for reporting consensus methods (n = 26) [19].

Next, the Steering Committee voted on whether the preliminary checklist items (n = 56) should be included in the Delphi via 2 anonymous online surveys conducted using Microsoft Forms (See S2 Text). There were 5 voting options: “Strongly disagree,” “Disagree,” “Agree,” “Strongly agree,” and “Abstain/Unable to answer.” NH processed the results in Excel, and WTG provided feedback and therefore neither voted. Items that received sufficient support (i.e., >80% of respondents voted “Agree”/“Strongly agree”) were included in the Delphi, while the rest were discussed by the Steering Committee for potential inclusion or removal. During the first survey, Steering Committee members could propose additional items based on their knowledge and expertise. These new items were voted on in the second Steering Committee survey. Upon completion of this process, the Steering Committee approved and updated the preliminary draft checklist, which was then prepared for voting on by the Delphi panel. Items were clustered or separated as necessary for clarity.

Delphi panel composition

Using an anonymous survey (June 9–13, 2022), the Steering Committee voted on the desired profile of Delphi panelists for the ACCORD project. There was unanimous agreement that geographic representation was important, and the aim was to recruit from all continents (thereby covering both Northern and Southern hemispheres) and include participants from low-, middle-, and high-income countries to account for potential differences in cultural and ideological ways of reaching agreement. The aim was to include a broad range of participants: clinicians, researchers experienced in the use of consensus methods and in clinical practice guideline development, patient advocates, journal editors, publication professionals and publishers, regulatory specialists, public health policymakers, and pharmaceutical company representatives. As described in the ACCORD protocol [29], there are no generally agreed standards for the panel size in Delphi studies, although panels of 20 to 30 are common. The target panel size (approximately 40 panelists) was therefore guided by the desired representation and to ensure an acceptable number of responses (20, assuming a participation rate of 50%) in the event of withdrawals or partial completion of review.

Delphi panel recruitment

Potential participants for the Delphi panel were identified in several ways: from the author lists of publications included in the systematic review, from invitations circulated via an EQUATOR Network newsletter (October 2021) [34] and at the European Meeting of ISMPP in January 2022, and by contacting groups potentially impacted by ACCORD (e.g., the UK National Institute for Health and Care Excellence [NICE]). Individuals were also invited to take part through the ACCORD protocol publication [29], and the members of the Steering Committee contacted individuals in their networks to fill gaps in geographical or professional representation. To minimize potential bias, none of the Steering Committee participated in the Delphi panel.

Invitations were issued to candidate panelists who satisfied the inclusion criteria. While participants were not generally asked to suggest other panel members, in some cases, invitees proposed a colleague to replace them on the panel. Only the Steering Committee members responsible for administering the Delphi had access to the full list of ACCORD Delphi panel members. Panelists were invited by email, and reminder emails were sent to those who did not respond. Out of the 133 panelists invited, 72 agreed to participate. No panelists or Steering Committee members were reimbursed or remunerated for taking part in the ACCORD project.

Planned Delphi process

The Delphi method was chosen to validate the checklist, in line with recommendations for developing reporting guidelines [32]. A 3-round Delphi was planned to allow for iteration, with the option to include additional rounds if necessary. Panelists who agreed to take part received an information pack containing an introductory letter, a plain language summary, an informed consent statement, links to the published protocol and systematic review, and the items excluded by the Steering Committee (see S3 Text). Survey materials were developed by PL and PB in English and piloted by WTG and NH. Editorial and formatting changes were made following the pilot stage to optimize the ease of use of the survey. In an amendment to the protocol, the order of candidate items was not randomized within each manuscript section. The Jisc Online Survey platform (Jisc Services, Bristol, United Kingdom) was used to administer all Delphi surveys, ensuring anonymity through automatic coding of participants. Panelists were sent reminders to complete the survey via the survey platform, and one email reminder was sent to panelists the day before the deadline for each round.

The Delphi voting was modified to offer 5 voting options: “Strongly disagree,” “Disagree,” “Neither agree nor disagree,” “Agree,” and “Strongly agree.” Votes of “Neither agree nor disagree” were included in the denominator. The consensus threshold was defined a priori as ≥80% of a minimum of 20 respondents voting “Agree” or “Strongly agree.” Reaching the consensus threshold was not a stopping criterion. For inclusion in the final checklist, each item was required to achieve the consensus criteria following at least 2 rounds of voting. This ensured that all items had the opportunity for iteration between rounds (a central tenet of the Delphi method) [6] and enabled panelists to reconsider their voting position in light of feedback from the previous round.

In Round 1, panelists had the opportunity, anonymously, to suggest new items to be voted on in subsequent rounds. Panelists were also able to provide anonymous free-text comments in each round to add rationale for their chosen vote or suggest alterations to the item text. After each voting round, the comments were evaluated and integrated by WTG, PL, PB, and NH and validated by the Steering Committee. If necessary, semantic changes were made to items to improve clarity and concision.

Feedback given to participants between rounds included the anonymized total votes and the percentage in each category (see example in S4 Text) to allow panelists to assess their position in comparison with the rest of the group, as well as the relevant free-text comments on each item. Items that did not achieve consensus in Rounds 1 and 2 were revised or excluded based on the feedback received from the panelists. Items that were materially altered (to change their original meaning) were considered a new item. All wording changes were recorded. Panelists received a table highlighting wording changes as part of the feedback process so that they could see modifications to checklist items (for example feedback documents, see S5 Text).

Items reaching consensus over 2 rounds were removed from the Delphi for inclusion in the checklist. Items achieving agreement in Round 1, which then fell into disagreement in Round 2 were considered to have “unstable” agreement. These unstable items were revised based on qualitative feedback from the panel and were included for revoting in Round 3.

Steering Committee checklist finalization process

Consistent with the protocol [29], following completion of the Delphi process, the Steering Committee was convened for a series of three 2-hour virtual workshops (March 7, 14, and 16, 2023) to make decisions and finalise the checklist. For each item, WTG, PL, PB, and NH presented a summary of voting, comments received, and a recommended approach. The possible recommended approaches are shown in S6 Text.

All recommendations (for example, to keep approved items, confirm exclusion of rejected items, etc.) were followed by an explanation of why WTG, PL, PB, or NH felt this would be the most appropriate action and a discussion between Steering Committee members in which the suggested action could be challenged and changed.

Grammatical changes were also considered at this stage but only where they did not change the meaning of an approved item. Following review of all items, the order of the checklist items was evaluated by WTG, PL, PB, and NH.

Standardized terminology

After the consensus meetings, NH updated and standardized the terminology according to the type of information requested in the item to ensure consistency between items, and this was approved by the Steering Committee. This standardization of terminology incorporated rules established for the use of terms common in reporting guidelines, as shown in S7 Text, such as the difference between using “state” or “describe.” All but 2 items (R5 and O1) contain a verb from S7 Text.

Delphi panel demographics

The Delphi panel included a diverse group of panelists, representing a wide range of geographical areas and professions (Table 3). Of the 72 participants who indicated their willingness to participate in the Delphi panel, 58 (81%) completed Round 1 and were invited to Round 2. Fifty-four participants completed Round 2 and were invited to Round 3, which was completed by 51 participants.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Self-identified demographics of the Delphi panelists, per voting round.

https://doi.org/10.1371/journal.pmed.1004326.t003

Delphi results

The updated preliminary draft checklist presented to the Delphi panel for voting contained 41 items. The changes in the number of checklist items over the Delphi voting rounds are illustrated in Fig 1. After Round 1, 7 new items were added, and 1 item was lost by combining with another item, resulting in 47 items being included in Round 2. Only items that were unstable (n = 4) or were modified sufficiently to be considered new (n = 6) were voted on in Round 3. After Round 2, 33 items achieved consensus, and a further 3 items achieved consensus after all 3 rounds of voting. Therefore, at the end of the Delphi process, consensus was reached on 36 items. The results of the Delphi process, showing the iteration of items and level of agreement at each round, are summarized in S8 Text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A flow diagram to show the development of checklist items.

*Potential items from relevant information beyond the predefined data extraction form [19]. ^†New item (T1) proposed at checklist review meeting.

https://doi.org/10.1371/journal.pmed.1004326.g001

Finalization by Steering Committee

One item rejected by the Delphi panel was restored to the checklist (M10, becoming item M5), and 3 highly approved (>90%) items were modified by combining with other items during the Steering Committee finalization workshops. S8 Text contains the iterations of the rounds of the Delphi voting demonstrating the changes made in each round and showing how items evolved.

Restored item (Delphi M10 > Final M5)

Delphi item M10 (patient and public involvement) failed to achieve stable consensus during the voting process (Round 1, 87.5%; Round 2, 73.1%; Round 3, 76.0%; see S8 Text). The comments from the panel led the Steering Committee to conclude that panelists had not reached agreement on reporting patient and public involvement due to the item being essential in some—but not all—consensus processes (“Depends on the topic of Delphi consensus, should be optional”; “For me this rests on the topic of the exercise”) and because of disagreements about preferred terminology (“The difference between lay and patient and public partners is potentially confusing”; “DO NOT change ‘participants’ to ‘partners’”). However, the Steering Committee identified many situations where the inclusion of patients would be considered essential. Priority-setting and core outcome identification are just 2 areas where patient participation in consensus exercises is becoming standard [35–37]. Based on unanimous agreement (11/11), the Steering Committee decided to reinstate M10 as reporting item M5, while taking into account the most consistent comments regarding wording (notably, that “lay” should not be used).

Items with high level of agreement that were modified

Three original items, R3, R6, and R7, overlapped by all covering aspects of which data were reported from the Delphi voting rounds. During the checklist finalization workshops, the Steering Committee discussed these 3 items and combined them to create 2 final items, R3 (quantitative data) and R4 (qualitative data). In addition, the Steering Committee noted an overlap between original items M22 and R8 related to modifications made to items or topics during the consensus process (see S8 Text). These 2 items were combined to create the final item R5. Finally, M13 was revised to remove a conceptual overlap with M12 and to use clearer language.

Final checklist

The final ACCORD checklist comprised 35 items that were identified as essential to ensure clear and transparent reporting of consensus studies. The finalized ACCORD checklist is presented in Table 4 and is available to download and complete (S9 Text).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. The final ACCORD checklist for the reporting of consensus methods.

https://doi.org/10.1371/journal.pmed.1004326.t004

Why ACCORD was needed

The need for optimal reporting of consensus methods has been documented for decades [19,24]. The absence of a reporting guideline that encompasses the range of consensus methods may contribute to poor reporting quality [5], and this prompted the development of the ACCORD checklist.

There are 2 EQUATOR-listed reporting guidelines that provide guidance for specific projects that typically include consensus exercises: AGREE-II has only 1 item, “Formulation of Recommendations,” relating to the method used to obtain consensus [27]. COS-STAR includes only 3 items around the definition of consensus and a “description of how the consensus process was undertaken” [28]. In addition, CREDES [26] is a method- and specialty-specific guideline aimed at supporting the conduct and reporting of Delphi studies in palliative care. None of these guidelines is suitable as a comprehensive and general tool for reporting any type of consensus exercise. ACCORD addresses the breadth of methods used to attain consensus (including the Delphi method) and should be complementary to AGREE-II where a clinical practice guideline also includes a formal consensus development process. Another reporting guideline currently under development, DELPHISTAR [25], is Delphi specific and covers medical and social sciences. ACCORD extends beyond Delphi methods and encompasses a wide range of consensus methods in various health-related fields.

Although familiarity with ACCORD is likely to be useful to ensure relevant elements are considered when designing a consensus study, it is a reporting guideline and not a mandate for study conduct. The methodological background to the items and published examples of what we consider to be good reporting will be discussed in the ACCORD Explanation and Elaboration document (manuscript in preparation).

Strengths and limitations

ACCORD was conducted through an open, collaborative process with a predefined, published protocol [29]. It started with a systematic review [19] using robust methods of searching, screening, and extraction, which led to the identification of common gaps in reporting consensus methods. Only 18 studies were eligible for inclusion in the systematic review, and data extraction generated 30 potential checklist items. An additional 26 items were identified that were not covered by the data extraction list. Following this thorough process, these 56 potential items were supplemented by a further 9 proposed by the Steering Committee, with an additional 7 proposed by Delphi panelists.

The ACCORD checklist involved input from participants with a wide range of expertise, including methodologists, patient advocates, healthcare professionals, journal editors, publication professionals, and representatives from the pharmaceutical industry and bodies such as NICE and the Scottish Intercollegiate Guidelines Network. With a few exceptions reported here, their recommendations were fully adopted and integrated into the final checklist. ACCORD was developed to assist everyone involved in consensus-based activities or research. It will assure participants that methods will be accurately reported; guide authors when writing up a publication; help journal editors and peer reviewers when assessing a manuscript for publication; and enhance trust in the recommendations made by consensus panels. Our hope is that ACCORD will ultimately benefit patients by improving the transparency and robustness of consensus studies in healthcare.

A limitation of the ACCORD initiative is that the panel was not as diverse as we hoped. ACCORD was a meta-research project drawing on work from many countries, but our view is that diversity of expertise and personal experience always strengthens consensus discussions. Our aim was to broaden the diversity of contributors to ACCORD by recruiting a panel more diverse than the Steering Committee in geography and experience to mitigate the perpetuation of and dilute any biases held by the Steering Committee. Although invitations were sent to potential panelists in South America, Asia, Africa, and Oceania, few responses were obtained, leading to limited participation from these continents and a panel that was largely drawn from Europe and North America. Similarly, the professional diversity of the ACCORD panel was not as broad as we hoped, with patient partners and policymakers relatively underrepresented compared with clinicians. Therefore, in the future, greater efforts should be made to recruit panelists with experience in consensus from a broader range of professions as well as other regions and countries with different cultures and health systems. For example, although some experience of clinical psychology exists in the Steering Committee, inclusion of more behavioral scientists with experience of the process of decision-making could be a helpful addition. Similarly, the inclusion of more policymakers would strengthen the representation of their perspective on consensus reporting to ensure it was relevant and reliable and, therefore, acceptable to be referenced and inform policy. Although these biases were not fully mitigated, future revisions or extensions to ACCORD will aim to improve in this regard.

Members of the ACCORD Steering Committee did not vote in the Delphi surveys. In our process, the virtual workshops held to finalize the ACCORD checklist did not include the Delphi panel. This might be seen as a limitation by some, especially those involved in reporting guidelines development, as a consensus meeting including some expert members of the Delphi panel is usually conducted according to the guidance issued by the EQUATOR Network [32]. However, our process held the Steering Committee and Delphi panel separate: the Steering Committee did not participate in the Delphi panel, and the Delphi panelists did not participate in the final consensus discussions. We suggest that this could in fact be seen as a strength of our process since, while the larger Delphi panel did not reach consensus on 1 particular item, discussion among the Steering Committee led to its inclusion in the final checklist without full approval of the Delphi panel (see results and commentary for item M5). If the panelists had been part of the final consensus meeting, this may have resulted in the omission from the final checklist of this item, which related to patient participation in consensus studies. However, the experience represented by the Steering Committee recognized the value of patient participation in consensus recommendations, the importance of which is reported in the literature [38], and voted to include this item.

Stability of agreement indicates when consensus is present among a group. There are several methods to assess for stability, but ACCORD adhered to a simple definition of achieving the a priori agreed threshold for agreement over a minimum of 2 voting rounds [39].

Another limitation that consensus and survey specialists may note is that the items in our Delphi survey were not presented to panelists in a random order. Since ACCORD was proposing content items for the sections of a scientific manuscript (title, introduction, methods, results, and discussion), we preferred to present items in these sections in the order that they usually appear to enhance comprehension and avoid confusion. This is something that may affect all reporting guidelines development. In fact, several panelists provided feedback on how to order the items.

The implementation of the ACCORD reporting guideline

Many reporting guidelines are published without initiatives to facilitate implementation. Only 15.7% of guidelines on the EQUATOR Network website mentioned an implementation plan [33]. An implementation study to inform an Explanation and Elaboration document has been completed and the results submitted for presentation at a conference. The full ACCORD implementation plan and supporting materials are being developed and will be available on the ACCORD website (https://www.ismpp.org/accord).

The future of ACCORD

Robust reporting is particularly important for studies using consensus methods given that so many methods exist and researchers frequently make modifications to “standard” methods. We anticipate that updates of the ACCORD checklist will be necessary, as technology and consensus methods continue to evolve.

Besides updates, ACCORD could have extensions developed in areas such as nonclinical biomedical studies, health economics, or health informatics and artificial intelligence, and even beyond healthcare, with input from appropriate experts. The Steering Committee welcomes feedback and interest from other researchers in these areas.