The Delphi Process
Also called the Delphi method, this is a structured way to get a group of experts to reach a consensus on a complex topic. Widely used in medicine (such as during development of the Phoenix Sepsis Score criteria), it mitigates the logical fallacies such as groupthink and halo effect that may befall a traditional in-person group meeting. The Delphi process renders all interactions anonymous, overseen by a facilitator, and sometimes even remote. The goal is to "sift" through the participants’ collective knowledge and find areas of objective agreement.
For example, during development of the Phoenix Sepsis Score, panel participants were first provided large amounts of data detailing the strength of association between a number of abnormalities and pediatric mortality. They were then subjected to a series of votes on matters such as which abnormalities should be included for the scoring system’s development and whether the name “Phoenix criteria/score" was appropriate. Items meeting a predetermined threshold underwent further consideration. For example, 'elevated creatinine' reached the predetermined ≥80% early in the study an underwent further consideration whereas the incidence of dialysis received less than the previously set threshold and is not included in today’s Phoenix Sepsis Score. However, as you can see, elevated creatinine is not considered in today’s Phoenix Sepsis Score—in later rounds of voting, the panel found that a four organ system score was just as sensitive and specific as a similar scoring system considering eight organ systems.
Pitfalls to the Delphi process include the following:
Facilitator Bias: The entire process is controlled by a facilitator who may leadingly phrase initial questions or summarize and re-phrase the expert feedback from each round in a way that steers the group a predetermined way.
Panel Attrition (Dropout): Delphi studies are slow, taking months or even years. Busy, high-level experts may get "survey fatigue" and simply stop responding. If too many people drop out, it can weaken the validity of the final consensus and introduce a survivorship bias where only the most persistent or opinionated experts remain.
Defining "Consensus": What does consensus actually mean? Is it 80% agreement? 75%? 90%? This threshold is often set by the facilitator and can feel arbitrary.
Alternatives to the Delphi process include the nominal group technique as well as the traditional expert panel/consensus conference.