What is this technique about
It is one of the most widely used techniques for prospective analysis by using expert assessment since it overcomes the biases and limitations of a single individual and allows to rely on the intersubjective judgment of an experts’ panel. It is a particularly versatile technique, very suitable for technological prediction but also to face situations of uncertainty or when there is a lack of objective information on a subject.
Where does it come from
The Delphi method was described for the first time in 1962, in the memorandum entitled “An experimental application of the Delphi method to the use of experts”, a simplified version of the study initiated in 1948 by the Rand Corporation, led by Norman Dalkney and Olaf Helmer, as an instrument to make predictions about nuclear catastrophe scenarios for the United States Army.
For which purposes it is used (why in your engineering teaching)
The technique is based on a successive series of consultations with a group of experts on a topic, identifying the key variables that influence its development and the probability with which certain changes can be produced in each one of these variables. It is particularly useful for problem solving and computerized problem solving.
How to use it
Implementation includes between 2 and 5 successive questionnaires to a group of around 25 people (between 15 and 100 individuals, depending on the target group) chosen among experts on the subject under investigation or people directly involved in an issue.
The process begins with the selection of the participants, which is the key element of the project, since they must have sufficient knowledge to be able to make relevant predictions and the necessary involvement to maintain their participation throughout the research process.
From that point, there are two main implementation sequences, depending on the objective and target group involved,
- Expert panel for technical & technological prediction:
- Identify the key variables affecting the subject (i.e. the 3 or the 5 most important variables).
- Develop a combined list of the variables.
- Check the combined list and evaluate the relative importance of each identified variable (rank, rating, etc.).
- Obtaining responses from the experts and creation of a ranked list of variables (averages).
- Show the results of the consultation of step D and discuss agreement-disagreement with results and other relevant measures (i.e. relevance, validity, etc.).
- Use for problem identification and problem solving.
- Divide the participants in small collaborative groups.
- Ask each individual to list the most relevant aspects (problems, uncertainties, etc.) affecting the subject or topic.
- Then ask each small group to analyse the individual responses, creating categories and subcategories. Reach group consensus about the relative importance (and/or priority, relevance, etc.) of each identified category (rank, punctuation, etc.).
- All small groups share their work with the large group. Each small group discuss the global results, developing possible scenarios/ alternatives.
- Final round in which small groups evaluate the scenarios or alternatives developed by all groups, discussing their agreement-disagreement with them (i.e. probability, desirability, etc.) until reaching consensus.
- Expert panel for technical & technological prediction:
How to implement this techniques online
Similar steps are to be taken in preparation of the use of the Delphi technique in an online setting as for one which is based on a face-to-face experience. That is, you start with the objectives of the exercise and the elaboration of the questions to be included in the questionnaires. In case of the on-line implementation, these questionnaires are provided using and online survey tool. The analysis of the results, depending on the design of the exercise this sequence is repeated until the results are discussed among the participants.
The results from the questionnaire phase are presented for discussion among the participants, and depending on the aforementioned implementation sequences, there are several things to bear in mind when preparing and implementing the activity.
For the expert panel used for prediction, the participants will gather in one single video-conferencing space, which allows the facilitator to present the results and the participants to interact. The approach is an open panel discussion in which the facilitator acts as a moderator.
For the problem solving exercise, the facilitator needs to ensure that the chosen videoconferencing tool allows for breakout sessions, and that these breakout rooms have the possibility of using different tools which the small groups use for their discussion (such as tools which simulate the “post-its” of a face to face activity). The facilitator needs to provide clear instructions on how the small groups should proceed, as the facilitator cannot have one single overview of all groups (and their body language) as in a physical environment, clear instructions on objective and process are pivotal and need to be prepared with great care. Processes for groups (or even an individual) to ask for a quick intervention and resolution of doubts while the facilitator is not “present” in the particular breakout room need to be defined.
If you work large groups and/or geographically dispersed participants, an on-line Delphi allows for a more efficient use of time and resources. Also if you want avoid social pressure (e.g. due to the topic under scrutiny) an anonymous on-line Dephi can be a good solution, participants should then be able to login anonymously and use avatars for the group discussion.
There are several (commercial) on-line tools for Delphi, but you can also opt for the more economic options such as google forms.
Examples and/or testimonials
The project “ENERPHI” frunded by the EC´s EIT on Energy promotes a students’ Web-Delphi platform for participatory processes in Energy decision-making contexts. By making use of the a specific technology for on-line Delphi exercises (WELPHI), students implement the Delphi method in their own projects, retaining the Delphi main features (anonymity, iteration, controlled feedback and statistical aggregation) and creating questionnaires for collecting the views and opinions of different participants that may be geographically dispersed, and therefore avoiding the need for them to physically meet. It is aligned within the ‘Decision Support Models’ (with application to Energy) course, a mandatory course run to the EIT InnoEnergy Masters at Instituto Superior Técnico de Lisboa. Nowadays, the involvement of decision-makers, experts, stakeholders and/or policy-makers in the development of tailor-made models is core, and therefore the teaching of participatory processes is highly relevant.
Enerphi Website: http://enerphi.tecnico.ulisboa.pt/
You will need a platform to share screens and communicate with the participants, such as for instance Zoom, MS Teams or similar.
A variety of other tools can be used in the activity, from sharing documents, brainstorming tools and polling activities.
- Concept Board
eDelphi.org. Edelphi 2022. Delphi Method Software. Available at: https://www.edelphi.org/
Welphi. The survey app to build consensus. Available at: https://www.welphi.com/en/Home.html
Mesylab Srl. Mesydel – Better Decisions Through Collective Intelligence. Available at: https://mesydel.com/en
Enerphi. Enerphi Homepage. Available at: http://enerphi.tecnico.ulisboa.pt/
Jason Zagami. (2022, May 11). Delphi Studies. [Video]. YouTube. Available at: https://youtu.be/zkzEwxD54eo
Online PM Courses – Mike Clayton. (2021, December 9). What is the Delphi Method? And How to Use the Delphi Method. [Video]. YouTube. Available at: https://youtu.be/48LlvwFGhC8
Khodyakov, D., Grant, S., Denger, B., Kinnett, K., Martin, A., Peay, H., Coulter, I. (2020). Practical Considerations in Using Online Modified-Delphi Approaches to Engage Patients and Other Stakeholders in Clinical Practice Guideline Development. In: The Patient – Patient-Centered Outcomes Research, 13, pp. 11-21. DOI: 10.1007/s40271-019-00389-4. Available at: https://link.springer.com/article/10.1007/s40271-019-00389-4
Haynes, C. A., Shelton, K. (2018). Delphi Method in a Digital Age: Practical Considerations for Online Delphi Studies. In: Handbook of Research on Innovative Techniques, Trends, and Analysis for Optimized Research Methods, pp. 132-151. DOI: 10.4018/978-1-5225-5164-5.ch009. Available at: https://www.igi-global.com/gateway/chapter/197733