Start Submission Become a Reviewer

Reading: Establishing Internationally Accepted Conceptual and Operational Definitions of Social Presc...

Download

A- A+
Alt. Display

Methodology papers

Establishing Internationally Accepted Conceptual and Operational Definitions of Social Prescribing Through Expert Consensus: A Delphi Study Protocol

Authors:

Caitlin Muhl ,

School of Nursing, Faculty of Health Sciences, Queen’s University, Kingston, Ontario, CA
X close

Kate Mulligan,

Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, CA
X close

Imaan Bayoumi,

School of Medicine, Faculty of Health Sciences, Queen’s University, Kingston, Ontario, CA
X close

Rachelle Ashcroft,

Factor-Inwentash Faculty of Social Work, University of Toronto, Toronto, Ontario, CA
X close

Christina Godfrey

School of Nursing, Faculty of Health Sciences, Queen’s University, Kingston, Ontario, CA
X close

Abstract

Introduction: There is currently no agreed definition of social prescribing. This is problematic for research, policy, and practice, as the use of common language is the crux of establishing a common understanding. Both conceptual and operational definitions of social prescribing are needed to address this gap. Therefore, the aim of the study that is outlined in this protocol is to establish internationally accepted conceptual and operational definitions of social prescribing.

Methodology: A Delphi study will be conducted to develop internationally accepted conceptual and operational definitions of social prescribing with an international, multidisciplinary panel of experts. It is anticipated that this study will involve approximately 40 participants (range = 20-60 participants) and consist of 3-5 rounds. Consensus will be defined a priori as ≥80% agreement.

Discussion: Not only will these definitions serve to unite the social prescribing community, but they will also inform research, policy, and practice. By laying the groundwork for the formation of a robust evidence base, this foundational work will support the advancement of social prescribing and help to unlock the full potential of the social prescribing movement.

Conclusion: This important work will be foundational and timely, given the rapid spread of the social prescribing movement around the world.

How to Cite: Muhl C, Mulligan K, Bayoumi I, Ashcroft R, Godfrey C. Establishing Internationally Accepted Conceptual and Operational Definitions of Social Prescribing Through Expert Consensus: A Delphi Study Protocol. International Journal of Integrated Care. 2023;23(1):3. DOI: http://doi.org/10.5334/ijic.6984
111
Views
8
Downloads
11
Twitter
  Published on 25 Jan 2023
 Accepted on 17 Jan 2023            Submitted on 02 Jul 2022

Introduction

There is growing interest in social prescribing around the world, with initiatives in Asia (e.g., China, Japan, Singapore, South Korea), Europe (e.g., Denmark, Finland, Germany, Ireland, the Netherlands, Portugal, Spain, Sweden, the United Kingdom (UK)), North America (e.g., Canada, the United States (US)), and Oceania (e.g., Australia, New Zealand) [1]. While the term originates from the UK and dates back to almost a century ago [1, 2], it is mostly within the past decade that social prescribing has entered into the lexicons of health systems across the globe [1]. During this time, there have been remarkable developments in the social prescribing movement. In 2015, the Social Prescribing Network was established [3], which was followed by the launch of the National Academy for Social Prescribing in 2019 [4]. In 2021, as a mark of the unprecedented momentum around the social prescribing movement at the global level, the National Academy for Social Prescribing partnered with the World Health Organization, the World Health Innovation Summit, and the United Global SDG Index Institute Foundation to establish the Global Social Prescribing Alliance, which is a global working group that is dedicated to the advancement of social prescribing through promotion, collaboration, and innovation [5]. Alongside these exciting developments, a global network of student champions has emerged to build the social prescribing student movement, with student groups in Australia, Canada, Japan, Portugal, Singapore, the UK, and the US, which collectively developed the Social Prescribing International Student Movement Framework in 2021 [2] and subsequently launched the Global Social Prescribing Student Council in 2022. Finally, as a way to foster the implementation of social prescribing, the World Health Organization recently released a social prescribing toolkit [6] and online training module [7], both of which highlight the importance of social prescribing as a vital tool in achieving integrated care. This sentiment is echoed by social prescribing experts around the world, who emphasize that social prescribing is a holistic approach to health care that fosters collaboration across health, social, and community sectors to provide integrated care [1].

While it is impressive to witness the explosion of interest in social prescribing across the globe, the fact of the matter is that there is currently no agreed definition of social prescribing [1, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. The reality is that the concept is considered nebulous and open to different interpretations [8]. There is little consistency in how social prescribing is defined across studies, with many studies not even providing a definition, meaning efforts to generate robust evidence on social prescribing are significantly hindered by the lack of consensus around the definition [9, 10]. This is problematic for research, policy, and practice, as the use of common language is the crux of establishing a common understanding, hence the need to establish an agreed definition of social prescribing [9, 10, 11, 12].

Both conceptual and operational definitions of social prescribing are needed to address this gap. Richards [28] emphasizes the importance of having both types of definitions – a conceptual definition outlines what a concept means but it does not explain how to measure it, whereas an operational definition outlines how to measure a concept but it does not explain what it means. Therefore, the aim of the study that is outlined in this protocol is to establish internationally accepted conceptual and operational definitions of social prescribing.

Methodology

Study Design

A Delphi study will be conducted to develop internationally accepted conceptual and operational definitions of social prescribing with an international, multidisciplinary panel of experts. The Delphi technique is a method of gaining consensus on a particular topic through multiple rounds of questioning of experts in the field, who remain anonymous and receive feedback between each round [29, 30]. This consensus method is widely used by health science researchers to achieve expert consensus [31], particularly to establish agreed definitions [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68]. Thus, the Delphi technique is an appropriate method to achieve the aims of this study.

Study Overview

The research team will conduct an online Delphi study. The Delphi surveys will be administered through Welphi (www.welphi.com), which is an online survey platform that is specifically designed for Delphi studies. Given the importance of the quality of the online survey platform to the success of this study, the research team carefully reviewed and tested several different options prior to selection.

At the start of each round, participants will receive an email with a link to the online survey. Important information about conceptual and operational definitions will be provided to ensure that there is common understanding of these terms. Each round will take approximately one month to complete. Participants will have two weeks to complete each round. Reminder emails will be sent out one week, two days, and one day before the closure of each round, as well as the day of the closure of each round. After two weeks, an email will be sent out to non-responders to give them an additional three days to complete the survey – if they do not respond by this time, they will be removed from the study. The research team will have the remaining two weeks of each round to complete data analysis, survey development, and pilot testing.

Participants

There is a lack of standard guidelines and agreement in the literature as to what constitutes an expert for Delphi studies [29, 30]. According to Niederberger and Spranger [31], the definition of an expert is either based on expertise or lived experience. This is reflected in the conceptualization of experts in this study, as experts will be defined according to the following criteria: (1) Person involved with the Social Prescribing Network; or (2) Person involved with the Social Prescribing Youth Network; or (3) Person involved with the Global Social Prescribing Alliance; or (4) Person involved with the National Academy for Social Prescribing; or (5) Person involved with the Canadian Institute for Social Prescribing; or (6) Student involved with any national social prescribing student group; or (7) Author of academic or grey literature on social prescribing, even if not labelled as “social prescribing”; or (8) Researcher involved in social prescribing, even if not labelled as “social prescribing”; or (9) Health care provider involved in social prescribing, even if not labelled as “social prescribing”; or (10) Link worker involved in social prescribing, even if not labelled as “link worker” or “social prescribing”; or (11) Patient involved in social prescribing, even if not labelled as “social prescribing”; or (12) Health care administrator or manager tasked with overseeing the use of social prescribing, even if not labelled as “social prescribing”. It is anticipated that this criteria will result in a heterogeneous group of experts for this study, which will increase the validity of the findings [29]. Furthermore, since this study will be in English, only those who are able to speak, read, and write English will be eligible to participate in this study. Determination of eligibility will be at the discretion of the interested party.

There is also a lack of standard guidelines and agreement in the literature as to the appropriate size of the expert panel for Delphi studies [29, 30, 31]. However, the average number of experts that health science researchers include in Delphi studies is 40 [31], which aligns with the fact that most Delphi studies that have been conducted by health science researchers to establish agreed definitions have had between 20–60 people on the expert panel [34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54]. Therefore, it is anticipated that approximately 40 participants (range = 20–60 participants) will be involved in this study.

Recruitment

During targeted recruitment, the research team will develop a comprehensive list with the names and contact information of experts (e.g., authors of academic or grey literature on social prescribing) and send out invitations to this group via email. This information will be obtained from public sources (e.g., corresponding author details). During open recruitment, the research team will send out invitations to experts via relevant communication channels (e.g., Social Prescribing Network newsletter) and advertise invitations to experts via relevant social media platforms (e.g., Twitter). Consistent with snowball sampling, experts will be asked to disseminate the call to relevant contacts. Throughout the recruitment process, the acquisition of an international, multidisciplinary panel of experts will be prioritized. The research team will use a matrix to ensure that there is diversity amongst experts in terms of country, job title, expertise in social prescribing, and years of experience with social prescribing. The registration survey will be closely monitored, and invitations will be sent out to experts from groups that are underrepresented.

Experts will receive an email with a link to the registration survey, which will be administered through Qualtrics (www.qualtrics.com). The first page of the survey will be the Letter of Information. Experts will be informed that by proceeding to the next page, they are consenting to participate in the study. Sociodemographic data, including name, email address, country, job title, organization, expertise in social prescribing, and years of experience with social prescribing, will be collected through the registration survey. Once enough experts have registered for the study, the first round will begin. Participants will receive an email one week prior to the launch of the first round to notify them and provide them with necessary information.

Consensus

While there is a lack of standard guidelines and agreement in the literature as to what constitutes consensus for Delphi studies [29, 30, 31, 69], most Delphi studies define consensus as a certain percentage of participants being in agreement, with the level of agreement ranging from 51–100% [29]. Consensus will be defined a priori as ≥80% agreement for this study. This threshold was chosen as an appropriate cutoff given that most health science researchers that have conducted Delphi studies to establish agreed definitions have defined consensus as either ≥70%, ≥75%, or ≥80% agreement [33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 59, 60, 61, 62, 63, 64, 65, 66, 67], with ≥80% being the most stringent level of agreement.

Rounds

While Delphi studies can consist of any number of rounds, they commonly involve 3–5 rounds [30]. In fact, most Delphi studies that have been conducted by health science researchers to establish agreed definitions have consisted of 3–5 rounds [32, 33, 34, 35, 36, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 63, 64, 65, 66, 67]. Based on this trend, and the format of each round in this study, it is anticipated that there will be 3–5 rounds.

Data Collection

Round 1

For the first round of this study, participants will complete an open-ended survey to gather information through open-ended questions, which aligns with the original Delphi technique [29, 30]. The first question will ask participants to list key elements that are essential to the conceptual definition of social prescribing. The second question will build on this by asking participants to provide corresponding statements that operationalize each of the key elements.

Round 2

For the second round of this study, participants will rate items from the first round through a structured survey, which will consist of two sections. In the first section, findings from the first question will be presented, and participants will be asked to rate their agreement with these items for inclusion in the conceptual definition of social prescribing. In the second section, findings from the second question will be presented, and participants will be asked to rate their agreement with these items for inclusion in the operational definition of social prescribing. Ratings will be given on a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree). The Likert scale was selected because it is commonly used to collect rated responses [29], and more specifically, the 5-point Likert scale was chosen because it is one of the most common scales that health science researchers use in Delphi studies [31]. A free-text box will also be provided to add comments, and participants will be encouraged to do so.

Round 3 and Beyond

For the third round and beyond, participants will complete a structured survey to rate their agreement with the conceptual and operational definitions of social prescribing. Consistent with the second round, ratings will be given on a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree). A free-text box will also be provided to add comments, and participants will be encouraged to do so. If consensus is not reached for both definitions by the end of the fifth round, a meeting will be held with participants via teleconferencing software to achieve consensus through discussion. While meetings are a common modification to the original Delphi technique, it is important to acknowledge that this gathering would violate one of the fundamental characteristics of this consensus method, in that participants would no longer be anonymous to the group [29], so this meeting will only take place if it is necessary. Once consensus is reached for both definitions in the third round or beyond, the study will be terminated, which will signify the successful development of internationally accepted conceptual and operational definitions of social prescribing.

Data Analysis

The data that is collected from the registration survey will be analyzed with the latest version of Microsoft Excel (www.microsoft.com). With respect to the Delphi surveys, the online survey platform will generate the pooled results for each round. Quantitative data will be expressed in percentages as statistical group response. Qualitative data will be presented for each survey item, and it will be analyzed through qualitative content analysis, which is “an approach to text analysis which combines strict rulebound interpretive category assignments with quantifications of category occurrences” [70 p. 209]. This will be done with the latest version of QCAmap (www.qcamap.org), which is an online software program that is specifically designed for qualitative content analysis. QCAmap interactively guides users through the steps of qualitative content analysis [70], which will ensure that each step is carried out in this study.

Round 1

The research team will conduct qualitative content analysis to analyze participants’ responses. The findings will be used to create a structured survey for the next round. At the start of the next round, participants will receive feedback from this round, which will consist of a summary of participants’ responses. All responses will be anonymized.

Round 2

Since consensus will be defined a priori as ≥80% agreement, items where ≥80% of participants rate their agreement to include them as 4 (agree) or 5 (strongly agree) on the Likert scale will be included in the preliminary conceptual and operational definitions of social prescribing. The research team will conduct qualitative content analysis to analyze participants’ responses. Based on the findings, preliminary conceptual and operational definitions of social prescribing will be developed. At the start of the next round, participants will receive feedback from this round. Quantitative feedback will consist of the percentage of agreement and the individual response of each participant in relation to the group response, and qualitative feedback will consist of a summary of participants’ responses. All responses will be anonymized.

Round 3 and Beyond

For consensus to be reached, ≥80% of participants will need to rate their agreement with the conceptual and operational definitions of social prescribing as 4 (agree) or 5 (strongly agree) on the Likert scale. If this threshold is not met for both definitions, another round will be completed. The research team will conduct qualitative content analysis to analyze participants’ responses. Based on the findings, modifications will be made to the conceptual and operational definitions of social prescribing. At the start of each round, participants will receive feedback from the previous round. Quantitative feedback will consist of the percentage of agreement and the individual response of each participant in relation to the group response, and qualitative feedback will consist of a summary of participants’ responses. All responses will be anonymized.

Quality and Transparency

This study will be conducted and reported in accordance with the Conducting and REporting DElphi Studies (CREDES) guideline [71]. This protocol has been registered on Open Science Framework (osf.io/pfyqg) and published in this journal to promote transparency.

Ethics

This study has been reviewed for ethical compliance by the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board (NURS-540-22).

Discussion

The Delphi study that is outlined in this protocol will be conducted to develop internationally accepted conceptual and operational definitions of social prescribing with an international, multidisciplinary panel of experts. This protocol has been published in this journal to promote transparency. It adds to the growing number of Delphi study protocols.

There are several strengths of the Delphi technique, such as its flexible approach to gathering the opinions of experts on a particular topic [29, 69]. The most significant strengths of this consensus method are the three characteristics that make it unique, namely participant anonymity, multiple rounds of questioning, and provision of feedback between each round, which reduce bias in the process of gaining consensus [29, 31, 69]. Another important strength is that the online format is well suited to gathering the opinions of experts who are geographically dispersed [29, 30], which is particularly relevant given the challenges imposed by the COVID-19 pandemic and the desire to acquire an international panel of experts to be able to develop internationally accepted definitions. There are also some limitations of the Delphi technique, such as its reputation for being complex and time consuming [30, 31, 69]. The most significant limitation of this consensus method is the lack of standard guidelines and agreement in the literature [29, 30, 31, 69]. This issue was mitigated in this protocol by making key decisions based on the literature, and it will be addressed in this study, as this research will be conducted and reported in accordance with the CREDES guideline [71]. Another important limitation is the risk of panel attrition [29, 30, 69]. This issue was mitigated in this protocol by taking this risk into account when determining the size of the expert panel, and it will be addressed in this study, as this research will include various retention strategies (e.g., sending out reminder emails to non-responders, providing quick turnaround times between rounds of questioning).

It is anticipated that this research will have extensive and enduring impacts on the social prescribing community. Not only will these internationally accepted conceptual and operational definitions of social prescribing serve to unite the social prescribing community, but they will also inform research, policy, and practice. By laying the groundwork for the formation of a robust evidence base, this foundational work will support the advancement of social prescribing and help to unlock the full potential of the social prescribing movement, not to mention that it will answer the call that was made by the Social Prescribing Network in 2016 for an agreed definition of social prescribing [16].

Conclusion

This paper outlines the protocol for a Delphi study that will be conducted to develop internationally accepted conceptual and operational definitions of social prescribing with an international, multidisciplinary panel of experts. This important work will be foundational and timely, given the rapid spread of the social prescribing movement around the world.

Reviewers

Chris Drinkwater, Emeritus Professor of Primary Care Development, Northumbria University, UK.

David Price, Professor, Department of Family Medicine, McMaster University, Canada.

Competing Interests

The authors have no competing interests to declare.

References

  1. Morse DF, Sandhu S, Mulligan K, Tierney S, Polley M, Chiva Giurca B, et al. Global developments in social prescribing. BMJ Glob Health. 2022 May; 7: e008524. DOI: https://doi.org/10.1136/bmjgh-2022-008524 

  2. Global Social Prescribing Alliance. Social prescribing: International student movement framework [Internet]. Global Social Prescribing Alliance; 2021. Available from: https://www.gspalliance.com/student-movement-framework. 

  3. Social Prescribing Network. About [Internet]. Social Prescribing Network; 2022 [cited 2022 Apr 22]. Available from: https://www.socialprescribingnetwork.com/about. 

  4. National Academy for Social Prescribing. About us [Internet]. National Academy for Social Prescribing; 2021 [cited 2022 Apr 22]. Available from: https://socialprescribingacademy.org.uk/about-us/. 

  5. Global Social Prescribing Alliance. Global Social Prescribing Alliance: International playbook [Internet]. Global Social Prescribing Alliance; 2021. Available from: https://www.gspalliance.com/gspa-playbook. 

  6. World Health Organization. A toolkit on how to implement social prescribing [Internet]. World Health Organization; 2022. Available from: https://www.who.int/publications/i/item/9789290619765. 

  7. World Health Organization. Social prescribing [Internet]. World Health Organization. [cited 2022 May 30]. Available from: https://openwho.org/courses/social-prescribing-WPRO. 

  8. Malby B, Boyle D, Wildman J, Smith S, Omar S. The asset based health inquiry: How best to develop social prescribing [Internet]. London South Bank University; 2019. Available from: https://www.lsbu.ac.uk/__data/assets/pdf_file/0018/251190/lsbu_asset-based_health_inquiry.pdf. 

  9. Costa A, Sousa CJ, Seabra PRC, Virgolino A, Santos O, Lopes J, et al. Effectiveness of social prescribing programs in the primary health-care context: A systematic literature review. Sustainability. 2021 Mar 3; 13(5): 2731. DOI: https://doi.org/10.3390/su13052731 

  10. Zurynski Y, Vedovi A, Smith K-l. Social prescribing: A rapid literature review to inform primary care policy in Australia [Internet]. NHMRC Partnership Centre for Health System Sustainability Australian Institute of Health Innovation, Macquarie University; 2020. Available from: https://chf.org.au/sites/default/files/sprapidreview_3-2-20_final.pdf. 

  11. Rempel ES, Wilson EN, Durrant H, Barnett J. Preparing the prescription: A review of the aim and measurement of social referral programmes. BMJ Open. 2017 Oct; 7(10): e017734. DOI: https://doi.org/10.1136/bmjopen-2017-017734 

  12. Islam MM. Social prescribing—An effort to apply a common knowledge: Impelling forces and challenges. Front Public Health. 2020 Nov 27; 8: 515469. DOI: https://doi.org/10.3389/fpubh.2020.515469 

  13. Bickerdike L, Booth A, Wilson PM, Farley K, Wright K. Social prescribing: Less rhetoric and more reality. A systematic review of the evidence. BMJ Open. 2017 Apr; 7(4): e013384. DOI: https://doi.org/10.1136/bmjopen-2016-013384 

  14. Booth A, Wilson P, Bickerdike L. Evidence to inform the commissioning of social prescribing [Internet]. Centre for Reviews and Dissemination, University of York; 2015. Available from: https://www.york.ac.uk/media/crd/Ev%20briefing_social_prescribing.pdf. 

  15. Carnes D, Sohanpal R, Frostick C, Hull S, Mathur R, Netuveli G, et al. The impact of a social prescribing service on patients in primary care: A mixed methods evaluation. BMC Health Serv Res. 2017 Dec; 17(835): 1–9. DOI: https://doi.org/10.1186/s12913-017-2778-y 

  16. Dixon M, Polley M. Report of the annual Social Prescribing Network Conference [Internet]. Social Prescribing Network; 2016. Available from: https://42b7de07-529d-4774-b3e1-225090d531bd.filesusr.com/ugd/14f499_9ba1233600eb454ab836b1c6424feed3.pdf. 

  17. Elston J, Gradinger F, Asthana S, Lilley-Woolnough C, Wroe S, Harman H, et al. Does a social prescribing ‘holistic’ link-worker for older people with complex, multimorbidity improve well-being and frailty and reduce health and social care use and costs? A 12-month before-and-after evaluation. Prim Health Care Res Dev. 2019; 20(e135): 1–10. DOI: https://doi.org/10.1017/S1463423619000598 

  18. Gibbons A, Howarth M, Lythgoe A. Social prescribing in Greater Manchester [Internet]. University of Salford; 2019. Available from: https://usir.salford.ac.uk/id/eprint/52968/1/SocialPrescribingGMFullReport2019.pdf. 

  19. Kimberlee R. What is social prescribing? Adv Soc Sci Res J. 2015 Jan 25; 2(1): 102–110. DOI: https://doi.org/10.14738/assrj.21.808 

  20. Kinsella S. Social prescribing: A review of the evidence [Internet]; 2015. Available from: https://www.scie-socialcareonline.org.uk/social-prescribing-a-review-of-the-evidence/r/a11G000000CTdhKIAT. 

  21. Lejac B. A desk review of social prescribing: From origins to opportunities [Internet]. Support in Mind Scotland; 2021. Available from: https://www.rsecovidcommission.org.uk/a-desk-review-of-social-prescribing-from-origins-to-opportunities/. 

  22. Moffatt S, Steer M, Lawson S, Penn L, O’Brien N. Link worker social prescribing to improve health and well-being for people with long-term conditions: Qualitative study of service user perceptions. BMJ Open. 2017 Jul; 7(7): e015203. DOI: https://doi.org/10.1136/bmjopen-2016-015203 

  23. Pescheny J, Pappas Y, Randhawa G. Facilitators and barriers of implementing and delivering social prescribing services: A systematic review. BMC Health Serv Res. 2018 Dec; 18(86). DOI: https://doi.org/10.1186/s12913-018-2893-4 

  24. Pescheny J, Gunn LH, Randhawa G, Pappas Y. The impact of the Luton social prescribing programme on energy expenditure: A quantitative before-and-after study. BMJ Open. 2019 Jun; 9(6): e026862. DOI: https://doi.org/10.1136/bmjopen-2018-026862 

  25. Pilkington K, Loef M, Polley M. Searching for real-world effectiveness of health care innovations: Scoping study of social prescribing for diabetes. J Med Internet Res. 2017 Feb 2; 19(2): e20. DOI: https://doi.org/10.2196/jmir.6431 

  26. Polley M, Fleming J, Anfilogoff T, Carpenter A. Making sense of social prescribing [Internet]. University of Westminster; 2017. Available from: https://waystowellness.org.uk/site/assets/files/1317/making-sense-of-social-prescribing_2017_2.pdf. 

  27. Westlake D, Elston J, Gude A, Gradinger F, Husk K, Asthana S. Impact of COVID-19 on social prescribing across an Integrated Care System: A Researcher in Residence study. Health Social Care Comm. 2022 Mar 30; 1–9. DOI: https://doi.org/10.1111/hsc.13802 

  28. Richards W. The zen of empirical research [Internet]. Vancouver, BC: Empirical Press; 1998. Part 1, Scientific research. Available from: https://www.sfu.ca/personal/archives/richards/Zen/Pages/zentoc.html. 

  29. Chalmers J, Armour M. The Delphi technique. In: Liamputtong, P (ed.), Handbook of Research Methods in Health Social Sciences [Internet]. Singapore: Springer Singapore; 2019 [cited 2021 Aug 30]. p. 715–35. Available from: http://link.springer.com/10.1007/978-981-10-5251-4_99. DOI: https://doi.org/10.1007/978-981-10-5251-4_99 

  30. Puig A, Adams C. Delphi technique. In: The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation [Internet]. 2455 Teller Road, Thousand Oaks, California 91320: SAGE Publications, Inc.; 2018 [cited 2021 Aug 30]. Available from: https://methods.sagepub.com/reference/the-sage-encyclopedia-of-educational-research-measurement-and-evaluation. 

  31. Niederberger M, Spranger J. Delphi technique in health sciences: A map. Front Public Health. 2020 Sep 22; 8(457). DOI: https://doi.org/10.3389/fpubh.2020.00457 

  32. Metzelthin SF, Rostgaard T, Parsons M, Burton E. Development of an internationally accepted definition of reablement: A Delphi study. Ageing and Society. 2020 Sep 4; 1–16. DOI: https://doi.org/10.1017/S0144686X20000999 

  33. Tait AR, Geisser ME. Development of a consensus operational definition of child assent for research. BMC Med Ethics. 2017 Dec; 18(41). DOI: https://doi.org/10.1186/s12910-017-0199-4 

  34. Allen J, Brenner M, Hauer J, Molloy E, McDonald D. Severe Neurological Impairment: A Delphi consensus-based definition. European Journal of Paediatric Neurology. 2020 Nov; 29: 81–6. DOI: https://doi.org/10.1016/j.ejpn.2020.09.001 

  35. Avdagic E, Wade C, McDonald M, McCormack D, Dakin P, Macvean M, et al. Resilience in young children: A Delphi study to reach consensus on definitions, measurement and interventions to build resilience. Early Child Development and Care. 2020 Oct 2; 190(13): 2066–77. DOI: https://doi.org/10.1080/03004430.2018.1556211 

  36. Campbell KA, Wood JN, Lindberg DM, Berger RP. A standardized definition of near-fatal child maltreatment: Results of a multidisciplinary Delphi process. Child Abuse & Neglect. 2021 Feb; 112: 104893. DOI: https://doi.org/10.1016/j.chiabu.2020.104893 

  37. Beune IM, Damhuis SE, Ganzevoort W, Hutchinson JC, Khong TY, Mooney EE, et al. Consensus definition of fetal growth restriction in intrauterine fetal death: A Delphi procedure. Archives of Pathology & Laboratory Medicine. 2021 Apr 1; 145(4): 428–36. DOI: https://doi.org/10.5858/arpa.2020-0027-OA 

  38. Ben-Chetrit E, Gattorno M, Gul A, Kastner DL, Lachmann HJ, Touitou I, et al. Consensus proposal for taxonomy and definition of the autoinflammatory diseases (AIDs): A Delphi study. Ann Rheum Dis. 2018 Nov; 77(11): 1558–65. DOI: https://doi.org/10.1136/annrheumdis-2017-212515 

  39. Guseva Canu I, Marca SC, Dell’Oro F, Balázs Á, Bergamaschi E, Besse C, et al. Harmonized definition of occupational burnout: A systematic review, semantic analysis, and Delphi consensus in 29 countries. Scand J Work Environ Health. 2021 Mar 1; 47(2): 95–107. DOI: https://doi.org/10.5271/sjweh.3935 

  40. Dekker J, Amitami M, Berman AH, Brown H, Cleal B, Figueiras MJ, et al. Definition and characteristics of behavioral medicine, and main tasks and goals of the International Society of Behavioral Medicine—An international Delphi study. IntJ Behav Med. 2021 Jun; 28(3): 268–76. DOI: https://doi.org/10.1007/s12529-020-09928-y 

  41. Khalil A, Beune I, Hecher K, Wynia K, Ganzevoort W, Reed K, et al. Consensus definition and essential reporting parameters of selective fetal growth restriction in twin pregnancy: A Delphi procedure. Ultrasound Obstet Gynecol. 2019 Jan; 53(1): 47–54. DOI: https://doi.org/10.1002/uog.19013 

  42. Luquin MR, Kulisevsky J, Martinez-Martin P, Mir P, Tolosa ES. Consensus on the definition of advanced Parkinson’s disease: A neurologists-based Delphi study (CEPA study). Parkinson’s Disease. 2017; 2017: 1–8. DOI: https://doi.org/10.1155/2017/4047392 

  43. Monnier AA, Eisenstein BI, Hulscher ME, Gyssens IC, the DRIVE-AB WP1 group, Adriaenssens N, et al. Towards a global definition of responsible antibiotic use: Results of an international multidisciplinary consensus procedure. Journal of Antimicrobial Chemotherapy. 2018 Jun 1; 73(6): vi3–16. DOI: https://doi.org/10.1093/jac/dky114 

  44. Schols AMR, Dinant GJ, Hopstaken R, Price CP, Kusters R, Cals JWL. International definition of a point-of-care test in family practice: A modified e-Delphi procedure. Family Practice. 2018 Jul 23; 35(4): 475–80. DOI: https://doi.org/10.1093/fampra/cmx134 

  45. Van den Steene H, van West D, Glazemakers I. Towards a definition of multiple and complex needs in children and youth: Delphi study in Flanders and international survey. Scandinavian Journal of Child and Adolescent Psychiatry and Psychology. 2019; 7: 60–7. DOI: https://doi.org/10.21307/sjcapp-2019-009 

  46. van der Horst N, Backx F, Goedhart EA, Huisstede BM. Return to play after hamstring injuries in football (soccer): A worldwide Delphi procedure regarding definition, medical criteria and decision-making. Br J Sports Med. 2017 Nov; 51(22): 1583–91. DOI: https://doi.org/10.1136/bjsports-2016-097206 

  47. Wong HS, Curry NS, Davenport RA, Yu L, Stanworth SJ. A Delphi study to establish consensus on a definition of major bleeding in adult trauma. Transfusion. 2020 Dec; 60(12): 3028–38. DOI: https://doi.org/10.1111/trf.16055 

  48. Zanker J, Scott D, Reijnierse EM, Brennan-Olsen SL, Daly RM, Girgis CM, et al. Establishing an operational definition of sarcopenia in Australia and New Zealand: Delphi method based consensus statement. J Nutr Health Aging. 2019; 23(1): 105–10. DOI: https://doi.org/10.1007/s12603-018-1113-6 

  49. Adams B, Sereda M, Casey A, Byrom P, Stockdale D, Hoare DJ. A Delphi survey to determine a definition and description of hyperacusis by clinician consensus. International Journal of Audiology. 2020 Dec 11; 1–7. DOI: https://doi.org/10.1080/14992027.2020.1855370 

  50. Keller S, Salinas A, Williams D, McGoldrick M, Gorski L, Alexander M, et al. Reaching consensus on a home infusion central line-associated bloodstream infection surveillance definition via a modified Delphi approach. American Journal of Infection Control. 2020 Sep; 48(9): 993–1000. DOI: https://doi.org/10.1016/j.ajic.2019.12.015 

  51. Lakhani BK, Giannouladis K, Leighton P, King AJ. Seeking a practical definition of stable glaucoma: A Delphi consensus survey of UK glaucoma consultants. Eye. 2020 Feb; 34(2): 335–43. DOI: https://doi.org/10.1038/s41433-019-0540-x 

  52. van Helsdingen CP van, Jongen AC, Jonge WJ de, Bouvy ND, Derikx JP. Consensus on the definition of colorectal anastomotic leakage: A modified Delphi study. WJG. 2020 Jun 21; 26(23): 3293–303. DOI: https://doi.org/10.3748/wjg.v26.i23.3293 

  53. Lagacé-Legendre C, Boucher V, Robert S, Tardif PA, Ouellet MC, de Guise E, et al. Persistent postconcussion symptoms: An expert consensus-based definition using the Delphi method. Journal of Head Trauma Rehabilitation. 2021 Mar; 36(2): 96–102. DOI: https://doi.org/10.1097/HTR.0000000000000613 

  54. Sudore RL, Lum HD, You JJ, Hanson LC, Meier DE, Pantilat SZ, et al. Defining advance care planning for adults: A consensus definition from a multidisciplinary Delphi panel. Journal of Pain and Symptom Management. 2017 May; 53(5): 821–832.e1. DOI: https://doi.org/10.1016/j.jpainsymman.2016.12.331 

  55. Bell DR, Snedden T, Biese K, Nelson E, Watson A, McGuine T, et al. Consensus definition of sport specialization in youth athletes using a Delphi approach. J Athl Train; 2021 Mar 31. DOI: https://doi.org/10.4085/1062-6050-0725.20 

  56. Kaur N, Pluye P. Delineating and operationalizing the definition of patient-oriented research: A modified e-Delphi study. Journal of Patient-Centered Research and Reviews. 2019 Jan 28; 6(1): 7–16. DOI: https://doi.org/10.17294/2330-0698.1655 

  57. Julià-Torras J, Cuervo-Pinna MÁ, Cabezón-Gutiérrez L, Lara PC, Prats M, Margarit C, et al. Definition of episodic dyspnea in cancer patients: A Delphi-based consensus among Spanish experts: The INSPIRA study. Journal of Palliative Medicine. 2019 Apr; 22(4): 413–9. DOI: https://doi.org/10.1089/jpm.2018.0273 

  58. Feo R, Conroy T, Jangland E, Muntlin Athlin Å, Brovall M, Parr J, et al. Towards a standardised definition for fundamental care: A modified Delphi study. J Clin Nurs. 2018 Jun; 27(11–12): 2285–99. DOI: https://doi.org/10.1111/jocn.14247 

  59. D’Souza N, de Neree tot Babberich MPM, d’Hoore A, Tiret E, Xynos E, Beets-Tan RGH, et al. Definition of the rectum: An international, expert-based Delphi consensus. Annals of Surgery. 2019 Dec; 270(6): 955–9. DOI: https://doi.org/10.1097/SLA.0000000000003251 

  60. Sacco S, Braschinsky M, Ducros A, Lampl C, Little P, van den Brink AM, et al. European headache federation consensus on the definition of resistant and refractory migraine: Developed with the endorsement of the European Migraine & Headache Alliance (EMHA). J Headache Pain. 2020 Dec; 21(76). DOI: https://doi.org/10.1186/s10194-020-01130-5 

  61. Schaap T, Bloemenkamp K, Deneux-Tharaux C, Knight M, Langhoff-Roos J, Sullivan E, et al. Defining definitions: A Delphi study to develop a core outcome set for conditions of severe maternal morbidity. BJOG: Int J Obstet Gy. 2019 Feb; 126(3): 394–401. DOI: https://doi.org/10.1111/1471-0528.14833 

  62. Lawrence M, Asaba E, Duncan E, Elf M, Eriksson G, Faulkner J, et al. Stroke secondary prevention, a non-surgical and non-pharmacological consensus definition: Results of a Delphi study. BMC Res Notes. 2019 Dec; 12(823). DOI: https://doi.org/10.1186/s13104-019-4857-0 

  63. Casellas-Grau A, Jordán de Luna C, Maté J, Ochoa C, Sumalla EC, Gil F. Developing a consensus definition of psychosocial complexity in cancer patients using Delphi methods. Pall Supp Care. 2021 Feb; 19(1): 17–27. DOI: https://doi.org/10.1017/S1478951520000784 

  64. Katz RV, Dearing BA, Ryan JM, Ryan LK, Zubi MK, Sokhal GK. Development of a tongue-tie case definition in newborns using a Delphi survey: The NYU–tongue-tie case definition. Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology. 2020 Jan; 129(1): 21–6. DOI: https://doi.org/10.1016/j.oooo.2019.01.012 

  65. Smith ME, Hardman JC, Mehta N, Jones GH, Mandavia R, Anderson C, et al. Acute otitis externa: Consensus definition, diagnostic criteria and core outcome set development. PLoS One. 2021; 16(5): e0251395. DOI: https://doi.org/10.1371/journal.pone.0251395 

  66. Spinelli A, Anania G, Arezzo A, Berti S, Bianco F, Bianchi PP, et al. Italian multi-society modified Delphi consensus on the definition and management of anastomotic leakage in colorectal surgery. Updates Surg. 2020 Sep; 72(3): 781–92. DOI: https://doi.org/10.1007/s13304-020-00837-z 

  67. Yousif A, Forget A, Beauchesne MF, Lemière C, Dugré N, Fénélon-Dimanche R, et al. Development of an operational definition of treatment escalation in adults with asthma adapted to healthcare administrative databases: A Delphi study. Respiratory Medicine. 2021 Aug; 185: 106510. DOI: https://doi.org/10.1016/j.rmed.2021.106510 

  68. Tack J, Boardman H, Layer P, Schiefke I, Jayne D, Scarpignato C, et al. An expert consensus definition of failure of a treatment to provide adequate relief (F-PAR) for chronic constipation – An international Delphi survey. Aliment Pharmacol Ther. 2017 Feb; 45(3): 434–42. DOI: https://doi.org/10.1111/apt.13874 

  69. Barrett D, Heale R. What are Delphi studies? Evid Based Nurs. 2020 Jul; 23(3): 68–9. DOI: https://doi.org/10.1136/ebnurs-2020-103303 

  70. Mayring P. Qualitative content analysis: A step-by-step guide. SAGE; 2021. Available from: https://uk.sagepub.com/en-gb/eur/qualitative-content-analysis/book269922. 

  71. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017 Sep; 31(8): 684–706. DOI: https://doi.org/10.1177/0269216317690685