Attachment 5 GuidelineForGuidelines Approved 08262022.pdf

Department of Veteran Affairs Department of Defense Veterans Health Administration Defense Health Agency Washington, DC 20420 Falls Church, VA 22042

GUIDELINE FOR GUIDELINES

Guideline Development and Approval Process:

1. New Guideline Request: A clinician or other group may request the development of a new Department of Veterans Affairs (VA)/Department of Defense (DoD) guideline utilizing the following process:

1.1. An application is completed and submitted to VA/DoD Evidence Based Practice Work Group (EBPWG) through the submitter’s respective Veterans Affairs Central Office (VACO) or Department of Defense chain of command, to include respective VA or DoD EBPWG Co-Chair.

At a minimum, the application will include: a description of the guideline, identify end-users of the guideline and perceived gaps in care and/or identify changes in performance to be driven by the guideline (See Appendix A.: Application Form). To the extent possible, data substantiating the need for the guideline will be presented.

1.2. The applicant will also submit a brief structured review of the literature.

1.3. A funding source for the guideline will be included with the application.

1.4. The VA/DoD Evidence-Based Practice Work Group may also suggest topics/areas for guideline development, particularly as they relate to the frequency of occurrence and uniqueness of our military and veteran population or as mandated by Congress or public law (e.g. Suicide, Opiate CPGs).

2. Application review and approval: The EBPWG will review complete applications, vote to approve or disapprove the development of a new CPG, and prioritize it for development if it is approved.

2.1. The respective VA or DoD Evidence Based Program office will acknowledge receipt of each application within 7 days.

2.2. The EBPWG will consider the following issues: High incidence or prevalence, risk and cost of the disease or condition in the general veteran/military population or sub-populations targeted by Special Emphasis Programs, potential for reduction of clinically significant variations in the prevention. The diagnosis, treatment, or clinical management of a disease or condition will also be considered when establishing priorities.

2.3. After discussion with a quorum of EBPWG voting members, the EBPWG Co-Chairs will notify the applicant of the outcome of the review.

3. Identification of Clinical Champions: When a topic has been approved for guideline development, designees of the DHA Clinical Quality Improvement Program/Clinical Practice Guidelines (CQI/CPG) and the VA Offices of Quality and Patient Safety (QPS) will identify Clinical Champions, and/or CPG Work Group Representatives. Specifically, the DHA CQI/CPG and VA QPS representatives will:

3.1. Identify clinical leaders (without conflict of interest) who will champion the guideline development.

3.2. Assure there is representation from primary care and, as needed, specialty services.

3.3. Invite members of related VA HSR&D (Health, Service, Research, and Development)

Center groups (i.e. QUERI) to participate, if available.

3.4. The VA and DoD program offices will convene a group of not more than 20 work group members;

ideally, 10 from the VA and 10 from the DoD, to evaluate the evidence and develop the guideline. At a minimum, each CPG work group will include representatives from primary care, nursing, pharmacy, social services.

3.5. The VA/DoD program offices, along with the contracted physician facilitator, will serve as evidence chaperones to maintain the integrity of the process. Third party subject matter experts will be utilized if needed.

3.6. Assign representatives from the VA & DoD Evidence Based Program offices to monitor the development process.

4. Key Question Development

4.1. VA and DoD Champions and work group members meet face-to-face/teleconference, as needed, with the contracted physician facilitator to identify key questions formulated in the PICO(TS) framework:

Population – Characteristics of the target patient population Intervention – Exposure, diagnostic, or prognosis Comparison – Intervention, exposure, or control used for comparison Outcome – Outcomes of interest to be answered by the evidence Time (if applicable) - Describes the duration of time that is of interest Setting (if applicable) – Describes the setting or context of interest

4.2. This is an iterative process and may require face to face and/or conference call discussions to complete the task.

4.3. Veteran/DoD Patient Focus Groups. The Veteran/DoD Patient Focus Group will be a convenience sample of no more than nine participants in accordance with General Accounting Office (GAO) guidance. The purpose of the focus group is to inform Key Question development.

4.4. Initial boundaries for admissible evidence will also be set, recognizing that no two CPGs will be the same and that additional data requirements may be discovered through the iterative process of CPG development. For example, questions of the efficacy of interventions usually means that randomized controlled trial data will be sought. In other instances, epidemiologic, pharmacoepidemiologic, case reports or research letters may contain applicable data.

5. Potential Conflicts of Interest: The VA/DoD has adopted a policy of transparency, disclosing potential conflicts, and competing interests of all individuals who participate in the development, revision, and review of the VA/DoD clinical practice guidelines.

5.1. Champion(s) and other key clinical leaders/CPG workgroup members involved with this effort will be asked to submit disclosure statements to reveal any areas of potential conflict of interest (See Appendix B) for the preceding 24 months. Conflict of Interest statements will be sent to VA and DHA Evidence Based Program office.

5.2. Verbal disclosures of conflict of interest: verbal affirmations are conducted at each meeting, and a signed disclosure statement is required annually.

5.2.1 Members may be subject to random web-based surveillance (i.e. CMMS open payments or ProPublica).

5.2.2 If there is a positive (yes) conflict of interest response (actual or potential) then a determination is made by the co-chairs and evidence-based practice program office based on level and extent of involvement to mitigate conflict of interest. Determination may range from restricting participation and/or voting on section related to conflict, up to removal from the work group. Recusals are determined by the individual, co-chairs and/or evidence-based practice program office.

5.2.3 Co-chairs/champions and the evidence-based practice program offices of the VA and DoD are responsible for monitoring conflict of interest compliance.

6. Systematic Review of the Literature Based on the Questions Identified in Step Five is

Conducted & Tables of Evidence are Produced:

6.1. When the initial Key Questions have been developed, the group will convene to: Review the Key Questions to assure that they are on track and address the Key Questions that will lead to a comprehensive, systematic review of the literature pertaining to the topic.

6.2. A systematic review of the literature, by a disinterested party, will be performed to minimize bias, collect all appropriate evidence available, and assess its potential applicability to the clinical question under consideration.

2.6.1. The first step in gathering the evidence is to see if a suitable, recent systematic review has already been published. If a current systematic review is not available, an original systematic review will be done using an established protocol, such as those of the Cochrane Collaboration, Evidence Synthesis Program, or the US Preventive Services Task Force (USPSTF). At a minimum, systematic reviews will use explicit, reproducible methods to: Identify relevant, eligible studies, assess the quality of each study and the body of evidence, critically appraise key studies, synthesize results.

2.6.2. To grade the quality of individual studies, the reviews will apply the USPSTF criteria for quality (Harris, Helfand, & Woolf, 2001), adapting those to specific clinical areas.

The Work Group will work with staff from the VACO Office of Evidence Based Practice to ensure conformity to prevailing standards for conducting high-quality systematic literature reviews.

6.3. Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) will be used to assess overall strength of evidence and clinical recommendations. (Guyett, et.al., 2008)

6.4. Prior to posting the reviews, the facilitator, Champion(s), (and the Evidence Chaperone as needed), will convene to ensure the adequacy of the evidence reviews.

7. Evidence Review Recommendation Development Meeting: Convened once the evidence tables have been completed.

7.1. The CPG Work Group will meet face to face to review and grade the evidence and begin development of clinical recommendations.

7.2. Prior to the Evidence Review Recommendation Development meeting, work group members will be asked to re-submit another disclosure statement regarding any potential conflicts of interest.

These statements will be reviewed in advance to assure the integrity of the group that is forming.

7.3. Each meeting will begin with a brief session that will permit full disclosure to the group of any conflicts related to the guideline.

7.4. Key points of the guideline will be identified.

7.5. A contracted physician facilitator will ensure that the meeting stays focused and that the evidence remains the driving force behind the guidelines.

7.6. Each guideline will include a clinical algorithm outlining step-by-step decision points in the disease management process.

7.7. The strength of each recommendation and the quality of evidence are provided at the end of the discussion section for each Recommendation in the guideline per the GRADE criteria (Appendix E).

7.8. The review of the evidence will summarize the quality and consistency of the evidence and the magnitude of benefits and harms.

7.9. The VA and DoD Champions will lead discussions to develop Recommendations with the clinical experts. The discussion will include interpretation the evidence, assessment of its ability to be applied in the clinical setting, its applicability to the population of interest, and an assessment of the overall strength of the evidence for the Recommendation.

7.10. Recommendations based solely on clinical judgment and experience will be thoroughly scrutinized to eliminate bias and self-interest.

Work group members will grade the evidence using the evaluation system established by the USPSTF and the recommendations using the GRADE format.

8. The USPSTF system is described in USPSTF Methods and Process, August 2012. See Appendix D

8.1. Work group members will rate the level of evidence using the terms shown in Table 1.

8.2. Based on the ratings of the level of evidence and the magnitude of net benefit, the clinical experts will assign a grade to each recommendation using the definitions in Appendix E.

8.3. The overall strength of each body of evidence that addresses a particular Key Question is then assessed. The number, quality, and size of the studies, as well as the consistency of results between studies and the directness of the evidence will be considered in assigning an overall quality [QE] of the evidence (i.e., good, fair, or poor) (see Table 2). Consistent results from several higher-level studies [LE] (see Table 1) that have been conducted across a broad range of populations support a high degree of certainty that the results of the studies are true. In such case the entire body of evidence would be considered ‘‘good” quality.

8.4. The quality of the body of evidence is considered ‘fair” when the results could be due to true effects or to biases present across some or all of the studies. For a ‘‘poor” quality body of evidence, any conclusion is uncertain due to serious methodological shortcomings, sparse data, or inconsistent results. For interventions that were supported by studies of ‘Fair’ or “Good” quality, the clinical experts evaluate the benefits and the potential harms as demonstrated by the results of the studies.

9. Recommendations will be graded using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system established by the World Health Organization

9.1. GRADE offers two categories of recommendations: “for” or “against”. Within each of those categories the Recommendation can be graded as “strong” or “weak” based on the strength of the evidence, balance of benefits and harm, and provider/patient preference. The recommendation and narrative should reflect the quality of the evidence. The contracted facilitator will ensure that the recommendation and the narrative are consistent.

9.2. GRADE is described in the series of tables in Appendix E.

10. Follow Up Conference Calls will be Conducted to Discuss Unresolved Issues and

Compile the Annotations of the Guideline.

10.1. The resulting product is the first draft of the guideline that will be distributed to members of the work group.

10.2. Prior to this review, the Champions and the Facilitator confirm the timeline and assure that the recommendations are consistent with the evidence.

10.3. Beginning with Draft 2, the EBPWG members will be provided with the website link and CPG draft copy for content feedback to the CPG work group.

11. The Third Draft of the Guideline will be posted on a Development Website for Field Review and Public Comment: Veteran/DoD Patient focus group participants will be invited to comment.

12. The third draft of the guideline is also sent to outside national experts who have agreed to perform an independent review via the identified website for each guideline.

12.1. This independent review is directed towards an evaluation of the content of the guideline, as well as the format and usability of the guideline.

12.2. The reviewer’s comments and recommendations regarding the content of the guideline will be provided to the champions / the executive panel of the working group.

12.3. All reviewers will be asked to identify any Conflicts of Interest.

13. DHA Clinical Community Advisory Council (CCAC) and the VA Network Clinical

Managers will solicit feedback from a broader group of end users, to include patients.

14. VA Network designated staff and DoD end users will be asked to review the guideline and provide feedback to the guideline contractor and/or directly to the VA and DoD program offices via the wiki web page that is available for online comment. This portion of the field test is more specifically directed towards an evaluation of the content and the logic and flow of the guideline.

14.1. Comments and recommendations regarding proposed changes to the content of the guideline must be supported by evidence.

14.2. The VA/DoD Guideline Champions will integrate comments and suggestions into the guideline as appropriate. The guidelines contractor will provide the EBPWG a copy of the document with comments and how they were adjudicated.

15. Presentation of Guideline to full VA/DoD EBPWG for Approval:

15.1. An electronic copy of the guideline along with a summary of the comments from the reviewers will be provided to the entire VA/DoD EBPWG at least two weeks in advance of the meeting

15.2. The VA/DoD EBPWG again reviews comments from independent reviewers and verifies that all appropriate suggestions have been incorporated into the final document.

15.3. When the EBPWG is convened, the Champion(s), and representatives of the guidelines contractor, will present the guideline to the EBPWG.

15.4. Following the presentation, EBPWG members will have the opportunity to ask questions of the Champion(s) and provide feedback that will be entered into the minutes.

15.5. The Guideline will then be either approved or further modifications will be made.

15.6. Once approved, the contractor/vendor will put the CPG and associated tools into final format.

16. Process of Defining the Role of Authors and Contributors for Subsequent Publications Emerging from the Main VA-DoD Evidence-Based Clinical Practice Guideline (CPG) Documents

16.1. Eligibility and Establishing Authorship Order

1.16.1. The work group members should refer to the International Committee of Medical Journal Editors (ICMJE) for details on the Roles and Responsibilities of Authorship and Contributors.1

1.16.2. Based on the above guideline, developing the Evidenced Based Practice key evidence questions framework including the Patient, Population or Problem; Intervention (or Exposure); Comparison; Outcome; Timing (if applicable); and Setting (if applicable) components referred to as the PICO(TS) process, writing the guideline content, including drafting, and editing the manuscript would be sufficient to meet authorship criteria. It should be noted any CPG work group member may elect to opt out at any stage of manuscript development, depending on their availability and interest in conferring authorship.

1.16.3. All designated champions will be designated as contributors unless opting out from being distinguished as an author. An appendix will be included in the manuscript with names of all contributing work group members, including those who opted out from authorship on the summary manuscript(s). When and if the VA-DoD CPGs are published, any work group member not participating in the PICO(TS) process including writing the guideline, drafting, and editing the manuscript will not be designated as an author but will instead be considered as a contributor to the CPG development upon recommendations of the Champion and the VA-DoD Evidence Based Practice Workgroup (EBPWG) members.

1.16.4. The champions shall determine the authorship role, as to whom would serve as first, middle, or senior, as well as corresponding author on the manuscript, and that decision should rely on their interest, willingness, and availability to complete the work required to fulfill these roles as defined in the aforementioned guidelines (ICMJE).

16.2. Disseminating the Information - To provide additional impetus and motivation to serve in the CPG development process, ample opportunities for the CPG work group member to be informed and opt in or out to participate in writing any summary manuscripts that follows the main CPG publication will be provided in the following ways:

2.16.1. At outset, as part of the initial written invitation letter

2.16.2. Announced verbally at the initial meeting and depict that information on a slide to be shown during meeting breaks

2.16.3. Announced verbally and depict the information on a slide at the second meeting

(and any subsequent meetings), as questions may arise, and for the benefit of absentee members at the previous meeting(s)

16.3. Conducting the work and draft review

3.16.1. Convene a first meeting of the identified authors to discuss the manuscript(s) and potential journals for publication

3.16.2. Discuss organization/content of the manuscript, journal for publication, distribute tasks and set a mutually agreed timelines for completion/return of assigned parts

3.16.3. If a manuscript related to the systematic evidence review is also under consideration, it will be coordinated with any other manuscripts related to the guideline.

3.16.4. It is the responsibility of the first and senior author to follow-up with remainders for task completion, draft the document according to journals instructions, and to circulate it for group’s review

3.16.5. Set a deadline date for a response to each draft version from ALL authors

16.4. Finalizing manuscript

4.16.1. Convene another group meeting to discuss the final draft of the manuscript and to discuss all edits/comments made by the EBPWG members

4.16.2. Reconfirm the authorship list and if any changes are needed, based on the actual contributions

16.5. Submission

5.16.1. Obtain final approval of the CPG from EBPWG members

5.16.2. Send the final draft of manuscript to the non-author work group members contributors, for awareness

5.16.3. As soon as feasible, send the final manuscript to the planned peer review journal for peer review and potential publication

5.16.4. Inform of and forward the final submission to the co-authors

16.6. Revision(s)

6.16.1. The first and senior authors should assume the leading roles in the revision process

6.16.2. These could be organized, conducted, and timed in the same manner as steps 3-

16.7. Reference: International Committee of Medical Journal Editors. Defining the Role of Authors and Contributors [Online}. Available at: http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html (Accessed: October 23, 2021).

17. The Guideline and Other Related Tools are Posted on the Office of Quality and

Performance internet and intranet and the DoD internet sites

DoD Internet: https://www.qmo.amedd.army.mil/pguide.htm

VA Internet: http://www.healthquality.va.gov/ https://www.qmo.amedd.army.mil/pguide.htm http://www.healthquality.va.gov/

VA Intranet: http://vaww.oqsv.med.va.gov/functions/mindfulness/cp/clinicalPractic.aspx .

All guidelines placed on the Web will conform to the requirements described in Section 508 of the Rehabilitation Act of 1973, as amended. 29 U.S.C. §798 (see http://www.access-board.gov/sec508/guide/act.htm

18. Guideline Adaptation: The overall objective of adaptation is to take advantage of existing guidelines to enhance the efficient production and use of high-quality adapted guidelines. Cultural and organizational differences can lead to legitimate variations in recommendations, even when the evidence base is the same. However, with a systematic approach to guideline modification adaptations can be used as an alternative to de novo guideline development. Adaptation of an existing guideline will ensure the validity of the resulting recommendations.

18.1. The adaptation process is based on the following core principles:

17.1.1 Respect for the evidence-based principles of guideline development

17.1.2 Reliable and consistent methods to ensure quality of the adapted guideline

17.1.3 Participative approach involving key stakeholders, to foster acceptance and ownership of the adapted guideline

17.1.4 Explicit consideration of context during adaptation to ensure organizational relevance for practice

17.1.5 Transparent reporting to promote confidence in the recommendations of the adapted guideline

17.1.6 Format consistent with VA/DoD guideline development

17.1.7 Accountability to the primary guideline sources

18.2. A panel of at least four members including the VA/DoD CPG work Group Champions, will utilize the AGREE II Instrument (www.agreetrust.org) to assess the quality of the proposed CPG and adaptability for VA/DoD specific population use.

18.3. Following the consensus process the panel, along with a facilitator, may decide the following:

17.3.1 Reject the whole guideline: After reviewing all the assessments, the panel decides to reject the complete guideline. The decision will reflect how the panel weighs the assessment (e.g., poor AGREE scores, guideline is out of date, or the recommendations do not apply to the panels context).

17.3.2 Accept a whole guideline and all its recommendations: After reviewing all the assessments, the panel accepts the guideline as is.

17.3.3 Accept specific recommendations: After reviewing the recommendations from the guideline the panel decides which recommendations to accept and which to reject (e.g. those recommendations needing major modification would be rejected).

17.3.4 Modify specific recommendations: After reviewing the recommendations from the guideline, the panel decides which are acceptable but need to be modified (e.g., new data may be added to the original recommendation, or the wording might be changed to better reflect the panel’s context). (ADAPTE Collaboration, 2009). Care must always be taken when modifying existing guidelines and/or recommendations not to change the recommendations to such an extent that they are no longer in keeping with the evidence upon which they will be based.

18.4. Based on the above decisions, the panel can create an adapted guideline acceptable for VA/DoD specific clinical practice guidelines. Note: All adapted guidelines shall conform to the VA/DoD CPG standard to include algorithmic format. Adapted guidelines follow the same VA/DoD CPG process as identified from step 10 forward.

http://vaww.oqsv.med.va.gov/functions/mindfulness/cp/clinicalPractic.aspx http://www.agreetrust.org/

Guideline Update and Approval Process:

19. Evidence Based Practice Work Group Approves Schedule for Update of Clinical

Practice Guidelines: The immediate update of guidelines will be triggered if any recommendation contained in a guideline is identified as harmful to patients (i.e., pharmaceutical or device recall, etc.) Routine guideline updates will ideally occur every three to five years. The process that will be followed mirrors that of guideline development. It is recognized that there may be areas of significant evidence advancement in between update periods. Guideline champions may bring focused update requests forward to the EBPWG at any time for consideration.

19.1. EBPWG considers request for focused update.

19.2. If approved, then convene a small work group consisting of the champions and 1-2 subject matter experts.

19.3. Focused evidence reviews (typically limited to Medline, Cochrane library).

19.4. Results and recommendations from the focused review will be presented to the

EBPWG for approval.

19.5. Once approved by EBPWG results will be posted to the electronic version of the CPG as an addendum.

19.6. CPG focused update will be posted to ECRI Guidelines Trust.

Appendix A VA/DoD Evidence-Based Clinical Practice Guidelines

Guideline Project Submission Form

Project Name Project Description

Project Champion Last Name First Name Title Service/Organization/Command Address

City State Zip Code Phone Fax E-mail

MAKING A CASE FOR CHANGE – Provide narrative to support guideline development.

Perceived gap in health status:

[Is there new information from the medical literature? What about current outcomes (e.g., prevalent conditions, diagnosis)? Are there clinical areas for improvement suggested by clinicians? Are there benchmarks available that suggest a need to change practice? Are there existing evidence-based guidelines on this subject? What is the impact of this guideline on patient outcomes?]

Perceived gap in patient satisfaction:

[Is there survey information available addressing patient satisfaction that indicates an opportunity for improvement? Are there benchmarks available that suggest a need to change practice?]

Perceived gap in provider satisfaction:

[Are there surveys or suggestions addressing provider satisfaction that indicate an opportunity for improvement? Are there benchmarks available that suggest a need to change practice?]

Perceived gap in cost/utilization:

[Are there areas of care with high utilization? Is there significant variation or an opportunity for improvement in utilization patterns (e.g. drug utilization, lab utilization, referral rates, or local variation)?

Are there benchmarks available that suggest a need to change practice? Rational and supporting evidence of relevance/importance of topic to the VA and/or DoD population?]

Perceived organizational issues:

[Are there political or organizational reasons why a change in practice might be warranted? Are there benchmarks available that suggest a need to change practice? Is the implementation of this project feasible? Is there evidence available to support evidence-based guideline development?]

Appendix B

DISCLOSURE STATEMENT

The VA/DoD Evidence-Based Clinical Practice Guideline (EBCPG) Workgroup members (voting and non-voting), as well as developers, reviewers, and others involved in the clinical practice guideline (CPG) process, are asked to sign a disclosure statement annually to detail involvement, of any kind, with manufacturers that may benefit from the inclusion or recommendation of their products within a VA/DoD CPG. This includes, but is not limited to, pharmaceuticals, diagnostic products/equipment, and monitoring supplies.

Please list the various projects you are involved with over the past two years in regard to the following areas:

1 Do you participate in research funded by pharmaceutical manufacturers?

YES NO

If YES, please list the company(ies), product(s), or disease state(s):

2 Do you serve on a Speakers Bureau? YES NO If YES, please list the company(ies), product(s), or disease state(s):

3 Do you receive remuneration for activities (such as board member or member of an advisory council) for any company or product that is coming to market?

YES NO

If YES, please list the company(ies), product(s), or disease state(s):

4 Do you have financial holdings (to include, but not limited to, company stock, bonds, or other shares, etc.) of said companies and/or products?

YES NO

If YES, please list the company(ies), product(s), or fund(s):

I affirm, to the best of my knowledge, the above statement is inclusive of my functions with said product(s), company(ies), and disease state(s). I acknowledge that if my involvement changes, I am to contact the respective VA or DoD EBCPG Workgroup co-chair and update this disclosure form immediately. I will recuse myself from voting on guideline selection, development, adaptation, or tool kit development matters concerning issues where a conflict of interest (or appearance of a conflict of interest) may exist.

SIGNATURE __________________________________ DATE________________

Printed Name: ___________________________________________________________

Appendix C-External Reviewer Form VA/DoD CLINICAL PRACTICE GUIDELINES

(Guideline Rating Tool 4-1-2010)

Reviewer_____________________________________________Date_____________________________

Title of the Guideline______________________________________________________________________________

Do you have any conflict of interest or potential conflict of interest in reviewing this guideline?

No Yes (Specify if yes.)

SCOPE AND PURPOSE

Strongly Agree

Agree Disagree Strongly Disagree

1. Targeted patient population is specified.

2. Intended users of guideline are specified.

3. Guideline addresses a documented gap in performance, safety, or quality.

B. COMMENTS

PRESENTATION

Agree Disagree Strongly

4. The guideline is clearly written.

5. Guideline defines unfamiliar terms and those that are critical to applying the recommendations.

6. The recommendations are specific and unambiguous.

PRESENTATION

Agree Disagree Strongly

7. The algorithm is logically complete and internally consistent.

C. COMMENTS

SYSTEMATIC REVIEW METHODS

Agree Disagree Strongly Disagree

8. Systematic methods were used to search for evidence.

The criteria for selecting the evidence are clearly described.

10. The quality of the studies was explicitly assessed.

SYSTEMATIC REVIEW METHODS YES NO NOT

SURE

11. Eligible studies were summarized in evidence tables.

COMMENTS

INTERGRATING EVIDENCE INTO

RECOMMENDATIONS

Strongly Agree

Agree Disagree Strongly Disagree

12. The methods used to formulate the recommendations are clearly described?

13. There is an explicit link between the recommendations and the supporting evidence.

14. Was sufficient information provided to understand the rationale behind key or controversial recommendations?

COMMENTS (on D. Integrating the Evidence)

BENEFITS, HARMS AND OUTCOMES

Agree Disagree Strongly Disagree

15. All important benefits and harms of recommended treatments or procedures are specified.

16. Benefits and harms of recommended treatments and procedures are quantified.

17. The effect of the recommended interventions on health care costs is quantified.

AUTHORSHIP

Agree Disagree Strongly Disagree

18. The guideline clearly notes author(s).

19. The guideline clearly notes the authors’ conflicts of interest.

20. All relevant disciplines are represented including primary care?

G. TESTING AND REVIEW

Agree Disagree Strongly Disagree

21. The guideline has been evaluated by field testing.

22. An expiration date or procedure for updating the guideline is specified.

FLEXIBILITY

Agree Disagree Strongly Disagree

23. The guideline clearly indicates the intended flexibility of the recommendation(s).

24. The role of patient preferences is discussed.

25. The guideline addresses special patient populations when appropriate.

FEASIBILITY OF IMPLEMENTING THE

GUIDELINE

Agree Disagree Strongly

26. The guideline recommendations are feasible to implement in all intended care settings (consider organizational characteristics, implementation costs, opportunity costs.)

OVERALL ASSESSMENT

27. Describe the predominant method(s) used to develop this guideline:

Evidence-based (key recommendations are supported by fair or good evidence with explicit estimation of benefits and harms)

Evidence-based (all recommendations are supported by fair or good evidence)

Structured consensus with systematic literature reviews

Global subjective judgment or consensus panel

Other (describe)

28. Would you recommend these guidelines for use in practice?

STRONGLY RECOMMEND

RECOMMEND

WOULD NOT RECOMMEND

UNSURE

COMMENT: (What is this guideline’s specific strengths? What is this guideline’s specific weaknesses?

Use additional space as necessary.)

Additional Review Comments: How can/might this guideline be improved?

Name of Reviewer ______________________________________________________________________

Address ______________________________________________________________________________

Phone ________________________________________________________________________________

E-Mail _______________________________________________________________________________

Table 1: Level of Evidence (LE) I At least one properly done RCT

II-1 Well-designed controlled trial without randomization

II-2 Well-designed cohort or case-control analytic study, preferably from more than one source

II-3 Multiple time series evidence with/without intervention, dramatic results of uncontrolled experiment

III Opinion of respected authorities, descriptive studies, case reports, and expert committees

Table 2: Overall Quality [QE] Good High grade evidence (I or II-1) directly linked to health outcome

Fair High grade evidence (I or II-1) linked to intermediate outcome;

or Moderate grade evidence (II-2 or II-3) directly linked to health outcome

Poor Level III evidence or no linkage of evidence to health outcome

USPSTF Methods and Process.

http://www.uspreventiveservicestaskforce.org/methods.htm, August 2012.

Evidence-based Practice Centers Overview. November 2012. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/clinic/epc/ http://www.ahrq.gov/clinic/epc/

Appendix E: GRADE Evaluation of Recommendations

Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y. Alonso-Coello, P.

Schünemann, H. J. & the GRADE Working Group. (2008). GRADE: going from evidence to recommendations. BMJ, 336, 1049-1051.

Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y. Alonso-Coello, P.

Schünemann, H. J. & the GRADE Working Group. (2008). GRADE: going from evidence to recommendations. BMJ, 336, 1049-1051.

Quality of evidence and definitions

High quality— Further research is very unlikely to change our confidence in the estimate of effect

Moderate quality— Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate Low quality— Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate Very low quality— Any estimate of effect is very uncertain”

From: Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y. Alonso-Coello, P.

Schünemann, H. J. & the GRADE Working Group. (2008). GRADE; An emerging concensus on rating quality of evidence and strength of recommendations. BMJ, 336, 924-926.

Criteria for assigning grade of evidence: Type of evidence

“Randomized Controlled Trial = high Observational study = low Any other evidence = very low Decrease grade if:

• Serious (− 1) or very serious (− 2) limitation to study quality

• Important inconsistency (− 1)

• Some (− 1) or major (− 2) uncertainty about directness

• Imprecise or sparse data (− 1)

• High probability of reporting bias (− 1) Increase grade if:

• Strong evidence of association—significant relative risk of > 2 (< 0.5) based on consistent evidence from two or more observational studies, with no plausible confounders (+1)46

• Very strong evidence of association—significant relative risk of > 5 (< 0.2) based on direct evidence with no major threats to validity (+2)

0. Evidence of a dose response gradient (+1)

• All plausible confounders would have reduced the effect (+1)”

Grade Working Group. (2004). Grading the quality of evidence and strength of recommendations.

BMJ, 328.

Imprecise or sparse data

“There is not an empirical basis for defining imprecise or sparse data. Two possible definitions are:

• Data are sparse if the results include just a few events or observations and they are uninformative

• Data are imprecise if the confidence intervals are sufficiently wide that an estimate is consistent with either important harms or important benefits These different definitions can result in different judgments. Although it may not be possible to reconcile these differences, we offer the following guidance when considering whether to downgrade the quality of evidence due to imprecise or sparse data:

• The threshold for considering data imprecise or sparse should be lower when there is only one study. A single study with a small sample size (or few events) yielding wide confidence intervals spanning both the potential for harm and benefit should be considered as imprecise or sparse data

• Confidence intervals that are sufficiently wide that, irrespective of other outcomes, the estimate is consistent with conflicting recommendations should be considered as imprecise or sparse data”

Grade Working Group. (2004). Grading the quality of evidence and strength of recommendations.

BMJ, 328.

A computer program exists to assist in developing GRADE recommendations:

Brozek, J., Oxman, A., Schünemann, H. (2008). GRADEpro. [Computer program].

Version 3.2 for Windows. http://www.ims.cochrane.org/revman/other-resources/gradepro .

http://www.ims.cochrane.org/revman/other-resources/gradepro http://www.ims.cochrane.org/revman/other-resources/gradepro

Appendix F - Journal of Clinical Epidemiology 66 (2013) 726 - 735

GRADE guidelines: 15. Going from evidence to recommendation – determinants of a recommendation's direction and strength Jeffrey C. Andrewsa,*, Holger J. Sch€unemannb,c, Andrew D. Oxmand, Kevin Pottiee, Joerg J. Meerpohlf,g, Pablo Alonso Coelloh,i, David Rindj, Victor M. Montorik, Juan Pablo Britok, Susan Norrisl, Mahmoud Elbarbarym, Piet Postn, Mona Nassero, Vijay Shuklap, Roman Jaeschkec, Jan Brozekb, Ben Djulbegovicq,r, Gordon Guyattb,c aVanderbilt Evidence-based Practice Center, Vanderbilt University, #27166-719 Thompson Lane, Nashville, TN 37204-3195, USA bDepartment of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario L8N 3Z5, Canada cDepartment of Medicine, McMaster

University, Hamilton, Ontario L8N 3Z5, Canada dNorwegian Knowledge Centre for the Health Services, PO Box 7004, St. Olavs plass, Oslo 0130, Norway eDepartment of Family Medicine, University of Ottawa, Ottawa, Canada fGerman Cochrane Center, Institute of Medical Biometry and Medical Informatics, University Medical Center Freiburg, Berliner Allee 29, 79110

Freiburg, Germany gDivision of Pediatric Hematology and Oncology, Center for Pediatrics and Adolescent Medicine, University Medical Center Freiburg, Mathildenstrasse 1, 79106 Freiburg, Germany hIberoamerican Cochrane Center, CIBER de Epidemiolog'ıa y Salud P'ublica, IIB, Sant Pau, Barcelona 08041, Spain iEpidemiology and Public Health CIBER (CIBERESP), Hospital de la Sant Pau, Creu i Sant Pau, Barcelona 08041, Spain jHarvard Medical School, Beth Israel Deaconess Medical Center, Healthcare Associates-E/Shapiro 6, 330 Brookline Aveen, Boston, MA 02215, USA kMayo Clinic, 200 1st SW St, Rochester, MN 55905, USA lDepartment of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, OR 97239-3098, USA mKing Saud University for Health Sciences, Riyadh, Saudi Arabia nPost Voor Zorg, Delft, The Netherlands oPeninsula College of Medicine and Dentistry, Universities of Exeter and Plymouth, The John Bull Building, Tamar Science Park, Plymouth, PL68BU, UK pCanadian Agency for Drugs and technologies in Health (CADTH), 600-865 Carling Avenue, Ottawa, Ontario K1S 5S8, Canada qDivision and

Center for Evidence-Based Medicine and Health Outcomes Research, University of South Florida, Tampa, FL, USA rH. Lee Moffitt Cancer Center & Research Institute, Tampa, FL, USA

Accepted 3 February 2013; Published online 6 April 2013

Abstract

In the GRADE approach, the strength of a recommendation reflects the extent to which we can be confident that the composite desirable effects of a management strategy outweigh the composite undesirable effects.

This article addresses GRADE’s approach to determining the direction and strength of a recommendation. The GRADE describes the balance of desirable and undesirable outcomes of interest among alternative management strategies depending on four domains, namely estimates of effect for desirable and undesirable outcomes of interest, confidence in the estimates of effect, estimates of values and preferences, and resource use. Ultimately, guideline panels must use judgment in integrating these factors to make a strong or weak recommendation for or against an intervention. © 2013 Elsevier Inc. All rights reserved.

Keywords: GRADE; Quality of evidence; Strength of evidence; Guideline development; Recommendation; Evidence

The GRADE system has been developed by the GRADE Working Group.

The named authors drafted and revised this article. A complete list of contributors to this series can be found on the Journal of Clinical Epi-demiology web site.

* Corresponding author. Tel.: (615) 343-5700.

E-mail address: jeff.andrews@vanderbilt.edu (J.C. Andrews).

0895-4356/$ - see front matter © 2013 Elsevier Inc. All rights reserved.

http://dx.doi.org/10.1016/j.jclinepi.2013.02.003

1. Introduction

In prior articles in this series devoted to the GRADE approach to systematic reviews and practice guidelines, we have dealt with the process before developing recommendations, namely framing the question and choosing critical and important outcomes [1], rating the confidence in effect estimates for each outcome [2e8], mailto:jeff.andrews@vanderbilt.edu http://dx.doi.org/10.1016/j.jclinepi.2013.02.003 dealing with resource

J.C. Andrews et al. / Journal of Clinical Epidemiology 66 (2013) 726-735 727 use [9], rating the confidence in effect estimates across out-comes [10], and creating an evidence profile and a Summary of Findings table [11-13]. The immediately previous article described GRADE’s approach to classifying the strength and direction of recommendations and discussed the implications of strong and weak recommendations, and the options for presentation and wording [14]. The present article presents GRADE’s approach to moving from evidence to recommendations. As we did in the previous article, we will refer to guideline developers as ‘‘the panel.’’

1.1. Globalizing evidence and localizing decisions

The pithy summary by Eisenberg [15] on the relation- ship between evidence and recommendations, ‘‘globalize the evidence, localize the decisions,’’ provides fundamental guidance for those working to produce evidence-based recommendations [15]. Summaries of evidence regarding alternative management strategies from the medical literature should ideally be very similar, no matter the site of the application of the recommendation.

Rating of confidence in estimates of effect (quality of evidence) may, however, differ for a variety of reasons.

First, desirable and undesirable outcomes may be valued differently, leading to different thresholds of acceptability.

This could lead to different judgments regarding imprecision, as we have highlighted in the article in this series dealing with imprecision [5].

Second, differences in values and preferences could lead to differences in the overall balance of desirable and undesirable outcomes and the rating of confidence in estimates: an outcome judged as critical by one panel (and thus included in the rating of overall confidence in estimates) may be judged important but not critical by another (and thus not included in the overall rating).

Finally, ratings of confidence may also differ as a result of uncertainties in the risk profile of untreated populations (baseline risk). We may be very confident of baseline risk in one setting but not at all confident in another. This could lead to rating down confidence in estimates for indirectness. Continued rapid uptake of GRADE by organizations that produce systematic summaries of evidence will greatly facilitate the production of transparent evidence summaries. If or- ganizations work together to produce summaries, there will be an enormous gain in efficiency [16]-even if, in the end, judgments about confidence in estimates will differ across settings, for reasons described in the preceding paragraphs. We now turn to a systematic presentation of the determinants of direction and strength of recommendations.

2. Determinants of direction and strength of recommendations

GRADE has identified six determinants of the direction and strength of recommendations, namely the magnitude of estimates of effect of the interventions on important out-comes, confidence in those estimates, estimates of typical values and preferences, confidence in those estimates, var-iability of values and preferences, and resource use. In the presentation here, we will present these six determinants in four domains. We package magnitude of effect and typical values and preferences together with the label balance of desirable and undesirable consequences or ‘‘trade-offs.’’ We also include uncertainty regarding typical values, and variability in values, in a single domain (Table 1).

Alternative groupings may work better, depending on the circumstances. We believe that the approach we present here is best for presenting the rationale for the recommendations to the guideline consumer audience. In developing recommendations, panels may want to keep all six determinants separate or group the three values and preferences determinants together.

Ultimately, guideline panels must integrate these six determinants to make a strong or weak recommendation for or against an intervention. Table 2 illustrates how the elements of the GRADE framework for moving from evidence to recommendations can be applied in making strong and weak recommendations, and Table 3 provides an example of the application in the management of chronic obstructive pulmonary disease.

2.1. Trade-offs between desirable and undesirable

consequences of alternative management strategies

When we consider the balance between desirable and undesirable outcomes (“trade-offs”), we are considering two domains. The first is our best estimates of the magnitude of desirable effects and the undesirable effects. If a guideline panel has adhered to the GRADE process, they will find the best estimates of effect in the evidence profiles that they have prepared or accessed.

The second element that determines the balance among desirable and undesirable outcomes is the typical values that patients - or a population - apply to those outcomes.

This can be otherwise conceptualized as the relative preferences for those outcomes-and thus the term we generally use, values and preferences (Box 1).

Ideally, to inform estimates of typical patient values and preferences, guideline panels will conduct or identify systematic reviews of relevant studies of patient values and preferences [18]. Given the paucity of empirical examinations of patients’ values and preferences, however, well- resourced guideline panels will usually complement such studies with consultation with individual patients and patients’ groups. The panel should discuss whose values these people represent, namely representative patients, a defined subset of patients, or representatives of the general population.

For example, the Canadian Collaboration for Immigrant and Refugees Health (CCIRH) guidelines sought to advance understanding of immigrant patient perspectives in

728 J.C. Andrews et al. / Journal of Clinical Epidemiology 66 (2013) 726-735

Table 1. Domains that contribute to the strength of a recommendation

Domains that contribute to the strength of a recommendation Comment

Balance between desirable and undesirable outcomes (estimated effects), with consideration of values and preferences (estimated typical) (trade-offs)

Confidence in the magnitude of estimates of effect of the interventions on important outcomes (overall quality of evidence for outcomes)

The larger the differences between the desirable and undesirable consequences, the more likely a strong recommendation is warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely a weak recommendation is warranted The higher the quality of evidence, the more likely a strong recommendation is warranted

Confidence in values and preferences and variability The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely a weak recommendation is warranted Resource use The higher the costs of an intervention (the more resources consumed), the less likely a strong recommendation is warranted two ways, namely they searched and synthesized evidence for immigrant perspectives in relation to each health condition, and worked closely with a community-based organization representing 18 ethnic groups to inform perceptions of immigrant patient perspectives [19]. Less well-resourced panels, without systematic reviews of values and preferences or consultation with patients and patient groups, must rely on unsystematic reviews of the available literature and their clinical experience of interactions with patients.