QA_TeamSTEPPS_FINAL_06-27-2023.pdf

Government Response to Questions RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 1 of 11

Government Response to Questions:

RFQ Number: AHRQ-23-10010;

Titled: TeamSTEPPS Implementation and Assessment Please see RFQ Amendment No. 1 for amendments made in response to industry questions.

1. Question: (Page 1, Quote Due Date) Question: Would AHRQ approve a 14-day extension for the quote deadline to 07/28/2023?

Government Response:

No solicitation amendment necessary.

2. Question: (Page 3, Purchase Order Pricing) Question: Purchase Order Pricing – Under, “Note.” Would AHRQ please confirm that travel for contractor staff is approved under the contract? If contractor travel is not approved, does AHRQ envision virtual-only training and implementation?

Government Response:

No solicitation amendment necessary. See 31 USC § 1345 for appropriations restrictions. Also see response to Q9.

3. Question: (Page 3, Purchase Order Pricing) Question: Purchase Order Pricing – Under, “Firm Fixed Price.” The CLINs of page 3 suggests that the contractor is paid for completing 55 trainings and assessment using the updated TeamSTEPPS training curriculum and for 60 trainings and assessment using the new TeamSTEPPS® for Diagnosis Improvement Course; however, on page 29, it states that “offerors may propose a milestone-based payment schedule.” Are offerors paid by the delivery and assessment of training as specified in the CLINs or may the offeror propose a payment schedule as indicated on page 29?

Government Response:

No solicitation amendment necessary. CLIN 00001 and 0002 on page 3, represent the maximum amount that the government would pay per CLIN (Price x Qty). However, the offeror is to propose how those payments are made in the deliverable/milestone-based payment schedule.

4. Question: (Page 3, Purchase Order Pricing) Question: “…The Contractor will be paid for the quantity of sites that complete the implementation/training…” How does AHRQ define a site that has completed TeamSTEPPS implementation for a health care organization?

See the solicitation amendment.

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 2 of 11

5. Question: (Page 3, Training/Pricing) Question: This section states that "the Contractor will be paid for the quantity of sites that complete the implementation/training. The three types of targeted entities are systems, schools, and private institutions, all of which may have multiple sites. Please clarify whether providing a training to a single system that included persons from 55 hospitals or primary care practices would count as 55 sites or as a single site.

6. Question: (Price 3, Training/Pricing) Question: This section states that "the Contractor will be paid for the quantity of sites that complete the implementation/training." If a healthcare organization is recruited for both the updated TeamSTEPPS training and the Diagnostic Improvement training, will that individual organization be counted towards both the updated TeamSTEPPS goal of 55 organizations and the new Diagnostic Improvement goal of 60 organizations?

7. Question: (Page 3, Training/Pricing) Question: This section states that "the Contractor will be paid for the quantity of sites that complete the implementation/training. A training might last as little as a single hour while implementation of all aspects of TS or all parts of the TS for diagnostic improvement might require months. Please clarify how AHRQ will determine whether an implementation/training is "complete" for the purpose of approving the fixed price payment that is linked to it.

See response to Q3.

8. Question: (Page 3, Training/Pricing) Question: Would AHRQ please provide an estimated number of hours this contract is expected to require. Without an estimate some bids might be far more modest than AHRQ expects (e.g., a single one hour virtual training that includes attendees from 55 health systems, schools and private institutions for the new TS curriculum and another training for 60 attendees for the TS for Diagnostic Improvement. Alternatively, a bidder could propose 55 2-day, onsite trainings for the new TS curriculum and 60 additional TS trainings for systems, schools and private institutions for TS for diagnostic improvement. Without a reasonable estimate from AHRQ for the overall level of effort, AHRQ may have bids so different from each other that comparisons between them will be difficult.

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 3 of 11

9. Question: (Page 3, Training/Pricing) Question: This section states that "the Contractor will be paid for the quantity of sites that complete the implementation/training." Is there an expectation for the percentage of training that shall be done in-person, as opposed to virtual training sessions?

Government Response:

No solicitation amendment necessary. This is a statement of objectives; it is up to the offeror to propose how to conduct the training to meet the Government’s objective.

10. Question: (Page 4, Training/Objectives) Question: Will recruitment be ongoing throughout the period of the contract, or is there a specific date in which all recruitment must be completed?

11. Question: (Page 4, Training/Objectives) Question: Would AHRQ consider alternative technical approaches proposed by the contractor, provided that the approach addresses the degree to which the updated TeamSTEPPS program and the new TeamSTEPPS® for Diagnosis Improvement Course enhances users’ experience, improves team effectiveness, streamlines team communication, accelerates implementation, and increases healthcare professionals’ commitment to interdisciplinary teamwork?

12. Question: (Page 4, Training/Objectives) Question: If AHRQ considers virtual training an acceptable format, is asynchronous, or "on-demand" virtual training acceptable for some or all TeamSTEPPS training?

13. Question: (Page 4, Training/Objectives) Question: AHRQ uses the term "healthcare organization" and then further specifies "healthcare system and affiliated primary care medical offices". Can AHRQ please define the elements of a "health system"? Do "affiliated primary care offices" need to be "owned" by the health system? For rural and other practices, in particular, affiliated practices may not have common ownership.

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 4 of 11

14. Question: (Page 4, Training/Objectives) Question: Is there any minimum number of persons AHRQ expects to receive the training from each organization? System-wide implementations may require training hundreds of staff, but the process might begin by sending a single person (or several) to an initial training. What expectations does AHRQ have for how many persons from each organization must be trained?

15. Question: (Page 4, Training/Objectives) Question: AHRQ states, “…implement the updated TeamSTEPPS training curriculum in at least 55 healthcare organizations and the new TeamSTEPPS for Diagnostic Improvement Course in at least 60 healthcare organizations.” On page 5 AHRQ states the “TeamSTEPPS training” should be implemented in 75 healthcare systems and primary care medical offices, 25 medical and nursing schools, and 15 private sector institutions. Can AHRQ clarify exactly where each TeamSTEPPS curriculum is to be tested and at how many sites? For example, page 5 implies 25 medical schools and 25 nursing schools for a possible total of 50.

16. Question: (Page 4, Training/Objectives) Question: "The primary objective of this project is to implement the updated TeamSTEPPS training curriculum in at least 55 healthcare organizations and the new TeamSTEPPS® for Diagnosis Improvement Course in at least 60 healthcare organizations.” Could the healthcare organizations be Government or Military Health?

17. Question: (Page 4, Training/Objectives) Question: If the same site implemented the updated TeamSTEPPS training curriculum and the new TeamSTEPPS® for Diagnosis Improvement Course, would that count for both the updated TeamSTEPPS training curriculum and the new TeamSTEPPS® for Diagnosis Improvement Course towards the total number of healthcare organization requirement?

See response to Q6.

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 5 of 11

18. Question: (Page 4, Training/Objectives) Question: Is the updated TeamSTEPPS training curriculum a pre-requisite for the new TeamSTEPPS® for Diagnosis Improvement Course?

No solicitation amendment necessary. These are 2 separate components.

19. Question: (Page 5, Training/Objectives) Question: A stated objective is to "test the short and long term impact of the TS program”. Please clarify what duration of time following the training would be needed to test its long term impact.

20. Question: (Page 5, Training/Objectives) Question: "This project shall implement TeamSTEPPS training and conduct testing across various healthcare organizations and institutions in the United States including: at least 75 health care systems and affiliated primary care medical offices, at least 25 medical and nursing schools, at least 15 private sector institutions that offer TeamSTEPPS program.” How does AHRQ define a health care system? Can AHRQ provide a list of health care systems?

See the answer to Q13.

21. Question: (Page 5, Training/Objectives) Question: Given that a primary objective of this contract is to assess the effect of the updated TeamSTEPPS program and the new TeamSTEPPS® for Diagnosis Improvement Course of participant knowledge, skills, and attitudes as well as organizational outcomes, what is AHRQ’s rationale for the number of sites to which training must be delivered?

22. Question: (Page 5, Training/Objectives) Question: AHRQ is proposing a fixed price contract that is linked to the number of sites that are trained. Please clarify whether the assessment for each site must also be complete before payment will be made. If payment is due following the training then what (if any) payment will be linked to the assessment? Alternatively, if payment does not occur until after a short and long term assessment of impact is complete, then the successful bidder may not receive payment for some services for months after their training costs have been incurred. For a small business, this would be very problematic.

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 6 of 11

23. Question: (Page 5, Training/Objectives) Question: Does training a health care system and affiliated primary care medical offices require training personnel from all system location and affiliated primary care medical offices, one of each, or something else. Additional clarity would allow bidders to be more responsive to AHRQ's needs.

Government Response:

No solicitation amendment necessary. This is a statement of objectives; it is up to the offeror to determine and propose the best method to meet the Government’s objectives.

24. Question: (Page 5, Training/Objectives) Question: What does it mean to train a medical or nursing school? Is this training a single class a single time? Is it training the teachers that use some aspects of TS in a class offered in their school, or is it something else. Clarity will enable bidders to submit more responsive proposals.

25. Question: (Page 5, Training/Objectives) Question: For the purpose of this procurement, can AHRQ define a "private sector institution"?

Many health systems and schools are in the private sector. Professional societies, trade associations, and quality improvement organizations may or may not be in the private sector. So how does this category differ from the first two?

26. Question: (Page 5, Training/Objectives) Question: Do the private sector institutions need to offer a TeamSTEPPS program on a continuous basis already or can these institutions be considering offering one in the future or can they incorporate aspects of TeamSTEPPS into improvement initiatives when that is appropriate?

27. Question: (Page 3, 4, and 5, Training/Objectives) Question: The RFQ uses many terms to describe the “healthcare organizations” for training/implementation of TeamSTEPPS courses. Further page 3 - #1 implies that the contractor is paid by the quantity of sites. Can AHRQ clearly define what is meant by “site?” For example, is it a health care system, a clinic, a hospital, or a unit within a hospital?

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 7 of 11

28. Question: (Page 6, Training/Period of Performance/Payment) Question: Please explain how the four specified tasks will be linked to payment. If the only thing that AHRQ is paying for are "the quantity of sites that complete the implementation/training" then how will the contractor be reimbursed for these activities?

There are no tasks. The Government provided a Statement of Objectives. See response to Q3.

29. Question: (Page 6, Training/Period of Performance/Payment) Question: What specific information will AHRQ require to document that each of the 115 organizations have been trained and their training has been assessed? If there is a single assessment report due a month before the end of the contract period, then does this mean that AHRQ will not approve any payments until after this report is received?

30. Question: (Page 20, Limitation on Subcontracting) Question: For a "similarly situated entity" clause (b) (2) states "Is considered small for the size standard under the North American Industry

Classification System (NAICS) code the prime contractor assigned to the subcontract." Does this mean a non-profit entity, which does not meet small business standard simply by virtue of its non-profit status, but otherwise meets the size standards for the NAICS code applicable to the proposal qualifies as a "similarly situated entity"?

Government Response:

No solicitation amendment necessary. A non-profit organization does not meet the definition of a small business (even if they are within the NAICS size standard). A business concern must be organized for profit to meet the definition of a small business. Please see 13 CFR 121.105(a)(1).

31. Question: (Page 25, Technical Response Requirements) Question: AHRQ requests that the offeror “…demonstrate a thorough understanding of TeamSTEPPS methods and practices as described in the updated TeamSTEPPS program…” Would AHRQ share the updated “consolidated” curriculum and assessment materials, which are not publicly available?

The TeamSTEPPS updated curriculum will be released on June 29, 2023.

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 8 of 11

32. Question: (Page 29, Proposed Payment Schedule) Question: If AHRQ intends to link fixed payments to the number of organizations trained and assessed, how would a milestone-based payment schedule be possible? Will AHRQ confirm that it intends to pay the same amount for each of the 115 planned trainings, regardless of system size, number of participants, duration of training, and whether the training in onsite or virtual? We are concerned that this approach would encourage bidders to submit low-ball bids based on a mix of training lengths and types, followed by providing the cheapest forms of training possible to maximize contract profits.

33. Question: (Page 29, Proposed Payment Schedule) Question: Suppose a contractor provides a training to an organization. The training is ineffective, evaluation feedback is neutral to negative, and the organization makes few or no changes as a result. Would documentation that the training occurred and was assessed be sufficient to meet the requirements for payment even if the training did not achieve its stated objectives? Alternatively, if the training is excellent, but the organization is impacted by an environmental change (e.g., significant leadership change, accreditation/survey jeopardy) and never can implement TeamSTEPPS, would this meet the requirements for payment? Please specify exactly what standard must be met and what documentation must be provided in order to justify reimbursement.

Government Response:

Payment tied to the contractor delivering the training and collecting assessment is not contingent on favorable assessment or likelihood of future implementation.

34. Question: (Page 3, Training/Pricing) Question: Please confirm unit of quantity is site.?

35. Question: (Page 3, Training/Pricing) Question: You have stipulated that contractors provide a milestone-based payment schedule, and that “Contractor will be paid for the quantity of sites that complete the implementation/training.”

Given the long lead time before sites can be identified, much less complete training, and the wait for OMB clearance, we assume that interim milestones such as “site sampling plan” may be proposed?

See response to question #3.

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 9 of 11

36. Question: (Page 5 Training/Objectives) Question: Contractor will implement the updated TeamSTEPPS training curriculum at least 55 healthcare organizations and the new TeamSTEPPS® for Diagnosis Improvement Course in at least 60 healthcare organizations. Implementation will be at least 75 health care systems and affiliated primary care medical offices, • at least 25 medical and nursing schools, • at least 15 private sector institutions that offer TeamSTEPPS program. Is there a requirement to implement any specific number of the two individual courses at these three different kinds of sites?

37. Question: (Page 5 Training/Objectives) Question: It appears that contractors are 100 percent responsible for recruiting and enlisting sites to participate in the program and contractors are to distribute those costs, including necessary oversampling, over the “per site” fee. Please confirm.

38. Question: (Page 5 Training/Objectives) Question: Will the government provide lists of approved sites from which to recruit participants or is that up to the contractor?

39. Question: (Page 5 Training/Objectives) Question: What assistance will the government provide to encourage sites to participate?

40. Question: (Page 5 Training/Objectives) Question: What is your budget for this project?

See response to Q8.

41. Question: (Page 5 Training/Objectives) Question: Does the government have any specific effectiveness metrics in mind or are contractors to construct and propose those metrics?

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 10 of 11

42. Question: (Page 5 Training/Objectives) Question: Is the effectiveness assessment required to have any specific degree of statistical significance?

43. Question: (Page 7, Section 4.3 Key Personnel) Question: Are contractors to designate any staff as key personnel in their proposals?

44. Question: (Page 24) Question: Organizational Experience is to be included in the 25-page Technical volume, correct?

45. Question: (Page 24) Question: Could you please clarify what you want us to include in “Appendix 2: HHS 508 product accessibility template”?

46. Question: (Page 24) Question: The TeamSTEPPS curriculum is an online program. Will our implementation be performed remotely/online, or will we be performing implementation onsite?

See the response to Q9.

47. Question: (Page 24) Question: It is expected that OMB clearance will be required for this project. Is it possible to fast-track this process due to the short contract performance period?

RFQ Number: AHRQ-23-10010 (TeamSTEPPS Implementation and Assessment) Page 11 of 11

48. Question: (Page 24) Question: Could you please provide examples of “private sector” institutions that offer the TeamSTEPPS program?

49. Question: (Page 24) Question: Is there a budget for this project?

See response to Q8.

50. Question: (Page 24) Question: Are the trainings listed in the “Non-Severable Commercial Items” table the same? Should offerors input an average/blend?

51. Question: (Page 24) Question: Are there suggested or preferred tasks to use in the staff loading chart? Is there a min or max number of tasks? Should offerors create and input task breakouts as they develop them?

Government Response:

Yes. It is up to the offeror to propose tasks based on their individual approach to meeting the Government’s objectives.

52. Question: (Page 24) Question: Regarding implementation of TeamSTEPPS, is the expectation to implement the Fundamentals Course (Modules 1-7) or the Master Trainer/Coach Course (Modules 1-12) or both?

See response to Q31.