Attachment S1 - STO1 - Performance Work Statement_revised_4-18.docx

SAMPLE TASK ORDER 1 (STO1)

PERFORMANCE WORK STATEMENT (PWS)

ATTACHMENT S1: STO1 - PWS

DEPARTMENT OF VETERANS AFFAIRS

Office of Health Informatics (OHI) Clinical Informatics and Data Management Office (CIDMO) Human Systems Integration (HSI)

Human Systems Services Electronic Health Record Modernization Support

Contents

BACKGROUND

The Office of Health Informatics (OHI) is integral to Veterans Health Administration (VHA) programs that provide patient-centered health care to Veterans, offering advanced and secure enterprise data systems, sophisticated analytic and measurement solutions, decision support, business intelligence and web communications. OHI facilitates evidence-based decisions for individual Veterans and their families, patient populations, clinicians, and those managing health care delivery systems. The Human Systems Integration (HSI) directorate exists within the Department of Veterans Affairs (VA), VHA, OHI, Clinical Informatics and Data Management Office (CIDMO). The Human Systems Integration (HSI) Division seeks to improve usability, safety, and supportability of VA systems through tailored analysis and timely insight into design tradeoffs related to human performance, staffing, training and systems safety, with the goal of maximizing system performance and minimizing lifecycle costs. HSI applies agile best practices to provide value incrementally to sponsors and business owners and works collaboratively within multi-disciplinary teams.

HSI employs a holistic systems view for understanding the human component of healthcare systems, and offers a framework for understanding how the configuration of human healthcare elements shape healthcare delivery. The division helps make design tradeoffs related to humans explicit, so decisions can be made intentionally.

HSI encourages and offers human-centered design (HCD) services (including task analysis, evaluation of context of use, workflow assessments, and user interface design and usability assessments) for a variety of VA-developed and acquired products and services. HSI requires multi-disciplinary engineering, management, and technical support for numerous projects to carry out their mission.

As a key domain of HSI, the HSI division accepts "human factors" as the scientific discipline concerned with understanding the interactions among humans and other elements of a system (Work system design for patient safety: the Systems Engineering Initiative for Patient Safety (SEIPS) model), and human factors engineering is the profession that applies theory, principles, data and methods to design in order to optimize human well-being and overall system performance (International Ergonomics Association).

The VHA requires a rigorous human centered, total systems approach in operational settings to ensure that HIT (acquired or developed) and corresponding elements of the health care socio-technical system (STS) are jointly optimized for performance and safer operations while providing a satisfying user experience. The HSI office optimizes both employee (e.g., clinicians, administrators) and Veteran/Caregiver user experience with VHA systems, processes, and products, including HIT, medical devices and consumer products (e.g., tablets, smartphones, wearables) via an iterative HCD approach offered in the International Organization for Standardization (ISO) 9241: plan the process, understand context of use, specify requirements, design, and test user interfaces to ensure they improve user effectiveness, minimize mental workload, support decision making, maximize clinical workflow efficiency, reduce system error and latent design issues, improve patient safety, and enhance quality of care. HSI collaborates with other administration and departmental resources to meet this enterprise objective of adoption of HCD, such as the Office of Healthcare Transformation (OHT), Office of Healthcare Innovation and Learning (OHIL), the Veterans Experience Office (VEO), the Office of Research and Development (ORD), and Office of Quality and Patient Safety (QPS)- which includes the National Center for Patient Safety (NCPS).

While all work undertaken applies human factors methods and techniques, projects vary significantly in terms of the role of end users, the number and type of HIT systems included, and the level of effort required to complete. For example, projects range from literature reviews performed by human factors experts only, to a training needs analysis to understand and document specific learning objectives by user role that may require observing and interviewing dozens of clinical staff members.

Given the nature of OHI’s mission, work will require expertise supporting projects within the following Functional Categories and Capability Areas:

☐Informatics Issue Management & Resolution

☐Application Support

☐Informatics Systems Use & Training

☐Informatics Information Security

☐Standards & Interoperability

☐Informatics Change Control

☐Informatics Change Management

☐Clinical Decision Support (CDS)

☒Agile Development Methodology

ReFERENCE DOCUMENTS The Contractor shall follow guidance in the following documents, in addition to the documents in PWS 8.0, in the performance of this effort:

A. ISO Standard: Ergonomics of human-system interaction -- Usability: Definitions and concepts (ISO 9241-11:2018) B. ISO Standard: Human-centered design processes for interactive systems (ISO 9241-210:2009) C. ISO Standard: Ergonomics of human-system interaction -- Usability methods supporting human-centered design (ISO/TR 16982:2002) D. ISO Standard: Ergonomics — General approach, principles and concepts (ISO 26800:2011) E. ISO Standard: Ergonomics of human-system interaction—Principles for the presentation of information (ISO 9241-112:2017) F. ISO Standard: Ergonomics of human-system interaction—Dialogue principles (ISO 9241-110:2008) G. ISO Standard: Ergonomics of human-system interaction— Guidance on visual user-interface elements (ISO 9241-161:2016) H. ISO Standard: Ergonomics of human-system interaction— Ergonomic principles for the design and evaluation of environments of interactive systems (ISO 9241-500:2018) I. ISO Standard: Ergonomics principles in the design of work systems (ISO 6385:2016) J. ISO Standard: Ergonomics of human-system interaction— Forms (ISO 9241-143:2012) K. ISO Standard: Human resource management – Learning and development (ISO 30422:2022) L. International Electrotechnical Commission (IEC) Medical device software — Software life cycle processes (IEC 62304:2006) M. ISO/IEC Standard: Systems and software engineering -- Software product Quality Requirements and Evaluation (SQuaRE) -- Common Industry Format (CIF) for usability: User needs report (ISO/IEC 25064:2013) N. ISO/IEC Standard: Systems and software engineering -- Systems and SQuaRE -- CIF for usability: Context of use description (ISO/IEC 25063:2014) O. National Institute of Standards and Technology (NIST) Interagency/Internal Report (NISTIR) 7742: Customized Common Industry Format Template for Electronic Health Record Usability Testing P. Office of the National Coordinator Health Information Technology (ONC) 2015 Edition of Safety-enhanced Design (URL)

SCOPE OF WORK

The purpose of this requirement is to enable HSI to expand services and execute key program areas, while also increasing consistency within performance and customer service. Success would include clear guidance and resources for HSI staff as well as adopted enterprise standards and policies to improve systems development and maximize performance of the HIT within the enterprise, while improving user experience. The Contractor shall provide overall program management services with specified deliverables and shall provide specified services performed by qualified resources to support key program Areas.

Dedicated Resources are individuals that provide at least a 60% full-time employee equivalent (FTEE), where they are available for team ceremonies and collaborative sessions Monday-Friday, between 7AM Eastern Standard Time (EST) and 6 PM EST, and other contiguous US time zones as applicable for nationally resourced teams.

1. Core Requirements

a. Service Expectations

i. The Contractor shall identify all resources capable of being tasked, in alignment with Key Personnel (marked by *) for their technical skillsets, for specific roles required in execution of the requirements in their Contractor Project Management Plan (CPMP). The Contractor shall provide the Key Personnel as Dedicated Resources that provide Full-time support during regular business hours from any contiguous time zone within the US, for continuity purposes.

ii. The Contractor shall support the HSI office in the provision of Human Factors Engineering and Human Systems Engineering Services to the numerous customers and sponsors engaged with OHI that aim to optimize overall system performance. The Contractor shall provide knowledgeable resources that shall have appropriate training, education, expertise and credentials to conduct the studies in accordance with HSI and VA policies and procedures. This includes supporting the usability, macro-ergonomics, human factors, instructional systems design, systems safety, applied analysis, cognitive engineering and interaction design functions in collaboration with HSI to perform studies for assessments, evaluations, and analyses to provide recommendations for interactive systems to improve their usefulness and usability.

iii. The Contractor shall support the analysis, design, development, testing, and implementation of various clinician-facing and Veteran/Caregiver-facing health IT applications. The Contractor shall provide support services across the project lifecycle utilizing the HCD approach, and at times provide services that spread across numerous activities concurrently.

iv. The Contractor shall develop, submit, and execute scientifically valid studies that include objectives, background, methodology, data analysis and results discussion that is consistent with the nature of the study. The Contractor shall apply human factors methods and analysis, mastering the fundamental concepts of observational, experimental, and correlational research methods that include both qualitative and quantitative data.

v. Below are representative methods that the Contractor shall conduct in accordance with published human factors engineering practices:

1. Literature reviews

2. Clinical observation studies

3. Time and motion studies

4. Contextual inquiries

5. Structured, semi-structured, and unstructured interviews

6. Questionnaires

7. Focus groups

8. Diary studies

9. GOMS (Goals, Operators, Methods, and Selection) modeling

10. Job and work analysis

11. Hierarchical task analysis

12. Cognitive task analyses

13. Affinity diagramming

14. Qualitative and quantitative data collection and analysis

15. Design thinking

16. Brainstorm sessions

17. Patient experience mapping

18. Story mapping

19. Conceptual modeling

20. Workflow reengineering / cognitive system reengineering

21. Storyboarding

22. Participatory design

23. Parallel (A/B alternative) design

24. User interface design (wireframes, functional prototypes)

25. Information architecture (Card sorting, information design)

26. HIS configuration

27. Survey development

28. Rapid ethnography

29. Thematic Analysis

vi. The Contractor shall be able to recommend and execute usability studies, to include:

1. Usability inspections/ design audit

2. Usability walkthroughs

3. Cognitive walkthroughs

4. Formative usability testing

5. Keystroke level modeling

6. Summative usability testing

7. In-situ usability testing

8. Usability observations

9. Usability and design inspections

vii. The Contractor shall have expertise and skill with data analyses, to include parametric and nonparametic statistical analysis, meta analyses, and qualitative data coding.

viii. The Contractor shall have experience in administering the NASA TLX, SUS, PSSUQ, Health-ITUEs and other standard usability and satisfaction instruments. They shall also advise on the best tool(s) to meet Sponsors’ desires for data.

ix. The Contractor shall execute all activities in accordance with approved HSI office methods, processes, and report formats as documented in the HSI Program SharePoint and following guidance in the following documents, in the performance of this effort: See Applicable Resources.

x. The contractor shall have experience in capturing workflows to be used for various purposes including re-engineering and optimization.

xi. The contractor shall have knowledge in the application of physical and cognitive ergonomic principals.

b. Roles

i. The Contractor shall collaborate with other Resources on each assigned study activity, which may include Government staff, other contractors, and Intergovernmental Personal Assignment (IPA) staff.

ii. The Contractor shall follow, support, and endorse the application of HCD practices within HSI and VA.

iii. The Contractor shall ensure that personnel are reliable/have experience in a domain or work with the government to change personnel if the methods change.

c. Work Execution

i. The Contractor shall perform study kick-off meetings to document the context of need for a study request for any assigned study activity and load the materials as a Working Document on the Effort Tracker.

ii. The Contractor shall propose use of an industry standard, or minimally, a recommended approach supported by published evidence.

iii. The Contractor shall acquire agreement from the Government HSI Lead and Sponsor (Business Owner Point of Contact) on format prior to execution and identify and communicate the key characteristics required to meet each stakeholder’s objectives for the study.

iv. The Contractor shall identify all key stakeholders for executed studies.

v. The Contractor shall deliver the Government HSI Lead approved proposal and study plan to the Sponsor upon approval.

vi. The Contractor shall perform work as part of a Study- this includes a study proposal, study plan, and study report. Study Reports and Study Plans are publicly accessible on the VA Network, while Working Products (e.g., drafts, After Action Reviews [AARs], study kick-offs, study proposals) are only accessible to HSI Resources on the Effort Tracker.

vii. The Contractor shall benchmark user performance measures on the as-is system and iterate the design and configuration of any new UI and work system as appropriate.

viii. The Contractor shall lead, manage, monitor, control and execute all aspects of a study; define test goals, recruit, and screen participants, establish test plans and scripts, conduct assessments, perform data analysis, create reports, communicate findings, and make design and functionality recommendations to improve performance and system reliability.

ix. The Contractor shall identify, outreach and consult with users, business leads, user experience, user interface and agile team members to lead, plan and execute a variety of studies and assessments applying the methods listed in Section 3.0.1.a.v.

x. The Contractor shall identify and define the gaps, risks, limitations, capabilities, system constraints, functionalities, affordances, findings, lessons, and factors contributing to the efficiency, effectiveness, comfort, motivation, safety and satisfaction of users.

xi. The Contractor shall apply a multi-disciplinary approach to human factors testing, with knowledge from HF, HCI, HSI, usability, ethnography, and cognitive, social and organizational psychology to produce assessments and reports, and other elements as appropriate to specified study objectives.

xii. Most studies shall be conducted virtually using the VA network and internet resources, although some studies shall be conducted at VA Medical Centers (VAMCs) or Community-Based Outpatient Clinics (CBOCs) in-situ, simulated in a lab, or in other face-to-face settings.

xiii. The Contractor shall provide expertise for evaluating user experience with HIS and to assess and validate the usability of a VA application or system in the context of its intended use in a production environment in accordance with usability testing standards in the International Organization for Standardization (ISO) standard 9241 part 11 (ISO 9241-11) and HF practices and procedures.

xiv. The Contractor shall coordinate the appropriate access requests and complete any necessary training to effectively conduct testing on the targeted systems for any study.

xv. The Contractor shall plan, prepare, conduct, analyze, report, and debrief the results of assessments and HF evaluations.

xvi. The Contractor shall plan for and provide the appropriate study logistics and coordinate the necessary data collection requirements in advance, such as recording voice, video, screen capture, keystroke monitoring, facial expressions, or eye-tracking.

xvii. The Contractor shall follow HSI practices and appropriate regulatory policies, such as Paperwork Reduction Act (PRA), regarding usability testing, interviews, questionnaires, and consent for use of picture and/or voice.

xviii. The Contractor shall coordinate studies with appropriate facility approvals, including any potential union approvals that require advanced notice and responding to revisions.

xix. For both formative and summative usability tests, the Contractor shall provide a Study Report containing Findings which shall be written using the HSI report template which adheres to NIST Interagency/Internal Report (NISTIR) 7742 Common Industry Format (CIF), an established standard format for reporting the results of summative usability tests.

xx. The Contractor shall include the test script, details regarding the number and types of participants, task success criteria, and other project-specific parameters (e.g., test objectives, test procedures, and scenarios) as part of the study plan; the raw data and preliminary findings (e.g., time on task measurements, mouse clicks, keystrokes) and findings such as themes produced from an initial analysis of the data, with a preliminary review of the data as part of the study report; and any updates based on the disposition with the customer for the final study report.

xxi. The Contractor shall also perform an After-Action Review (AAR) and submit entries to the Lessons Learned Journal (LLJ) per HSI standard operating procedure.

xxii. The Contractor shall attend meetings as members of project or study teams and coordinate the discovery (scope definition), preparation (kick-off), execution (status), disposition, and closeout briefings for HSI studies during the project lifecycle.

xxiii. The Contractor shall facilitate sprint retrospectives and/or AARs per HSI guidelines and also if requested by HSI or a customer.

xxiv. The Contractor shall work with each customer on preferred meeting and communication cadence.

xxv. The Contractor shall recommend standards to which projects should adhere, as part of the baseline metric discussions.

d. Work Products

i. Regardless of format, the Contractor shall provide deliverables that are 508 Accessibility compliant and text editable by the Government, including reusable components for future work.

ii. All materials, including written and visual products produced by the Contractor shall be owned by the Government and be submitted using approved templates or branding.

iii. The Contractor shall create and include a proposal (concurred on by the HSI Government Lead and sponsor before study work can begin), a study plan, and the study report for each study-related task performed, each of which the Contractor shall upload to the HSI Effort Tracker SharePoint (VA Network URL provided upon award, referred to as Effort Tracker throughout) as created.

iv. The Study Proposal shall minimally include:

1. Introduction

2. Provide an overview of the study being proposed. Shall include the intent of the study.

3. Objectives

4. Describe the purpose of the study.

5. Mention what is not in scope or evaluated

6. Study Design

7. Include the study method used.

8. If possible, provide a link to the method details.

9. Provide a brief description of the test data being used, test measures, and study participants.

10. Describe the high-level expected results of the study.

11. Schedule (High Level)

12. Create the high-level schedule and include the logical phases.

13. Describe the duration of the project and any schedule constraints (i.e. holidays or other events)

14. Assumptions/Dependencies

15. Describe any organizational or study dependencies.

16. Consider any developer or contractor requirements for supporting study efforts.

17. Identify what assumptions are being made about the study and supporting elements.

18. Concurrence

19. Collected via email, with signatures and date.

20. Concurrence indicates that both HSI and the sponsor agree on the objectives of the study before work begins to ensure that the results of the study address program office needs, including timeline.

21. As the Contractor shall collaborate with other HSI Resources, some studies may be executed where the Study Proposal was crafted by another HSI Resource, but all related study materials shall be loaded to the same Effort Tracker entry.

v. The Study Plan shall minimally include:

1. Purpose of Study

2. Study Objectives

3. Applicable Methods, tools, and resources

4. Session script(s) or questions for users (questionnaire)

5. Target number and types of participants (for various user roles)

6. Data collection and analysis approach

7. Task success criteria if performing tasks

8. Project Timeline and Milestones

9. Sponsor connection points to review progress, risks, and responsibilities

10. Other project-specific parameters, such as:

11. Test procedures

12. Scenarios

13. Personas

14. The Study Report format (visual or written) shall vary depending on the study type (such as wireframes for design related studies) and customer preferences and can range from an Adobe InDesign, Adobe PDF, Excel workbook, a PowerPoint presentation, a Word document, diagrams, web content or other visual format.

vi. The HSI minimum expectation for written reports shall follow this structure, adapted from the Human Factors and Ergonomics Society (HFES) conference proposal requirements (http://cms.HSIs.org/Events/Annual-Meeting/Call-for-Proposals.aspx#Forms), and are expected to be associated with a Word document:

1. Introduction: General statement to orient the reader to the specific problem and context, study personnel, background information, and relevant literature.

2. Method: How the study was conducted, including participants, apparatus, and procedure (e.g., any tasks completed by participants).

3. Results: Summarize the data collected, list any findings1, and/or include visual products2 such as mock-ups

4. Discussion: Describe your analysis of the results, the implications, pertinent evidence-based recommendations, and study limitations and next steps.

5. References: Provide full references for all citations used in the proposal in current edition American Psychological Association (APA) style.

vii. The Contractor shall produce other products, to include briefings, note summaries, Visio models, Excel workbooks, findings/process summary reports, patient journey maps, health service blueprints, personas, value stream maps, and user stories.

viii. The Contractor shall create design patterns, guidelines, and similar resources to be used to promote evidence-based products, quality, and consistency.

ix. For support roles filled by Dedicated Resources, the Contractor shall submit meeting summary notes in tables, with actionable outcomes, key decisions, assumptions, upcoming milestones and deliverables, and any other takeaways and actions that are critical to communicate to HSI government staff.

x. The Contractor shall provide analysis of meeting tone and style (e.g., frustration, confusion, dominated by one organization) along with notes when meeting attendance and report back is tasked.

xi. The Contractor shall provide evidence that their solutions meet approved acceptance criteria and/or the “definition of done” or explain deviance.

e. Tracking and Coordination

i. The Contractor shall update and maintain the VA Program Manager (PM) approved CPMP throughout the PoP.

ii. The Contractor shall identify the primary contact point for all programmatic issues/concerns/status.

iii. The Contractor shall complete a Problem-Issue + Resolution (PI+R) document or a decisions, expectations (assumptions), risks, actions, and issues log (DERAIL) document to track decisions for each effort. See Applicable Resources. The DERAIL or Problem-Issue tracker and Communication Schedule shall be maintained on the Effort Tracker by the Contractor as a Working Product.

iv. The Contractor shall document throughout the PoP and communicate weekly the PI+R or DERAIL aligned with each activity during study planning, execution, and closing phases.

v. The Contractor shall establish and maintain a recurring Communication Schedule with stakeholders aligned with each assigned activity’s objectives.

vi. The Contractor shall document all initiated work, provide updates, and load the final Study materials in the HSI Effort Tracker SharePoint and the Project Management (PM) Tracker SharePoint.

vii. The Contractor shall be responsible for ensuring all materials are loaded to the Effort Tracker and the LLJ, and shall notify the Government HSI Lead of known gaps of content, sourced or attributed as work products from other entities.

viii. Microsoft Teams is VA’s official collaboration tool and is used extensively for VA-initiated meetings and is available on the VA Network and shall be the primary teleconferencing tool used during performance of studies.

ix. The Contractor shall perform and document White Glove (WG) reviews and formal Peer Reviews (PR) on assigned work activities and submit as a Working Product to the relevant Effort Tracker entry.

x. The Contractor shall provide project coordination services across multiple, simultaneous activities of varying complexity.

xi. The Contractor shall perform VHA employee (such as clinicians, not Veterans) participant recruitment and scheduling coordination, technology set-up for virtual studies, logistics planning for in-person or remote sessions, data recording, coding and analysis, report writing, technical editing/proofreading for formal PR and WG, meeting scheduling and agenda preparation, study recommendations disposition tracking, engagement summaries preparation, publishing support, and graphical design.

xii. The Contractor shall take notes that will include key information discussed, decisions and their justification, outstanding questions, and action items. If all parties agree, the call may be recorded, but a call transcript is not a suitable substitution for this notes document, which shall include appropriate grammatical standards for secondary use.

xiii. On calls they lead or represent HSI on, the Contractor shall track action items, discussion points, and decisions and review them prior to the call ending for concurrence.

xiv. The Contractor shall create agendas and facilitate calls for work they lead.

xv. The Contractor shall monitor their action items for completion and report status to the government.

xvi. When reviewing deliverables, the Contractor shall make notes of government requested changes, note how each was met (e.g., in a table or in MS Word comments) and submit that prior to next review, to include rationale for changes.

xvii. The Contractor shall use data (e.g., Jira reports or adherence to an agreed-on schedule) to make recommendations for resource changes to achieve project or sprint objectives.

xviii. The Contractor shall obtain government approval before proposing any specific work to customers.

f. Operational Support

i. The Contractor shall offer guidance and materials to HSI on how to align their processes with industry best practices.

ii. The Contractor shall offer guidance and materials to HSI on promoting the science of HF within VA.

iii. The Contractor shall support the continued creation, publication, and curation of HSI and usability.

iv. The Contractor shall support the management and enhancement of knowledge materials on sharing portals, such as SharePoint.

PERFORMANCE DETAILS

PERFORMANCE PERIOD

The Period of Performance (PoP) shall be 12-months from date of award, with one 12-month option period.

There are eleven (11) Federal holidays set by law (USC Title 5 Section 6103) that VA follows:

Under current definitions, four are set by date:

New Year's Day January 1

Juneteenth June 19

Independence Day July 4

Veterans Day November 11

Christmas Day December 25

If any of the above falls on a Saturday, then Friday shall be observed as a holiday. Similarly, if one falls on a Sunday, then Monday shall be observed as a holiday.

The other six are set by a day of the week and month:

Martin Luther King's Birthday Third Monday in January

Washington's Birthday Third Monday in February

Memorial Day Last Monday in May

Labor Day First Monday in September

Columbus Day Second Monday in October

Thanksgiving Fourth Thursday in November

PLACE OF PERFORMANCE

Efforts under this contract shall be performed at Contractor facilities. The Contractor shall identify the Contractor’s place of performance in their Task Execution Plan (TEP) submission.

TRAVEL OR SPECIAL REQUIREMENTS

The Government anticipates travel under this effort to perform the tasks associated with the effort, as well as to attend program-related meetings or conferences through the PoP. Travel shall be Cost Reimbursable with an NTE ceiling on travel expenses. Travel expenses shall be invoiced and paid against the travel CLIN, which will have an NTE price specified.

Travel shall be in accordance with the Federal Travel Regulations (FTR) and requires advance concurrence by the Contracting Officer’s Representative (COR). Each Contractor invoice must include copies of all receipts that support the travel costs claimed in the invoice. Contractor travel within the local commuting area will not be reimbursed. General and Administrative expenses will not be reimbursed. These costs will be considered to be reasonable and allowable only to the extent that they do not exceed the maximum per diem rates in effect at the time of travel as set forth in the FTR.

The FTR can be referenced at:

https://www.gsa.gov/policy-regulations/regulations/federal-travel-regulation

Travel may occur, but is not limited to, the below locations for performance of activities detailed in the PWS.

SPECIFIC TASKS AND DELIVERABLES

PROJECT MANAGEMENT

The Contractor shall summarize all their planned sprint activities for each Area below for each team, every month in a Monthly Execution Plan (MEP). The Contractor shall provide invoices covering each MEP timeframe that will align with Quality Assurance Surveillance Plan (QASP) measures and include % Commitments Completed and Planned Effort / Performed Effort Ratio. The Contractor shall deliver the CPMP, and update the document as needed as an appendix to the MEP. The Contractor shall deliver a Summary Package that includes the final update for performance to the last MEP and the transition materials for each task.

The Contractor shall provide reports or meetings outlined below for each Area as needed by the VA Government Program Manager (GPM). The Contractor shall provide the Decisions, Expectations/Assumptions, Risks, Actions, Issues Log (DERAIL) or a Problem-Issue Resolution document on the VA Effort Tracker as a working document that is kept up to date for all efforts. The Contractor shall upload deliverables and related components to the HSI Effort Tracker SharePoint as created. The Effort Tracker is the single authoritative project and study deliverables and working documents repository. The Project Management (PM) Tracker is the task listing where all existing commitments and planned efforts are tracked with assigned resources. The Contractor shall provide weekly status updates on the PM Tracker. The Contractor shall support effective communication of commitments through shared project workspaces when collaborating with other interdisciplinary teams, such as Jira. The Contractor shall support material sharing on the Teams channel for collaboration. The Contractor shall apply agile principles to the work effort and be familiar with Scaled Agile Framework techniques. The Contractor shall support product increment planning efforts as appropriate to the sprint, epic, and feature objectives.

The Contractor shall prepare and deliver a CPMP that lays out the Contractor’s approach, timeline and tools to be used in execution of each task. The CPMP shall also include how the Contractor shall coordinate and execute planned, routine, and ad hoc data collection reporting requests as identified within each task.

The Contractor must have network access to complete this task (Effort and PM Trackers)., any absence of network access will lead to a stop work order, and initial award may be delayed in order to support necessary onboarding activities. The Contractor shall expedite onboarding in accordance with VHA practices. The Contractor shall expeditiously undergo sufficient background checks and complete required training and forms in order to be granted access to VA’s network and key resources used and managed by the HSI team within three (3) business days of receiving the notice of award. The Contractor shall communicate on an on-going manner the current status of contracting on-boarding including; last actions taken, next steps, and anticipated completion dates.

The initial baseline CPMP shall be concurred upon and shall include a labor category staff roster identifying the following:

1. Employee Name

2. Personnel PIV Expiration

3. Personnel clearance level and adjudication date

4. Role

5. Project affiliation and assigned activities log

6. Personnel duty location address, alternate email, phone number

The Contractor shall support sprint (2-week work cycle) planning that detail commitments for the coming 2 weeks, across all the Areas, and detail them to the GPM on a Workload Assessment Meeting (WAM). The Contractor shall review the backlog by Area to inform commitments. The Contractor shall summarize engagements / activities via the MEP as well as specify key performance indicators (KPIs) and intended outcomes for each project. At the end of each sprint, the Contractor shall document the percentage done.

The Contractor shall be paid for the percentage of work that has been accepted by the Government from each MEP period (i.e., monthly).

Work submitted shall follow VA, VHA, OHI, and HSI’s processes and procedures, such as AAR and LLJ.

The Contractor shall continuously monitor performance and report any deviation to the COR and VA GPM. The Contractor shall lead a WAM with the VA GPM every two weeks for project summary and status discussions in preparation for the MEP submission.

The weekly WAM shall include the following information:

1. Overall high-level assessment of progress

2. Works in progress (WIP) with planned deliverables and working documents

3. Activities recently completed since the last meeting

4. Planned projects to begin in the next two (2) weeks, when applicable

5. Barriers to progress or inefficiencies experienced

6. New assignments and opportunities

The MEP shall incorporate the prior WAM details for the prior sprint’s efforts, including corresponding changes made during the current reporting period, such as “Original target date for study report MM/DD/YYYY revised to MM/DD/YYYY on MM/DD/YYYY because of … concurred upon by HSI Government Lead”. Government feedback on prior reports shall be addressed in follow-on deliveries, with rationale provided and details about the corrective actions taken.

The Contractor shall hold a technical kickoff meeting within 10 business days after award to summarize the CPMP and execution strategy. The Contractor shall present, for review and approval by the Government, the details of the intended approach, work plan, and project schedule for each task.

The Contractor shall specify kickoff dates, location (often virtual), agenda (shall be provided to all attendees at least five (5) calendar days prior to the meeting), and meeting minutes including all presented materials (shall be provided to all attendees within three (3) business days after the meeting).

The Contractor shall invite the Contracting Officer (CO), Contract Specialist (CS), COR, the VA GPM, and the HSI Government Lead.

The Contractor shall acquire tooling necessary to complete the taskings below, making full use of the VHA software center tooling already available, and only with software that is approved by VHA. Any purchases must be coordinated with the COR in advance and cannot exceed the allotted amount for General Other Direct Costs (ODCs). The Government anticipates access required for up to 10 individuals to perform the usability testing and simulation taskings.

Deliverables:

A. Kick-off Materials (Due 3 business days after meeting) B. Contractor Performance Management Plan (Due 30 calendar days after award, updated in the MEP as needed) C. Monthly Execution Plan (Due on the 1st business day of each month, starting the beginning on the start of the second month after award) D. Summary Package (Due 10 calendar days before the expiration of the Task Order) RISK ANALYSIS AND MITIGATION, ENGINEERING SOLUTIONS, ISSUE RESOLUTION, SIMULATION, AND SYSTEM TESTING The Contractor shall provide support for five (5) Workstreams (Risk, Engineering Solutions, Issues, Simulation, and Test) with multiple cross-functional teams to perform system testing, simulation and forecasting, issue resolution, risk analysis and mitigation, and solution engineering. The goal of this effort is to determine the user needs by role and perform structured usability testing for critical functionality, including providing baselining and post-implementation assessments, while providing evidence for risk-informed decision-making. The Contractor shall provide technical expertise and support for the councils, with multiple teams to perform the solution development activities required for issue resolution, risk assessment and mitigation, and analysis of alternatives. The Contractor shall support emerging studies resulting from OHI priorities and council engagement. Teams will often interact with each other, and the Contractor shall communicate all necessary details to support information sharing.

Risk- The Contractor shall proactively identify and mitigate potential issues.

Engineering Solutions- The Contractor shall support rapid cycle improvements.

Issues- The Contractor shall provide HSI expertise to support the Councils.

Simulation- The Contractor shall provide modeling data to advise VHA leadership.

Test- The Contractor shall perform usability testing to inform VHA leadership.

The Contractor shall be able to surge support for multiple cross-functional teams for each Workstream. The Contractor shall provide a total of seven (7) teams to support the identified Workstreams. Each Workstream will have at least one (1) active team per increment, and no more than three (3) active teams per Workstream during an increment. An increment is approximately a quarter of a year, with increment planning and retrospective commitments distributed over approximately a 15-week period. Planning and retrospective events will overlap with the adjoining increments. The Contractor shall participate in agile events as needed.

The Contractor shall provide for each team, Key Personnel and Dedicated Resources in the form of a Study Lead (Lead) and Study Support so that each sprint executed will have a lead and support person. The Contractor shall provide appropriate System/Data expertise on an as needed based for defined applications and data domains (to include Oracle Health’s Cerner Millenium and VistA data expertise). The Contractor shall provide Clinical expertise for sprint consultation within the domain under assessment, including specific VHA procedural and operational insights gathered from direct experience in the VHA environment. Given the pace of sprints, System/Data and Clinical expertise is a critical requirement that the Contractor shall be able to provide within a 24-hour timeframe from identified need.

The Lead shall execute, manage, and coordinate with the team the various activities needed to produce a plan, define specifications (e.g., scenarios), capture objectives, design the strategy, perform risk assessment, evaluation alternatives, and assess targeted increment objectives. The Lead shall maintain a strategy to support the teams’ increment objectives. The Contractor shall apply standards and best practices in execution of these activities.

The Lead shall maintain a strategy to support the team’s increment objectives. The Contractor shall apply standards and best practices in execution of these activities. The Contractor shall provide recommendations on any existing or emerging technologies, tools or methods that can be leveraged to maximize effectiveness and reduce risks and associated costs.

The Contactor shall provide resources that can be Dedicated to only one team given the cadence and cycle time. A series of sprints able to support broader initiatives may occur. End-user Sessions are generally 60 minutes in length, and overlapping is avoided, when possible, but the Contractor shall be responsive to clinician participants’ availability. Elaboration sessions can generally be combined in up to a 3-hour workshop (e.g., morning session/afternoon session). The Contractor shall lead site-visit planning and follow the most current site-visit checklist in coordination with other OHI resources.

The Contractor shall provide support for each planned go-live facility and perform testing, simulation, issue resolution, risk mitigation, and solution engineering for each council and facility prioritized needs for a group of core users from each planned go-live facility.

The Contractor shall provide support for each post-implementation facility and perform testing, simulation, issue resolution, risk mitigation, and solution engineering for each council and facility prioritized needs for a group of core users from each post-implementation facility.

The Contractor shall provide support for baselining and evaluating intervention effectiveness, using pre- and post- adoption sites with an emphasis on validated measures for Functionality, Usability, Morale/Satisfaction, Productivity, Access to Care, and Cost associated with the Federal Electronic Health Record (EHR) implementation.

The Contractor shall support technical and user experience measures that should be captured continually or timed to specific interventions.

The contractor shall support dashboard (re)design and reporting through evidence-based solutions that adhere to best practices for information display for requested dashboards.

The Contractor shall support on-site and virtual (remote) activities and align efforts with the Informatics Readiness Critical Path (IRCP). The IRCP is a VA-developed document that prepares sites for the Federal EHR implementation. It depicts the key activities (greater than 300) of the Federal EHR implementation and provides a matrix mapping of the current core activities and timelines for the necessary informatics functions that should be integrated into the process for successfully completing each activity. This helps VHA OHI determine the functions necessary to support the implementation process and ensure adoption of the system. It enables the organization to identify gaps or areas that may require additional resources or support, leading to improved efficiency and effectiveness of the Federal EHR implementation, and ultimately resulting in greater satisfaction. This framework is not public, and therefore a copy will be provided post-award.

The Contractor shall be responsible for all coordination efforts to ensure the appropriate site engagement occurs in alignment with HSI best practices for site-visits and remote sessions, which includes outreach to VISN and local executive leadership teams and updating the OHI EHRM Integrated Master Schedule (IMS) and calendar. The Contractor shall coordinate site engagement collectively across other OHI efforts to reduce or mitigate local burden, which includes supporting ‘joint’ assessments with other contract and program offices on integrated sessions (e.g., Business Architecture E2E). The Contractor shall lead the negotiation with site resources to mitigate the impact to care delivery and optimize the overall assessment opportunity, including data collection and analysis with diverse participant groups.

Due to the limited facility space, site-visits are expected to occur over a 5-day week with up to two teams to avoid disruption to ongoing care delivery. HSI aims to have at least five (5) representative users per targeted role, to the extent possible, generally with no more than six (6) roles, or 30 participants over 5 days when on site per team. Individual study objectives and applied methods may vary and impact these participant expectations, including potential site variation, such as community-based outpatient clinics. The Contractor shall provide both Intro and Exit briefs to senior site leadership. The Government anticipates up to two teams to travel per month across VA facilities for the workstreams.

The Contractor shall, for all activities, support reducing errors and improving safety and usability (efficiency, effectiveness, and satisfaction).

The Contractor shall acquire the necessary software for usability testing and simulation activities for the teams, that must be approved by VA and listed on the TRM (Technical Reference Manual).

In doing the work, the Contractor shall perform the following activities as appropriate for the project engagement and customer:

a. Conduct environmental scans, research best practices and the existing scientific literature to provide evidence-based practices on evaluations from an end-user perspective.

b. Leverage survey development best practices to develop a repeatable semi-structured interviews survey with both quantitative (validated, closed-ended questions) and more qualitative (open-ended items) to collect comprehensive data that can be scaled to other VA facilities and are applicable for other VA roles and positions.

c. Develop an interview guide that includes details on the quantitative items (i.e., validated quantitative performance baseline usability metrics, for example, time on task; usability errors and user satisfaction) that will provide a reference point to determine the quantitative impact of the applied intervention and data collection process and a series of semi-structured questions and other relevant qualitative methods for the data collection.

d. Establish Scenarios and maintain a library for testing key user…