36C77619R0010-004.pdf

PERFORMANCE WORK STATEMENT (PWS)

DEPARTMENT OF VETERANS AFFAIRS

Office of Health Informatics (OHI)

Clinical Informatics and Data Management Office (CIDMO) Human Factors Engineering (HFE)

Human Factors Services (HFS)

Indefinite Delivery, Indefinite Quantity (IDIQ)

Attachment A1 - HFS IDIQ Performance Work Statement

Contents

Table of Contents

PERFORMANCE WORK STATEMENT (PWS)

1.0 BACKGROUND

2.0 APPLICABLE DOCUMENTS

3.0 SCOPE OF WORK

3.1 ORDER TYPE

4.0 PERFORMANCE DETAILS

4.1 PERFORMANCE PERIOD

4.2 PLACE OF PERFORMANCE

4.3 TRAVEL OR SPECIAL REQUIREMENTS

4.4 GOVERNMENT FURNISHED PROPERTY

4.5 SECURITY AND PRIVACY

4.5.1 POSITION SENSITIVITY

5.0 SPECIFIC TASKS AND DELIVERABLES

5.1 PROJECT MANAGEMENT

5.1.1 CONTRACTOR PROJECT MANAGEMENT PLAN

5.1.2 REPORTING REQUIREMENTS

5.1.3 TECHNICAL KICKOFF MEETING

5.2 HUMAN FACTORS ENGINEERING SERVICES

5.2.1 SPECIFYING HIS CONTEXT OF USE AND USER REQUIREMENTS

5.2.2 DESIGNING HIS SOLUTIONS

5.2.3 EVALUATING HIS USABILITY

5.3 COORDINATING ENGAGEMENTS, PROJECTS, AND STUDIES

5.4 MATURING CONTENT FOR THE USER EXPERIENCE (UX) GUIDE

Rights in Deliverables

6.0 GENERAL REQUIREMENTS

6.1 PERFORMANCE METRICS

6.2 SECTION 508 – ELECTRONIC AND INFORMATION TECHNOLOGY (EIT) STANDARDS

6.2.1 EQUIVALENT FACILITATION

6.2.2 COMPATIBILITY WITH ASSISTIVE TECHNOLOGY

6.3 ENTERPRISE AND IT FRAMEWORK

6.4 INFORMATION TECHNOLOGY USING ENERGY-EFFICIENT PRODUCTS

ADDENDUM A – General Notes

B2. ACCESS TO VA INFORMATION AND VA INFORMATION SYSTEMS

B3. VA INFORMATION CUSTODIAL LANGUAGE

B4. INFORMATION SYSTEM DESIGN AND DEVELOPMENT

B5. INFORMATION SYSTEM HOSTING, OPERATION, MAINTENANCE, OR USE

B6. SECURITY INCIDENT INVESTIGATION

B7. LIQUIDATED DAMAGES FOR DATA BREACH

B8. SECURITY CONTROLS COMPLIANCE TESTING

1.0 BACKGROUND

The Office of Health Informatics (OHI) is integral to Veterans Health Administration (VHA) programs that provide patient-centered health care to Veterans, offering advanced and secure enterprise data systems, sophisticated analytic and measurement solutions, decision support, business intelligence and web communications. OHI facilitates evidence-based decisions for individual Veterans and their families, patient populations, clinicians, and those managing health care delivery systems. The Human Factors Engineering (HFE) directorate exists within Department of Veterans Affairs (VA), VHA, Office of Health Informatics (OHI), Clinical Informatics and Data Management Office (CIDMO). HFE seeks to increase awareness and application of human factors principles, improving performance and safety of VA health information systems (HIS) by optimizing the end-user experience. HFE offers human-centered design (HCD) services (including user interface design and usability assessments) for a variety of VA-developed and acquired products and services. HFE requires engineering support for numerous projects that range from understanding user needs through implementation, amongst a diverse population of Veteran, Caregiver, Administrative and Clinician Users to ensure that the health information technology (HIT) provided for their use is helpful and relevant to them. A best practice in HCD is to encourage end-user feedback at all stages of the development lifecycle.

Beyond the name of the HFE organization, ‘human factors engineering’ is the scientific discipline concerned with the understanding of interactions among humans and other elements of a system, and the engineering profession that applies theory, principles, data and methods to design in order to optimize human well-being and overall system performance (International Ergonomics Association). A ‘system’ is a network or group of interdependent components and operational processes that work together to accomplish the objectives and requirements (Design for Safety).

The VHA requires a rigorous human factors engineering approach in operational settings to ensure that all HIT (acquired or developed) and corresponding elements of the health care socio-technical system are jointly optimized to provide a user experience with high performance and satisfaction. The HFE office optimizes both employee (i.e., clinicians) and Veteran user experience with HIT, medical devices and consumer products (e.g., tablets, smartphones, wearables) via an iterative HCD approach offered in the International Organization for Standardization (ISO) 9241: plan the process, understand context of use, specify requirements, design, and test user interfaces to ensure they improve user effectiveness, minimize mental workload, support decision making, maximize clinical workflow efficiency, reduce human error, improve patient safety, and enhance quality of care.

The HFE office responds to requests for collaboration and partnership from multiple channels (e.g., VA/VHA Initiatives, National Program office objectives, field inquiries, system platform sponsors) for human factors engineering services in support of analysis of user needs, (human-centered) design, re-design, and usability testing. Depending on objectives, these services may be provided for a particular activity, intermittently, or in concert over an extended period of time. While all work undertaken applies human factors methods and techniques, projects vary significantly in terms of the role of end users, the number and type of HIT systems included, and the level of effort required to complete. For example, projects range from a heuristic evaluation of a mobile app performed by human factors experts only, to a rapid ethnography study to understand and document clinical workflows that may require observing and interviewing dozens of clinical staff members.

Certain engagements are continually supported (e.g., legacy systems such as Veteran’s Health Information Systems and Technology Architecture (VistA) / Computerized Patient Record System (CPRS), VA’s in-house developed electronic health record (EHR)), while other needs will emerge throughout the Period of Performance (PoP).

Examples of ad hoc support provided in recent months include end user testing of a Comprehensive Suicide Risk Assessment template (clinical documentation tool) and its use in the context of EHR clinical workflow and evaluation of a web-based dashboard (decision support tool) used by Veteran Crisis Line staff.

2.0 APPLICABLE DOCUMENTS

The Contractor shall follow guidance in the following documents, in the performance of this effort:

A. ISO Standard: Ergonomics of human-system interaction -- Usability: Definitions and concepts (ISO 9241-11:2018) B. ISO Standard: Human-centered design processes for interactive systems (ISO

9241-210:2009) C. ISO Standard: Ergonomics of human-system interaction -- Usability methods supporting human-centred design (ISO/TR 16982:2002) D. ISO Standard: Ergonomics — General approach, principles and concepts (ISO

26800:2011) E. ISO Standard: Ergonomics of human-system interaction—Principles for the presentation of information (ISO 9241-112:2017) F. ISO Standard: Ergonomics of human-system interaction—Dialogue principles

(ISO 9241-110:2008)

G. ISO Standard: Ergonomics of human-system interaction— Guidance on visual user-interface elements (ISO 9241-161:2016) H. ISO Standard: Ergonomics of human-system interaction— Ergonomic principles for the design and evaluation of environments of interactive systems (ISO 9241- 500:2018)

I. ISO Standard: Ergonomics principles in the design of work systems (ISO 6385:2016)

J. ISO Standard: Ergonomics of human-system interaction— Forms (ISO 9241- 143:2012)

K. International Electrotechnical Commission (IEC) Medical device software — Software life cycle processes (IEC 62304:2006)

L. ISO/IEC Standard: Systems and software engineering -- Software product Quality Requirements and Evaluation (SQuaRE) -- Common Industry Format (CIF) for usability: User needs report (ISO/IEC 25064:2013)

M. ISO/IEC Standard: Systems and software engineering -- Systems and SQuaRE -- CIF for usability: Context of use description (ISO/IEC 25063:2014)

N. National Institute of Standards and Technology (NIST) Interagency/Internal Report (NISTIR) 7742: Customized Common Industry Format Template for Electronic Health Record Usability Testing

O. Office of the National Coordinator Health Information Technology (ONC) 2015 Edition of Safety-enhanced Design (URL)

3.0 SCOPE OF WORK

The purpose of this contract is to obtain human factors engineering support services for the HFE office. The Contractor shall provide human factors engineering support from knowledgeable resources with appropriate training, education, expertise and credentials to conduct the studies in accordance with HFE and VA policies and procedures. The Contractor shall develop, submit, and execute scientifically valid studies that include objectives, background, methodology, data analysis and results discussion that is consistent with the nature of the study. While occasional organizational changes may occur, the scope of this effort is intended to include projects and programs listed below as well as new ones that may emerge:

1. Legacy VistA/CPRS: VA’s current electronic health record (EHR) system. Includes the Joint Legacy Viewer (JLV) and CPRS support on national releases.

2. Electronic Health Record Modernization (EHRM): a new electronic health record system that will facilitate interoperability with the Department of Defense (DoD) and other health care partners to enhance patient-centered, team-based and evidence-based care by giving health care providers a complete picture of a patient’s care and treatment history.

3. Connected Care: tools for Veterans, family caregivers, and clinicians provided through a number of modalities including smartphones, tablets, kiosks and websites.

My HealtheVet, Web and Mobile Solutions, and Telehealth are part of this suite.

4. Community Care: refers eligible Veterans for care in the community, collecting data about the care provided and patient outcomes achieved, and paying for the care.

5. VA Innovation Office: identifies and funds innovations that show promise for improving care and benefits delivery and Veteran and VA staff experiences.

Courtney, Mary G. (OGC) Insert the earlier scope paragraph here.

Courtney, Mary G. (OGC) This reads more like an introduction to the tasks the contractor will perform.

Blasko, Robert (VHACLE) moved

6. Clinical Informatics and Data Management Office (CIDMO): entities such as Veterans Health Information Exchange, Knowledge-Based Systems, Informatics Patient Safety and Bar Code Medication Administration that periodically request services from HFE.

7. Other VA administrations and program offices: HFE conducts ad hoc reviews of websites, commercial software products and a variety of other systems, including support for Veteran Benefits Administration (VBA), Office of Information and Technology (OI&T/OIT), Veterans Experience Office (VEO) and Digital Service at

VA (DSVA).

VA VISN and Field Offices: VISN and Field offices develop and deploy solutions for local or VISN-wide use and occasionally request HFE consultation and/or review.

The Contractor may be required to provide a range of services from an isolated activity (for example, planning and executing a standalone performance-based usability test) to a collection of services (for example, conducting a user needs assessment that leads to iterations of User Interface (UI) designs that then undergo usability testing). HFE responds to numerous project and programs’ requests for Human Factors Services within VA and VHA, and the variability of the needs change from each activity based on the agreed upon study objectives. The Contractor shall provide the following human factors engineering services, outlined in the below PWS sections:

3.1 ORDER TYPE

This is a Multiple Award Indefinite Delivery/Indefinite Quantity (IDIQ) contract. The task orders will be awarded on a Fixed-Firm Price (FFP) basis except for travel which shall be on Reimbursable basis with a Not To Exceed (NTE) amount per Task Order (TO) under Time and Materials.

4.0 PERFORMANCE DETAILS

4.1 PERFORMANCE PERIOD

The Period of Performance (PoP) shall be an ordering period for up to five (5 years).

The Period of Performance of the Task Orders shall be determined and defined at the Task Order level.

4.2 PLACE OF PERFORMANCE

Place of performance shall be identified in individual task orders, but it is anticipated that a majority of the efforts under this contract shall be performed at Contractor facilities. The Contractor shall identify the Contractor’s place of performance in their Task Execution Plan (TEP) submission.

4.3 TRAVEL OR SPECIAL REQUIREMENTS

Travel shall be in accordance with individual task order requirements. The Government anticipates travel under this effort to perform the tasks associated with the effort, as well as to attend program-related meetings or conferences through the PoP. Travel shall be Reimbursable with an NTE ceiling on travel expenses per TO. Travel expenses shall be

Courtney, Mary G. (OGC) This is your scope paragraph not your background.

Blasko, Robert (VHACLE) moved

Courtney, Mary G. (OGC) The use of cost-reimbursement contracts is prohibited for the acquisition of commercial items (see parts 2 and 12). See FAR 16.301-3 You need to have it as T&M

Blasko, Robert (VHACLE) Accepted corrections

Kurt Ruark Cost reimbursable was just for the travel- I thought that was FAR, travel only reimbursed up to cost (per diem)?

invoiced and paid against the travel CLIN, which will have an NTE price specified on each TO.

Travel shall be in accordance with the Federal Travel Regulations (FTR) and requires advance concurrence by the Contracting Officer’s Representative (COR). Each Contractor invoice must include copies of all receipts that support the travel costs claimed in the invoice. Contractor travel within the local commuting area will not be reimbursed. These costs will be considered to be reasonable and allowable only to the extent that they do not exceed the maximum per diem rates in effect at the time of travel as set forth in the FTR.

4.4 GOVERNMENT FURNISHED PROPERTY

Government Furnished Property may be provided and shall be identified in the individual task orders. The VA HFE office will provide access to the tools listed below.

The number of Government-furnished licenses available for Contractor use is listed in parentheses. If the mandated toolset changes during the ordering period, then the Government will provide access to the new toolset.

• Balsamiq (1)

• Morae (2)

• Survey Monkey (1)

• PSPP (open source version of SPSS)

In addition to the licensed tools above, HFE also uses no-cost versions of web-based tools such as Trello and AirTable. Lync/Skype for Business is VA’s official collaboration tool and is used extensively for VA-initiated meetings, and is available on the VA Network and shall be the primary teleconferencing tool used during performance of studies. The Contractor shall ensure contract staff have access and are knowledgeable with Microsoft (MS) Office based products (Word, Excel, PowerPoint, Project, Publisher, Access, etc.) and common Design, User Experience, Usability and User Interface tools (e.g., Adobe InDesign, Axure, SnagIt, OptimalSort, etc.).

The Government has determined that remote access solutions involving Citrix Access Gateways (CAG) have proven to be an unsatisfactory access method to complete some of the tasks required. The Government will evaluate the need for CAG on a case by case basis. The Government also understands that Government Furnished Equipment (GFE) is limited to Contractors that may require direct access to the network in order to ensure usability testing software compliance with VA network environments;

upload/download/ manipulate content on SharePoint Server; and configure and assess secure applications, such as NATREM (NATional REMinder Test System) and CPRS.

The usage of GFE will be on a case by case basis and must be approved by the COR when sufficient. Each instance where CAG is not sufficient shall be justified to the COR prior to issuing. Based on the Government assessment of remote access solutions and the requirements of this TO, the Government estimates that ‘standard laptops’ with charging cables will be required by this TO.

The Contractor is responsible for providing any other computer accessories (e.g., keyboards, mice, cables) as necessary and any VA installation required for accessories shall be coordinated with the COR. The Contractor shall document, track, and monitor all GFE, and any GFE can be audited by the COR or local VA resource. The Contractor shall ensure a current list of GFE to include the name, computer make and model, serial number, and previous PM/COR (if existing issued device is used) is provided to the COR and Contracting Officer.

Regardless of use of CAG or GFE, the Contractor shall be accessible on the VA Network account for communication purposes, including via Outlook and Skype/Lync, and for tool access (i.e., SharePoint, NATREM, etc.).

4.5 SECURITY AND PRIVACY

4.5.1 POSITION SENSITIVITY

Position Sensitivity

Background Investigation (in accordance with Department of Veterans Affairs 0710 Handbook, “Personnel Suitability and Security Program,” Appendix A)

Low / Tier 1

Tier 1 / National Agency Check with Written Inquiries (NACI) A Tier 1/NACI is conducted by OPM and covers a 5-year period. It consists of a review of records contained in the OPM Security Investigations Index (SII) and the DOD Defense Central Investigations Index (DCII), FBI name check, FBI fingerprint check, and written inquiries to previous employers and references listed on the application for employment. In VA it is used for Non-sensitive or Low Risk positions.

Moderate / Tier 2

Tier 2 / Moderate Background Investigation (MBI) A Tier 2/MBI is conducted by OPM and covers a 5-year period. It consists of a review of National Agency Check (NAC) records [OPM Security Investigations Index (SII), DOD Defense Central Investigations Index (DCII), FBI name check, and a FBI fingerprint check], a credit report covering a period of 5 years, written inquiries to previous employers and references listed on the application for employment; an interview with the subject, law enforcement check; and a verification of the educational degree.

High / Tier 4

Tier 4 / Background Investigation (BI) A Tier 4/BI is conducted by OPM and covers a 10-year period. It consists of a review of National Agency Check (NAC) records [OPM Security Investigations Index (SII), DOD Defense Central Investigations Index (DCII), FBI name check, and a FBI fingerprint check report], a credit report covering a period of 10 years, written inquiries to previous employers and references listed on the application for

The position sensitivity and the level of background investigation commensurate with

I created this- We need to clearly detail they will be required to be on GFE, and access to NATREM- they also have to put deliverables on the Effort Tracker, which requires VA Access.

Testing applications requires VA test environment.

Courtney, Mary G. (OGC) This is from T4NG. It does not seem applicable to me.

Read PM’s comment above the required level of access for the following tasks within the Performance Work Statement are:

Position Sensitivity and Background Investigation

Requirements

Task Number Low/NACI Moderate/MBI High/BI

5.1 5.2 5.3 5.4 5.5 5.6

The Tasks identified above and the resulting Position Sensitivity and Background Investigation requirements identify, in effect, the Background Investigation requirements for Contractor individuals, based upon the tasks the particular Contractor individual will be working. The submitted Contractor Staff Roster must indicate the required Background Investigation Level for each Contractor individual based upon the tasks the Contractor individual will be working, in accordance with their submitted proposal.

5.0 SPECIFIC TASKS AND DELIVERABLES

The Contractor shall follow, support, and endorse the application of HCD practices within HFE and VA. The Contractor shall support the usability, macro-ergonomics, human factors, cognitive engineering and interaction design functions in collaboration with HFE to perform studies for assessments, evaluations, and analyses to provide recommendations for interactive systems to improve their usefulness and usability. The Contractor shall provide use of a collaboration tool such as WebEx, during the PoP that is both compatible with Morae and can be used by clinician and Veteran participants both within and outside the VA firewall.

A ‘Study’ is the use of one or more methods to achieve an objective. Sample study methods include heuristic evaluations, focus groups, and performance-based usability test. The Contractor shall perform a mix of face-to-face and virtual methods to conduct studies. The Contractor shall align the proposed method to the agreed upon study objectives. Studies may contain ‘Participants’ or individuals representing an end user, as aligned with the executed method, and be performed over a series of ‘Sessions’. A Session is a single event attended by one or more Participants. For example, a focus group could consist of a single session with 9 Participants, where a performance-based usability test could consist of 9 sessions with a single Participant in each. Each project may have one or more individual studies established to meet project objectives, and engagements may have one or more projects underway, prepped in the back log for future work, or previously completed. Individual projects have targeted completion dates, while engagements do not and represent the partnership to mature the HCD

Courtney, Mary G. (OGC) This is where I think the discussion under scope about studies and participants seems appropriate.

approach throughout VA.

The Contractor shall be responsible for managing assigned projects and executing established processes to identify and plan effective and timely HCD activities across various engagements throughout the system development lifecycle. The Contractor shall support the continued provisioning of human factors services with HFE for external requests as well as for HFE-initiated studies. When HFE’s work is performed for an external partner, they are identified as the Sponsor or the primary customer of HFE. The Contractor shall identify all key stakeholders for executed studies.

Tasks may require Access to local and national VHA Data, such as Protected Health Information, Sensitive Personal Information and Personally Identifiable Information, Individually Identifiable Information (PHI/SPI/PII/III) during the course of projects, for multiple purposes as an Administrative or Special User including but not limited to:

quality assessment and improvement, operational reporting, compliance and oversight, and general administrative duties not generally associated with clinical care. The Contractor shall ensure all contractor-provided resources are covered under a Business Associate Agreement (BAA) as a covered entity.

The initiatives supported range in size and duration from a single assessment of a wireframe or prototype to an ongoing, long-term engagement providing human factors support for legacy VistA and CPRS workflows and improvements. While TO deliverables are expected to be actionable and highly operational in nature, some of them may also be synthesized with other data and used to inform strategic direction.

HFE has a defined set of processes, methods (services) and reports across key areas of engagement and strives for continuous improvement. The Contractor shall execute human factors engineering activities in accordance with approved HFE office methods, processes, and report formats as documented in the HFE UX Guide and HFE Program SharePoint.

The Contractor’s level of involvement on projects and studies shall vary, as the Contractor shall support the overall HFE team and complete portions of study activities in support of the overarching objectives. The Contractor shall provide a Study Lead and/or a Study Support role for individual studies as agreed upon with the Government HFE Lead. The Contractor shall collaborate with other HFE Resources on each assigned study activity, which may include Government staff, other contractors, and Intergovernmental Personal Assignment (IPA) staff.

The Contractor shall propose use of an industry standard, or minimally, a recommended approach supported by published evidence when an existing HFE process is not yet established. The Contractor shall perform study kick-off meetings to document the context of need for a study request for any assigned study activity, and load the document as a Working Product. From the study kick-off meeting and other related information, the Contractor shall propose to the HFE Government Lead a Functional Point Estimate (FPE) for the scope of each assigned task, while referencing the Government provided Nominal Descriptions to support the FPE. The Contractor shall confirm acceptance with the HFE Government Lead of the agreed-upon Functional Points in the Study Proposal review. The Contractor shall create and include a proposal

Courtney, Mary G. (OGC) What does this mean and how does it impact both the work performed as well as the price. There is nothing in this PWS that defines how an FPE is calculated.

Attachment that details ‘nominal descriptions’ provides the calculation examples.

(concurred on by the HFE Government Lead and sponsor before work can begin), a study plan, and the study report for each Study Package, each of which the Contractor shall upload to the HFE Effort Tracker SharePoint (VA Network URL provided upon award, referred to as Effort Tracker throughout) as created. The Effort Tracker is the single authoritative project and study repository. The Contractor shall perform and document White Glove (WG) reviews and formal Peer Reviews (PR) on assigned work activities, and submit as a Working Product to the relevant Effort Tracker entry. The Study Package shall be provided for task performed in PWS 5.2, 5.3, 5.4 and 5.5. For taskings focused on clinical process analysis and redesign, the deliverables will come in a Clinical Study Package, maintaining the same format but separated for fiscal tracking purposes.

The Study Proposal shall minimally include:

1) Introduction

a) Provide an overview of the study being proposed. Be sure to include the intent of the study.

2) Objectives

a) Describe the purpose of the study

b) Mention what is not in scope or evaluated

3) Study Design

a) Include the study method used

b) If possible, provide a link to the method details

c) Provide a brief description of the test data being used, test measures, and study participants

d) Describe the high level expected results of the study

4) Schedule (High Level)

a) Create the high level schedule and include the logical phases

b) Describe the duration of the project and any schedule constraints (i.e. holidays or other events)

5) Assumptions/Dependencies

a) Describe any organizational or study dependencies

b) Consider any developer or contractor requirements for supporting study efforts

c) Identify what assumptions are being made about the study and supporting elements.

6) Concurrence

a) Collected via email, with signatures and date

b) Concurrence indicates that both HFE and the sponsor agree on the objectives of the study before work begins to ensure that the results of the study address program office needs, including timeline

As the Contractor shall collaborate with other HFE Resources, some studies may be executed where the Study Proposal was crafted by another HFE Resource, but all related study materials shall be loaded to the same Effort Tracker entry.

The Study Plan shall minimally include:

Courtney, Mary G. (OGC) Is all this information setting up how the work is to be performed and 5.2,5.3,.5.4 and 5.5 is the actual work?

Kurt Ruark Yes, these are the requirements applied throughout.

Courtney, Mary G. (OGC) Earlier you state all task orders will be 12 months, yet each study will vary in the timeline depending on what it is.

PoP section above changed to define TO PoP at the TO level

1) Purpose of Study

a) Study Objectives

b) Applicable Methods, tools, and resources

2) Session script(s) or questions for users (questionnaire)

a) Target number and types of participants (for various user roles)

3) Data collection and analysis approach

a) Task success criteria if performing tasks

4) Project Timeline and Milestones

a) Sponsor connection points to review progress, risks, and responsibilities

5) Other project-specific parameters, such as:

a) Test procedures

b) Scenarios

c) Personas

The Study Report format (visual or written) shall vary depending on the study type (such as wireframes for design related studies) and customer preferences and can range from an Adobe InDesign, Adobe PDF, Excel workbook, a PowerPoint presentation, a Word document, diagrams, web content or other visual format. Regardless of format, the Contractor shall provide deliverables that are 508 Accessibility compliant and text editable by the Government, including reusable components for future work. All materials, including written and visual products produced by the Contractor shall be owned by the Government and be submitted using approved templates or branding.

Studies including briefings and disposition of findings are expected to be associated with a PowerPoint presentation to support collaborative decision-making to identify solutions to usability problems. Studies including data analysis and collection of numerous quantifiable information elements are expected to be associated with an Excel workbook providing raw data and analyses summaries. The HFE minimum expectation for written reports shall follow this structure, adapted from the Human Factors and Ergonomics Society (HFES) conference proposal requirements (http://cms.hfes.org/Events/Annual-Meeting/Call-for-Proposals.aspx#Forms), and are expected to be associated with a Word document:

• Introduction: General statement to orient the reader to the specific problem and context, study personnel, background information, and relevant literature.

• Method: How the study was conducted, including participants, apparatus, and procedure (e.g., any tasks completed by participants).

• Results: Summarize the data collected, list any findings1, and/or include visual products2 such as mock-ups.

• Discussion: Describe your analysis of the results, the implications, pertinent evidence-based recommendations, and study limitations and next steps.

• References: Provide full references for all citations used in the proposal in current edition American Psychological Association (APA) style.

The Contractor shall acquire agreement from the Government HFE Lead and Sponsor on format prior to execution and identify and communicate the key characteristics http://cms.hfes.org/Events/Annual-Meeting/Call-for-Proposals.aspx#Forms) required to meet each stakeholder’s objectives for the study. For internal HFE operations work, the Government HFE Lead will determine if study proposals and plans are required in addition to the work item (visual or written report) itself. The Contractor shall document all initiated work, provide updates, and load the final Study Package in the HFE Effort Tracker SharePoint. The Contractor shall deliver the Government HFE Lead approved proposal, study plan, and study report to the Sponsor, and perform a study debriefing call to answer questions should the Sponsor require. Sponsors have 30 days to review the study report before studies are marked as ‘Completed’. Studies are scheduled once the study plan is approved by the Government HFE Lead. A study plan is established after Government HFE Lead and Sponsor concurrence of the study proposal is acquired.

The Contractor shall also perform an After-Action Review (AAR) and submit entries to the Lessons Learned Journal (LLJ) for all applicable opportunities for improvement for each study activity which the Contractor was actively engaged, including facilitating the AAR meeting, and document the materials in a Working Product, and include the study sponsor in the session.

1 The HFE basic definition of a finding has been adopted from the Nielsen Norman Group’s, to include Takeaway, Prioritization, Description, Participant quote, Analysis, Evidence Based Recommendation, and Ownership and Measurement. (Nielsen Norman Group. (2017). Anatomy of a Usability Finding. UX Deliverables) 2 Visual products are as described elsewhere in the PWS.

Courtney, Mary G. (OGC) Where are these listed as deliverables?

Kurt Ruark Under sub sections

Courtney, Mary G. (OGC) Is this a deliverable?

For each study, as applicable, on the Government SP site.

The AAR meeting shall make use of the proposal as a basis for discussion. Suggested questions for brainstorming include:

• Were we able to meet the objectives? Why or why not?

• Did we have the right stakeholders identified initially? If not, what might have allowed for that upfront?

• Were there technology issues that were preventable? If so, how?

• Was our mitigation of risks successful?

• Did our assumptions pan out?

• Did scope creep affect our outcomes? Could that have been avoided? If so, how?

• Were we able to keep to our original schedule? If not, why? Were there actions that could have been done to keep to the schedule better?

• Was communication within the team such that everyone was aware of the status, timeline, and issues?

The LLJ is a SharePoint list maintained on HFE’s main SharePoint site that the Contractor shall submit the title, problem, and recommendations for each lesson learned. The Contractor shall be responsible for ensuring all materials are loaded to the Effort Tracker and the LLJ, and shall notify the Government HFE Lead of known gaps of content, sourced or attributed as work products from other entities. The Effort Tracker entry is not marked complete until the LLJ content submitted, if applicable.

LLJ field Description Notes

Title Give the lesson a descriptive title.

Problem

Describe the problem encountered. Why was it a problem? What were the (potential) effects on the project/study? Was any course of action pursued during the execution phase?

Was anything tried that didn’t work?

Recommendations

Provide a recommendation from the project/study team to avoid the problem in the future.

Problem: The discussion shall consider both positive (“do more of this”) and negative (“avoid this in future”) lessons. For example, if you successfully mitigated a risk or

Courtney, Mary G. (OGC) You need something here to tell me what this is. Is this a format for something that the contractor is to follow? Switched the paragraph that identifies LLJ to precede this Table to give context to it.

Yes, format/process. The Contractor’s submission goes in this column.

effectively navigated a timeline change, please share those as well.

Recommendations: Recommendations shall be detailed enough that the Practice Committee or other Practitioners can put them to use. For example, if the team feels an item should be added to the study checklist, provide verbiage for that item.

Recommendations can be ideas for avoiding a problem in the future or they can reflect a good practice that mitigated a problem during the execution phase.

Study Package deliverables are publicly accessible on the VA Network, while Working Products (e.g., drafts, AARs, study kick-offs) are only accessible to HFE Resources on the Effort Tracker.

5.1 PROJECT MANAGEMENT

5.1.1 CONTRACTOR PROJECT MANAGEMENT PLAN

The Contractor shall prepare and deliver a Contractor Project Management Plan (CPMP) that lays out the Contractor’s approach, timeline and tools to be used in execution of this TO effort. The CPMP shall describe the Contractor’s plans to support HFE studies and projects. The CPMP shall incorporate the tasks described in the PWS.

The CPMP should take the form of both a narrative and graphic format that displays the schedule, milestones, risks and resource support. The CPMP shall also include how the Contractor shall coordinate and execute planned, routine, and ad hoc data collection reporting requests as identified within the PWS. The initial baseline CPMP shall be concurred upon and updated in accordance with Section B of the Contract. The Contractor shall update and maintain the VA Program Manager (PgM) approved CPMP throughout the PoP. The Contractor shall identify the primary contact point for all programmatic issues/concerns/status.

The Contractor shall document throughout the PoP and communicate weekly the decisions, expectations (assumptions), risks, actions, and issues log (DERAIL) aligned with each activity during study planning, execution, and closing phases. The Contractor shall establish and maintain a recurring Communication Schedule with stakeholders aligned with each assigned activity’s objectives. The DERAIL and Communication Schedule shall be maintained on the Effort Tracker by the Contractor as a Working Product.

The Contractor shall expeditiously undergo sufficient background checks and complete required training in order to be granted access to VA’s network and key resources used and managed by the HFE team within three (3) business days of receiving the notice of award. The Contractor shall communicate on an on-going manner the current status of contracting on-boarding including; last actions taken, next steps, and anticipated completion dates.

The CPMP shall also include a labor category staff roster identifying the following:

1. Employee Name

2. Personnel PIV Expiration

3. Personnel clearance level and adjudication date

4. Role

5. Project affiliation and assigned activities log

6. Personnel duty location address, alternate email, phone number

The Contractor shall continuously monitor performance and report any deviation to the COR or VA PgM.

Deliverable:

A. Contractor Project Management Plan

5.1.2 REPORTING REQUIREMENTS

The Contractor shall participate in, coordinate portions of, or lead meetings and briefings on assigned projects and studies while minimally documenting summary updates on the Effort Tracker. HFE is commonly a representative on an integrated project team, voicing the HCD approach and human factor expertise to a wide audience. Since the frequency and volume of associated meetings will differ by study, project, and engagement, the depth of reporting is variable. The Contractor shall maintain all materials created for individual studies as Working Products on the Effort Tracker.

During study execution, updates will be kept in the DERAIL, updated on the Effort Tracker as a Working Product that is maintained throughout project execution for each assigned project or study. The Contractor shall participate in joint reviews, track action items, schedule resources, and summarize technical and/or administrative problems encountered with recommendations for remediation. Each meeting or briefing will typically last one hour or less and most often be performed remotely via Skype/Lync, occurring once every week or every other week per project, depending on individual study complexity. The Contractor shall attend meetings as members of project or study teams and coordinate the discovery (scope definition), preparation (kick-off), execution (status) and closeout briefings for HFE studies during the project lifecycle. On all assigned recurring meetings, the Contractor shall be actively engaged, offering expert advice, suggestions, and recommendations on HFE issues as appropriate, including agreed upon timely responses to stakeholders for action items. On all assigned HFE coordinated briefings, the Contractor shall ensure the appropriate invites have been sent, a recommended agenda crafted, any materials shared at least two business days in advance (unless specified otherwise in the PWS), and appropriate follow-up on action items defined and distributed. The Contractor shall not commit the HFE Government team to additional studies or scope, but report such requests during the weekly Workload Assessment Meeting (WAM) and ensure potential requests for HFE services are forwarded to the appropriate channel (HFE group email to be provided upon award).

The Contractor shall summarize engagements with study activities via the Monthly Performance Report (MPR) that documents activities related to current studies, as well as anticipated upcoming engagements that are either planned or in the backlog. The Contractor shall ensure any impediments to ongoing work are readily visible and communicated. The Contractor shall seek remedies to impediments relative to their impact and urgency as appropriate. The Contractor shall communicate the impediments as they occur, not waiting for the monthly report, and shall track resolutions in the MPR.

The Contractor shall document the Functional Points assigned to active work, previously assigned Functional Points to completed work, and remaining Functional Points available on the Contract at each WAM.

The Contractor shall lead a WAM with the VA PgM each week for project summary and status discussions in preparation for the MPR submission. The weekly WAM shall include the following information:

1. Overall high-level assessment of study/project progress

2. Works in progress (WIP) with planned Study Package deliverables

3. Studies or projects recently completed since the last meeting

4. Planned studies or projects to begin in the next two (2) weeks, when applicable

5. Barriers to progress or inefficiencies experienced

6. New assignments

The MPR shall incorporate the prior WAM details for the prior full month’s efforts, including corresponding changes made during the current reporting period, such as “Original target date for study report MM/DD/YYYY revised to MM/DD/YYYY on MM/DD/YYYY because of … concurred upon by HFE Government Lead”. Government feedback on prior reports shall be addressed in follow-on deliveries.

A. Monthly Performance Report

5.1.3 TECHNICAL KICKOFF MEETING

The Contractor shall hold a technical kickoff meeting within 10 days after award. The Contractor shall present, for review and approval by the Government, the details of the intended approach, work plan, and project schedule for each effort. The Contractor shall specify dates, location (expected to be virtual), agenda (shall be provided to all attendees at least five (5) calendar days prior to the meeting), and meeting minutes including all presented materials (shall be provided to all attendees within three (3) business days after the meeting). The Contractor shall invite the Contracting Officer (CO), Contract Specialist (CS), COR, the VA PgM, and the HFE Government Lead.

A. Kickoff Agenda B. Kickoff Meeting Minutes

5.2 HUMAN FACTORS ENGINEERING SERVICES

The Contractor shall support the HFE office in the provision of Human Factors Engineering Services to the numerous customers and sponsors engaged with OHI. The contractor shall support the design, development, testing, and implementation of various clinician-facing and Veteran-facing health IT applications including, but not limited to, electronic health record systems, clinical reminders, consults, templates, order sets, alerts, patient discharge summaries, patient websites, print materials, and mobile applications. For some projects, the Contractor shall support the re-engineering of the clinical workflow or care delivery process into which the health IT applications must integrate. The Contractor shall provide support services across the project lifecycle utilizing the HCD approach, and at times provide services that spread across numerous activities concurrently, including: Specifying HIS Context of Use and User Requirements (5.2.1); Designing HIS Solutions (5.2.2); and Evaluating HIS Usability (5.2.3).

Regardless of the effort, the Contractor shall provide Human Factors Engineering Services that aim to optimize overall system performance using the scientific method.

The Contractor shall consider and assess factors such as cognition, distractions, physical demands, and environments impact on patient care. The Contractor shall apply the definition of ‘cognition’ as the mental processes that enable users’ memory, attention, and abilities to make decisions and problem solve. The Contractor shall consider the process limitations and recommend changes to improve cognition.

Distractions are unavoidable in healthcare environments, and the Contractor shall assess means of limiting elements that divert attention from a user’s focus. Physical demands include the stressors that inhibit physical and cognitive abilities and technical skills, for which the Contractor shall assess the physical demands users experience and their impacts on task performance during workflow analysis. The Contractor shall incorporate these elements into their findings and recommendations. The Contractor shall assess environmental factors associated with the care setting, including temperature, sound and lighting. The Contractor shall conduct human factors assessments that consider the environmental conditions of the care setting and its impact on safe and effective delivery.

The Contractor shall perform an analysis of targeted health information systems before implementation of a technology solution into a complex, safety-critical, healthcare delivery organization.

The Contractor shall perform requirements gathering for HIS that includes understanding and specifying the context of system use. The Contractor shall analyze clinical workflows, and shall analyze user jobs, tasks, and cognitive work (utilizing methods appropriate for improving patient-centered care). For example, cognitive task analysis is one family of methods that can be used to reveal the thinking involved in performing tasks in real-world contexts, and is well suited to understanding aspects of patient-centered care. The Contractor shall conduct various types of Time and Motion Studies to investigate and baseline current human-computer system interactions. The Contractor shall identify the Socio-technical factors likely to impact HIS usability, and those commonly addressed in system re-engineering using, for example, published practice guidelines for EHR implementation.

The Contractor shall support clinical workflow re-engineering and HIS design/customization activities to standardize care across VHA and between legacy and modernized EHR systems. The Contractor shall apply evidence-based guidance to engineer workflows that standardize clinical protocol, terminology, and processes to support team-based patient care, clinical decision-making, error tolerance, and system reliance. The Contractor shall apply evidence-based guidance to engineer workflows that enable knowledge management and the utilization of clinical data to inform care improvement and interoperability between VA Health Care systems and with the community. The Contractor shall perform an iterative design and assessment process to optimize workflow efficiency by streamlining clinical tasks and documentation, maintaining clinical data integrity and context, and minimizing workflow redundancies.

The Contractor shall follow industry practices for evaluating, testing, and validating the usability, physical and cognitive ergonomics of the EHR, HIS and other work systems in a holistic manner. The Contractor shall benchmark user performance measures on the legacy system and iterate the design and configuration of the new UI and work system as appropriate. The Contractor shall assess the HIS for compliance with evidence-based standards to support usability and to support effective clinician communication.

The Contractor shall utilize and refine methods to evaluate and optimize the human-system interactions of a planned or deployed HIS.

The Contractor shall lead, manage, monitor, control and execute all aspects of a study;

define test goals, recruit and screen participants, establish test plans and scripts, conduct assessments, perform data analysis, create reports, communicate findings, and make design and functionality recommendations to improve performance and system reliability. The Contractor shall adhere to published, evidence-based practices necessary to implement safe, highly usable HIS that supports effective and efficient delivery of patient-centered healthcare. “Published” or “evidenced-based” refers to materials published in a peer-reviewed journal, a human factors textbook, or by non-profit or governmental organization with a central mission to improve biomedical or health technology (i.e., AAMI, AMIA, ONC, HIMSS, etc.). Much of this work will include the novel application of HF related professional engineering practices and operations research functions on highly complex problems plaguing the health care system in both a tactical and strategic manner. The Contractor shall consult with users, business leads, user experience and user interface team members to lead, plan and execute a variety of studies and assessments including usability studies, ethnographic studies, error analysis, and cognitive walkthroughs, among others.

The Contractor shall apply human factors methods and analysis, mastering the fundamental concepts of observational, experimental, and correlational research methods that include both qualitative and quantitative data. The Contractor shall have expertise and skill applying statistical analysis, interviews, contextual inquires, focus groups, inspections, evaluations, survey development, rapid ethnography, systems design, data coding and literature reviews. The Contractor shall identify and define the gaps, risks, limitations, system constraints, functionalities, findings, lessons, and factors contributing to the productivity, comfort, motivation, safety and satisfaction of users.

The Contractor shall apply a multi-disciplinary approach to human factors testing, with knowledge from HF, HCI, usability, ethnography, and cognitive, social and organizational psychology to produce assessments and reports, and other elements of the study package as appropriate to specified study objectives.

Most studies shall be conducted virtually using the VA network and internet resources, although some studies shall be conducted at VA Medical Centers (VAMCs) or Community-Based Outpatient…