SB1341-15-RQ-0505_Attachment_1_Voluntary_Voting_System_Guidelines_Performance_Work_Statement_(Ref._15-03073).pdf

Performance Work Statement Request for Quotation: SB1341‐15‐RQ‐0505

A. BACKGROUND

The mission of the National Institute of Standards and Technology (NIST), Software and Systems Division (SSD) is to develop software testing tools and methods that improve quality, conformance to standards and correctness. The Division also participates with industry in the development of forward‐looking standards relative to the healthcare industry. The Division's programs also respond to industry, consortia, and standards groups’ needs by concentrating on key areas at the forefront of technology and assisting in the transfer of technology and expertise to industry.

This task order seeks to contribute to the general state of the art for voting systems as well as contribute to the healthcare industry specifically, by advancing healthcare information standards that are complete and testable and by providing documentation of conformance tests and test tools. There is a real need for formal and informal coordination of these efforts to leverage the synergy of the various efforts, to harmonize documentation to promote consistent testing and certification programs across and within organizations.

The Voluntary Voting System Guidelines (VVSG) is a requirements specification for voting systems that contains requirements and guidance both for voting system manufacturers and for Voting System Test Laboratories (VSTLs ). The VVSG is used by many states to define how voting systems must operate during an election.

Further, conformance testing to the VVSG is required by these states and is often achieved through the Election Assistance Commission’s (EAC) voluntary conformance testing program whereby a manufacturer contracts with a VSTL to test a voting system, which consists of a series of voting devices that interoperate to provide a defined level of capability for conducting elections. Voting systems that pass the conformance tests are certified by the EAC. See www.eac.gov for further information.

The NIST SSD requires project leadership and technical staff support from the Contractor in the areas of standards and technology development and adoption, measurement, and development of test methodologies and software testing tools to improve the quality and correctness of software used in our nation’s critical infrastructures, such as Healthcare and Voting Systems. This task order focuses on testing for Meaningful Use of Healthcare technologies including documentation. The Health IT Standards Testing is a collaborative effort to help ensure interoperability among all healthcare providers, hospitals, clinical laboratories, pharmacies, payers, medical device manufacturers and others via the meaningful use and healthcare domain criteria. NIST has developed testing tools to support healthcare domain groups, the meaningful use testing criteria, as well as healthcare domains which have not yet (i.e., via ‘stages’) been identified within the meaningful use directives.

We require the Contractor to furnish advanced research and development expertise in standards and software testing to assist (SSD) in providing leadership and technical expertise in analyzing the VVSG specification, development of test assertions, harmonization of these test assertions across VSTL’s and voting system manufacturers, and mapping specific VSTL and NIST developed tests to the test assertions. The mission of the (SSD) Division is to develop software testing tools and methods that improve quality, conformance to standards and correctness. The Division's programs also respond to industry, consortia, and standards groups needs by concentrating on key areas at the forefront of technology and assisting in the transfer of technology and expertise to industry.

B. PROPOSED RESEARCH AREAS AND TASKS

1. Task 1: Meaningful Use Test Tool Support

a. Task 1A: Meaningful Use Conformance Tool Documentation

The Contractor shall support NIST SSD in documenting the conformance tools and Architecture/Workflow being developed for the following tools used in the Meaningful Use Certification program:

IGAMT Tool ‐ Develop/Update documentation for Implementation Guide Authoring and management Tool (IGAMT) o Draw/revise architecture and workflow diagrams o Update User’s Guide

Edge transport Testing ‐ Modify documentation for Edge Testing Tool o Draw/revise architecture and workflow diagrams o Update User’s Guide

Laboratory Orders/Results Tools o Draw/revise architecture and workflow diagrams o Update User’s Guides

Immunization Tool o Draw/revise architecture and workflow diagrams o Update User’s Guide

Electronic Delivery of Service (eDOS) Tool o Draw/revise architecture and workflow diagrams o Update User’s Guide

Deliverables:

Documentation of Laboratory Orders and Results Tools Architecture and workflow diagrams.

Documentation of the Electronic Delivery of Service (eDOS) and workflow diagrams.

Documentation of the Implementation Guide Authoring and Management Tool (IGAMT).

Documentation of the Edge Testing Tool

Formatting and incorporation of documentation for conformance tools into “test procedure packets” for appropriate test procedures for public posting (in the same format as MU Stage 2 test procedures were posted at: http://www.healthit.gov/policy‐researchers‐implementers/2014‐ edition‐final‐test‐method).

Meeting with NIST technical staff monthly or more often to learn about conformance tools, review documentation and support other project needs

Development of agendas, summary PowerPoint presentations and other materials in support of status update meetings with NIST, ONC and other stakeholders on a monthly basis for the duration of the project.

b. Task 1B: Organize and Support Monthly Status Update Meetings The Contractor shall schedule and organize monthly status update meetings with NIST and other applicable stakeholders. The Contractor shall prepare agendas, summary PowerPoint presentations and other materials in support of the meetings. In addition, the Contractor shall provide the COR with a monthly status report outlining project deliverable status, funds expended and planned activities (for the upcoming month). Meetings will be a combination of in‐person (at NIST) and virtual, to be determined.

The Contractor shall deliver the following:

Up to monthly in‐person and virtual meetings for the duration of the project

Hard and soft‐copies of agendas, summary PowerPoints and other materials to support meetings (distributed at least 24 hours prior to the meetings).

Monthly update for COTR delivered via email during the 1st week of the month

2. Task 2: Direct Implementation and Certificate Authority Support NIST SSD is collaborating with industry, standards organizations, consortia, and government agencies to build tools and prototypes to advance the adoption of Information Technology (IT) within healthcare systems.

As requested, the Contractor shall support NIST SSD in the collaboration with industry, healthcare informatics‐ related standards organizations, consortia, and government agencies to build tools and prototypes to advance the adoption of IT within healthcare systems. Specific objectives include:

• Maintain the NIST version of the Direct Reference implementation installed on a NIST server.

• Maintain the NIST Certificate Authority CA.nist.gov, installed on a NIST server.

a. Task 2A: Support the Meaningful Use Certificate Authority (CA.nist.gov) To support the testing of Electronic Health records, NIST is acting as a Certificate Authority and issuing certificates for testing, we have installed the software for this tool on a NIST server (hit‐ testing.nist.gov).

The Contractor shall assist NIST staff in supporting the tool. The Contractor shall assist NIST staff to:

• Receive and respond to requests from the public inquiries on the tool via the Google Group set up by NIST (https://groups.google.com/d/forum/transport‐testing‐tool)

• Help NIST staff with Java programming support of the tool.

• Provide bug fixes to the tool based on industry or NIST input.

• Provide any needed modifications to the tool based on industry/NIST input.

The Contractor shall deliver/maintain a working version of the Certificate Authority (CA) and source code, to be maintained on the NIST server.

b. Task 2B: Support the Meaningful Use instance of Direct for Test Tool Validation The Direct Project focuses on the technical standards and services necessary to securely push content from a sender to a receiver and not the actual content exchanged. However, when these services are used by providers and organizations to transport and share qualifying clinical content, the combination of content and Direct‐Project‐specified transport standards may satisfy some Stage 2014 Meaningful Use requirements. For example, a primary care physician who is referring a patient to a specialist can use the Direct Project to provide a clinical summary of that patient to the specialist and to receive a summary of the consultation. The NIST Transport Testing Tool verifies an implementation of Direct complies with the Direct Specification. NIST requires and implementation of the Direct Reference implementation from the Direct Project(http://www.healthit.gov/policy‐researchers‐implementers/direct‐ project) and (http://wiki.directproject.org/) be installed to use as a test instance for the NIST Transport Testing Tool.

The Contractor shall assist NIST staff in maintaining the Direct Instance. The Contractor shall assist NIST staff to:

Maintain the Direct Reference Implementation on a NIST server (hit‐testing.nist.gov) to be used for testing against NIST Transport Test Tool.

Help NIST staff with Java programming support and configuration of Direct.

Provide any needed modifications to Direct based on industry input.

Update Direct to current version at NIST direction (may need to stay below latest version for technical reasons)

Test this Direct instance Monthly to make sure it is in proper working order by sending and receiving messages to the NIST Transport Testing Tool.

Meet/Email NIST project leader once per month on status of tools.

3. Task 3: Test Assertion Framework for the Voluntary Voting System Guidelines (VVSG) The NIST Voting Test Assertion Framework (TAF) provides conformance requirements, or assertions, for the Voluntary Voting Systems Guidelines (VVSG). These assertions serve the purpose of unambiguously stating how each specification requirement or, as applicable, sections of specification requirements in the VVSG are to be tested. This will, in turn, provide a framework for organizing the device‐specific tests developed by EAC‐ accredited Voting System Test Laboratories (VSTLs). In general, each assertion may link to one or more device‐ specific tests. Each assertion will describe in plain language how a specification requirement(s) is to be tested while the VSTL‐developed test(s) that are linked will show how the testing will be applicable to and carried out for specific voting devices.

Conflicts can arise over the interpretation of VVSG specification requirements between the VSTLs, the EAC, and manufacturers who have submitted voting systems under test. This can be further complicated if different VSTLs interpret the same specification requirements in different ways. It is the test assertions, therefore, that will provide the common understanding of the VVSG specification requirements and their associated testability, and serve as a “bridge” between the specification requirements and the VSTL‐developed device‐specific tests.

See vote.nist.gov for further information.

The NIST Software and Systems Division (SSD) is requesting project leadership and technical staff support from the Contractor in developing and analyzing the new VVSG specification, developing test assertions, and harmonizing these test assertions across VSTL’s and voting system manufacturers.

A key goal of these efforts is the development of new guidelines, harmonized test assertions, and a mapping to associated test methods and ultimately to the tests that are used in conformance testing programs for voting systems. NIST works with the EAC, election officials, advocacy groups, academia, the VSTL’s, and voting system manufacturers in the development of guidelines and conformance testing resources for these efforts.

The Contractor shall provide technical leadership in harmonizing these past efforts on the development of the VVSG and participate in the development of a new version of the VVSG. In addition, the Contractor shall participate in the development of new test assertions and an effort to update existing test assertions to be consistent with the recently approved VVSG 1.1. These test methods shall also be integrated with those used by the National Voluntary Laboratory Accreditation Program (NVLAP) which recommends qualified voting system test laboratories for accreditation by the EAC.

The revised VVSG must take into account on‐going work in the development of a NIST Roadmap for Usability and Accessibility, input from the EAC Future VVSG working group, input from the National Association of State Election Directors (NASED) subcommittee on Future VVSG, and input from the Technical Guidelines Development Committee. Recent activities across these groups calls for a restructuring of the VVSG that allows for the use of commercial‐off‐the‐shelf components, component‐based testing, and innovative solutions to voting. The resultant test assertions must unambiguously describe how specific requirements or associated collections of requirements in the VVSG are to be interpreted for the purposes of testing, and to provide a linkage to the VSTL‐developed device‐specific tests.

The VSTLs will be responsible for associating their device‐specific tests with the test assertions. NIST has already begun development of a web‐based test assertion framework; the Contractor will be responsible for developing, harmonizing, and populating test assertions for this framework.

There has been prior work in the area of test assertions and a test assertion framework for the VVSG, which is available at a website accessible at a NIST location in Gaithersburg, MD. NIST has also developed tests for the requirements in the 2007 TGDC Recommendations to the EAC; these tests may be applicable in the development of test assertions in that many of the requirements in the 2007 document are restatements of requirements in the existing EAC VVSG. The 2007 TGDC Recommendations also included limited requirements for electronic pollbooks that may be applicable to full requirement analysis and development. See www.eac.gov for further information on the 2007 TGDC Recommendations.

The Contractor shall review other standards from W3C, OASIS, IEEE, and ISO, and recommend a structure for the VVSG that is consistent with overall objectives of the constituents of the voting community. In addition, the Contractor shall develop additional sets of test assertions that correspond to specific VVSG requirements on a one‐to‐one basis or, as determined by the contractor, correspond to 2 or more interrelated VVSG requirements. In addition, the Contractor shall contribute to the harmonization of the test assertions across VSTL’s, the EAC, NIST, and voting system manufacturers. The Contractor shall report on parts of the VVSG that are ambiguous and/or not fully specified.

a. Task 3A: Develop test assertions that correspond to VVSG 1.1 requirements.

1. Develop a prioritized three‐phased plan for remaining test assertion development for VVSG 1.1, Volume I, including performance, functional, hardware, software, and telecommunications.

2. Develop phase 1 test assertions.

3. Develop phase 2 test assertions.

4. Develop phase 3 test assertions.

Detailed three‐phase plan is due within 30 days of date of the award, followed by phase 1 assertions due 4 months from date of award, phase 2 assertions due 8 months from date of award and phase 3 assertions due 12 months from the date of award.

• Microsoft Word document detailing a prioritized three‐phase plan for further development of test assertions.

• Harmonized XML‐encoded document of test assertions for phase 1 test assertions.

• Harmonized XML‐encoded document of test assertions for phase 2 test assertions.

• Harmonized XML‐encoded document of test assertions for phase 3 test assertions.

b. Task 3B: Test Assertion Harmonization

1. Participate in testing discussions (TCons and F2F/live; may involve travel) – as approved by the

COR to harmonize test assertions with EAC, NIST, VSTL’s, and voting system manufacturers.

2. Present test assertions and relevant background information to describe use cases in testing discussions and as feedback into the standards setting process.

3. Develop and deliver professional findings, recommendations, and status in various formats

(e.g., MS PowerPoint, MS Word, etc.).

4. Contribute toward research/findings papers and proposals to advance Voting systems testing.

5. Propose and make recommendations to project advancements (ad‐hoc as knowledge base grows) to project lead (and team members as appropriate)

• PowerPoint/Slide sets (as needed).

• Ad‐hoc Recommendation Documentation (MS Word)

• Ad‐hoc Documentation (MS Word) as needed

c. Task 3C: Develop tools for a revised VVSG

1. Review existing standards templates from W3C, OASIS, IEEE, and ISO, and recommend a method for developing a revised VVSG that allows mandatory/optional components, the use of COTS hardware, and innovation clauses.

2. Review new technical material and compare with existing VVSG 1.1. Make recommendations revising the existing VVSG, highlighting consistency issues.

• MS‐Word document recommended a template approach to a revised VVSG.

• Ad‐hoc documentation (MS Word/PowerPoint) on best practices for modifying the VVSG.

C. PERIOD OF PERFORMANCE

Period of Performance is September 30, 2015 through September 29, 2016.

D. CONTRACTOR QUALIFICATIONS

All Contractor Key Personnel must meet the following minimum qualifications per stated for the specified task.

Task 1: Research Scientist The Contractor must have formal training and/or demonstrated experience in the following areas:

• More than five (5) years of experience as a Technical Writer

• Prior experience documenting conformance tools, specific to Healthcare.

• More than five (5) years of experience with Systems Requirements Gathering and Document Creation

• Prior experience writing User Guides for Software Testing Tools

• Test Plan Development Experience

• Prior experience on Edge Testing tool Documentation

• Experience with the NIST conformance Tools

Task 2: Research Scientist Contractor Personnel assigned to this task shall possess the following minimum qualifications:

Minimum of 3 years of experience with Direct Messaging

Minimum of 3 years of experience with django.

Minimum of 3 years working with Direct Certificates.

Minimum of 2 years of experience with NIST Certificate Authority Tool (CA.nist.gov)

Minimum of 2 years of experience as a Direct Expert

Minimum of 3 years of experience with OpenSSL

Minimum of 3 years of experience with Django and Python

Minimum of 3 years of experience in Certificates.

Minimum of 3 years of experience in Certificate authorities.

Minimum of 1 years working with NIST Direct Test Tool

Minimum of 3 years working with RedHat Linux OS

Minimum of 3 years working with Apache Web server

Task 3: Research Scientist Contractor Personnel assigned to this task shall possess the following minimum qualifications:

It is expected that this task order will require the work of one Contractor employee working on a part basis.

The estimated level of effort required to successfully complete the tasks outlined in this document is approximately 20 hours per week for the contractor key personnel.

The staff proposed for this task must have formal training in the following areas or demonstrated experience in the areas. Contractor will submit resumes of proposed staff.

10 or more years’ experience in the management and development of standards and guidelines for IT products

10 or more years’ experience in EAC conformity testing evaluation for voting devices and voting systems including in‐depth knowledge of VVSG requirements

7 or more years’ experience in EAC VSTL‐specific testing programs and test procedures

THIS SPACE INTENTIONALLY LEFT BLANK

E. DELIVERABLES

Task

Number Description Format Qty.

Due Date / Frequency

Task 1A Documentation of Conformance Tools/Workflows and Architectures

MS Word, MS PowerPoint, and

HTML

6 Conformance Tool User Guides Power Point Deck for each tool

12 months from award

Task 1B Organize and Support Monthly Status Update Meetings

MS PowerPoint and In Person (when required)

Once per month Monthly

Task 2A Support the Meaningful Use Certificate Authority (CA.nist.gov)

Software and PowerPoint Presentation to Project Leader on software upgrades

Once per Month Monthly

Task 2B Support the Meaningful Use instance of Direct for Test Tool Validation

Software and PowerPoint Presentation to Project Leader on software upgrades Email

Once per Month Monthly

Task 3A Test Assertions

• Prioritized three‐phase approach MS Word 1 1 month after award

• Phase 1 assertions MS Word 1 4 months after award

• Phase 2 assertions MS Word 1 8 months after award

• Phase 3 assertions MS Word 1 12 months after award

Task 3B Test Assertion Harmonization

• Recommendations MS Word 1 12 months after award

• PowerPoint Slide Sets MS PowerPoint 1 12 months after award

• Monthly Presentation MS PowerPoint 1 Monthly

Task 3C Develop tools for a revised VVSG

• Recommended structure MS Word 1

1 month after award

• Best practices for revising VVSG MS Word 1 Monthly

F. PLACE OF PERFORMANCE

The Contractor’s staff must work primarily on‐site at NIST Gaithersburg but may also work by arrangement at the contractor’s primary business location, Task 2 may be entirely at contractors’ business location.

G. INSPECTION AND ACCEPTANCE CRITERIA

As duties and responsibilities are completed, the Contractor shall request review and inspection by the NIST COTR. Tasks completed in an unacceptable manner may require corrective action by the Contractor. Within 5 business days of receipt of deliverables, the NIST COTR shall provide the Contractor of any required revisions or deliverable acceptance.

H. GOVERNMENT FURNISHED PROPERTY AND RESPONSIBILITIES

The Government will provide to the Contractor through the NIST COTR:

1. Office space to include desk and chair

2. All computing hardware, software and networking resources.

3. Access to NIST campus during working hours; non‐working hour access may be granted on a need basis.

I. EXHIBITS AND ATTACHMENTS

None.

J. TRAVEL

Task 1 – No Travel Required Task 2 – No Travel Required Task 3 – Contractor shall attend 2 CONUS meetings to be determined by COR.

K. INVOICES

The Contractor may submit an invoice each month based on percent of satisfactory work completed on any or all tasks.

L. QUALITY ASSURANCE SURVEILLANCE PLAN

The purpose of this plan is to provide a quality assurance surveillance plan for the work to be conducted under this contract. The plan provides a basis for the NIST COTR to evaluate the quality of the Contractor's performance. The oversight outlined in this plan will help to ensure that the topics and assessment are maintained at the required levels through the Period of Performance. Further, this plan provides the NIST COTR with a proactive way to avoid unacceptable or deficient performance.

Performance Standards:

1. Quality Level: By monitoring the Contractor, the NIST COTR will determine whether the Performance Standards set forth in the Purchase Order have been attained. Performance Standards for all tasks are specified in the Performance Requirements Summary of the Performance Work Statement.

2. Frequency: During the performance of this Purchase Order, the NIST COTR takes periodic measurements (i.e., conduct surveillance), as specified, and will analyze whether the frequency of measurement is appropriate for the work being performed. Adjustments may only be made by a modification to the Contract.

3. Management Responsiveness: The NIST COTR will determine whether the Contractor has managed the Contract effectively and efficiently with successful completion of the topics and assessment as specified in the Performance Standards set forth in the Performance Requirements Summary.

M. PERFORMANCE REQUIREMENTS SUMMARY

Task 1A: Documentation of Conformance Tools Desired Output: Documentation of conformance tools in support of Interoperability Testbed that are approved by NIST; properly formatted for public availability and use by general public. Input provided to Project Leader through meetings, email and other discussion opportunities. Meetings attended as required. Phone and email discussions held as needed.

Monitoring Method: Reviewed by COTR.

Performance Standard: Documentation and diagrams reflect the Testbed process and detail all the steps that vendors need to support testing. The Contractor responds promptly to phone calls and emails and is in attendance at meetings. Any written communication is in grammatically correct English and delivered on time.

Task 1B: Organize and support monthly status update meetings Desired Output: Monthly meetings and materials that concisely summarize the status and any barriers to completion of the project. Monthly summary reports that keep the COTR appraised current and planned project progress and expenditures

Monitoring Method: Reviewed by COTR.

Performance Standard: Meeting materials and invitations distributed at least 24 hours prior to meeting (electronically); COTR received monthly status report during 1st week of the month. Meetings start and end on time.

Task 2A: Support the Meaningful Use Certificate Authority (CA.nist.gov) Desired Output: Application is up and running and checked each month for correct operation. Questions and emails are answered promptly. Input provided to TIC through meetings, email and other discussion opportunities. Meetings attended as required. Phone and email discussions held as needed.

Monitoring Method: Reviewed by COR/TIC.

Performance Standard: The Contractor responds promptly to phone calls and emails and is in attendance at meetings. Any written communication is in grammatically correct English and delivered on time in correct format.

Task 2B: Support the Meaningful Use instance of Direct for Test Tool Validation.

Desired Output: Application is up and running and checked each month for correct operation. Questions and emails are answered promptly. Input provided to TIC through meetings, email and other discussion opportunities. Meetings attended as required. Phone and email discussions held as needed.

Monitoring Method: Reviewed by COR/TIC.

Performance Standard: The Contractor responds promptly to phone calls and emails and is in attendance at meetings. Any written communication is in grammatically correct English and delivered on time in correct format.

Task 3A: Develop test assertions that correspond to VVSG 1.1 requirements.

Desired Output: For each of the tasks identified in Task 1, the contractor shall develop a report in MS Word format.

Performance Standard: The Contractor responds promptly to phone calls and emails and is in attendance at meetings. Any written communication is in grammatically correct English and delivered on time.

Monitoring Method: Reviewed by COR.

Task 3B: Test Assertion Harmonization.

Desired Output: Paper‐based representations in MS Word and MS PowerPoint of test assertions and associated explanations/use cases.

Performance Standard: The paper‐based standard is easy to read and grammatically correct.

Monitoring Method: Reviewed by COR/TIC.

Task 3C: Develop tools for revised VVSG.

Desired Output: For the tasks identified in Task 3, the contractor shall develop a report in either MS Word or PowerPoint format.

Performance Standard: The Contractor responds promptly to phone calls and emails and is in attendance at meetings. Any written communication is in grammatically correct English and delivered on time. The paper‐ based guideline is easy to read and grammatically correct.

Monitoring Method: Reviewed by COR.