Eai-SOW Cogito SOW 2021-v-mo.docx

Statement of Work Cogito Support and Integrated Professional Services

1. GENERAL INFORMATION

1.1. Scope of Work

NAL needs to maintain Cogito software licensing, maintenance and technical support, and integrated professional IT services for continued support and enhancement of automated indexing production and services, and quality control. This includes a requirement to maintain technical support for the software integration into NAL workflows for NALT subject indexing and NALT curation, including development, testing, and maintenance of Cogito skill cartridges and annotation plans for indexing production and quality goals to be met, and to support anticipated NAL indexing enterprise expansion and relevance.

1.2. Background

Cogito automated indexing software and its predecessor versions have been in use at the National Agricultural Library (NAL) since circa 2012. The software is fully integrated into the indexing production workflows as a vital component to achieve the annual goal of 0.5 million articles tagged with NAL Thesaurus (NALT) agricultural concepts (terms) for the PubAg and AGRICOLA databases. The Cogito system currently supports automated indexing with NALT terms for USDA funded technical journal articles which is critical to USDA’s goal for transparency and data and information access for USDA funded research. The system also supports NALT subject indexing for NAL collections and non-USDA citations from publisher supplied data streams, as well as other items such ARS research projects for the ARIS database search, and USDA Forest Service literature for TreeSearch database search. Cogito underpins the “NALT Annotator”, which is a GUI and API web-service hosted by Expert System Enterprise (Cogito developer) for automated indexing of text and data mining of possible new concepts for NALT. The NALT Annotator is used by NAL patrons, internal NAL teams, the USDA Office of the Chief Information Officer (USDA-OCIO), and AgNIC (Agriculture Network Information Center) partners, and others.

2. CONTRACTOR REQUIREMENTS

2.1. Software Licensing, Maintenance and Technical Support:

Software licensing, maintenance and technical support provides NAL with any technical assistance required during the period of performance to maintain, update, and operate Cogito software provided, i.e. Cogito Studio Express, Cogito Discover, and NALT Annotator. This includes support to correct technical issues or unexpected breakdowns, maintenance services for delivery of product fixes and upgrades, and configuration procedures required to run the software per the Cogito software documentation, and software integration and optimization within NAL workflows.

2.1.1. Licensing

2.1.1.1. License for minimum of 10 users for Cogito Studio Express and Cogito Discover applications using the latest versions available.

2.1.1.2. NALT Annotator continued hosting, maintenance, and support.

2.1.1.3. Timely upgrades, updates, patches, etc. to Cogito Studio Express and Cogito Discover software and all related modules and components as they are released by the vendor.

2.1.1.4. NAL shall receive any upgraded application software entering general customer release during this period.

2.1.1.5. NAL shall receive any bug fixes, upgraded modules and related components as they are made available by the Cogito developer.

2.1.1.6.

2.1.1.7. Cogito Studio Express and Cogito Discover deployment on development/staging and production servers.

2.1.2. Maintenance and Technical Support of Cogito Studio Express, Cogito Discover NALT Annotator software for:

2.1.2.1. Integration into NAL workflows for content ingest, indexing, and publication

2.1.2.2. NAL workflows for file conversion and reversion from/to cogx (Cogito format) and ingest/return to NAL repositories in appropriate formats.

2.1.2.3. Reporting and tracking features of indexing and thesaurus curation to facilitate management assessment, decisions and reporting of the overall workflow, indexing production rate, thesaurus curation, and software applications functions (logs, error reporting, timeouts, etc.).

2.1.2.4. Automated feedback loops for continual enhancement and quality improvement of the classification and extraction, and of term proposals.

2.1.2.5. Annotation plan development, testing, deployment, and assessment.

2.1.3. NAL Workflows Integration with Cogito Studio Express and Cogito Discover software

2.1.3.1. Support and consult to maximize Cogito Studio Express and Cogito Discover software functionality, scalability, and flexibility to apply and adapt to state-of-the-art artificial intelligence approaches such as natural language processing, machine learning for classification and text extraction.

2.1.3.2. NAL Thesaurus curation integrated with the indexing or term extraction software.

2.1.3.3. Curation (new or updated terms), maintenance, quality control, persistent URI preservation, etc.

2.1.3.4. Integration from ingest/import to project thesaurus editing, to exporting and updating the thesaurus for deployment in Cogito Studio Express and Cogito Discover software.

2.1.4. Reporting and Quality Assessment support and development.

2.1.4.1. Tracking, analysis, and reporting integral to the system for indexing and NAL Thesaurus workflows and quality control.

2.1.4.2. Algorithms allow for machine aided feedback loops to improve indexing quality.

2.1.4.3. Human indexing validation per NAL specifications and quality standards.

2.1.4.4. Reporting/filtering of NAL Thesaurus updates, additions, other changes.

2.1.4.5. Export of NAL Thesaurus (skos, rdf, etc.) and indexing documents (cogx, xml, csv, etc.) for follow on routine or ad hoc analyses outside of Cogito Studio Express and Cogito Discover software.

2.1.5. Customer Support

2.1.5.1. Customer support shall include telephone, e-mail, and online or in-person meetings as needed.

2.1.5.2. Response time for support requests will be within 24 hours.

2.1.6. Applicable Specification, Software, and File Format Standards

http://www.redhat.com/rhel/

2.2. Hardware Requirements per USDA IT specifications

2.2.1. Cogito Studio Express and Discover software is currently implemented internally at NAL and housed on NAL servers. Contracted server software must run on Red Hat Enterprise Linux 7 or 8 x86 64-bit or on the equivalent CentOS 7 or 8. 64-bit native server application is preferred. Minimal hardware requirements not to exceed two Intel-compatible 2 GHz quad-core processors and 16 GB RAM. Hardware will be provisioned on the NAL VMware high-availability cluster environment or in the Azure Government Cloud. The software must allow parallel processing between two or more servers or instances.

2.2.2. Software is currently deployed on production and development/staging servers. The development servers are used to test skill cartridges, annotation plans, upgrades, patches, workflows, etc. so that there is minimal to no impact on production workflows.

2.2.3. Software must run securely on modern browsers.

2.2.4. NALT Annotator is currently deployed on a server hosted by Expert ai as a cloud-based environment at Amazon Web Services. Expert ai provides the set-up, configuration, server, API, and software maintenance, upgrades, and integration with Cogito software and NAL annotation plans and workflows, as well as technical support and integrated professional IT services.

2.3. Integrated Professional IT Services

The contractor shall provide professional services for NAL’s use of Cogito software as directed by the Government Contracting Officer’s Representative (COR). This includes routine (monthly at a minimum) testing/review for optimization of software deployment configurations and NAL workflow integration. Optimization activities include:

2.3.1. Cleaning up Cogito Studio Express projects,

2.3.2. Review and update of annotation plans deployed in Cogito Discover for Cogito Studio Express and NALT Annotator,

2.3.3. Configuration and performance optimization of the Cogito software database and servers,

2.3.4. Analysis of Cogito logs and reporting/correcting of any issues found,

2.3.5. Advising NAL of best practices for Cogito software use and integration into NALT workflows,

2.3.6. Provide recommendations on Cogito input/output file formats to optimize integration into NAL’s workflows.

2.3.7. Professional IT services include:

2.3.7.1. Technical tasks - such as programming, utilization of REST APIs, troubleshooting quality issues, liaison to product development on suggested improvements, configuring system and application software, developing new skill cartridges based on NAL provided knowledge, and writing scripts for systems integration. The contractor will work with NAL staff to test and deploy these configurations, modules, and scripts.

2.3.7.2. Training – for REST APIs, Cogito capabilities and use cases, Administrative interfaces, use and optimization of Cogito Studio Express, Discover, NALT Annotator, NAL workflow integration, or other web services if applicable.

2.3.8. Services will be provided as requested in the following areas:

2.3.8.1. Development of new skill cartridges and annotation plans to enhance indexing quality.

2.3.8.2. Enhancement of current skill cartridges and annotation plans.

2.3.8.3. Enhancement of NALT Annotator web application.

2.3.8.4. Troubleshooting quality issues.

2.3.8.5. Integration strategies and programming support.

2.3.8.6. Knowledge transfer to NAL staff on Cogito modules, skill cartridge and annotation plan development, strategies, and maintenance.

2.3.9. The contractor shall implement and support all services using a variety of communication methods, including email, teleconference, online and/or onsite face-to-face meetings. Face-to-face meetings may be requested by either the vendor or NAL and will be conducted at the NAL building or a mutually agreed location.

2.3.10. The contractor shall conduct any training onsite at the NAL building, unless otherwise is mutually agreed by the vendor and NAL.

2.3.11. As new requirements are identified for the Cogito system, NAL will consult with the contractor to determine potential solutions and the tasks necessary to implement these solutions.

2.3.12. When new tasks are requested by NAL, the contractor shall, within 10 working days, provide a brief, one-page task specification proposal describing the consulting and development services needed, a plan for the course of action, activities and deliverables required from NAL staff , and an estimate of the number hours required to complete the task.

2.3.13. Execution of each task is subject to approval by NAL.

2.3.14. Hours proposed for a given task must be preapproved by the NAL Government Contracting Officer’s Representative (COR). If hours to complete the task are over the amount approved by the COR, work must stop on the task until additional hours are approved by the COR.

2.3.15. The contractor shall provide a written Progress report to the COR within five working days of the end of the month. This report will include an audit of services provided and number of hours expended on each task at the end of each month.

2.3.16. Upon completion of each task, contractor will provide the NAL COR with a deliverable acceptance receipt within 10 working days.

2.3.17. Potential Professional IT Services Tasks

2.3.18. Potential Professional IT Services Tasks that may be assigned under this contract are listed below to provide a sense of the type and scope of potential tasks envisioned. This list is not a checklist of tasks that must be completed. The contractor shall be able to provide services that will accomplish, in cooperation with NAL, the type of specific tasks listed here:

3. DELIVERABLES, SUCCESS CRITERIA

Each deliverable created under this Statement of Work will be delivered to NAL with an NAL created Deliverable Acceptance Receipt. This receipt will describe the deliverable, user acceptance tests, and provide NAL with space to indicate if the deliverable is accepted, rejected, or conditionally accepted. Conditionally Accepted deliverables will contain a list of deficiencies that need to be corrected in order for the deliverable to be accepted by the NAL Government Contracting Officer’s Representative (COR).

3.1. When new tasks are requested by NAL, the contractor shall provide, within 10 working days, a brief, one-page task specification proposal describing the consulting and development services needed, a plan for the course of action, activities and deliverables required from NAL staff , and an estimate of the number hours required to complete the task.

3.2. Upon completion of each task, contractor will provide, within 10 working days, the NAL COR with a deliverable acceptance receipt.

3.3. Deliverables will be sent to the shared file server and/or be emailed to the COR at the address provided on the award.

3.4. The contractor shall provide a written Monthly Progress report to the COR within five working days of the end of the month. This report will include an audit of services provided and number of hours expended on each task at the end of each month.

3.5. Use cases, business deliverables, and success criteria

· Collaborative environment, so teams work together and can support/back up

· Individual project environment i.e. cannot inadvertently index the same batches

· Indexers understand where to get help within software

· Indexers have access to training, manuals, or other support by vendor

· Admins can import/export the projects easily

· Compatibility between file formats xml, cogx, skos, etc.

· NAL Thesaurus (NALT) integration into the workflow as needed for indexing, and for indexers to add terms to the official NALT, and for updating and publishing NALT (as skos or if more efficient, a skos compatible format such as nt, ttl) for import/export from/to Cogito.

· Optimization review, and production data capture for validation, analysis, tracking, and reporting of indexing quality

a. False/true positives, negatives, term usage rates, trends, etc.

b. Production rates by indexer, journal, cluster, etc.

c. Failures, system logs

d. Technical and training Support

e. Skills measures, enhancements, evaluations, gap analyses

NAL Evaluation: Survey of users, managers, IT support personnel for feedback/satisfaction, reporting features - data easily accessible for validation, analysis, tracking, and reporting.

· Develop/maintain/test/enhance annotation plans within Cogito system

· Multi-annotation plan features of Cogito software enabled for targeted indexing quality

· APs development/testing based on indexing performance by cluster (indexer annotation plan validation feedback)

· Develop/test current or new annotation plans by classification cluster

· Target current low F1 scores until they are 0.75 or better by cluster

· MI = Cogito only indexing at up to 6000 articles on average per batch (varies).

· HI = human detailed indexing in Cogito Studio Express of raw source cogx articles sampled from each MI batch. A team of indexers each receive the same documents in individual projects which are used to score/validate Cogito automated indexing

· HI% = Percent of articles randomly selected from MI batch for HI project,

· Target HI% = 5% of MI. July 2021 minimum HI% = 1%. (60 articles/HI project)

· Target HI% variable based on analysis by F1 cluster score, what is optimal HI% to achieve F1 = 0.75 per each cluster?

· Annual increase in HI% goal = +1% for this contract.

· Golden Set = merged individual annotations from HI (identical) projects from each indexer. The merged document is used within CSE for a face-to-face discussion/consensus on annotations. Once the team agrees on all annotations and the file modified to reflect their decisions, the golden set is complete.

· Evaluation, MI validation by cluster MI (same articles in HI) compared to Golden Set articles from that batch to calculate F1, precision, recall, accuracy, etc.

· Target F1 = 0.75 per cluster. July 2021 current F1 = 0.3 to 0.6

· Annual increase in F1 goal = +0.10 for all clusters for this contract.

NAL Evaluation: MI : HI = 100 : 5 (optimum based on analysis) for each cluster,

· Clusters indexed using MI, HI, HI% as outlined in #2 NAL Evaluation.

· Indexing tracking of false positives/negatives, true positives/negatives to evaluate recall, precision, F1 score

· Workflow design to include human indexers, expandable to SME partners (AWIC prototype) NAL Evaluation: as outlined in #2 NAL Evaluation: Evaluation scores/targets, survey of users, managers, IT support personnel for feedback/satisfaction, reporting features - data easily accessible for validation, analysis, tracking, and reporting.

· Accessible to the public as an online resource GUI and API for natural language processing extraction/annotation of text using the same or similar annotation plan(s) methodology as for NAL indexing.

· User tools for flexible usage of NALT and/or annotation plans.

· Capture of non-NALT terms that may be appropriate for addition to the NALT.

· Proposed non-NALT terms captured are sent to NALT manager for consideration, including metadata for context and to aid in discovery or research as to their appropriateness for NALT and their placement in the hierarchy.

· Web analytics – visits, referrals, bounce rate, etc.

4. GOVERNMENT FURNISHED PROPERTY

NAL will not provide equipment, hardware, software, or access directly to government systems.

5. SECURITY REQUIREMENTS

The contractor will not require any special passes or identification for physical access, security clearance, or other clearances for access to government IT systems.

6. PLACE OF PERFORMANCE

The contractor will generally perform task at their own facility. Training will be conducted onsite at the NAL building, unless otherwise is mutually agreed by the vendor and NAL. Face-to-face meetings will be conducted at the NAL Building, unless otherwise is mutually agreed by the vendor and NAL.

9. INFORMATION SENSITIVITY

The contractor understands and agrees that all information gathered for evaluation, system software versions and configurations, or any other technical fact, configuration or electronic file is the property of the Government.

10. SECTION 508 – ACCESSIBILITY OF INFORMATION AND COMMUNICATIONS TECHNOLOGY

(a) This SOW/PWS is subject to Section 508 of the Rehabilitation Act of 1973 (29 U.S.C. 794d) as amended by the Workforce Investment Act of 1998 (P.L. 105-220). Specifically, subsection 508(a)(1) requires that when the Federal Government procures Information and Communications Technology (ICT)[footnoteRef:2], the ICT must allow Federal employees and members of the public with disabilities comparable access to and use of information and data provided to Federal employees and members of the public without disabilities. [2: Please note that the term Information and Communications Technology (ICT) is synonymous with Electronic and Information Technology (EIT), the previously used term. The term ICT will be used to meet international standards after the release of the Section 508 Refresh. ]

(b) The ICT accessibility standards as 36 CFR Part 1194 were developed by the Architectural and Transportation Barriers Compliance Board (also known as the Access Board) and apply to contracts, task orders, and indefinite quantity contracts on or after June 25, 2001.

(c) Each Information and Communications Technology (ICT) product or service furnished under this contract shall comply with the Information and Communications Technology Accessibility Standards (36 CFR 1194), as specified in the contract, at a minimum. If the Contracting Officer determines any furnished product or service is not in compliance with the contract, the Contracting Officer will promptly inform the Contractor in writing. The Contractor shall, without charge to the Government, repair or replace the non-compliant products or services within a period of time specified by the Government in writing. If such repair or replacement is not completed within the time specified, the Government shall have the following recourses:

1) Cancellation of the contract, delivery, or task order, purchase or line item without termination liabilities; or

2) In the case of custom Information and Communications Technology (ICT) being developed by a contractor for the Government, the Government shall have the right to have any necessary changes made or repairs performed by itself or by another firm for the non-compliant ICT, with the contractor liable for reimbursement to the Government for any expenses incurred thereby.

(d) The contractor must ensure that all ICT products and services that are less than fully compliant with the accessibility standards are provided pursuant to extensive market research and are the most current compliant products or services available to satisfy the contract requirements.

(e) For every ICT product or service accepted under this contract by the Government that does not comply with 36 CFR 1194, the contractor shall, at the discretion of the Government, make every effort to replace or upgrade it with a compliant equivalent product or service, if commercially available and cost neutral, on either a contract specified refresh cycle for the product or service, or on a contract effective option/renewal date, whichever shall occur first.

Section 508 Compliance The contract shall comply with the standards, policies, and procedures below. In the event of conflicts between the referenced documents and this SOW/PWS, shall take precedence.

Section 508 Accessibility Standards

1) 29 U.S.C. 794d (Rehabilitation Act as amended)

2) 36 CFR 1194 (Section 508 standards)

3) http://www.access-board.gov/sec508/508standards.htm (Section 508 standards)

4) FAR 39.2 (Section 508)

5) http://www.ocio.usda.gov/document/departmental-regulation-4030-001 (USDA standards, policies, and procedures for Section 508)

In addition, all contract deliverables are subject to these standards.

All web content or communications materials produced, regardless of format (text, audio, video, etc.), must conform to the applicable Section 508 standards to allow Federal employees and members of the public with disabilities comparable access to and use of information and data provided to Federal employees and members of the public without disabilities. All contractors (including sub-contractors) and consultants responsible for preparing or posting content must comply with the applicable Section 508 accessibility standards and, where applicable, those set forth in the referenced policy or standards document above. Remediation of any materials that do not comply with the applicable provisions of 36 CFR Part 1194 as set forth in the SOW/PWS shall be the responsibility of the contractor or consultant. The following Section 508 provisions apply to the products and/or services identified in this SOW/PWS:

· 36 CFR Part 1194.21 provisions a-l

· 36 CFR Part 1194.22 provisions a-p

· 36 CFR Part 1194.23 provisions a-k[4]

· 36 CFR Part 1194.24 provisions a-e

· 36 CFR Part 1194.25 provisions a-j[4]

· 36 CFR Part 1194.26 provisions a-d

· 36 CFR Part 1194.31 provisions a-f

· 36 CFR Part 1194.41 provisions a-c The following Section 508 provisions apply for software development material identified in this SOW/PWS.

For software development, software applications, and operating systems the Contractor/Developer shall comply with the following standards, policies, and procedures:

Section 508 Accessibility Standards

1) 29 U.S.C. 794d (Rehabilitation Act as amended)

2) 36 CFR 1194 (Section 508 standards)

a. 36 CFR Part 1194.21 provisions a-l

b. 36 CFR Part 1194.31 provisions a-f

c. 36 CFR Part 1194.41 provisions a-c

For web-based applications (intranet, internet information and applications, 16 rules), the Contractor/Developer shall comply with the following standards, policies, and procedures:

Section 508 Accessibility Standards

1) 29 U.S.C. 794d (Rehabilitation Act as amended)

2) 36 CFR 1194 (Section 508 standards)

a) 36 CFR Part 1194.21 provisions a-l

b) 36 CFR Part 1194.22 provisions a-p

c) 36 CFR Part 1194.31 provisions a-f

d) 36 CFR Part 1194.41 provisions a-c

For telecommunication products and services, the Contractor/Developer shall comply with the following standards, policies, and procedures:

Section 508 Accessibility Standards

1) 29 U.S.C. 794d (Rehabilitation Act as amended)

2) 36 CFR 1194 (Section 508 standards)

a) 36 CFR Part 1194.23 provisions a-k

b) 36 CFR Part 1194.31 provisions a-f

c) 36 CFR Part 1194.41 provisions a-c

For video and multimedia applications (including training materials), the Contractor/Developer shall comply with the following standards, policies, and procedures:

Section 508 Accessibility Standards

1) 29 U.S.C. 794d (Rehabilitation Act as amended)

2) 36 CFR 1194 (Section 508 standards)

a. 36 CFR Part 1194.24 provisions a-e

b. 36 CFR Part 1194.31 provisions a-f

c. 36 CFR Part 1194.41 provisions a-c

For self-contained and closed products, the Contractor/Developer shall comply with the following standards, policies, and procedures:

Section 508 Accessibility Standards

1) 29 U.S.C. 794d (Rehabilitation Act as amended)

2) 36 CFR 1194 (Section 508 standards)

a. 36 CFR Part 1194.21 provisions a-l

b. 36 CFR Part 1194.25 provisions a-j

c. 36 CFR Part 1194.31 provisions a-f

d. 36 CFR Part 1194.41 provisions a-c

For desktop and portable computers, the Contractor/Developer shall comply with the following standards, policies, and procedures:

Section 508 Accessibility Standards

1) 29 U.S.C. 794d (Rehabilitation Act as amended)

2) 36 CFR 1194 (Section 508 standards)

a. 36 CFR Part 1194.21 provisions a-l

b. 36 CFR Part 1194.26 provisions a-d

c. 36 CFR Part 1194.31 provisions a-f

d. 36 CFR Part 1194.41 provisions a-c

For help desk and other support services, the Contractor/Developer shall comply with the following standards, policies, and procedures:

Section 508 Accessibility Standards

1) 29 U.S.C. 794d (Rehabilitation Act as amended)

2) 36 CFR 1194 (Section 508 standards)

a. 36 CFR Part 1194.31 provisions a-f

b. 36 CFR Part 1194.41 provisions a-c

If the help desk or other support services include training, Contractor/Developer must also comply with the following standards, policies, and procedures in addition to 36 CFR Part 1194.31 provisions a-f and 36 CFR Part 1194.41 provisions a-c:

a. 36 CFR Part 1194.21 provisions a-l (installable and web-based training)

b. 36 CFR Part 1194.22 provisions a-p (web-based software)

All Information and Communications Technology (ICT) subject to the 36 CFR 1194 standards will have a Section 508 usability and acceptance test where Section 508 compliance will be validated. This test must be administered by a Federal Section 508 Testing Center.

All maintenance for Information and Communications Technology that requires upgrades, modifications, installations, and purchases will adhere to the Section 508 standards and 36 CFR 1194.

WCAG 2.0 Compliance The Contractor shall comply with the standards, policies, and procedures below. In the event of conflicts between the referenced documents and this SOW/PWS the SOW/PWS shall take precedence.

Custom ICT Development Services When Contractor provides custom ICT development services pursuant to this contract, Contractor/Developer shall ensure the ICT fully conforms to the applicable Revised 508 Standards prior to delivery and before final acceptance.

Installation, Configuration, and Integration Services When Contractor/Developer provides installation, configuration, or integration services for equipment and software pursuant to this contract, the offeror shall not install, configure, or integrate the equipment and software in a way that reduces the level of conformance with the applicable Revised 508 standards.

Maintenance, Upgrades, and Replacements Contractor/Developer shall ensure maintenance upgrades, substitutions, and replacements to equipment and software pursuant to this contract do not reduce the original level of conformance with the applicable Revised 508 standards at the time of the contract award.

Service Personnel Contractor/Developer shall ensure the personnel providing the labor hours possess the knowledge, skills, and ability necessary to address the applicable Revised 508 standards defined in this contract and shall provide supporting documentation upon request.

Hosting Services When providing hosting services for electronic content provided by the agency, Contractor/Developer shall not implement the hosting services in a manner that reduces the existing level of conformance of the electronic content with applicable Revised 508 standards. Throughout the life of the contract, the agency reserves the right to perform testing on a vendor or contractor’s hosted solution to verify conformance with this requirement.

Validation for ICT Items When purchasing ICT where 1) 508 validation is not possible prior to award, 2) when ICT will be changed after the award, or 3) ICT will be hosted in a third-party environment, Contractor/Developer shall test and validate the ICT solution for conformance to the Revised 508 standards, in accordance with the requirement testing methods, as defined by the agency. Throughout the life of the contract, the agency reserves the right to perform testing to verify conformance with this requirement.

Documentation Contractor/Developer shall maintain and retain full documentation of the measures taken to ensure compliance with the applicable requirements, including records of any testing or demonstrations conducted.

Conformance Reporting Prior to acceptance, Contractor/Developer shall provide an Accessibility Conformance Report (ACR) for each ICT item that is developed, updated, configured for the agency, and when product substitutions are offered. The ACR should be based on the latest version of the Voluntary Product Accessibility Template (VPAT) provided by the Information Technology Industry Council (ITI). To be considered for award, an ACR must be submitted for each ICT item, and must be completed according to the instructions provided by ITI.

When the contractor is required to perform testing to validate conformance to the agency’s accessibility requirements, Contractor/Developer shall provide a Supplemental Accessibility Conformance Report (SAR) that contains the following information:

· Accessibility test results based on the required test methods.

· Documentation of features provided to help achieve accessibility and usability for people with disabilities.

· Documentation of core functions that cannot be accessed by persons with disabilities.

· Documentation on how to configure and install the ICT item to support accessibility.

· When an ICT item is an authoring tool that generates content (including documents, reports, videos, multimedia productions, web content, etc.)., provide information on how the ICT item enables the creation of accessible electronic content that conforms to the Revised 508 Standards, including the range of accessible user interface elements the tool can create.

· Before final acceptance, the contractor shall provide a fully working demonstration of the completed ICT Item to demonstrate conformance to the agency's accessibility requirements. The demonstration shall expose where such conformance is and is not achieved.

Before acceptance, the agency reserves the right to perform independent testing to validate that the ICT solution provided by the contractor conforms to the applicable Revised 508 standards.

Non-Compliance Before final acceptance of any ICT item, including updates and replacements, if Contractor/Developer claims its products or services satisfy the applicable Revised 508 standards specified in the contract vehicle, and the contracting officer determines that any furnished ICT item is not in compliance with such requirements, the contracting officer will promptly inform Contractor/Developer in writing of the non-compliance. Contratory/Developer shall, at no cost to the agency, repair or replace the non-compliant products or services within the period specified by the contracting officer.

11. Compliance with Internet Protocol Version (IPv6) This contract involves the acquisition of Information Technology (IT) that uses Internet Protocol (IP) Technology. In order to comply with OMB Memorandum M-05-22, Transition Planning for Internet Protocol Version 6 (IPv6) issued on August 2, 2005, the contractor agrees that: (1) all deliverables that involve IT that uses IP (products, services, software, etc.) will comply with IPv6 standards set forth in the USGv6 Profile (NIST Special Publication 500-267) and interoperate with both IPv4 and IPv6 systems and products; and (2) it has IPv6 technical support for such deliverables. If the contractor plans to offer a deliverable that involves IT that is not compliant with these requirements, the contractor agrees to obtain the Contracting Officer’s approval before starting work on the deliverable.