RFI_Attachment_3_-_DRAFT_Performance_Work_Statement.pdf

DRAFT PERFORMANCE WORK STATEMENT (PWS)

Business Oriented Software Solutions (BOSS)

1 BACKGROUND

1.1 Vision: The vision of the United States Patent and Trademark Office (USPTO), Office of the Chief Information Officer (OCIO) is to transform the agency with next generation technology and services.

The USPTO technology vision starts with understanding the USPTO businesses need and establishes a high level direction to meet present and future needs. The USPTO technology vision is to provide evolving services that quickly adapt to the growing needs of internal and external customers and takes advantage of industry developments. To reach this desired end state, the OCIO must move its technology, people, and processes forward in support of the USPTO’s core mission. The technical challenges faced by the USPTO are:

• Build services to be shared and re-used

• Build services that are always available

• Encourage a consistent, user-centered look and feel of web-based IT products

• Provide searchable information

• Write code once and deploy whenever needed

• Take an Agile approach to all IT projects

• Provide mobility and collaboration

1.2 Business Areas: The two primary USPTO business areas supported by OCIO are Patents and Trademarks, which include the following initiatives:

1.2.1 Patents End-to-End (PE2E): Web-based system that integrates patent application processing activities currently done in separate systems, including work management, application document viewing, searching, office actions, and saving reference information and electronic notes for easy retrieval. PE2E will streamline steps from the time a patent application enters the USPTO to final disposition.

1.2.2 Trademark Next Generation (TMNG): Will replace all legacy systems with one cohesive web application. TMNG will be an all-encompassing application with unique components to serve specific business needs such as trademark electronic filing, examination (domestic and international), and reporting and analytics.

1.3 Reusable & Open Source Software: Prior to analyzing, designing, developing, and deploying software products for USPTO business areas, OCIO first depends on our business partners to identify ways to improve their business processes. Next, coordinating closely with the Product Owner from the business area, OCIO will gain clarity on the requirements, including use of prototyping when appropriate. Once requirements for a software product are defined and prioritized, OCIO will leverage proven assets, including software re-use and the use of Commercial-Off-The-Shelf (COTS) software.

As its own software integrator, OCIO must increase software re-use via open source software consistent with OMB’s August 2016 policy “Federal Source Code Policy: Achieving Efficiency, Transparency, and Innovation through Reusable and Open Source Software”.

1.4 Software Development and Integration: In support of these efforts, the USPTO has an ongoing requirement for software development and integration IT services. The Business Oriented Software Solutions (BOSS) effort is intended to deliver software development and integration services for COTS products and customized software applications, database applications, and other solutions. The USPTO OCIO seeks technical capability along with innovative concepts, methodologies, and industry best practices to transform its legacy operations to the next generation.

1.5 Methodologies: In recent years, USPTO and its Contractor support have leveraged multiple methodologies to deliver required support (e.g., DevOps, Agile, User Centered Design, Scrum, Kanban, Scrumban, etc.). USPTO anticipates the continued use of these methodologies as well as evolving best practices and Contractor recommended innovative solutions to best meet USPTO’s needs including individual Contractor code check-in source control, automated builds and test, and reporting metrics by individual Contractor as well as at a Contract and Task Order Level.

2 OBJECTIVE

2.1 Advance the existing software development process and quality of the USPTO systems in support of the program offices and OCIO Roadmap Transformation initiatives. In support of this effort, the USPTO has identified the following IT objectives (in no specific order):

2.1.1 Promote innovation and deliver cost-effective and seamless next generation IT solutions.

2.1.2 Incentivize government/Contractor team performance in agile, continuous delivery, and infrastructure within a blended team environment.

2.1.3 Stabilize and provide optimal service on legacy systems to employees and public users.

2.1.4 Provide cost-effective and transparent operations, processes, and information.

2.1.5 Support organizational change initiatives in accordance with OCIO’s goal to transition to a best-in-class development environment (e.g., Agile, DevOps, etc.).

2.1.6 Work seamlessly with enterprise-wide Product Teams, particularly the Product Owners, given their critical role in the team’s ultimate success in delivering end-user capabilities.

2.1.7 Improve internal and external collaboration and information sharing.

2.1.8 Enhance the internal and external user experience by developing user-driven products, and allow flexibility to adapt to changing priorities and requirements.

2.1.9 Develop the capability to allow major USPTO legacy systems to be retired.

3 SCOPE

3.1 The BOSS effort encompasses the following scope of services:

• Program Management

• System and Software Development

• Architecture and Design

• Coding

• Unit, Integration, Performance, Security, and

Regression Testing

• Configuration and Release Management

• Defect Triage

• Production Support

• Maintenance

• User Experience Design

3.2 It is envisioned that this contract will issue a variety of task orders, including, but not limited to:

development; maintenance; operations; studies and planning; or any combination of the aforementioned.

4 TECHNICAL REQUIREMENTS/TASKS/OUTCOMES

The Contractor shall perform the technical requirements defined throughout PWS Section 4 below. In addition, the Contractor shall perform the following overarching requirements:

• Align with the U.S. Digital Services Playbook (https://playbook.cio.gov). The Contractor shall be familiar with the concepts in each play and emphasize them in their approaches and support.

• Recommend innovative solutions or technologies and their associated benefits to USPTO. USPTO will consider their use if they represent cost-effective and improved functionality that advances the initiatives of the OCIO to provide service excellence.

4.1 Program Management

To provide program management services, the Contractor shall:

4.1.1 Work collaboratively with both the government and other Contractor support staff.

4.1.2 Provide leadership on making timely decisions and engaging all relevant stakeholders.

https://playbook.cio.gov/

4.1.2.1 Perform proactive coordination and communication and interface with the USPTO staff and other Contractor support teams, to ensure accountability, mission accomplishment, and support of USPTO and BOSS services and operations.

4.1.2.2 Develop and maintain a communications plan.

4.1.3 Lead or participate in teams involving system stakeholders to facilitate the completion of quality and timely deliverables.

4.1.4 Identify, track and manage technical debt on each project.

4.1.4.1 Follow System and Software Development (PWS Section 4.2) and Coding Practices (PWS Section 4.4).

4.1.4.2 Document the technical debt (inherited or otherwise), determine how it occurred, and minimize and/or prevent it in the future.

4.1.5 Develop a project plan/roadmap and identify project dependencies at the task order level.

4.1.6 Provide weekly status reports to the Task Order Manager (TOM), Contracting Officer (CO), and Contracting Officer’s Representative (COR) via electronic mail if required on individual task orders and participate in status review meetings. The status reports shall include a summary of all Contractor work performed, including an assessment of technical progress, schedule status, resource changes, performance against metrics, and any Contractor concerns or recommendations for the previous period (see RFP Section 5 – Attachment 6).

4.1.7 Establish a Work Breakdown Structure (WBS) describing the project tasks to be executed if the project is utilizing a non-agile methodology.

4.1.7.1 The Contractor shall identify critical paths in project schedules.

4.1.8 Submit accurate and timely cost reports and invoices (see RFP Section 5 – Attachment 5).

4.1.9 Establish effective cost controls and seek opportunities to minimize costs to USPTO.

4.1.10 Integrate and coordinate all activities needed to execute the requirements. Follow USPTO procedures and policies referenced in this PWS as well as best practices to ensure requirements are delivered at the highest quality. Provide effective management of sub-contractors. Ensure customer satisfaction and professional and ethical behavior of all Contractor personnel.

4.1.11 Manage the timeliness, completeness, and quality of problem identification. Notify the USPTO's CO and COR of all problems that impact or potentially impact the contract, deliverable(s), or project schedule. Submit notifications in writing (email is acceptable) within one hour of the problem and within four hours once a schedule delay is realized. Submit this written report in accordance with the format and criteria contained in the Problem Notification Letter (see RFP Section 5 – Attachment 7). Provide corrective action plans.

4.1.12 Actively participate in transition activities for seamless transition (on-boarding or off-boarding) of support responsibilities to/from the government and other Contractors.

4.1.13 Possess professional proficiency, experience, knowledge, and skills to perform the required tasks. Ensure that its staff and subcontractors maintain any required professional certifications, accreditations, and proficiency relative to their areas of expertise.

4.1.14 Ensure that its staff prepare and provide deliverables after requisite reviews which meet the quality standards expected of project/product deliverables that meet relevant Industry/Government standards.

Table 1 provides a program management Performance Requirements Summary (PRS).

Table 1: Program Management PRS Performance Standard

Performance Objective Acceptable Quality Limit

(AQL)

Measurement Technique

Program Management

Timeliness of interactions, problem notification documented within 24 hours;

Score of 3 or Better on all survey questions

Survey conducted by government every two months (Bi-

Standard

Performance Objective Acceptable Quality Limit

(AQL)

Measurement Technique ability to perform on-time, within budget, quality and effectiveness of selecting and retaining staff

Monthly). See RFP Section 5 – Attachment 8

Key Personnel

Key Personnel changes follow RFP Section 4.2.4 guidelines for notification and replacement

Government is notified of key personnel change NLT 15 business days prior to departure at least 95% of the time. There is no gap in key personnel 100% of the time.

Random sampling

Accurate Reporting

Contractor Actual Reports and Status Reports are accurate and submitted on time

100% compliance 100% inspection

Invoicing

Invoices are accurate, match Cost/Status reports, and are submitted by 10th of each month

100% compliance 100% inspection

4.2 System and Software Development

To provide system and software development services, the Contractor shall:

4.2.1 Provide application software design, development, testing, production installation, and post installation support.

4.2.2 Provide automated unit test code and associated data compatible with Continuous Integration

Configuration Management (CICM) for delivery.

4.2.3 Ensure compliance with Enterprise Architecture and High Availability guidelines (see RFP

Section 5 – Attachments 9-10).

4.2.4 Support system integration across implementation teams (e.g., configuration, development, data, test, technology integration, etc.).

4.2.5 Design, document, and deliver functional and technical specifications for new and/or changed functionality or components of an Enterprise System.

4.2.6 Maximize use of existing software assets. New software assets developed shall be reusable vertically and/or horizontally.

4.2.7 For Agile projects, execute against the backlog which includes features, bug fixes, and functional system level upgrades for development (see RFP Section 5 – Attachments 11-12).

4.2.8 In accordance with the USPTO System Development Life Cycle (SDLC), create all necessary functional and technical design documentation. Examples include configuration documents and Reports, Interfaces, Conversions, Extensions, Forms, and automated tools. The automated tools, and/or manual controls may include user authorization (e.g., SSO, two factor authentication, etc.), role development and management, and user assignment planning. SDLC mandated documentation shall be prepared and reviewed as needed.

4.2.9 Collaborate with Government points of contact to analyze needs and identify gaps prior to proceeding with development of design specifications. Provide requirements traceability for designs, development, configurations, bug fixes, and scripts.

4.2.10 Purchase software and hardware, as needed, on behalf of the government. All products will become the property of USPTO.

4.2.11 Validate if systems are defined, designed, developed, and tested to validate complete system functionality and alignment with Government system requirements and identify any areas for improvement or remediation. Closure shall validate that the objectives are met.

4.2.12 Implement health check for all NextGen systems in accordance with USPTO’s NextGen

Applications Logging and Monitoring Guidelines.

4.2.13 Each individual developing the code is responsible for checking in that code into CICM – a separate team member shall not perform this function.

4.2.14 All coding and development shall adhere to USPTO standards and policies (see RFP Section 5

– Attachments 13-22).

4.2.15 All source code developed under a USPTO contract will be owned by USPTO. Source code development shall follow Office of Management and Budget (OMB) guidance on best practices.

4.2.16 Following the Federal Source Code Policy guidelines, all developed software shall be delivered to the USPTO repository (CICM/SVN) and the Government will retain the required rights of use for public release and re-use to other Government agencies.

4.2.17 Cloud aware applications will be architected as per the pattern for integrating business processes and supporting IT infrastructure wherein application components are decomposed into self-contained stateless restful services that communicate with each other using a communications protocol and a set of well-defined Application Programming Interfaces (APIs), independent of any vendor, product or technology.

4.2.18 Cloud applications will be designed to run its components on a shared Operating System.

Application Containers may be used but will be isolated from other Application Containers and share the resources of the underlying Operating System, allowing for efficient restart, scale-up or scale-out of applications across clouds allowing for multiple Applications to share a single system’s Operating System and underlying physical compute resources

Table 2 provides a system and software development PRS.

Table 2: System and Software Development PRS

Standard

Performance Objective AQL Measurement Technique

Sprint Cadence Team’s velocity is consistent sprint to sprint

Achieve stable velocity after no more than 3 sprints

Data obtained from Agile Central

Productivity Ability to meet Planned Velocity in a Sprint

Average velocity as a percentage of Planned Velocity will meet or exceed 95% for the Sprints.

Percentage = Actual Velocity/Planned Velocity

Data obtained from Agile Central

Productivity

Team accomplishments over time based on the percentage of points by work product (stories, defects, split stories) accepted

Value delivered vs. unplanned work (including splits).

Calculated as total points delivered in a quarter divided by accepted stories, split stories or defects

Data obtained from Agile Central

Predictability Consistency of work produced over time

Delivered points and stories within 10% of commitment or less

• Number of committed stories delivered / Number of committed stories

• Number of committed points delivered / Number of

Data obtained from Agile Central http://ptoweb.uspto.gov/ptointranet/cisd/cio/uea/docs/USPTO_Technical_Reference_Model_v8.1.doc https://www.whitehouse.gov/blog/2016/08/08/peoples-code

Standard

Performance Objective AQL Measurement Technique committed points

Technical Debt Minimize story rejection rates and story splitting

Stories that are split are not to exceed 1 Sprint

Data obtained from Agile Central

Iteration velocity

Uniform velocity variance for all development iterations throughout the period of performance

Overall variance percentage for all iterations not to exceed 10%

Data obtained from Agile Central

Task Order Development Deliverables

Completes development tasks within required schedule as specified in task order

At least 95% of tasks completed accurately and within the schedule outlined in the task order

Random sampling

System Design Document

(SDD)

Deliver SDD that accurately meets or exceeds the task order requirements and technical solution prior to each major deployment

SDD is approved prior to deployment of major release

100% inspection at each major deployment

Configuration Management Documents

Documentation is developed and maintained in accordance with USPTO CM Policy prior to each major deployment

100% of documentation follows policy

100% inspection at each major deployment

4.3 Architecture and Design

To provide architecture and design services, the Contractor shall:

4.3.1 Develop and document solution architectures in alignment with the enterprise architecture.

This includes storage, database designs, interfaces, services, technologies, and frameworks.

4.3.2 Ensure that remediation of applications or systems comply with all Office of the Chief

Technology Office (OCTO) principles, policies and guidelines stated in the USPTO System Development Life Cycle, Technical Stacks, and the Technical Catalog (see RFP Section 5 – Attachment 23).

4.3.3 Develop to-be solution architectures which enable minimization of custom code, use modular design and standards-based interfaces, and leverage best practices to provide ease of configuration updates, portability, maintainability, vendor independence, reusability, upgradeability, interoperability, and long-term supportability.

4.3.4 Develop solution architecture artifacts, which may include: application components, modules, databases, tools, interfaces, extensions, bolt-ons, models, and standards (see RFP Section 5 – Attachment 24 and 25).

4.3.5 Identify and document all interfaces and services in the design for USPTO review.

4.3.6 Adhere to all USPTO policies and standards for all design to include quality, performance, scalability, maintainability, accessibility, usability, security, and logging (see RFP Section 5 – Attachments 9 and 26-27).

Table 3 provides an architecture and design PRS.

Table 3: Architecture and Design PRS

Standard

Performance Objective AQL Measurement Technique

Architectural Documentation

Develop, document, and maintain all relevant architectural documentation

All required architecture views are accepted during the Architectural Review

Architectural Review

Architectural Data Flow

Create data flow diagram depicting data ingestion curation and presentation

Diagram must include Data registration, data Pre-ingestion cleansing, Data set staging

Random Sampling

Data Management

Create data architecture document that depicts Data Quality, Data Validation, Data profiling, and Data Verification

Evidence in the tool provides actual data consumption in terms of report, cube, etc.

Random Sampling

Security Compliance

Comply with USPTO and FISMA security guidance

All critical vulnerabilities detected and eliminated or mitigated prior to production deployment

Security vulnerability scans

4.4 Coding

To provide coding services, the Contractor shall:

4.4.1 Follow USPTO coding and development standards for all software development. Ensure

Contractor staff adheres to USPTO standards and policies (see RFP Section 5 – Attachments 13-22).

4.4.2 Each individual developing the code is responsible for checking in the code. A separate team member shall not perform this function.

4.4.3 Develop and provide build and deployment instructions, configuration, integration with automated builds and deployment.

4.4.4 Install scripts for server, database, and desktop deployments.

4.4.5 Provide deployment instructions for software releases.

4.4.6 Participate collaboratively in code reviews, and provide results from internal code reviews (see

RFP Section 5 – Attachment 19).

4.4.7 Review and resolve code issues highlighted by the quality dashboards (e.g. SonarQube) during the Sprint or develop a code analysis review report at the end of each sprint and perform corrective actions for any issues, as requested. Deficiencies discovered in the code review process that impact the quality and reliability of the deployed product must be addressed within the project cycle where the discovery was made.

Table 4 provides a coding PRS.

Table 4: Coding PRS

Standard

Performance Objectives AQL Measurement Technique

Automatic Static Code Analysis

No findings with severity level ‘Blocker’ 0 ‘Blocker’ Level SonarQube reports

4.5 Unit, Integration, Performance, Security, and Regression Testing

To provide unit, integration, performance, security and regression testing services, the Contractor shall:

4.5.1 Perform unit, integration, performance, security, and regression testing on all software developed for the USPTO.

http://ptoweb.uspto.gov/ptointranet/cisd/cio/uea/docs/USPTO_Technical_Reference_Model_v8.1.doc

4.5.2 Integrate the unit tests into the build per Configuration Management (CM) policies and monitor test results via the test automation dashboards and tools provided. The Contractor shall meet required unit test coverage specified at the task-order level.

4.5.3 Provide automated unit test code and associated data compatible with CICM for delivery.

4.5.4 Update test plans, documentation, and unit tests to reflect any required changes found.

4.5.5 Assist with development of test strategies and automated test plans for test events.

4.5.6 Create and maintain test data, develop automated test scripts, conduct and support test readiness reviews and test events, develop test stage gate criteria, produce test result reports, and maintain requirements traceability documentation.

4.5.7 Use Government-provided automated testing tools to detect errors, enable best practices, find security vulnerabilities, and remediate the applicable vulnerabilities within developed software source code.

4.5.8 Use Government furnished equipment (GFE) tools including automated testing software for test events when available. The Contractor may propose alternate or additional tools to support test events.

4.5.9 Prepare, schedule, coordinate, conduct, analyze, and document test events. Test events may include: network, connectivity, integration, functional, volume, stress, regression, auditability/ Federal Information System Control Audit Manual (FISCAM), security, user acceptance, backup, restore, and disaster recovery.

4.5.10 Identify, document, track, mitigate, manage, and resolve all defects discovered during test events in the USPTO designated system (see RFP Section 5 – Attachment 28 and 29).

Completion timeframes to be defined at the task order level.

4.5.11 Develop and execute test automation scripts and utilize current, if available, or develop new automation frameworks to be used for functional and regression testing of both batch processing and interactive applications.

4.5.12 Research tools, methods, and technology trends to support test automation objectives.

4.5.13 Contribute to the development and promotion of design and coding standards for automated testing scripts.

4.5.14 Provide tools expertise and design and coding assistance to USPTO staff tasked with developing, maintaining, and executing automated test scripts.

4.5.15 Develop reusable functions and components that can be used to maintain and extend automated tests for multiple projects with maximum reuse of code.

4.5.16 Design modular scripts that enable tests to be maintained or extended with minimal additional effort.

4.5.17 Provide in-depth technical expertise and advice to testing teams in the use of automation framework, to facilitate the use of automated testing across multiple projects and work streams.

4.5.18 Review User Interface specifications, and technical specifications to understand the system workflow and business requirements.

4.5.19 Review manual test cases, executing where necessary, to understand the low level detail and identify functions required to enable scripting/coding. Identify application components to be automated based on both the business priority and expected benefit of automated testing.

4.5.20 Develop a design approach for automated testing for assigned projects. Document the proposed approach and review it with the project team.

4.5.21 Participate in reviews and inspections that pertain to the inputs to test automation as well as the test automation code.

4.5.22 Develop test data in preparation for test execution.

4.5.23 Maintain test scripts, making changes where necessary in order to maintain their proper functioning as applications and data change.

4.5.24 Execute automated test scripts for both functional and regression testing cycles. Analyze and report test results.

4.5.25 Document application problems found using automated tests, including scripting steps and data needed to reproduce the problem and provide in a written report to the Task Order Manager.

4.5.26 Report test execution progress and test results to development team lead.

4.5.27 Perform regression testing on all software prior to deployment into a quality assurance environment.

4.5.28 Develop 508 compliant applications within the scope of USPTO adapted standards to perform automated and manual 508 testing to ensure compliance.

Table 5 provides a unit, integration, performance, security, and regression testing PRS.

Table 5: Unit, Integration, Performance, Security, and Regression Testing PRS

Standard

Performance Objective AQL Measurement Technique

Software Quality Employs techniques to develop defect free software

Defect rate not to exceed 8% of delivered functionality

Metrics and test results obtained from UAT/FQT/PVT test summary reports

Test Automation Incorporates use of automated testing tools

Automated testing tools used at least 95% of the time Periodic sampling

Quality of unit testing

All unit tests include meaningful assertions and execute critical paths or sections of code

100% of all unit test scripts are submitted for Government Code Review

Reviewed during regularly scheduled Government code reviews

Source Code Integration

Automated unit test coverage required for all code committed to CICM

A minimum of 80% of all software delivered to the USPTO CICM repository must have automated unit testing executed by the CICM system

Periodic review of SonarQube reports

4.6 Configuration and Release Management

To provide configuration and release management services, the Contractor shall:

4.6.1 Check code, scripts, and configuration files into the USPTO CM Repository as developed.

These activities shall be performed in accordance with CM policy, specifically the CICM User Guide, to develop Software on the CICM platform (see RFP Section 5 – Attachment 19).

4.6.2 Ensure that the USPTO can recreate all builds for every release exclusively from code in the CM Repository.

4.6.3 Provide detailed documentation describing how to build the delivered, tested software.

4.6.4 Perform system configuration to enable the to-be business processes in one or more of the enterprise applications. Perform periodic configuration audits and identify common configuration issues, share knowledge and best practices.

4.6.5 Provide system configuration support based on USPTO configuration plans and SDLC, that incorporate configuration scope, release cycles, test plans, data requirements, and associated development objects. System configuration includes creating supporting documentation, including processes and procedures, and performing audits. All of which shall conform to the USPTO configuration management policy and USPTO Release to Production procedures (see RFP Section 5 – Attachments 23).

4.6.6 In accordance with the SDLC, conduct all necessary deployment and release activities including fully preparing the sites for implementation; conducting pre-deployment site assessments; validating infrastructure readiness; providing end user identification, mapping, provisioning and implementation; and change management and communications activities.

Table 6 provides a configuration and release management PRS.

Table 6: Configuration and Release Management PRS

Standard

Performance Objective AQL Measurement Technique

Physical Configuration Audit (PCA) Compliance

Delivers all configuration items based on USPTO PCA Policies

100% prior to each major production deployment

Reviewed prior to major production deployment

Enterprise Configuration Management Plan (ECMP) Compliance

Provides system configuration information in accordance with the ECMP and as designated in TOs

100% compliance with documentation required by the ECMP data description prior to first project release

Beginning of Design Phase (Non-Agile) or Release (Agile) Phase

Compliance with Continuous Integration Continuous Delivery (CICD)

System configuration and build conforms to USPTO configuration management policy

All developed software and documentation is managed within the USPTO CICM system and in accordance with USPTO CM policy

Periodic sampling

Source Code and Automated Test Script Management

Comprehensive Source Code and Automated Test Script Daily Check-In

100% compliance Random sampling

4.7 Defect Triage

To provide defect triage services, the Contractor shall:

4.7.1 Perform defect triage following the USPTO Defect Management Plan using USPTO defect management tools.

4.7.2 Identify, correct, and document defects using the defect management tools.

4.7.3 Implement, maintain, and report on the causes for high risk issues and determine how to prevent them from being repeated. Include a root-cause analysis and written remediation report when requested by the COR or TOM within 5 business days of the findings. Create automated scripts that are generated to identify bugs and technical issues and creates corrective action plans for defect resolution when bugs and technical issues are generated.

Table 7 provides a defect triage PRS.

Table 7: Defect Triage PRS

Standard

Performance Objective AQL Measurement Technique

Defect Density

The defect density per iteration remains constant or decreases over time

Percentage of user stories developed in the iteration with directly associated defects shall not exceed 8%

Data obtained from CA Agile Central

Defect Backlog Growth

The defect Backlog growth rate is low over the Sprint time box

Difference between the number of defects opened versus the number of defects closed over the sprint timebox is less than 15% of the

Data obtained from Agile Central

Standard

Performance Objective AQL Measurement Technique total open Defect backlog

4.8 Production Support

Provide production support services to the applications/systems that are in active development cycle.

To provide production support services, the Contractor shall:

4.8.1 Maintain, sustain, update, and migrate system baselines for development, quality assurance, Continuity of Operations (COOP) and training. Maintain and update Plan of Action and Milestones (POAMs), and bugs.

4.8.2 Ensure that the USPTO can support any proposed software solution and complies with USPTO architecture (see RFP Section 5 – Attachments 9, 10, and 30).

4.8.3 Support on-site post deployment activities, which includes but is not limited to, end user training, data validation, data maintenance, prioritization and escalation of help desk tickets, financial compliance and validation, and translation of business processes in the enterprise environment.

4.8.4 Provide configuration and installation information for production needs, and document and provide deployment instructions for both COTS and developed systems.

4.8.5 Provide training and/or documentation as requested – level of effort will vary by task order.

4.8.6 Provide on-call operational support as required and specified in task order.

4.8.7 Provide emergency support for production issues as required and specified in task order.

4.8.8 Provide a root cause analysis for production software investigations, as required.

4.8.9 Provide support for outages on production systems in accordance with the Dynamic

Operational Support Plan (DOSP) for that AIS, as required.

4.8.10 Automate production tasks, when possible.

4.8.11 Provide, maintain, and update DOSPs.

Table 8 provides a production support PRS.

Table 8: Production Support PRS

Standard

Performance Objective AQL Measurement Technique

Incident Response time

Upon identification of software issue, correct the problem and restore Software services to operational readiness within required SLA specified in TOs

At least 95% of software services restored within the specified SLA

Incident Management reports

Maintenance Updates

Apply timely and planned maintenance updates to avoid disrupted/unplanned production outages

At least 98% of planned maintenance updates are completed on schedule

Periodic sampling

Problem Resolution

Provide root cause analysis for software system failures within required SLA specified in TOS

At least 99% of root cause analysis are detail with explanations and provided within the SLA set by the Task Order

Periodic sampling

4.9 Maintenance

Maintenance applies to applications/systems that are in containment and are not in active development cycle. To provide maintenance services, the Contractor shall:

4.9.1 Identify, plan, and conduct maintenance activities, including identification, isolation, and resolution of system problems to restore normal operations.

4.9.2 Recommend, as required, maintenance activities including systematic inspection, detection, and correction of problems. This support will help increase system maintainability and reliability, and prevent problems in the future (e.g., applying application or operating system patches).

4.9.3 Schedule maintenance, apply patches, and adhere to information assurance vulnerability alerts.

Plan for and manage multiple landscapes and transport paths and coordinate efforts across multiple products/programs.

4.9.4 Perform maintenance activities (frequency to be defined at the TO-level) designed to cope with changes in the software environment including the implementation of processing efficiencies, and/or considerations for additional delivered capabilities to enable existing and future requirements.

4.9.5 Monitor and report system and operational metrics against system-specific defined standards and parameters.

4.9.6 Complete operational and system performance measurement as specified in each Task Order.

4.9.7 Design, document, and implement policies, processes, and procedures to ensure that COOP is consistent with system availability requirements during all disaster recovery test events, and after a natural or manmade disaster renders a component of the technical landscape unusable.

Coordination includes communication with the hosting organization to ensure the requirements for system design and sustainment are synchronized.

4.9.8 As specified in each Task Order, track and resolve system incidents and problems identified in the USPTO tracking system. This includes problems that impact system functionality or availability, diagnostics, interface problems, performance-related problems, and collaboration with the COTS enterprise application vendor to resolve problems.

Table 9 provides a maintenance PRS.

Table 9: Maintenance PRS

Standard

Performance Objective AQL Measurement Technique

Restoration of Operations

Restore normal operations within required SLA specified in TOs

At least 95% of operations restored within the specified SLA

Incident management database reports

Maintenance Updates

Apply timely maintenance updates

At least 98% of maintenance updates completed Periodic sampling

Problem Resolution

Respond to system alerts, track and resolve system problems

At least 99% of maintenance actions are responded to within the SLA set by the Task Order

Periodic sampling

4.10 User Experience Design

To provide user experience design services, the Contractor shall:

4.10.1 Plan and conduct user research to determine stakeholder and end-user needs and preferences.

4.10.2 Determine usability metrics that need to be met.

4.10.3 Using the USPTO User Centered Design (UCD) methodology, develop conceptual

(wireframes, mockups) and logical (clickable wireframes, prototypes) designs that both meet usability metric goals and are technically feasible (see RFP Section 5 – Attachment 72).

4.10.4 Plan and conduct expert reviews (heuristic evaluations) and/or usability tests to evaluate and report on the usability of the product.

4.10.5 Maintain the Portfolio and Project Pattern Libraries and Style Guide to provide a consistent look and feel across USPTO applications. Collaborate with the User Experience Branch

(UXB) and maintain all documentation within the Enterprise Pattern Library as new designs are created (see RFP Section 5 – Attachment 31-33).

4.10.6 Develop testing plans, including scripts and metrics.

4.10.7 Iteratively conduct usability testing. Capture and maintain all documentation necessary for usability testing, including waivers.

4.10.8 Generate usability test reports outlining findings and recommendations from tests.

Table 10 provides a user experience design PRS.

Table 10: User Experience Design PRS

Standard

Performance Objective AQL Measurement Technique

Usability Documentation

Develops, documents, and maintains all usability documentation, plans, results, and waivers

100% of all documentation is maintained

Periodic sampling

Usability testing Conducts usability testing on software to ensure end user needs are met and design issues are uncovered

Conducts usability testing on at least 25% of new functionality

Periodic sampling

Design standards and adherence

Designs interfaces that follow USPTO UI design guidelines and best practices. Communicates necessary deviances to USPTO UX Lead.

100% of deviances are , documented, and approved bythe USPTO UX Lead

Random sampling;

Reviewed prior to major production deployment

Quality of design Provides User Experience design that is effective, efficient, and satisfying to end users

SUS score within 15% of USPTO product average

Periodic sampling

Software reuse Designed interactions and artifacts are captured in a meaningful way for software reuse

New designs and/or UX patterns created are provided in reusable code to the Enterprise Pattern Library

Periodic sampling;

Review of GitHub commits

5 SKILLS AND ABILITIES

Below is a representative list of required skills and abilities required for this PWS. The Task Orders will describe the required skills and abilities needed to meet individual Task Order requirements.

The Contractor shall provide expertise in the following skills (but not limited to):

Note: Bold means these are the required skills, methodologies, CICM CORE products, or solutions used most often.

Required Skills & Methodologies

• Scrum

• Automated systems performance, load, and stress testing

• Bootstrap

• Cloud paradigms (IaaS, PaaS, FaaS, CaaS)

• Data modeling

• Data warehousing

• Distributed architectures

• Service Oriented Architecture

(SOA)

• Relational database setup, distribution, & admin.

• Test Driven Development

Required Skills & Methodologies

• Data interface and security evaluations

• GitHub

• Software build and release management

• Web application development

• Continuous Integration

• Java

• EAI patterns

• Extract, transform and load processes

• Enterprise Architecture

• Unit Testing

• Networking protocols

• RESTful API Design

• SBX virtualization

• Threading & memory management

• VPN

• Web Services

• Service segmentation (data & business services) characterization, & definition

• Object-oriented design

The Contractor shall be competent with the following CICM CORE Products:

CICM CORE Products

• Jenkins

• SonarQube

• Nexus

• Apache Subversion

• SCM Manager

• Canary

• CloudBees Enterprise

The Contractor shall be competent with the CICM Supported Technologies. The various technology stacks in use at the USPTO catalogs the current technology products and standards approved for use in supporting the agency’s vision going forward. It is important that the products are used as intended within the business system.

Category Current Solution Continuous delivery Deployment automation Tools

• Puppet 1.3.1

• Ansible 1.6.6

• Cloudforms

Build management

• Apache Maven 2.2

• Apache Maven 3.0

• Apache Ant 1.6.2

• Apache Ant 1.7.1

• Apache Ant 1.8.3

• Apache Ant 1.9.2

• NPM

• Bower 1.7.2

• Grunt 0.4.1

• GULP 1.6.11

• Drush (Drupal)

Development framework

• Gradle 1.9

• Gradle 2.3

• Grails 2.3.7

• Java 1.4, 6 – 8

• Framework 2.5 –

4.0

• Node.js 0.12.2

• PhantomJS v2.0

• Ruby

• ImageMagick

• Bouncy Castle

• ComponentOne

Development Unit Testing

• Visual Studio .Net

• Framework 4.0

• Karma v0.13

• OpenCover

• Nunit

Automation Testing

• Selenium

• TestComplete

• SOAPUI

• LoadRunnerTestExecute

• WebInspect

6 DELIVERABLES

# Deliverable Due Date Deliverable Recipient

(include contact information)

Deliverable Format Reference

1 Communication Plan

Draft: As part of the proposal response Final: Within five (5) business days of award kick-off meeting

CO & COR Microsoft Word 4.1.2

2 Project Plan As specified in TO COR, CO, & TOM:

Defined at TO-level

Microsoft Project 4.1.5

3 Weekly Status Report

The first calendar day of each week COR, CO, & TOM Microsoft

Word 4.1.6

4 Work Breakdown Structure

Within five (5) business days of award of each TO

COR, CO, & TOM:

Defined at TO-level

Microsoft Project 4.1.7

5 Contractor Actual Templates

• High Level Planned Template: NLT 5 days after TO award or modification

• Detailed Planned Template: NLT 30 days after TO award or modification

• Actuals Template: 25th and 20th of the month

• Invoice Template: NLT the 20th of the month, prior to the invoice submission

COR,

<AMDContractorData SubmissionMailbox@

USPTO.GOV>

Microsoft Excel 4.1.8

6 Invoices Monthly Office of Finance, CO, COR PDF 4.1.8

7 Corrective Action Plan As specified by CO COR, CO, & TOM Microsoft

Word 4.1.10

8 Problem Notification Letter

Within one hour of the problem and within four hours once a schedule delay is realized

COR, CO, & TOM Microsoft Word 4.1.11

9 Transition Plan As specified in TO COR, CO, & TOM:

Defined at TO-level

Microsoft Word

4.1.12, 10.1.2

Functional and Technical Specifications

As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.2.5

11 Rally Reports and Metric Reports As specified in TO COR & TOM: Defined at TO-level Rally 4.2.6

12 Design Documentation As specified in TO COR & TOM: Defined at TO-level Microsoft Word 4.2.7

13 Source Code As specified in TO COR & TOM: Defined at TO-level

CICM

Repository

4.2.12, 4.2.13, 10.1.3

14 Functional Software As specified in TO COR & TOM: Defined at TO-level

CICM

Repository

4.2.9, 4.2.12, Deliverable Recipient

(include contact information)

Deliverable Format Reference

4.12.13

15 Solution Architecture As specified in TO COR & TOM: Defined at TO-level Microsoft Word 4.3.1, 4.3.3

Solution Architecture Artifacts

As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.3.4

17 Design Interfaces As specified in TO COR & TOM: Defined at TO-level

Microsoft Word

4.3.5

18 Build Instructions As specified in TO COR & TOM: Defined at TO-level 4.4.3, 4.6.2, 4.6.3

19 Deployment Instructions As specified in TO COR & TOM: Defined at TO-level 4.4.5

Internal Code Review Report and External Code Review Corrections

As specified in TO COR & TOM: Defined at TO-level 4.4.6, 4.4.7

Automated Unit Test Code and associated data

As specified in TO COR & TOM: Defined at TO-level 4.5.3

22 Test Plans As specified in TO COR & TOM: Defined at TO-level

Microsoft Word

4.5.4, 4.5.5, 4.10.6

23 Test Data As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.5.6

24 Test Scripts As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.5.6

25 Test Stage Gate Criteria

After Test (SIT) Prior to QA

(FQT/PVT)

COR & TOM: Defined at TO-level

Microsoft Word 4.5.6

26 Test Results As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.5.6

27 Requirements Traceability Matrix As Required COR & TOM: Defined at TO-level Microsoft Word 4.2.8, 4.5.6

28 Test Events As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.5.9

29 Automated Test Approach As specified in TO COR & TOM: Defined at TO-level Microsoft Word 4.5.20

System Configuration Documentation

As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.6.5

Dynamic Operational Support Plan

As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.6.6, 4.8.11

32 Defect Reporting As specified in TO COR & TOM: Defined at TO-level

Rally, Microsoft Word

4.7.2, 4.7.3

33 Deployment Plan As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.8.4

34 Root Cause Within five (5) business COR & TOM: Defined Microsoft 4.8.8

Deliverable Recipient

(include contact information)

Deliverable Format Reference

Analysis and Remediation Report days of the occurrence of the issue at TO-level Word

35 Deployment Instructions As specified in TO COR & TOM: Defined at TO-level Microsoft Word 4.4.3, 4.4.5

36 Training Documentation As specified in TO COR & TOM: Defined at TO-level Microsoft Word 4.8.1, 4.8.5

System and Operational Metrics

As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.9.5

User Experience Design Documentation

After initial user needs identification and as changes occur

COR & TOM: Defined at TO-level

Microsoft Word 4.10.5

39 Usability Test Report

As screens/interfaces are completed

COR & TOM: Defined at TO-level

Microsoft Word 4.5, 4.10.8

40 COOP Artifacts As specified in TO COR & TOM: Defined at TO-level

Microsoft Word 4.8.1, 4.9.7

41 Kick-Off Meeting NLT five (5) business days after contract and TO date of award

COR, CO, & TOM:

Defined at TO-level n/a 9.2.1

42 Agenda and Meeting Minutes

Meeting Minutes and Action Items - NLT one (1) business day after meeting

COR, CO, & TOM:

Defined at TO-level

Microsoft Word 9.2.2

7 PLACE/LOCATION OF PERFORMANCE/DELIVERY

The work specified under this PWS is to be performed primarily at the Contractor’s facility unless otherwise specified in the task order. Meetings will be held at USPTO at 600 Dulany Street, Alexandria, Virginia 22314. There may be occasions when the Contractor is asked to temporarily work at the USPTO site at Alexandria, Virginia or a Government approved alternate work location.

7.1 Hours of Operation

7.1.1 The core hours of operation will be between 0600 and 1800 Eastern Standard Time, Monday through Friday (except Federal Holidays). There may be occasions when Contractor employees will be required to work other than normal business hours including evenings, weekends and holidays to fulfill requirements under the individual task orders, which will required prior approval from the COR or TOM. The Contractor shall be available to meet and interact with USPTO personnel during the core hours, including daily standup meetings. These meetings may be held virtually or physically or a combination of the two.

7.1.2 As specified in the Task Order, hours of operation for Operational Support Hours may be up to 24 hours a day, seven (7) days a week. This can be supported through on-call support, or as specified in the Task Order.

7.2 Emergency Situations

7.2.1 The Contractor shall provide emergency support as designated by each task order. The Contractor shall follow USPTO emergency management and notification procedures as delineated in the DOSP for each Automated Information System (AIS). As directed by the

COR or CO, the Contractor shall continue performance in emergency or mission essential conditions. Additionally, the Contractor may be required to account for the whereabouts of their personnel should this information be requested by the COR or CO.

8 TRAVEL REQUIREMENTS

All travel requirements will be determined at the Task Order level and will require COR or TOM approval prior to the travel taking place.

9 SPECIAL REQUIREMENTS

9.1 Knowledge Transfer

9.1.1 As the USPTO prepares to complete a project with the assistance of a Contractor, it desires to preserve the knowledge that the Contractor has amassed over the duration of the project. This may be in addition to the requirements for the documentation required under the SDLC.

9.1.2 The Government plans for an eight to twelve week Contractor transition-out phase, during which the Contractor shall provide the minimum staff to perform necessary transition at the task order level.

9.1.3 The Contractor shall ensure all development efforts are performed on the CICM platform with frequent, daily source code check-ins to Subversion. All documentation marked as Configurable Items (CI’s) per the Enterprise Configuration Management Plan (CMP) will be checked into the CM repository on a daily basis or as required in Task Order project schedule.

CICM platform will be utilized for all development efforts with active monitoring of Jenkins and Sonar for code quality and quality assurance. USPTO shall be able to reproduce all production systems from the CM repository.

9.2 Administrative Requirements

9.2.1 The Contractor shall attend a Kick-Off Meeting with the CO and COR no later than five (5) business days after contract and each TO(s) date of award. The contract level kick-off will be scheduled by the CO. The task level kick-off(s) will be scheduled by the COR with the TOM in attendance as well.

9.2.2 Upon request and by the direction of the TOM, the Contractor shall provide meeting agendas and meeting preparation material no later than two (2) hours prior to a scheduled meeting, capture meeting minutes and action items during the meeting, and within one (1) business day distribute the minutes and action item list to the meeting attendees (or appropriate…