Attachment F - Key Performance Indicators.xlsx
XLSX spreadsheet 34 KB Posted
- Attached to
- DA10--Health Services DevSecOps, New Task/Delivery Order (VA-21-00044934) - New Requirement -
- Solicitation number
- 36C10B21Q0410
View the file
Other files
| File | Type | Posted |
|---|---|---|
| 36C10B21Q0410 0001_1.docx | DOCX document | |
| 36C10B21Q0410 0001.docx | DOCX document | |
| Attachment E - HFPL Architecture Standard Operating Framework 20210311.docx | DOCX document | |
| Health Services DevSecOps Draft PWS 29June2021.docx | DOCX document | |
| Attachment D - As-Is Intake Process for Health Product Support.docx | DOCX document | |
| Attachment G - Product Line Architecture (PLA) - VA Enterprise Architecture.docx | DOCX document | |
| Attachment A - First Party Lockbox High-Level Business Process Flow.docx | DOCX document | |
| Attachment B - Product List_Average Ticket Count.xlsx | XLSX spreadsheet | |
| Attachment C- DevSecOps Tools and the Delivery Process.pptx | PPTX presentation | |
| 36C10B21Q0410.docx | DOCX document | |
| RFI_Health Services DevSecOps.doc | DOC document | |
| Health Services DevSecOps Draft PWS 25June2021.docx | DOCX document | |
| 36C10B21Q0410.docx | DOCX document |
Show all 13
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
DevSecOpsKPIs
| Health Services DevSecOps PWS - Attachment F | Key Performance Indicators (KPIs) for Health Services Portfolio DevSecOps Support | |||||||||||||||
| KPI | Metric Name | Description | Notes on reporting frequency | Target Measure Progression 1 | * Target Date Progression 1 | Target Measure Progression 2 | * Target Date Progression 2 | Target Measure Progression 3 | * Target Date Progression 3 | |||||||
| Development | Information Radiators for each time segment identified available for all to review | |||||||||||||||
| Quality | Total Escaped Defect Count - TEST | How many defects made it to testers during the period of measure | Daily, Weekly, Monthly, Quarterly | Equal to or Less than 15% | 120 Days | Equal to or Less that 10% | 180 Days | Equal to or less than 5% | 300 Days | |||||||
| Quality | Total Escaped Defect Count - PROD | How many defects made it to production during the period of measure | Daily, Weekly, Monthly, Quarterly | Equal to or Less than 10% | 120 Days | Equal to or Less than 7% | 180 Days | Equal to or less than 3% | 300 Days | |||||||
| Quality | Critical Escaped Defect Count - PROD | How many critical defects made it to production during the period of measure | Daily, Weekly, Monthly, Quarterly | Equal to or Less than 1% | 120 Days | Equal to or less than 0.5% | 180 Days | Equal to or less than 0.25% | 300 Days | |||||||
| Efficiency | Code Coverage (Unit test) | Measured by a tool at build time - every release whether it goes to prod or not; Code coverage (or test coverage) shows which lines of the code were(or were not) being executed by the tests. It is also a metric which helps you find out the percentage of your covered (executed) code by the tests. | Daily, Weekly, Monthly, Quarterly | Equal to or greater than 85% | 120 Days | Equal to or greater than 95% | 180 Days | Equal to or greater than 99% | 300 Days | |||||||
| Velocity | Average Release Cadence | Average production release cadence (Days between production releases and production demand) | Daily, Weekly, Monthly, Quarterly | Equal to or less than 30 days | 120 Days | Equal to or less than 20 days | 180 Days | Equal to or less than 10 days | 300 Days | |||||||
| Efficiency | Mean time to restore | The time it takes to restore service for the product, from unplanned outage to restore (i.e. outage) | Daily, Weekly, Monthly, Quarterly | Equal to or less than 30 Minutes | 120 Days | Equal to or less than 15 Minutes | 180 Days | Equal to or less than 5 minutes | 300 Days | |||||||
| Quality | Release Fail % | The percentage (%) of releases to production that fail (i.e. back out/rollbacked).# of failed releases divided by (/) total releases | Daily, Weekly, Monthly, Quarterly | Equal to or less than 5% | 120 Days | Equal to or less than 2% | 180 Days | Equal to or less than 1% | 300 Days | |||||||
| Efficiency | Automated test coverage | Number of automated test cases (Unit, Smoke, Functional, Integration, Regression, Performance) divided by (/) total number of test cases | Daily, Weekly, Monthly, Quarterly | Equal to or greater than 25% of all test cases | 120 Days | Equal to or greater than 50% of all test cases | 180 Days | Equal to or greather than 70% of all test cases | 300 Days | |||||||
| Efficiency | Story Lead Time | From time a story is created in the product issue repository to the time it is Resolved/Completed. | Daily, Weekly, Monthly, Quarterly | Equal to or less than 45 days | 120 Days | Equal to or less than 30 days | 180 Days | Equal to or less than 15 days | 300 Days | |||||||
| Quality | Total Current Open CriticalHigh Defects in Prod by Iteration | Number of Open defects in defect management tool at the time of the end of the iteration by severity (1-Critical, 2-High, 3-Medium, 4-Low) and whether it is a Production Defect (A defect that was found by the end user and logged as an incident initially) or defects that were found by the product team but were not fixed by the date of the next iteration. | Daily, Weekly, Monthly, Quarterly | Equal to zero Critical, Equal to or less than 1 - High, Equal to or less than 2 - Medium | 120 Days | All medium to critical defects resolved each iteration. Zero exceptions. | 180 Days | |||||||||
| Effectiveness | Defect Lead Time | From time a story is created in the product issue repository to the time it is Resolved/Completed. | Daily, Weekly, Monthly, Quarterly | Equal to or less than 10 days | 120 Days | Equal to or less than 3 days | 180 Days | Equal to or less than 1 day | 300 Days | |||||||
| Effectiveness | Flow efficiency | Process cycle time efficiency [%] = Value - added time / Lead time*100 . can identify when waste is increasing or decreasing in your processes. Flow Efficiency is the ratio of active time vs. wait time out of the total Flow Time. | Daily, Weekly, Monthly, Quarterly | Equal to or greater than 30% | 120 Days | Equal to or greater than 40% | 180 Days | Equal to or greater than 50% efficiency | 300 Days | |||||||
| Effectiveness | Flow Distribution | How are resources and capacity being balanced against the flow items. Feature = New Value added to drive a business result; Defect = Fixes to quality problems that affect the customer experience; Debt = Improvement of software architecture and operational infrastructure; Risks = work to address security, privacy, and compliance exposures | Daily, Weekly, Monthly, Quarterly | Equal to or greater that 75% of the teams accurately reporting Flow Distribution. | 120 Days | Equal to or greater that 90% of the teams accurately reporting Flow Distribution. | 180 Days | 100% compliance | 300 Days | |||||||
| Effectiveness | Flow Time | Measurement of how long it takes for the core four flow items to get from starting point to the finish (Predictability) . Flow Time measures the time it takes for Flow Items to go from ‘work start’ to ‘work complete’, including both active and wait times. | Daily, Weekly, Monthly, Quarterly | Equal to or greater that 75% of the teams accurately reporting Flow Distribution. | 120 Days | Equal to or greater that 90% of the teams accurately reporting Flow Distribution. | 180 Days | 100% compliance | 300 Days | |||||||
| Effectiveness | Flow Load | Calculating demand against velocity. | Daily, Weekly, Monthly, Quarterly | Equal to or greater that 75% of the teams accurately reporting Flow Distribution. | 120 Days | Equal to or greater that 90% of the teams accurately reporting Flow Distribution. | 180 Days | 100% compliance | 300 Days | |||||||
| Effectiveness | Flow Velocity | The number of each of the following items completed over a particular period of time. Feature = New Value added to drive a business result; Defect = Fixes to quality problems that affect the customer experience; Debt = Improvement of software architecture and operational infrastructure; Risks = work to address security, privacy, and compliance exposures | Daily, Weekly, Monthly, Quarterly | Equal to or greater that 75% of the teams accurately reporting Flow Distribution. | 120 Days | Equal to or greater that 90% of the teams accurately reporting Flow Distribution. | 180 Days | 100% compliance | 300 Days | |||||||
| Efficiency | Program Increment Objectives Report | Each Program Increment there should be Objectives set up that are Specific, Measurable, Achievable, Realistic, and Time-Bound. There should be an anticipated Business value provided by the business owners. At the end of the Program Increment there should be an actual value provided by the business owners based on what is achieved. | Daily, Weekly, Monthly, Quarterly | Equal to or greater that 75% of the teams accurately reporting Objectives | 120 Days | Equal to or greater that 90% of the teams accurately reporting Objectives | 180 Days | 100% compliance | 300 Days | |||||||
| Efficiency | Architecting for releasability (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | Different parts of the solution require different release strategies. The solution must be designed to enable various incremental release strategies and shift them over time based on business demand. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of the test environments match production | 300 Days | |||||||
| Effectiveness | Full-Stack Telemetry | The ability to monitor for problems across the full stack that a system covers | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 6/25/21 | 95% of teams compliant | 180 Days | 100% of the test environments match production | 300 Days | |||||||
| Effectiveness | Clearly defined features in the backlog | Clearly defining features that fit in a Program Increment is critical for releases to align on what is needed and for teams to be able to plan. The backlog also reflects important security requirements. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Effectiveness | Iteration | |||||||||||||||
| Status Report | Identifies the planned user stories, accepted user stories, open impediments, and newly accumulated or resolved technical debt. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | ||||||||
| Effectiveness | Iteration Plan | The purpose of iteration planning is to organize the work and define a realistic scope for the iteration. Each Agile Team agrees on a set of stories for the upcoming iteration (the iteration backlog) and summarizes those stories into a set of iteration goals. The iteration backlog and goals are based on the team’s capacity and allow for consideration of each story’s complexity, size, and dependencies on other stories and other teams. At the end of planning, the teams commit to the goals of the iteration and adjust stories as necessary to achieve the larger purpose. In return, management does not interfere or adjust the scope of the iteration, allowing the team to stay focused. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Effectiveness | Story Board (Kanban) for tracking | Tracking iteration progress requires visibility into the status of user stories, defects, and other team activities. For this purpose, most teams use a Big Visible Information Radiator (BVIR) on a wall in the team room. Kanban teams use their Kanban board, while ScrumXP teams would use a storyboard. With this simple storyboard, the team just moves the red ribbon to the current day, providing an immediate and easy-to-understand visual assessment of iteration progress. The storyboard can be shared with remote participants or stakeholders using a webcam, email, wiki, or Agile project management tooling. But this is usually in addition to the physical BVIR. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Effectiveness | Iteration Review | The Iteration Review is a cadence-based event, where each team inspects the increment at the end of every Iteration to assess progress, and then adjusts its backlog for the next iteration. During the iteration review, each Agile Team measures and then demonstrates its progress by showing working stories to the Product Owner and other stakeholders to get their feedback. Teams demo every new Story, Spike, Refactor, and Nonfunctional Requirement (NFR). The preparation for the iteration review begins during Iteration Planning, where teams start thinking about how they will demo the stories to which they have committed. ‘Beginning with the end in mind’ facilitates iteration planning and alignment, fostering a more thorough understanding of the functionality needed, ahead of iteration execution. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Effectiveness | Iteration Review | The Iteration Review is a cadence-based event, where each team inspects the increment at the end of every Iteration to assess progress, and then adjusts its backlog for the next iteration. During the iteration review, each Agile Team measures and then demonstrates its progress by showing working stories to the Product Owner and other stakeholders to get their feedback. Teams demo every new Story, Spike, Refactor, and Nonfunctional Requirement (NFR). The preparation for the iteration review begins during Iteration Planning, where teams start thinking about how they will demo the stories to which they have committed. ‘Beginning with the end in mind’ facilitates iteration planning and alignment, fostering a more thorough understanding of the functionality needed, ahead of iteration execution. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Effectiveness | Iteration Retrospective | The Iteration Retrospective is a regular event where Agile Team members discuss the results of the Iteration, review their practices, and identify ways to improve. At the end of each iteration, Agile teams that apply ScrumXP (and many teams who use Kanban) gather for an iteration retrospective. Timeboxed to an hour or less, each retrospective seeks to uncover what’s working well, what isn’t, and what the team can do better next time. Each retrospective yield both quantitative and qualitative insights. The quantitative review gathers and evaluates any metrics the team is using to measure its performance. The qualitative part discusses the team practices and the specific challenges that occurred during the last iteration or two. When issues have been identified, root cause analysis is performed, potential corrective actions are discussed, and improvement Stories are entered into the Team Backlog. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Effectiveness | Kanban for WIP management | Kanban, which means ‘visual signal,’ is a method for visualizing and managing work. While there are many interpretations of how to apply Kanban in development, most would agree that the primary aspects include the following: The system contains a series of states that define the workflow. The progress of items is tracked by visualizing all work. Teams agree on specific WIP limits for each state and change them when necessary to improve flow. Policies are adopted to specify the management of work. Flow is measured—work items are tracked from the time they enter the system to the time they leave—providing continuous indicators of the amount of work in process and the current lead time. In other words, how long, on average, it takes an item to get through the system. Classes of service are used to prioritize work based on the Cost of Delay (CoD). A team’s Kanban board evolves iteratively. After defining the initial process and WIP limits and executing for a while, bottlenecks should surface. If not, the team refines the process or further reduces some WIP limits until it becomes evident that a workflow state is ‘starving’ or is overloaded. Other changes might include merging, splitting, or redefining workflow states. This helps the team continually adjust the process to optimize flow. Kanban teams use objective measures, including average lead time, WIP, and throughput to understand their process and improve flow. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Effectiveness | Release Governance (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | Release planning is part of the PI planning process. But that’s the easy part; the hard part is coordinating the implementation of all the capabilities and features over multiple iterations within a release. This is especially true when new issues, roadblocks, dependencies, and gaps in Vision and backlogs are uncovered. Due to these challenges, the scope of each release must be continually managed, revalidated, and communicated. Primary considerations include: Ensuring that the organization’s release governance is understood and adhered Communicating release status to external stakeholders Ensuring that an appropriate deployment plan is in place Coordinating with marketing and with Product and Solution Management on internal and external communications Validating that the solution meets relevant solution quality and Compliance criteria Participating in Inspect and Adapt to improve the release process, value stream productivity, and solution quality Providing final authorization for the release Acting as a liaison with Lean Portfolio Management, as appropriate Participating in, and overseeing the final release activities. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Efficiency | Languages | Number of unsupported languages in the pipeline framework, | Daily, Weekly, Monthly, Quarterly | 120 Days | 180 Days | 300 Days | ||||||||||
| Efficiency | Tools | Pipeline shall support all the tools necessary to accomplish task in Section 5.5 (e.g. SonarQube, Fortify, 508 Testing, Dynamic Security Testing, etc.). | Daily, Weekly, Monthly, Quarterly | 120 Days | 180 Days | 300 Days | ||||||||||
| Efficiency | Test environment matches production | Does the test environment match the functionality of the production environment in regards to platform, OS, applications, interfaces, data, and simulated networking? - Production like testing environment | Daily, Weekly, Monthly, Quarterly | 90% of the test environments match production | 120 Days | 95% of the test environments match production | 180 Days | 100% of the test environments match production | 300 Days | |||||||
| Efficiency | Continuous Delivery Pipeline (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | The Continuous Delivery Pipeline (CDP) represents the workflows, activities, and automation needed to shepherd a new piece of functionality from ideation to an on-demand release of value to the end user. The SAFe continuous delivery pipeline model shows the flow of value through four aspects: continuous exploration, continuous integration, continuous deployment, and release on demand. The CDP enables organizations to map their current pipeline into a new structure and then use relentless improvement to deliver value to customers. Feedback loops that exist internally within and between the aspects, and externally between the customers and the enterprise, fuel improvements. Internal feedback loops often center on process improvements, while external feedback often centers on solution improvements. Collectively, the improvements create synergy in ensuring the enterprise is ‘building the right thing, the right way’ and delivering value to the market frequently. The paragraphs below describe each aspect. Continuous Exploration (CE) focuses on creating alignment on what needs to be built. In CE, design thinking is used to ensure the enterprise understands the market problem / customer need and the solution required to meet that need. It starts with an idea or a hypothesis of something that will provide value to customers, typically in response to customer feedback or market research. Ideas are then analyzed and further researched, leading to the understanding and convergence of what is needed as either a Minimum Viable Product (MVP) or Minimum Marketable Feature (MMF). These feed the solution space of exploring how existing architectures and solutions can, or should, be modified. Finally, convergence occurs by understanding which Capabilities and Features, if implemented, are likely to meet customer and market needs. Collectively, these are defined and prioritized in the Program Backlog. Continuous Integration (CI) focuses on taking features from the Program backlog and implementing them. In CI, the application of design thinking tools in the problem space focuses on refinement of features (e.g., designing a user story map), which may motivate more research and the use of solution space tools (such as user feedback on a paper prototype). After specific features are clearly understood, Agile Teams implement them. Completed work is committed to version control, built and integrated into a full system or solution, and tested end-to-end before being validated in a staging environment. Continuous Deployment (CD) takes the changes from the staging environment and deploys them to production. At that point, they’re verified and monitored to make sure they are working properly. This step makes the features available in production, where the business determines the appropriate time to release them to customers. This aspect also allows the organization to respond, rollback, or fix forward when necessary. Release on Demand (RoD) is the ability to make value available to customers all at once, or in a staggered fashion-based market and business needs. This provides the business the opportunity to measure hypothesis and learn, based on the customer response to the value released, what it needs to do next. Although it is described sequentially, the pipeline isn’t strictly linear. Rather, it’s a learning cycle that allows teams to establish one or more hypotheses, build a solution to test each hypothesis, and learn from that work, | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Efficiency | Continuous Exploration (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | Continuous Exploration (CE) is the process that drives innovation and fosters alignment on what should be built by continually exploring market and customer needs, and defining a Vision, Roadmap, and set of Features for a Solution that addresses those needs. During CE, new ideas are raised, refined, and prepared as a list of prioritized features in the Program Backlog. They are pulled into implementation during PI Planning, which kicks off the continuous integration process. After that, the continuous deployment cycle pulls the features into production, where they are validated and prepared for release. There are four expected activities of Continuous Exploration: Hypothesize describes the practices necessary to identify ideas, and the measurements needed to validate them with customers Collaborate and research describes the practices required to work with customers and stakeholders to refine the understandings of potential needs Architect describes the practices necessary to envision a technological approach that enables quick implementation, delivery, and support of ongoing operations Synthesize describes the practices that organize the ideas into a holistic vision, a roadmap, a prioritized program backlog, and supports final alignment during PI Planning. From their understanding of the marketplace—as well as from the Strategic Themes, Portfolio Vision, and the Solution Roadmap—Product and Solution Management have notions of what needs to be developed. These ideas, however, should not be considered facts that drive the teams. Instead, they should be considered as hypotheses. We don’t know if our ideas will work. To create a compelling and differentiated vision, Product Management facilitates a continuous and collaborative process, soliciting input from a diverse group of stakeholders. Conducting research enables the organization to further refine their processes and create artifacts that clearly express their emerging understanding of the problem space. With a clear understanding of the problem, CE moves into the solution space, starting with architecture. Architects serve the business and the customer by ensuring the architecture has sufficient Architectural Runway to delivery the required functionality and that the architecture is designed to enable the continuous delivery pipeline. Synthesize distills the knowledge gained into a new future state for the solution. The vision, roadmap, and prioritized backlog of features align the appropriate teams to a common, shared direction. Synthesis is focused on ensuring these assets are ready for PI planning. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Efficiency | Continuous Deployment (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | Continuous Deployment (CD) is the process that takes validated Features in a staging environment and deploys them into the production environment, where they are readied for release. Continuous deployment separates the deployment and release process. There are four activities of Continuous Deployment. Deploy to production describes the practices necessary to deploy a solution to a production environment Verify the solution describes the practices needed to make sure the changes operate in production as intended before they are released to customers Monitor for problems describes the practices to monitor and report on any issues that may arise in production Respond and recover describes the practices to rapidly address any problems that happen during deployment. Deployment is the migration of changes into a production environment. In the Continuous Delivery Pipeline, such changes are deployed continuously. Thus, partial functionality—i.e., deploying some of the stories that comprise a feature—can be implemented into production. Before being released to end users, deployments must be verified for functional integrity and robustness. When they’re coupled, deployment and release have to happen almost instantaneously, as decisions must be made immediately about whether to rollback or not. When they’re decoupled, however, there’s room to test new functionality extensively in production before approving it for release. Immediately following the migration to production, solutions undergo a final round of testing. Typically, this is conducted through smoke testing and/or light user acceptance testing, but also as a stress and performance test, which can only be done in production. This provides a critical sanity check that tests the behavior of the solution in an actual production environment. Verifying that deployed features didn’t break on their way into production is an essential pre-release quality check. However, teams also need to ensure they can measure a feature’s performance and value over its lifespan. The insights that drive this critical feedback loop come primarily from robust monitoring capabilities, which must be in place before release. Effective monitoring requires that full-stack telemetry is active for all features deployed through the Continuous Delivery Pipeline. This ensures that system performance, end-user behavior, incidents, and business value can be determined rapidly and accurately in production. That information allows tracking and monitoring of each feature, which increases the fidelity of the assertions about business value delivered, as well as increased responsiveness to production issues. While some business-value metrics cannot be collected until release, teams need to make sure they know how to measure them once the decision to release occurs. Three practices help support this: Full-stack telemetry – the ability to monitor for problems across the full stack that a system covers Visual displays – tools that display automated measurements Federated monitoring – consolidated monitoring across applications in the solution that creates a holistic view of problems and performance. The goal of respond and recover is to identify potential issues before they turn into incidents and to prevent them from affecting business operations. This requires the ability to detect difficulties internally before end users discover them, quickly identify root causes, and restore services with well-rehearsed procedures. In contrast, making hasty, reactive changes directly to production systems—‘just to keep the lights on’—invites configuration drift, unverified changes, and long-term risk. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | |||||||
| Continuous Security Monitoring | Security as code and penetration testing focus on preventing known vulnerabilities from getting to production. But it’s also important to test services continuously for newly discovered and reported vulnerabilities and to detect intrusions and attacks on services and infrastructure. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 9/24/21 | ||||||||
| Release on Demand (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | Release on Demand is the process that deploys new functionality into production and releases it immediately or incrementally to customers based on demand. Release on Demand is the final aspect in the four-part Continuous Delivery Pipeline of Continuous Exploration (CE), Continuous Integration (CI), Continuous Deployment, and Release on Demand. The four practices associated Release on Demand: Release describes the practices necessary to deliver the solution to end users, all at once or incrementally Stabilize and operate describes the practices needed to make sure the solution is working well from a functional and non-functional perspective Measure describes the practices necessary to quantify whether the newly-released functionality provides the intended value Learn describes the practices needed to decide what should be done with the information gathered and prepare for the next loop through the continuous delivery pipeline. When the Solution is in production and has been verified as operable, the time has come to make it available to customers. This is a crucial business decision, as releasing value too early or too late can have negative economic repercussions. Product management, in collaboration with other stakeholders, establishes policies that govern this process, ranging from automatically allowing qualified code to be released to customers or establishing a more formal review process with a manual gate. The more complex the system, the more likely there will be a manual gate to determine the answers to the critical questions presented earlier (what to release, to whom, and when). The changes to the solution have been verified after they were deployed, but once customers have access to them, new problems might arise. These new problems not only may be due to the increase in usage, but also due to unexpected usage patterns. Incidents and security threats must be resolved quickly and within agreed to Service Level Agreements (SLAs). The first activity of continuous exploration is to hypothesize—and as value is released to customers, it’s time to use the application telemetry to measure the hypothesis and the business value delivered. The information gathered from release is used to close the loop in the continuous delivery pipeline. Product management will use these data to make a choices about Epics, determining if the Epic hypothesis was proven true, if a pivot is warranted, or if the Epic should be stopped and development effort applied elsewhere. This is also where learning about the process comes into effect, and the information on how value flowed must be used to improve the continuous delivery pipeline. release governance facilitates the activities needed to help internal and external stakeholders receive and deploy the new solution. It also ensures that the most critical governance quality elements are appropriately addressed before deployment—particularly internal and external security, regulatory, and other compliance guidelines. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Training | Percent of agile teams trained on using the DevSecOps framework, | Daily, Weekly, Monthly, Quarterly | 85% of teams trained on using Framework | 90 Day | 100% of teams trained on using Framework | 180 Days | ||
| Efficiency | Training | Response to training requests should be completed within 10 business days of initial request. | Daily, Weekly, Monthly, Quarterly | Equal to or greater than 85% of all training requests completed within 10 business days | 90 Days | Equal to or greater than 95% of all training requests completed within 10 business days | 180 Days | Less than 2 exceptions per quarter | 300 Days |
| Efficiency | Canary Releases | These provide a mechanism for releasing the solution to a specific Customer segment and measuring the results, before expanding and releasing to more customers | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Value stream Mapping (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | An essential tool to improve the flow of value across the pipeline is value stream mapping. This tool provides the visibility needed to identify bottlenecks and problem areas to flow, as well as design a future state and create Enablers to improve the pipeline. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Decoupled Release Elements (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | This technique identifies specific release elements, each of which can be released independently. Even simple solutions will have multiple release elements, each operating with different release strategies. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Architecting for testability (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | Systems that can’t be easily tested can’t be readily changed. Systems designed and architected in a modular way enable continuous testing. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Separating deploy and release (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | In order to continuously deploy, the ability to release may need to be separate from the work of deploying to production. This requires architectural enablers that will allow functionality to be in production but hidden from Customers. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Architecting for Operations (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | Operational needs must be considered. Build telemetry and logging capabilities into every application and the solution as a whole. Allow services to be downgraded or even removed in times of high loads or in response to incidents. Build capabilities for fast recovery and fix-forward. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Threat Modeling (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | Information security considerations should start early, identifying threats to proposed architecture, infrastructure, and applications. And record important security requirements as Non-Functional Requirements to influence backlogs. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Solution Vision | The vision serves as the cornerstone for teams to understand the why for the features being developed. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Solution Roadmap | The solution roadmap provides a view into the near future for the release. It helps Product Management prioritize the work, enables System Architects to prioritize the runway, and provides visibility for Business Owners. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Clearly defined features in the backlog | Clearly defining features that fit in a Program Increments is critical for releases to align on what is needed and for teams to be able to plan. The backlog also reflects important security requirements. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Behavior Driven Development | BDD fosters collaboration between Product Management, Product Owners, and Agile Teams, which clarifies requirements by adding acceptance criteria. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Visual Displays | Tools that display automated measurements | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Federated Monitoring | Consolidated monitoring across applications in the solution that creates a holistic view of problems and performance | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Proactive Detection (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | A practice for proactively creating faults in the solution to identify potential problems and situations before they occur | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Cross-team collaboration (Documented evidence of this practice should be readily available and visible to any agency stakeholders as requested) | A mindset of cooperation across the Value Stream to identify and solve problems as they arise | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Dark Launches | The ability to deploy to a production environment without releasing the functionality to end users | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Feature Toggles | A technique to facilitate dark launches by implementing toggles in the code, which enables switching between old and new functionality | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Monitor Non-Functional Requirements (NFRs) | To avoid service disruptions, system attributes such as reliability, performance, maintainability, scalability, and usability must be continuously monitored. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Deployment Automation | The ability to deploy a tested solution automatically from check-in to production | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Selective Deployment | When automation deployment is not fully implemented, self-service deployment allows a single command to take solutions from staging to production | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Blue/green Deployment | A technique that permits automatic switching between two environments, one for deployment and one that is live | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Continuous Security Monitoring | Security as code and penetration testing focus on preventing known vulnerabilities from getting to production. But it’s also important to test services continuously for newly discovered and reported vulnerabilities and to detect intrusions and attacks on services and infrastructure. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Session Replay | The ability to replay end-user sessions to research incidents and identify problems | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Failover/Disaster recovery | Failures will occur. It’s vital to develop a failover mechanism to allow service to resume quickly, or even avoid service interruption. Disaster recovery must be planned, architected into the service, and practiced. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Rollback and Fix Forward | The ability to both rollback a solution quickly to a previous environment, or to fix a problem quickly through the pipeline without the need to rollback | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Immutable Infrastructure | this concept recommends that teams never change the elements of the production environment in an uncontrolled manner, but instead manage infrastructure changes through the Continuous Delivery Pipeline | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Version Control | environments should be maintained under version control in order to rollback quickly | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days |
| Efficiency | Metrics Radiator | For the Continuous Delivery Pipeline, it will be expected that the contractor tracks the following metrics on a Radiator shared with all stakeholders for review if they wish: | |||||||
| Flow Velocity: Flow Velocity is the number of Flow Items of each type completed over a particular period of time. This is also referred to as throughput. | |||||||||
| Flow Time: Identifies when Time to Value is getting longer. Flow Time measures the time it takes for Flow Items to go from ‘work start’ to ‘work complete’, including both active and wait times. | |||||||||
| Flow Efficiency: Flow Efficiency is the ratio of active time vs. wait time out of the total Flow Time. | |||||||||
| Flow Load: Flow Load measures the number of Flow Items currently in progress (active or waiting) within a particular value stream. | |||||||||
| Flow Distribution: Flow Distribution measures the ratio of the four Flow Items completed over a particular window of time. | Daily, Weekly, Monthly, Quarterly | 75% of teams compliant | 120 Days | 95% of teams compliant | 180 Days | 100% of teams compliant | 300 Days | ||
| Efficiency | Reviews | All review requests responded to within 10 business days. | Daily, Weekly, Monthly, Quarterly | Equal to or greater than 85% of all review requests completed within 10 business days | 90 Days | Equal to or greater than 95% of all review requests completed within 10 business days | 180 Days | Less than 2 exceptions per quarter | 300 Days |
| *Days indicated in the Target Date Progression Columns (F, H, J) refer to the number of days after acceptance of Product intake |
| ** future KPIs will be developed with Rally Dashboard PWS 5.1.2 |
File details come from the government source that posted it. Updated .