CBL-IPF_Capability_Delivery_SOP--Oct2021.docx

UNCLASSIFIED

(U) National Geospatial-Intelligence Agency (NGA)

Integrated Program Office-Foundation (IPF)

Foundation GEOINT Capability Delivery Standard Operating Procedure (SOP)

21 SEP 20

Version 1.0

(U) Document Change Record

Table is UNCLASSIFIED

Table of Contents

Tables

Figures

1.0 (U) Introduction

1.1 (U) Purpose

(U) Provide policies and guidance for improving Scaled Agile Framework (SAFe) and DevSecOps execution in the Foundation GEOINT (FG) Integrated Program Office (IPF). Implementing the policies and guidance in this SOP will ensure IPF develops and delivers capabilities using appropriate processes, best practices, and tools to assist its projects in achieving the maximum value from their development and sustainment contracts.

1.2 (U) IPF Goals

· (U) Align IPF Agile and DevSecOps practices with NGA’s Technology Strategy

· Build better software faster using modern software engineering practices

· Identify and mitigate blockers in the Continuous Integration/Continuous Delivery pipelines (Ex: Authority to Operate-ATO)

· (U) Support the selection of the most appropriate pipeline to meet each projects’ needs

· (U) Increase awareness, sharing, and implementation of best business practices

· Ex: tools usage, automation, process execution, contract language, and business intelligence

· (U) Adopt and implement appropriate policy guidance across IPF Projects

· Ex: Maximizing use of NGA provided pipeline tools, services, and infrastructure

· (U) Promote a culture of continuous improvement using IPF-defined metrics

1.3 (U) Scope

(U) This SOP applies to all IPF Projects, their contractors, and subcontractors executing capability development, deployment, operations, and sustainment activities on IPF contracts. It is a living document that is posted SOP Location and will be updated regularly as new innovations, tools, or processes are adopted.

1.4 (U) Intent

(U) This SOP describes policies, processes and agile development methodologies to direct development, integration, and operations and sustainment activities in all IPF contacts. It should also be used by contractors to describe how they will accomplish the requirements identified in IPF Statements of Work (SOW) and Performance Work Statements (PWS). The Government requires Agile Development Management Plans (ADMP) as a guide to implement a SAFe approach to delivering requirements provided through IPF contracts. Plans shall describe the activities and roles to support release trains of all FG agile efforts, to include earth modeling, vector, cartographic, publications, and sustainment services as required.

(U) The document, which uses call out boxes throughout, is a bidders’ library mandatory artifact in all IPF solicitations. Specific policy direction is highlighted in blue boxes as IPF Policy statements. The SOP also articulates direction where IPF prefers projects and developers migrate to but will allow flexibility to innovate to the maximum extent possible. These areas are highlighted in yellow boxes as IPF Preferred statements. The green boxes are areas from the NGA Technology Strategy that we are similarly emphasizing to ensure broader alignment with agency direction and priorities.

(U) This SOP also serves as a way to inform and educate the IPF team over time in order to maintain a more consistent level of capability delivery from our projects. To that end, projects should consider this document required reading for all new teammates, regardless of position.

2.0 (U) Agile Development and Deployment Approach

2.1 (U) Overview

(U) IPF projects use the SAFe methodology and DevSecOps best practices for the development and sustainment of capabilities required by their contracts. DevSecOps practices can be considered the implementation phase of the SAFe methodology because IPF teams use its processes to continuously deliver into operation new application code in frequent and repetitive cycles.

(U) Descriptions of SAFe/Agile activities in this SOP are meant to be informative and not prescriptive for IPF projects. This allows each project and their developer(s) the flexibility to innovate as SAFe/Agile continues to transform.

2.2 (U) SAFe Big Picture

(U) IPF follows the latest version of the SAFe framework and operates primarily at the levels depicted in Figure 1 below, where individual projects plan and execute their agile release trains (lower box) and the IPF Director and Mission Owner operate at the portfolio level (upper box). SAFe/Agile execution is done primarily at the Project level with the goal of increasing IPF-level Portfolio planning and execution in the future.

(U) As the SAFe framework continues to evolve, the expectation is that IPF Project GPMO’s and their contract development and sustainment teams will adapt their processes to stay current with relevant changes.

(U) Figure 1 - SAFe Big Picture

2.3 (U) Identification of Requirements

(U) IPF Projects manage requirements for new and existing capabilities using Agile Project Management and Continuous Collaboration Tools. Agile Project Management tools serves as the common system (e.g. JIRA) for updating and maintaining a prioritized backlog for planning, developing, and delivering software developed using a SAFe approach while Continuous Collaboration software (e.g. Confluence) provides a team workspace for knowledge sharing. As IPF Projects move to the Enterprise JIRA, one of the benefits will be increased visibility through mapping of their requirements to NGA’s Baseline.

(U) Figure 2 below from NGAs Systems Engineering Plan (SEP), depicts this mapping, and the requirements decomposition constructs of Solution Epic Lean Business Case (LBC), Solution Epic Minimum Viable Product Function (MVPf), Program Epic MVPf, and features. IPF is moving towards a more complete implementation of this model to realize greater synergy across its projects.

(U) NGA’s Enterprise Release Management (ERM) Team facilitates system development and integration efforts within and across IPOs and Strategic Initiatives to meet enterprise objectives for the NSG Enterprise. IPF will continue to collaborate with ERM to ensure we meet further objectives of common look and feel of project/program artifacts, top-to-bottom traceability of development activities and normalized Project and Program roadmaps.[footnoteRef:2] [2: NGA Systems Engineering Plan (SEP), Revision A, 9 OCT 2019, Section 3.10]

(U) Figure 2 - Requirements and Mission Needs Taxonomy JIRAIPF Policy: Projects and their development and/or sustainment contractors will use NGA-provided instances of JIRA and Confluence to the maximum extent possible. IPF GPMOs can grant waivers to this policy but only if a compelling business case is presented showing that use of the agency tools is demonstrably less efficient than a contractor’s solution.

2.3.1 (U) Build the Right Product with Customers

(U) IPF projects should leverage User Centered Design process, to ensure comprehensive customer engagements in creating “working software.” Projects should follow https://usability.gov for user experience (UX) best practices and guidelines. Below is the User-Centered Design Process Map which is composed of several methods and tasks. The type of site or product that is being developed, the requirements, team, timeline, and the environment in which development is taking place will determine the tasks to perform and the order in which to perform them.

(U) Figure 3 - User-Centered Design Process Map

NGA Technology Strategy: Focus more directly on customer needs and requirements. Build an iterative on products customers love and that meet their evolving needs. Reduce complexity and improve the experience for all users. Improve UX design will provide user-centric capabilities.

lists user research focuses on understanding user behaviors, needs, and motivations through observations techniques, task analysis, and other feedback methodologies.

2.4 (U) Release Planning

(U) IPF projects conduct a Release Planning Event (RPE) every 10 weeks to design the most valuable set of user stories balanced against schedule, budget, and developer capacity. The RPE is an iterative process of estimation and negotiation between the SF Product Owners, scrum teams, and Product Management resulting in a commitment from the scrum team to deliver a set of user stories during the release.

(U) For contracts with a lead integrator (e.g. FGS-IOS), the IPF Integrator coordinates with supported developer teams to synchronize each team’s release planning event. The integrator conducts a pre-release planning meeting with each Release Train to capture their support requirements for the Release. Then the Integrator presents a final Release plan after all the RPEs are completed. The below example represents a 10-week block with one release finishing and another beginning, complete with RPE’s and End of Sprint Reviews.

(U) Figure 4 - Release Planning Schedule w/ an Integrator supporting Development Teams

(U) Release planning requires the following activities, concepts, or deliverables:

(U) Story Point Estimation – Each development scrum team solicits feedback from other support scrum teams (providing expertise in development, architecture, automation, deployment, operations, security, and engineering) to develop an understanding of JIRA features and the derived user stories sufficient to produce a “t-shirt” sizing (small, medium, large) estimate for user stories. User story estimates for development scrum teams include tasks for test support, defect work off, bug fixes and other work required to deliver a fully functional capability at the end of each release. Estimating story points must be normalized so that estimates for features or epics that require support from multiple teams are based on the same story point definition. This leads to a shared basis for economic decisions. Tools like planning poker or other agile assessment strategies help the scrum team find story points for the tasks. (https://cardsmith.co, https://www.pointingpoker.com). IPF project teams follow the SAFe guidance for release and sprint planning.

(U) For Operations and Sustainment (O&S) teams, Kanban is an alternative agile method as opposed to scrum. Kanban consists of team members pulling from a common pool of items to perform reactive work such as upgrades, patches, and DR work-offs. This type of work does not lend itself to planning in a development release because it is difficult to anticipate the amount of work that will arrive in each period. Metrics that can be used to measure O&S team performance are provided in section 3.6.

(U) Negotiation / Prioritization - Each scrum team has a projected, estimated velocity (refined over time) for an upcoming release. Scrum teams review priorities from the team’s backlog during weekly CPM evaluations. Scrum teams reserve a percentage of their velocity (typically 20%) for unplanned activities that may occur during the release[footnoteRef:3]. Scrum teams work with Product Owners to cycle through the following activities until the scrum teams’ planned capacity is reached. [3: Unplanned technical tasks and new requirements (user stories) must be sized, prioritized and coordinated with the Product Owner for approval to work during a current release as they will impact existing Release commitments]

· Determining the teams’ velocity for each sprint based on historical velocity and member availability. Nominally each member can perform 1 point of work a day and can perform 8 points of development work per sprint (normally 2 weeks), although these will vary between teams. Points should be deducted for holidays and planned absences.

· Prioritizing the team's backlog in descending order of relative importance.

· Sum the story point estimates of the highest priority issues from the backlog that equals the scrum team's estimated capacity/velocity for the upcoming release.

(U) Product Backlogs (PB) – JIRA provides issue and project tracking for software development teams to improve code quality and the speed of development. Projects can manage product backlogs, sprint planning and tracking each scrum team through the entire release process. Each team can be aligned with one or more Product Backlogs, but this may vary between projects.

(U) Scrum Team Release Backlog – The JIRA planning feature lets the teams commit to stories from the Product Backlogs by a drag-n-drop capability of a story into the Release Backlog. The number of commits is automatically computed until the team’s velocity is reached. The Release Backlog is groomed throughout the Release during CPMs to support adaptive planning at each RPE.

(U) Release Commitment - The scrum team documents (e.g. JIRA) their delivery commitments by moving the issues from the Product Backlog to the Scrum Team’s Release Backlog. The Release Commitment is accomplished after Release Planning and prior to Sprint 1 planning.

(U) Definition of Done – The definition of done is a checklist of the value-added steps that the team is expected to successfully complete before it can declare its work to be potentially shippable. Backlog items have specific acceptance criteria specified by the Product Owner or user representative. Backlog items must pass acceptance criteria (verified through acceptance test) and the definition of done to be accepted by the Product Owner.

(U) Agile Release Train Planning – During the Agile Release Train (ART) planning session, features are derived from the business or architectural epic that are drafted by the SF Technical Executive (TX), and/or Product Owner, along with IPF System Architect/Engineering when required.

(U) Final Release Planning Event – Program-level release planning is culminated in a Final Release Planning Event (FRPE). Product owners and a scrum master are identified for each agile team to participate in the FRPE. All team members are present, the teams know their estimated velocity, and can perform story point estimating. The product owner and scrum master negotiate their commitments to develop Features for the next (ART) workload. The teams make their final commitments to a defined set of objectives to develop the features identified for the upcoming ART at the Feature Commitment meeting.

2.5 (U) Sprint Planning

(U) Sprint planning consists of identification and decomposition of issues from the release backlog into tasks that Scrum Teams commit to delivering over the coming sprint. Scrum Teams document their sprint commitments by assigning issues from the Scrum Team’s Release Backlog to their Sprint Backlogs. The scrum team works with their Product Owner to identify the issues they will commit to using the same general negotiation/prioritization approach identified in the previous section.

(U) Sprint deliveries are on a fixed time schedule. Costs are normally fixed for development and delivery capacity of story points but may be increased to handle surge requirements (e.g. to meet a critical delivery schedule). The trade-off is made by the Government Mission Owner for determining what requirements get developed within each Sprint.

(U) Commitment of specific capability for specific sprints is made by the Government with the responsible contractor agreement within each release planning cycle. Planning cycles are currently 10 weeks for IPF projects, which includes four 2-week sprints plus a 2-week Innovation and Planning (IP) iteration.

2.6 (U) DevSecOps and Agile- Melding Both To Deliver Capabilities

(U) In terms of software development, Agile improves the process of delivery; encouraging changes in the functions and practices of the Business and Development teams to better produce the project / product envisioned by the end-user, or customer. DevSecOps improves the lead time and frequency of delivery outcomes through enhanced engineering practices; promoting a more cohesive collaboration between Development, Security, and Operations teams as they work towards continuous integration and delivery. Both Agile and DevSecOps can be implemented to promote change and collaboration within their respective domains, resulting in a cultural shift in the practices of the individuals implementing them. In an ideal environment, an organization would employ both Agile and DevSecOps practices, however, it is important to note that DevSecOps can be implemented in any environment - Agile or otherwise.[footnoteRef:4] [4: GSA Tech Guides: Understanding the difference between Agile and DevSecOps; https://tech.gsa.gov/guides/understanding_differences_agile_devsecops/]

(U) NGA has endorsed both methodologies and IPF is committed to practicing each within their development and sustainment teams. Error! Reference source not found. below shows the DevSecOps process with the enabling information technology (IT) environment types aligned. Use of each environment by IPF projects may vary based on the type of project, available funding, and manning, etc. Error! Reference source not found.(following page) further describes each environment. NGA Technology Strategy: NGA’s pivot to a more consistent DevSecOps experience for builders and makers will significantly improve our ability to design, develop, and deliver applications that support current and future GEOINT capabilities.

(U) Figure 5 - DevSecOps Process with Environments Aligned

2.6.1 (U) Environment Types

(U) NGA has several development environment options available for both cleared and un-cleared developers. The environment selected will depend on a projects’ data requirements and the clearance level of the development team. Table 1 describes the environment types and the activities that occur in each. NGA has offerings in each of these categories which is posted to IPF’s DevSecOps SharePoint page: https://teams.intranet.nga.mil/sites/IPO_FG/eng/devops/Lists/EnvirOpts/AllItems.aspx

(U) These documents will be updated as each environment transforms. We welcome any IPF project inputs in an effort to keep the DevSecOps SharePoint site relevant.

(U) Table 1 - Environment TypesIPF Policy: Projects and their development and/or sustainment contractors will use NGA-provided environments to the maximum extent possible. IPF GPMOs can grant waivers to this policy but only if a compelling business case is presented showing that use of the agency provided environments has negative impacts (e.g. cost, schedule, or performance) to the contract.

2.6.2 (U) Continuous Integration and Continuous Delivery (CI/CD)IPF Policy: Projects and their development and/or sustainment contractors will use infrastructure automation tools (e.g. Cloud Formation, Ansible) when deploying software into production. This enables repeatable processes and consistency when releases are made. Automation scrips should be stored in an NGA approved source code repository (e.g. Gitlab) for version control and tracking purposes.

(U) IPF teams perform continuous integration and continuous delivery (CI/CD) of working products using DevSecOps best practices. IPF development scrum teams use a DevSecOps “pipeline” to continuously deliver into operation new application code in frequent and repetitive cycles; thereby executing the implementation phase of the SAFe methodology.

NGA Technology Strategy: Other crucial pieces to growing the builder and maker culture and moving to a service-centric architecture at NGA include (1) continuous integration and continuous deployments, including automated deployments to production; (2) active monitoring and alerting; and (3) automated unit, integration, and end-to-end testing in a deployment pipeline. All of these need to be in place so applications can be part of a service-centric architecture and used on a modern platform.

(U) CI/CD, highlighted in Error! Reference source not found.on the previous page, requires the following activities, tools, or deliverables:

(U) Software Build – IPF development scrum teams use tools to automate builds to ensure the quality and fidelity of the code. These tools perform testing such as static code analysis, memory leak analysis, load testing, unit testing and security scans. Developers use version control to check-in code daily to allow quicker response to integration issues.

(U) Code Transfer-Low to High – Since most of IPF development takes place on the low side with code being pushed high, a service called Code and Artifact Movement (CAM) aka Project Wormhole should be leveraged. This allows for automated code movement from XC -> UC -> SC -> TC. The CAM only moves code and artifacts from low to high and uses products such as GitLab for code repository and Nexus 3 for artifacts. The GitLab branch identified for transfer will be required to be a protected branch in order for things to be moved. Figure 5 depicts the NGA Enterprise Low to High DevOps Pipeline.

(U) Figure 6 - NGA Code Transfer from Low to High NGA Technology Strategy: (U) - Enable standard policies and implementation paths for building low and moving code high as the norm, rather than the exception.

(U) Software Deployment – IPF development scrum teams build reusable and automated deployment scripts for deployment of code into test, integration, and operational environments. These scripts consider the different configurations for each environment on multiple security domains to avoid expensive risky manual intervention in the deployment process. This automated approach to deployment ensures that fixes can be applied within 24 to 72 hours with a high level of confidence that no regressions are introduced to operations.

(U) Code Deployment Instructions - In IPF projects, the code developers and code deployers are often times different teams and/or vendors. To facilitate the deployment into QA, Test, and Operations, it is vital that the development teams write valid and complete deployment instructions. Development teams work with integration and deployment personnel to produce the deployment instructions outlined in (U) Appendix 3 IPF Deployment Instructions Template and Checklist. IPF Policy: (U) – IPF development scrum teams are required to test and deliver comprehensive code deployment instructions for use by personnel performing code deployment activities.

(U) Automated Tool Chain – An automated approach to integration and deployment is accomplished in part by building and applying an integrated automated ‘tool chain’ which typically contains the following categories of tools (See https://devx.pages.gs.mil/devsecops/tool-chain/tool-parity-dashboard/):

NGA Technology Strategy: Development Implementation Guidance for Application Development Teams Across NGA: For areas with multiple or conflicting tools, attempt to standardize on one tool or document which tool should be used in specific scenarios. For areas where no tools currently exist, attempt to standardize on a tool that most closely meets team needs.

· Application Lifecycle Management (ALM) – Application and Agile life cycle management tools create a standardized environment for communication and collaboration between development teams and related groups.

· Model-Based Systems Engineering tools - provide engineering and architecture models in a common, integrated environment that is machine readable and searchable.

· Artifact Management Repository – These tools provide a software repository for storing and versioning binary files and their associated metadata.

· Build – Build automation is used to script or automate the process of compiling computer source code into binary code.

· Testing – Automated testing tools include unit and acceptance testing, performance testing, and load testing.

· Continuous integration – CI tools automate the process of compiling code into a build after developers have checked their code into a central repository. After the CI server builds the system, it runs unit and integration tests, reports results, and typically releases a labeled version of deployable artifacts.

· Continuous delivery – Deployment tools automate application deployments through to the various environments. They facilitate rapid feedback and continuous delivery while providing the required audit trails, versioning, and approval tracking.

· Additional tools – There are numerous other important DevSecOps support tools and information for: configuration, logging, management and monitoring, provisioning, source code control, software versioning, continuous security, code review, and collaboration. More information can be found on the DevCorps site: https://devcorps.pages.gs.mil/Documentation/index.html

IPF Preferred: (U) – Projects should maximize use of NGA provided tool sets from Planning to Operations. This will promote interoperability not only in IPF, but across the agency. Other expected long term benefits include time and cost savings and increased efficiency for capabilities delivered.

2.7 (U) Sprint Execution

(U) Each scrum team works to implement the stories allocated to their iteration during sprint execution. Scrum teams deploy code to their Development Environment as products are created to support interaction with users, solicit feedback and take corrective actions. Two key standup meetings occur during sprint execution; scrum team standup meetings and the Scrum of Scrum (SoS) standup meetings.

(U) Scrum Standups: Each scrum team establishes a daily stand-up (no more than 15 minutes) during the sprint to assess, revise, and coordinate tasks between the team. Scrum team member attendance is mandatory, as the scrum team collectively discusses:

· What did I do yesterday?

· What will I be able to complete today?

· What’s preventing us from completing?

(U) Product Owners may attend to provide any additional comments to users. Product Owners occasionally may not attend daily. Other stakeholders may dial-in as needed but should hold questions, comments, or requests for a follow on discussion until after the daily stand-up is complete.

(U) Scrum of Scrum (SoS) Standups: Scrum teams, Scrum Masters will attend SoS standup meetings on the cadence determined by the supported activity when the teams are aligned with an ART. Scrum master’s attendance is mandatory at SoS meetings, as the scrum masters collectively discuss team progress, impediments, and identify future activities.

2.8 (U) Testing

(U) Testing is conducted to provide the highest quality product with the lowest number of possible problems, issues, and/or bugs introduced into production. Testing is achieved through both manual and/or automated means during each scheduled phase of the pipeline. There are many types of tests to ensure quality: test for code, interfaces, security, etc. The IPF GPMOs, in cooperation with their product owners, will work with the NGA Test Organization (NTO) to conduct IV&V and reach Cert-to-Field (CtF), when required.

(U) Appendix 4 FGMOD and FGS-IOS Testing Workflow Diagrams captures the complete testing workflows that the FG Modernization and FGS-IOS contractors have coordinated with IPF’s Sustainment Services GPMO and the NTO. These can be used as guides for other IPF projects when conducting testing. IPF Policy: (U) The following tests below will be conducted within each project unless IPF GPMOs approve alternative events (e.g. combining tests) to achieve their test objectives.

2.8.1 (U) Peer Code Review

(U) This is a process used for checking the work performed by one’s peers to ensure it meets specific criteria. In software development, a peer review is used in code development as a walk through, searching for errors quickly and easily. They speed up the time it takes for mistakes to be identified and corrected. The peer review should:

· Verify the work satisfies the specifications and/or expectations

· Identify any deviations from the ‘standards’ or accepted process

· Facilitates knowledge transfer and achieves a common understanding between team members

· Provide suggestions for improvements

(U) Any issues discovered are reviewed, prioritized, and depending on the severity of the issue, assigned to the appropriate developer(s) to correct.

(U) This verification ensures the code/package functions as expected and meets the recognized standards created and adhered to by the developers/leads on the project team.

(U) Peer reviews complement testing. IPF does not expect to replace test execution with peer reviews. For more information please visit: https://devcorps.pages.gs.mil/Documentation/process/code_review_guide.html

2.8.2 (U) Unit Testing (Development)

(U) Unit tests are normally automated and written and run by developers ensuring that a section (unit) of an application meets the validity of feature functionality. Developers should integrate unit test frameworks with the integrated development environment (IDE) and execution them along with the local IDE builds.

(U) Unit Tests also should be integrated with the CI pipeline in the IDE, to enable rapid re-execution with each code check-in; thereby quickly identifying any regressions to existing functionality and ensuring that architectural changes or refactoring does not introduce defects.

(U) Code review should include unit tests as a pre-condition. All code units (classes, function, procedures, etc.) developed should have unit test coverage. A developer should identify and implement all test scenarios to ensure proper system functionality based on user acceptance criteria and other non-functional requirements.

(U) Any bugs/issues that is discovered should be reviewed, prioritized, and code returned to the appropriate developer to be fixed.

2.8.3 (U) Integration Testing

(U) Once a build is complete and successfully deployed from development to QA with no conflict, integration testing begins. Integration testing is part of the development process where program units are combined and tested as groups of functionality designed and intended to interact with each other. Integration focuses primarily on the interfaces and the flow of data between each module of a system.

(U) The QA Test Engineer, with the support of the developer, executes this test in the QA Environment. The QA team verifies the individual package, relying on the developer’s knowledge of the package, and the expected behavior for each function as well as the combined behavior of the system as a whole. Any exposed issue is reviewed, and prioritized. Depending on the severity of the issue, it is assigned to the appropriate developers to correct.

2.8.4 (U) Security Test

(U) IPF DevSecOps teams will ensure code is stored in the NGA approved Source Control Management Tools (e.g. Gitlab). The developer runs their build through a series of steps using a Continuous Integration (CI) Server (e.g. Jenkins) such as fetching code and running the identified continuous security software (e.g. HP Fortify). If there are no identified critical or high findings after the scans complete, the developer may proceed to deploy into the appropriate staging environment. Once all testing is complete, including unit and user testing, the developer will use automated deployment wherever possible. IPF Policy: (U) IPF contractors shall use NGA approved products, such as HP Fortify for code quality inspection and security vulnerability (static and dynamic) in the NGA provided environments on each of the available security domains and report the findings and remediation steps to the GPMO so the code meets security assurance requirements and the findings can be stored for NGA’s further evaluation.

2.8.5 (U) Regression Testing – Customer Site Test

(U) Regression testing verifies that the software modifications did not cause any unintended or adverse side effects in the Build. This form of testing is often known as end to end testing. Manual and Automated (if available) test execution can begin once the expected build is deployed to the testing environment and the required test data is located. Test Files/Scripts shall be stored in the appropriate planning tool and are executed to meet the requirements of each release. QA personnel begin by testing each of the Bugs, and Stories (Product Owner and Technical requirements).

(U) Any issues discovered are reviewed, prioritized, and depending on the severity of the defect/bug, each is assigned to the appropriate future Hotfix or Release, as determined by the project team and the product owner.

2.8.6 (U) User Acceptance Testing (UAT) – Customer Site Test

(U) UAT verifies that the software meets the customer’s (real users) expectations. The expected build is deployed to an environment accessible by the customer on-site. The customer executes UAT and support is available from project team members as needed. User Acceptance Testers execute test plan and confirm noted Bug fixes and stories fulfill the customer’s expectations. Any issues discovered are reviewed, prioritized, and depending on the severity of the defect/Bug, each is assigned to the appropriate future Hotfix or Release, as determined by the project team and the product owner. Test Files/Scripts shall be stored in a code repository and are executed to meet the release requirements.

2.8.7 (U) NGA Test Organization (NTO)

(U) Integrated features/stories developed by IPF scrum teams are tested following the NTO Assisted-Service Test workflow and roles (next page) to achieve Cert-to-Field for all major software version deployments into operations. IPF PMs must submit separate Request for Service (RFS) for install and test which are created and tracked in the NTO VOLT system. If possible both RFS’ should be submitted prior to the start of the scheduled install. The Install RFS includes submitting the Version Description Document (VDD) for the baseline to test. During planning, the IPF PM coordinates the test configuration with the integrator/O&S contractor in charge of the system.

(U) IPF PMs must create a Test Readiness Reviews (TRR) presentation to be reviewed with testers prior to the start of IV&V. The TRR presentation is not necessary for minor releases or after the initial IV&V and/or initial deployment to Operations has been completed. Once approved, the Product Owner’s Team leads testing using the Test Plan. Upon completion, the Product Owner team provides test results (which include any open discrepancy reports plus an acceptance recommendation) to the PM for creation of the Assisted-Service Test Report. The PM then submits the report to the NTO for review and issuance of a Cert to Field (CtF). Self-Service Testing Planning/Execution Checklist is located in (U) Appendix 2 Assisted-Service Testing/Execution Checklist.

(U) Figure 7 – Assisted-Service Test Planning/Execution Workflow

· Two RFS’ are required

· NIIS Install RFS

· Assisted Service Test RDS (either IV&V or Regression)

· Write and verify DRs in conjunction with SF users

· Manage and execute testing

· Activities and outcomes requiring independent testers

· IV&V

· Emergency Regression Test

· Regression Test

· Certification to Field

· Respond to OCIO security scan findings (typically handled by Contractor team in charge of install)

· Create/complete Assisted Service Test Report

· Identify POC(s) for RFS Coordination

· Coordinates NIIS installs and Security scan reviews

· Manage resource de-confliction and scheduling (RAM process)

· Facilitate new and modified DR requests in Remedy

· Provide infrastructure services support throughout test

· Provides VDD and Test Results

· Support User test activities

(U) Table 2 - Assisted-Service IV&V Roles

2.9 (U) Sprint Closeout

(U) Sprints are completed and closed out when Sprint Reviews and Sprint Retrospectives are completed. Sprint Reviews are facilitated through Sprint Demonstrations that are conducted from the Development/Test environment.

(U) Sprint Demonstrations: These allow the team to demonstrate to the Product Owner how the acceptance criteria for each completed user story has been met. The Product Owner must approve each user story before the story is considered complete. Sprint Demonstrations include a demonstration of the working product to the Product Owner and other stakeholders; the format and extent of the demonstrations may vary depending on the scope of the working products, and audience. This creates a regular, short-interval feedback loop and allows problems, issues and other information to be raised even if insufficient value has been created to release the product to the customers at that time. During the sprint demonstration, the scrum team demonstrates to the Product Owner's satisfaction that the software design complies with established policies, satisfies requirements and supports an acceptable implementation.

(U) Sprint Retrospective: Scrum teams meet for a Sprint Retrospective at the end of every sprint to reflect on what went well, what didn’t, and what could be done better next time. If there is variance in the expected velocity, a brief summary should be documented to capture the reasons why. The sprint retrospective summary will help document process improvement ideas as well as protect the developers if there are external influences prohibiting work. The goal of the scrum teams, particularly the Scrum Masters, should be to remove an impediments as quickly as possible.

2.10 (U) Release Closeout

(U) The Release Closeout is the final review of the Release Backlog to ensure all open or in-progress items are returned to the appropriate Product Backlog and all resolved stories and tasks are closed out. Release Notes are published (e.g. Confluence), and final release metrics are computed and published to the Continuous Collaboration software (e.g Confluence). The Sprint Retrospective is also published to Confluence. The Agile Release Train (ART) closeout occurs in the last sprint, the IP[footnoteRef:5] sprint. [5: (U) “Scaled Agile FrameworkTM Innovation and Planning Iteration,” last modified November 1, 2017, WWW Link - http://www.scaledagileframework.com/innovation-and-planning-iteration/.]

3.0 (U) Agile Management Approach

(U) The following description of IPF’s SAFe/Agile approach is provided as guidance and not a mandate. Presently IPF executes SAFe primarily at the Program/Project and Team level with the goal of improving Portfolio-level planning in the future. To that end, we highlight the different levels of the SAFe framework and the activities that occur in each.

3.1 (U) Portfolio-level (IPF Mission Owner Level)

(U) The Portfolio level is where strategy is formulated, investment decisions are made, and where engineering planning and execution begins. Portfolio-level business or architectural Epics are value-descriptions for the big things NGA would like to do; large-scale development initiatives that realize the value of the Investment Themes for NGA which result in improving the GEOINT analytical and production processes. NGA stakeholder organizations (S, A, G, R, and CIO/T) identify new capability needs and capability enhancement needs both business and architectural. As Epics may be decomposed into features, each feature created will “stand on its own” as an “epic” that is engineered, managed, and developed during the agile engineering and development processes (currently executed at the IPF Project-level). An Epic may be developed by a single development team or it may be developed by multiple development teams. The Epic Owner may also be referred to as the Business Product Owner, or in IPO parlance, the Mission Owner.

3.2 (U) Program-Level (IPF Project Level)

(U) The Program-level is the engine of value delivery, and in IPF, this normally starts at the SF three-letter office level (e.g. SFA, SFH, SFN, etc). With a Technical Executive (TX)-provided roadmap and vision, IPF Projects and SF Product Managers (PdM), who are also the TXs, and SF Product Owners (PdO), work with the Development Teams to achieve the SF Office’s objectives. The Program-level value description and the key artifact is the Feature, which is derived from the Business or Architectural Epics received from the applicable SF TX. The Release Train Engineer is responsible for scheduling current approved Features into the Agile Release Train, which is a term used to describe a virtual organization of 5-12 teams that plans, commits, and executes together. The train delivers valuable Program Increments (PI) every 10 weeks (4-two week sprints plus a 2-week Innovation and Planning (IP) Increment) to the enterprise.

3.3 (U) Team-Level

(U) Task Orders, Agile Development teams and initiatives use their choice of development methods to manage their workload during the Agile Release Train. The Agile teams use the agile scrum and Extreme Programming (XP) methodologies and the power of software engineering to build working software.

(U) The Team-level value description and key artifact is the User Story, written in terms of what the user wants and what value the user gains from the solution. The Product Owners, which come from SF Offices drive the prioritization and the build-order and the Scrum Master drives the solution to market. IPF Projects schedule sprint demonstrations at the end of each sprint which is critical to gain user feedback. If the sprint demonstration isn’t accomplished every two weeks, then the Agile Teams are not driving agility. Once work is scheduled for an existing sprint, new work is only added unless approved by the Product Owner. Release Backlog grooming is acceptable at the end of each sprint and throughout the Release to facilitate adaptive planning.

(U) The IPF Release cadence is 10 weeks in duration. During the Release Planning Event the scrum teams commit to the work in their Product Backlogs based on their velocities for the upcoming Release Planning. Teams should also align with and adopt the programs they support. The development teams coordinate with their supporting integrator and enterprise services to document all dependencies and required GFE.

(U) Figure 8 - Program Increment Process

(U) The Team develops and delivers the code to meet the Features that they committed to during Release Planning. Completed code and DR/bug fixes (demonstrated to the Product Owner and approved as complete) are continually deployed for testing. See (U) Appendix 4 FGMOD and FGS-IOS Testing Workflow Diagrams for complete testing and deployment workflows.

3.4 (U) Configuration Management

(U) Configuration Management (CM) is a systems engineering process for establishing and maintaining consistency of a system’s performance, functional and physical attributes with its requirements, design and operational information throughout its life. CM provides change management and traceability for all changes made to the following Configuration Items (CI):

· All software, licenses and manuals (including COTS/GOTS) relating to IPF services.

· The software technical documentation baselines developed under IPF contracts, including the source code, build files, and related software documentation.

· All hardware and software provided by the Government (GFX) or Contractor Acquired Property (CAP) until returned to the Government. The Contractor will provide an annual listing of GFX/CAP hardware/software in a government-directed format.

· Other data developed as part of IPF services for use in development and sustainment.

· All work maintained in the JIRA backlog.

(U) Changes are monitored through the Product Backlog to manage the change and evaluate its impact on other parts of the applications. CM details the specifics of each CM function and addresses change escalation mechanisms.

(U) The four functions of Configuration Management include:

· Configuration Identification: Identify project items (called Configuration items (CIs)) that need to be managed by CM.

· Configuration Change Management: Identify and manage project changes.

· Configuration Status Accounting: Measure progress and accomplishments of the project's CIs.

· Configuration Verification and Audit: Independent verification that the project's work has been accomplished.

(U) Additional NGA Configuration Management (CM) documents to include the NGA Configuration Management Process (NCMP) Revision E, December 27, 2019 located at: https://intelshare.intelink.gov/sites/taef/mini-library/default.aspx ):

(U) As teams align to their programs cadence and backlog, the program is the arbiter of the build order and prioritization of the workload.

(U) During each release, the Release Manager addresses enterprise requirements for approval to deploy software (Software Approval Process (SWAP) and CIO/T whitelist). The deployment package will be handed off for direct deployment to operations.

3.4.1 (U) Change Impacts

(U) IPF may initiate a directive for a contractor to change a tool or process that may impact cost, schedule or performance. IPF recognizes that, short of a contract modification, all changes are not equal in terms of impacts and has categorized different levels of change and the time allowed for a contractor to implement the change.

(U) Table 3 - IPF Change Management Categories

3.5 (U) Risk and Opportunity Management

(U) Risk Planning is incorporated in all Release/Sprint Planning events and managed throughout the Agile Release Train (ART) during the Daily Scrum and Scrum of Scrum meetings. A key tenant of Agile is transparency, which is the concept of straightforward, recurrent communication as well as openness in methods and decision-making. Risk Mitigation relies on strong and frequent communications.

(U) SAFe recommends that the riskiest items are worked first and with full transparency. Large problems are decomposed into smaller, lower risk items that are more manageable.

(U) Agile Risk Management follows the Pareto Principle, or the 80/20 rule. For example, it can be generalized that 80% of a program’s risk is found in only about 20% of its issues. Conversely, 20% of requirements are going to provide the top 80% of functionality.

(U) Risks that cross Portfolios and Portfolio Initiatives should be addressed at the Portfolio level on a quarterly basis. Release Train level risks should be addressed every 2 weeks at the end of Sprint review. Release Trains present their risks at their RPE out brief. Team level risks should be addressed daily.

(U) Risks should be elevated when they cross ART’s and involve things that individual trains can’t solve.