AMA_QA_Report_(9.25.19).pdf

Question Report

Report Generated: 9/25/2019 12:41

Topic Webinar ID

DevOps Enterprise BPA Ask

Me Anything 534‐717‐715

Question Details

No. Question Answer(s): Additional Follow‐On Follow‐On Answer(s):

1 is there any audio? Yes, the panelists will have audio

Software DevSecOps BPA: Will you be responding to any emailed submitted questions? ‐ John

Gannon, GDIT

Yes ‐ questions submitted via email will be answered and posted

Can you expand what your are looking for in Volume, Por olio Review? Is it a tradi onal past performance write‐up? The source code we develop for DoD is at a classification that does not permit access via a web portal.

Please clarify it is the Government’s intent that all bidders submit the worksheet only, and that bidders are not requested to submit any associated narrative and/or justification for the proposed rates in order to demonstrate fair and reasonable pricing.

DevSecOps Question#1: Is the solicitation a brand new contract or is this a new contract to continue support of an existing contract whose period of performance is ending? If this is to establish a new contract due to a period of performance ending, would the existing vendor be eligible to compete for the new contract?

6 Will slides be posted to FBO?

yes ‐ session is recorded and will be posted to FBO along with the slides

Cloud Services Blanket Purchase Agreement

Solicitation Number: FA8307‐19‐R‐0135

RE Factor 1 – Management review

“Portfolio indicates numerous and strong partnerships with certified/authorized reseller and/or partner networks. Positive aspects of the portfolio outweigh any negative aspects.”

This leaves out CSP’s who directly offer cloud services. This implies that only FedRAMP/DISA Cloud services are available vis a partnership with CSP’s, which is not true.

There are several FedRAMP Authorized IaaS/PaaS CSP’s with a JAB P‐ATO Moderate and DISA IL 2/4 who directly offers services to the Government without the use of resellers.

How can a direct seller CSP who sells direcly be compliant with factor 1 when it is mandated that you must be a partner

Software DevSecOps Services BPA:

a. Shall rate build‐up be based on Government or Contractor site?

b. What city and state location shall the rate build‐up be based upon?

c. Are there labor category requirements and/or descriptions for Levels 1, 2, and 3?

d. Is labor category Level 3 the more senior/expert level?

e. Will Task Order potentially require higher level security clearances than outlined in Attachment G

"able to obtain secret"?

f. Please confirm Attachment 3 is Attachment B

g. Please confirm Attachment 3/Attachment B can be manipulated to be compliant with contractor's estimating structure/disclosed estimating practices

h. Please clarify whether the customer is expecting the contractor to establish 1 set of rates at the

BPA level to apply to FFP and T&M/LH call orders or if two separately priced ratecards should be priced.

j. Can all volumes be combined into a single PDF file for submission?

a. In relation to Factor 2: Portfolio Review, will the Government accept links to DI2E project spaces where the Government may need to obtain access to a given project in order to view associated data and details?

b. Will this contract support AFLCMC/HNCP’s Cyber Enterprise Services (CES) contract requirements at the conclusion of the CES contract’s period of performance of 20 Feb 2021?

Why is the government not using DI2E tool sets from the DoD overall effort‐ Travis Lewis NextGen

Federal Systems because we mandated compliance with the DoD Enterprise DevSecOps inititive and stack has to be containerized using DoD approved containers and using CNCF compliant Kubernetes. DI2E is working with us on leveraging the stack as well but the scale of this will go beyond their existing IC customers.

a. Cloud Services BPA: 1. Attachment 1 ‐ ITO_for_Cloud: Will the government consider providing an architecture example to use for the basis of estimate in order to provide a price that is evaluated using “comparison of proposed prices received in the response to the RFQ”?

b. Attachment 1 ‐ ITO_for_Cloud: Will the government provide the methodology of the “price analysis” used to determine price reasonableness and completeness.

c. Attachment 1 ‐ ITO_for_Cloud: “Volume I: Administrative” per 3. Evaluation Factors is not being evaluated. Part of the 2‐page requirement is the opportunity to provide “a statement that the company understands the requirements specified and will meet the performance standards and requirements”. The remaining requirements (cover letter with their quote, POC and contact info and the statement of exceptions, if any and the validity of 120‐day quote) will not require space of two pages. Will the government consider changing the Volume I: Administration” page count to 1, move the compa

12 Will there be one awardee for each BPA? or multiple awards for each BPA?

Multiple awards. We are looking at awarding up to 15 companies per BPA

DevSecOps BPA: Do you want Volumes 1‐3 submitted as a single document or separate documents? ‐

Kara Haynes, TWD

Leidos: Attachment_1_‐ITO_SW_DevSecOps in Section 2, Basis for Award, states “Offerors who receive a Highly Capable Technical rating, a higher than Acceptable Portfolio Review rating, and their prices are determined fair and reasonable will be invited to the BPA Event Day tentatively scheduled for 13 November 2019 as a continuation of the technical evaluation.” It is unclear how the

Government will consistently determine that proposed labor rates are fair and reasonable as

Attachment_G_‐_Labor_Category_LevelUP does not provide education and experience requirements for each labor category. This will result in bidders using their own interpretation of each labor category. The rates proposed will be fair and reasonable to the offeror but may not be fair and reasonable to the Government who may have a different interpretation of the education and experience required for each labor category. Will the Government consider revising Attachment G to include education and experience requirements?

15 How does the contracting office plan to manage audit rights for orders under these BPAs?

How can we submit additional requirements (tool sets) for the BPA 2? To clarify: to whom should we submit new requirements as Mr. Chaillan suggested?

Please submit everything to the

Contracting Officers: Ms. Dawn Davenport

& Ms. Christina Fernandez

17 Are there plans to run Jenkins as a container ‐‐ inside Kubernetes?

Leidos: DevSecOps BPA ‐ If the Government is seeking access to the bidder’s source code repository, this may be problematic for many bidders. Bidders whom perform work for the DoD are required to adhere to strict security controls. These controls combined with contractual requirements limiting disclosure of customer code make it difficult for DoD contractors to provide access to their source code repositories. This may result in an advantage for commercial or non‐DoD contractors whom may more easily provide such access. To ensure an equitable evaluation of positive aspects, what may bidders provide which is equivalent to providing Government access to a source code repository for evaluation in the context of “Offeror provided its code repository or equivalent …”? live answered

19 When will you post the slides from today’s Ask Me Anything on FedBiz Opps?

Slides will be posted along with the AMA on FedBizOpps by close of business Central

Standard Time, Wednesday 02 October along with other questions already

Cloud Services BPA

The list of applications includes MSBuild. MSBuild is an application that is part of Visual Studio. Does the government require pricing for Visual Studio as this is not an application that can be purchased as a standalone product?

The pricing and availability of Cloud Services is dynamic in nature. Cloud Service Providers continually add new services and modify existing services. How will these changes be handled under this BPA?

Steve Coleman ‐ Microsoft

There is a requirement Bidding vendors will need to prove that their company can process data securely at the second‐highest security level for Defense Department systems, impact level five.

Does this mean we have to have developed a system in the cloud that is rated level five?

Ben Chicoski ‐ How will using these BPAs fit in the "order of preference" compared to the priority given to DoD ESI contracts?

Question was addressed live, but to expand further, lienses and services

(together) are not provided on ESI

Thanks for the follow‐up. I believe ESI provides

"related services" (i.e., related to SW or HW), so

Within the product portion of the BPA, only DevSecOps Pipeline tools are mentioned. I do not see any platforms (PKS, OpenShift, etc) listed. What is the plan to provide pricing for the platform? live answered

SW‐DevSecOps ‐ For resource planning purposes, can you provide insight into the November 13 event date (ahead of the additional information planned for Nov 1)? What type of personnel should we plan to make available? – Zeyad Mobassaleh, Unisys

SW‐DevSecOps ‐ Volume III Portfolio Reviews ‐ Can you clarify what “offerer may provide information for several Government or commercial clients…”? Is this past performance, descriptive examples, code examples? – Zeyad Mobassaleh, Unisys

Are companies that are foreign owned, but fully FOCI‐compliant (per the foreign ownership control regulations), elligible to be a prime on a LevelUP BPA?

Leidos: DevSecOps BPA ‐ Attachment_1_‐ITO_SW_DevSecOps in Section 2, Basis of Award, states:

“Note that the BPA Event Day is a continuation of the technical evaluation and attendance is mandatory in order to receive a BPA. Award determination will be contingent on the results of the

BPA Event Day.” What are the expectations for the bidders participation in BPA Event Day, i.e. are bidders just required to attend, or are there additional evaluation criteria which will be based upon an oral presentation?

Can vendors submit quotes for a subset of the skillsets listed in the BAA (the case for small businesses that do not have all the skillsets/job descriptions that are required in the BAA?) or are small businesses recommended to go through a prime?

SW‐DevSecOps ‐ Volume III Portfolio Reviews ‐ How should we interpret the page limitation (3 pages) for the portfolio review? Is this a description of content that includes links to source or other

DevSecOps components? – Zeyad Mobassaleh, Unisys

Leidos: DevSecOps BPA ‐ If the Government is not looking for bidders to provide links to publically available Web Sites, will you update the BPA to clarify this evaluation factor? If the Government is requesting bidders include links to publically available Web Sites, essentially providing infinite page count for Factor 2. This infinite expansion of page count may make it difficult to assess all bidders fairly and may extend the Government’s acquisition timeline. Will the Government please update

Factor 2 such that only material provided within the allocated page count is evaluated?

31 Will the responses to the questions posed in this forum be released on FBO as well?

Yes ‐ this session is recorded and will be posted to FBO along with the slides

SW‐DevSecOps ‐ Volume III Portfolio Reviews ‐ We assume the repository is set up specifically for 1:1 sharing for this acquisition (i.e. access provided to government by invitation, not an existing public repository?) – Zeyad Mobassaleh, Unisys

Will any of these BPAs entail supporting workforce and organizational development and transformation to support benefits realization of these new DevSecOps capabilities? Are these services factored into these three BPAs as well?

'‐Will this work be done in a common development environment (I.E. GovCloud and/or C2S) ‐ or is there also an on‐prem requirement? If so ‐ OpenStack? VMWare?

‐For the software development factories/labs that are expected to be supported ‐ is there any information on platform (cloud native and containerized ‐ or is there transition work that will be happening?) Is there a requirement to migrate existing development shops to Agile and CI/CD, or will the development teams be Agile‐focused and bought in to the CI/CD and DevSecOps methodologies?

‐Are development labs and pipelines intended to be universal ‐ one pipeline, one path to production ‐ or is the intent to support multiple mission partners and sites with varying infrastructure requirements?

‐Is the intent that CATO be provided by adherence to the DevSecOps process (similar to

SpaceCAMP), or by virtue of being built into a common, accredited platform and infrastructure?

What is the government's intent with any source code repositories that are supplied? How do you assess code quality?

Are you looking for DevSecOps capabilities for end to chain of Command for artifacts generated deployed and the Configuration, Change and incident management for these artifacts (content trust)?

37 Does the contractor have to have experience with LevelUP? No

SW‐DevSecOps ‐ Volume II Technical capability ‐ Can the government provide additional guidance on the various site locations and provide additional definition of co‐location (i.e. is staff located on government site or at offeror site nearby?) – Zeyad Mobassaleh, Unisys

Will you only be looking at security tools native to the CSP's or will 3rd party vendors be looked at to provide additional security controls?

40 How many awards does the government anticipate to make to small businesses?

41 Can small business joint ventures submit?

For all BPAs: You indicate a plan to award up to 15 BPAs across Large and Small Businesses. Do you have a target split between large and small?

there is no target split between large and small businesses. based on market research there is a very good mix between the two to support these capabilities

Leidos: Pipeline BPA ‐ Attachment_3_Pipeline_Pricing_Sheet: Only one Labor category is provided.

What is the government’s anticipated use of labor on this BPA vs. actual license purchases? Will be awardees be allowed to provide value add labor to provide thought leadership to guide the USAF in broader aspects of this critical capability?

44 Any portion of the funds allocated to small business?

There are no funds allocated to small business. Small business set‐asides will be conducted at the task order level based on the determination that there are enough small businesses capable of meeting requirements

Ref BPA 2 ‐ Pricing Template ‐ Worksheet for Container/Licenses listing different example DSOP tools

‐ can you provide explanation and/or a sample illustration on product pricing for this work sheet?

IndraSoft: In order to level the playing field between offerors, will the government please provide labor category descriptions (including requirements for education and years of experience) for each labor category included in Attachment 3, Pricing Sheet? ‐ relevant for Software DevSecOps BPA

Regarding Cloud Services BPA

As a follow up to a previous question...

What is the format and page limit for submitting technical capability?

Hello! For the Software DevSecOps Services BPA, should a company price all of the developer roles identified in the Pricing Sheet or just the roles that reflect their company strengths? Is it ok to only apply for some of the Labor Category roles. Thank you!

If awarded a BPA, will the contractor be able to add approved software tools after the award is made? live answered

50 What are the Evaluation Criteria for DevSecOps SVCs? Please refer to the instructions to the RFQ.

51 Are vendors expected to support ALL CNCF compliant distributions? No, as long as they support one.

52 What levels of support are vendors expected to provide through the help desk? (L1‐L3)

53 Are you open to ideas/experience from the commercial business sector? Yes

54 Is IL6 and IL5 Fedramp required? Is IL4 acceptable? live answered

55 Will task orders awarded on these BPAs be set‐aside for small businesses?

The Government has the ability to conduct small business set‐asides at the task order level

56 What is the pricing evaluation process and criteria for both the BPA and the Task Orders?

If a contractor receives an award for the Services BPA, is that contractor eligible to also receive an award for the Cloud or Tool BPA?

IndraSoft: Many of the tools offer a wide range of licensing options based on levels of services (e.g, Standard, Professional, Enterprise), number of users, remote access, managed services, SaaS access and/or features desired. Can the government clarify specific software products, number of users, and licensing model desired? ‐ Relevant to the Pipeline & Platform Integration and Licensing Services

BPA

59 IBM‐ Can any of the work be done remote from secure locations? Yes, a lot of the work will be unclassified anyway.

Logiksavvy Solutions: DevSecOps BPA

1. Is there currently an incumbent in this role or is it a new position?

a. If an incumbent is in place, would the DoD prefer to keep them in place for this new award?

b. If an incumbent in in place, would we be able to speak to them so we could include them in our response?

c. Can you provide guidance on the labor category they are currently working under?

61 How many exemplars are desired for the Portfolio? Is more better yes

62 IBM ‐ You would expect source code or mostly links to our capabilities and demos?

Will there be a small business track?

Do you require bidders to have prime past performance?

Are you accepting CTAs and JVs?

64 Can the work be done remotly or do ryou require the hours to be complete onsite?

yes. a lot of the work will be unclassified anyways

Logiksavvy Solutions: DevSecOps BPA

1. Does the winning company need to be a cleared facility? live answered

Will the list of prospective tools/products continously evolve, or will the list be 'locked' at a certain point? Continuously involved.

What is the estimated number of awards to be made? Is there a set number or ratio of Large award and small awards? live answered

Cole‐ BoxBoat Can the work be done remotely or do you require the hours to be completed onsite? live answered

Logiksavvy Solutions: DevSecOps BPA

Does the professional supporting the contract need to have a Secret or Top Secret clearance?

Clearance levels will determined at the call level.

Are you looking for DevSecOps capabilities for end to end Chain of Command for artifacts generated deployed? Is there a recommended platform for Configuration, Change and incident management for the artifacts (content trust)?

71 Do we have to submit resumes for Software DevSecOps Services at the time of bid? live answered

Can you please repeat your response. Did not understand resumes will be submitted when a call is issued against a BPA

Software DevSecOps

On the bottom of page 2 of the SOO it mentions “Support a 24 hour operations and/or business hour

(0830‐1630) help desk”. If conducted during business hours, would this need to be EST, CST, or MST?

73 Are you looking for any content management system? live answered

Any tool that can be applied to DevSecOps and will potentially add at the

IBM ‐ would the answer to all the questions be provided after this call or before next call? We have many similar ones so do not want to duplicate.

yes ‐ answers to all questions will be provided and posted to FBO

Software DevSecOps ‐ Altamira: will the government support/post an interested parties list to support teaming discussions? No.

Leidos: Cloud BPA ‐ Attachment_5_‐_BPA_Guide_for_Cloud_Services states: “This BPA will leverage various contract types to include Time & Materials/Labor Hour and Firm‐Fixed price.” Is it the

Government’s intent that labor will be purchased form this BPA?

77 In other words ‐ small businesses should not even consider moving forward? live answered

78 Are these prime only bids or are you looking for team bids? live answered

Leidos: Cloud BPA ‐ The Government is requesting 11 year pricing; however cloud service providers and their offerings will evolve over this period. What provisions does the BPA have allowing BPA holders to update their rate card (rates and items) as the cloud landscape evolves?

Applied Information Sciences (AIS) ‐ As it relates to teaming partners, can vendors submit as a prime contractor and also support other small business as subcontractor?

DEVSEC ‐ Sofware Lic. Procure ‐ If we have tools that support/protect the Cloud environment/infrastructure, would this be the best BPA (FA8307‐19‐R‐0134) to submit against. If our tools only support IL2 FedRamp Moderate is that an issue?

Yes, your tool needs to support IL5 minimum.

Nicolas, what about for the protection of the unclassified development areas. We are already

82 Will the government allow teaming for BPA onboarding? Yes

83 what is clearance requirement for AFLCMC? live answered

84 will labor category qualifications be provided? education/experience? Please refer to the initial RFQ.

85 Cole ‐ Boxboat : What is the expected range of award size and length at the task order level?

Leidos: All 3 BPAs ‐ Will the Government consider providing a representative call order for this BPA?

A representative call order allows bidders to clearly understand the commitment being made to the

AF as a BPA participant. This ensures all bidders understand the level of support required to participate in the BPA meeting the Government’s expected 10 day turnaround for call order proposals. Providing the representative call order is also advantageous to the Government as bidders not willing to make the required commitment to the AF will not burden the Government with evaluation of a BPA proposal.

Will these BPAs be written against an existing contract, such as GSA Sch 70 or SEWP? Or will they be standalone BPAs? live answered they are standalone BPAs and are not against a

Regarding "BPA 3" (Cloud Service Provider‐focused), Will the Government please clarify if the products and services must be on GSA Schedule? In other words, are offers restricted to companies that have Letters of Supply (LOS) from the associated cloud service providers? Industry suggests that such a restriction will limit competition, as the major cloud service providers undoubtedly anticipated for use under this contract have very limited LOS programs.

You need to decide the FCL requirement at the BPA level, or awardees may not be able to bid at the

Call Level ‐ Software DevSecOps requires up to TS/SCI (see your draft DD254). If you award to companies that do not hold an FCL, then they will not be able to hold or process security clearances.

90 Will bidders be evaluated on Prime capability, or whole tram capability? Whole team

Leidos: DevSecOps ‐ Attachment_G_‐_Labor_Category_LevelUP contains the labor categories one would expect to use when staffing DevSecOps teams. However, labor categories do not exist for leadership positions such as Scrum Master and Product Manager. Based on this observation it is unclear who provides these leadership roles. Does the Government intend to perform these leadership activities with this BPA primarily serving as staff augmentation to Government led teams, i.e. does the Government anticipate letting call orders for support staff (FTEs) or for the design, development, deployment, or maintenance of a complete product or service?

Chris Stone: CSEngineering: The question about source code being shared in the Portfolio volume is important for those of us who are already doing DevSecOps, but exclusively in the classified community. We wouldn’t score “highly” if it is a requirement to share the source code. And thus, would not bid on the BPA.

93 Will there be any set‐aside goals for SDVOSB, HUBZone or other classifications?

Not at this time, small business set‐asides will be set‐aside at the call level.

For all BPAs: Does the Gov't anticipate providing a forecast pipeline of calls to facilitate planning activity?

IndraSoft: Relevant to Software DevSecOps and Pipeline/Platform BPAs ‐ On page 2 in Section F.B of

Attachment 1 ITO, the Government states “If an Offeror chooses to obtain an NDA, to facilitate that process the Government is providing Abacus Technology Corporation, MITRE, and P3I, Inc. points of contact, with address, to whom Offerors should submit both their and any subcontractor NDAs for signature:” however it appears that the points of contact are missing after the colon. Please provide the POCs, with address, for Abacus, MITRE, and P3I referenced so that we (and our subcontractors) may obtain an NDA with them.

96 Is there a preference for open source? live answered

OM Group: SW DevSecOps ‐ it is not clear from DD254 provided as attachment E ‐ is facilitily clearance requirred at the time of BPA response submission? if so at what level?

98 Leidos: All BPAs ‐ Can teaming occur after an award on a BPA? live answered

We will get back to you on this question.

What do you anticipate will occur at the Event Day in November for those invited for Software

DevSecOps Services?

those invited for the Software DevSecOps

Services will validate their capability documents to the evaluation team as part of their evaluation (think of it as a pitch).

The final portion of the evaluation will determine if they will receive a BPA

Sam Sustaita (Go Virtual Hub) For R‐0135 Cloud Services please provide the labor categories that are expected. Thank You

101 Will you provide Labor category qualifications (yrs of experience and education requirements)

102 I assume you have to be an approved vendor on the BPA in order to respond to calls? live answered

103 Can you provide more technical documentation about the CloudOne environment? live answered

104 Does the government intend to deliver a technical challenge as part of its process? Not at this time.

Logiksavvy Solutions: DevSecOps BPA Will there be any additional consideration for certified Woman‐Owned Small Businesses for BPA's? live answered

There will not be any preference during

How will the Government ensure that Small Businesses with strong capabilities have a fair opportunity at winning a BPA if there are several Large Businesses with extensive experience and past performance that are also bidding? We understand there not set number of Large and Small

Business awards, but, would the Government consider establishing a number of ratio of awards to

Small Businesses to even the playing field?

107 Can government customers apply SBIR funding to purchase software off the BPA?

DevSecOps BPA: The majority of our DevSecOps portfolio has been under DHS. All these apps are internal applications (hosted on DHS's AWS enclave) that are only accessible from DHS networks with proper credentials. Are you only evaluating public accessbile work? or is there a method to evaluate work hosted in private government enclaves? ‐ Kara Haynes, TWD

The ITO states that NDAs must be submitted "within seven work days after the date of this RFQ".

Please confirm that these are due within seven work days of proposal submission. If not, will the

Government extend the due date for these NDAs through October 4, 2019?

For all BPAs: You indicated you would be revising the pricing templates. Since you also indicated that work may be performed at either Gov't or Contractor locations, will the pricing template include both On‐site and Off‐site rate tables?

You did not answer the question concerning applications written for the Government, CI/CD pipelines written for the Government ‐ we can not provide access to or source code for the proposal.

How do you propose we provide you access?

Are you open to receive suggestions for other open sources tools to be included on the list? If so, how would you like to receive those suggestions?

yes ‐ please submit all suggestions to the

Contracting Officers: Ms. Dawn Davenport

& Ms. Christina Fernandez

For DevSecOps Svcs BPA: Does the Portfolio Review have to be provided for only the Prime or will

Subcontractors' portfolios be evaluated as well? live answered

Reference the Portfolio Review requirement for providing source code or access to previous work.

Our work is writing source code or pipelines for the Government. In terms of providing access to this code, how do you propose we do that?

If you have a custom product, we will not be asking for the source code but again we are not looking for turnkey CI/CD pipelines but tools to build one ourselves, owned by the USG.

115 How many minimum code repository are required as PP?

Can you provide any additional detail about what the BPA DAY Event might look like in DC? THANKS

A TON!

Location and draft agenda will be posted to FBO before the next AMA

117 How will cross doamin be handled? live answered

118 Xeenius ‐ Do we need a secure facility as a requirement for responding?

Not sure you understand the question on source code and pipelines. We do not own the source code or pipelines. They are Government owned. If is a question of releasability outside the

Government owner and not classification.

120 All BPAs ‐Will each of these questions be recorded for each of us to review after this call?

Can you confirm that all questions answered by COB Wednesday will also include all of the questions submitted previously via email using the Government‐provided template? confirmed if a small business graduates after onboarding, will they be considered a small business on the BPA or will they no longer be a small business?

Would you consider extending the deadline for submission, since revised pricing template and anwser to questions won't occur until next Wednesday?

124 A you using “containers” as a synonym for Docker containers?

OCI compliant container and Docker is one of the options.