VAULT Town Hall Meeting QA Session w_Chatbox Transcript.docx

Intent Collaborate with industry on the Visible Accessible Understandable Linked Trusted (VAULT) requirement. To gain a better understanding of industry standards/practices, the discussion will be framed around focused topics.

Q: Question D: Discussion A: Government Response Discussion and Q&As Q: Since this is a multi-award IDIQ, could the task orders be different contract types?

D: Time and Materials and FFP are the best contract types.

D: Time and materials will provide type the required level of skill when you need it.

D: Ex: 5 FTEs requirement and contractor authorizes 12 people and the contractor is required to stay under the ceiling of 5 FTEs.

D: The flexibility is required to move people in and out as the requirement ebbs and flows at different times of the requirement.

A: Thank you for your suggestion. Nothing set in stone yet (as far as contract type). The government is seeking feedback from industry.

Q: Will there be a small business set aside?

A: The government doesn’t know at this time. Market research will dictate that. The FAR says the rule of two applies. If there are two or more capable SBs. There’s new upcoming guidance that if you have a multi-award IDIQ the CO shall set aside task orders within the multi-award IDIQ. Ultimately, the FAR will be the governing document after market research indicates the best acquisition strategy.

Industry Comment: I agree with multiple contract types approach CPFF variance in work.

Government: Cannot do cost type contracts since the requirement is deemed commercial.

Industry Comment: Firm Fixed Price contracts have had success for large requirements. T&M has had some success. At this dollar amount FFP is ideal.

Industry Comment: We recommend changing the NAICS to 541611 Government: That NAICS does not apply to our requirement.

Industry Comment: Commercial acquisitions have had success with large requirements using T&M with incentive milestones. It encourages early/on time delivery. Set a date for a milestone and give an incentive for delivery for on time/early delivery.

Q: Have you considered setting a rate schedule at the IDIQ level?

A: Yes it would be easier and allow quick-turns on award. We won’t have to evaluate pricing [meaning professional employee compensation plans] at the Task Order (TO) level.

Industry Comment: FBI identified the ability to award task orders to small businesses using contract value as a determining factor.

Q: What can you share about the currently identified task orders? Are some enterprise wide? Can you discuss the initial task order?

A: Initial task order will most likely be a kickoff meeting or for SAF/CO. Some task orders will affect the entire AF enterprise.

Q: Will the government describe labor categories (LCAT) at IDIQ level or we (industry) be expected to propose?

Industry Comment: We recommend the government propose LCATs at IDIQ level. According to the PWS, there is a broad range of skills and LCATs needed due to varying range of each task order.

A: The government will provide LCATs in the request for proposals and we are looking for recommendations from industry. We suggest providing a “blank” LCAT to industry after award to provide flexibility in the case that the TO requires a support that an awardee feels a different SME would be better suited.

Q: How does the government expect the multi-step advisory (MSA) process submission?

A: FAR 15.202 explains the MSA process. Oral presentations may be utilized for the MSA process which is dependent on the current situation and availability.

Q: How many FTEs does the government expect for each task order?

A: The government has an approximate immediate need for 30+ offices. Each is estimated at 10 FTEs per task order.

Q: Is the Government open to receiving further info additional info?

A: Yes, we are willing to receive any related information until we solicit. We may not respond to all submissions.

Q: Are SME in data science service applied to data in a central data platform for this requirement?

A: Some task orders yes some no.

Q: Will some SME data services be assigned to multiple locations?

A: Yes Q: Is the government going to require fortune 500 experience?

A: The government has not identified that need at this time.

Q: What degree will government value work outside the AF?

A: Yes there is value added for commercial experience and capabilities. They are both equally valued. The emphasis is on technical capability.

Industry Comment: We recommend a data analytics challenge.

Industry Comment: We suggest setting this up as a product not a service.

Q: What do you project the average period of performance for each task order?

A: The average is unknown, but the government is looking to limit each task order to 2 years.

Q: Explain commercial product.

A: Commercial in this sense means a service that can be purchased by other than government entities. Commercial experience is speaking to non-government organizations related experience.

Q: What is the government looking for to award at the TO level?

A: Task order level looking for the technical solution, staffing plan, and price. It will be streamlined.

Industry Comment: My concern is about 2 year task orders. Current state of demand for Data Scientist is extremely high. Recommend building a bench of professionals in data science that stay on for the 5 year ordering period. There’s uncertainty in turnover if the task orders are less than the entire ordering period. Recommend those individuals are identified at the IDIQ level not TO level.

Q: Are we going to attract non-traditional small businesses (NTSB) in this requirement?

Q: Is the definition of NTS the same as for SBIR?

A: The government is interested in NTSBs but the definition is a little different than with other programs. “Nontraditional Small Business (NTSB) Contractor” means a small business entity that is primarily engaged in similar lines of activity as listed and described in the North American Industry Classification System (NAICS) manual, having met the small business standards for that industry, established by the Small Business Administration, that is not currently performing and *has not performed any contract or subcontract for the Federal Government. Any entity organized for profit (even if its ownership is in the hands of a nonprofit entity) with a place of business located in the United States or its outlying areas and that makes a significant contribution to the U.S. economy through payment of taxes and/or use of American products, material and/or labor, etc. A NTSB Contractor includes but is not limited to an individual, partnership, corporation, joint venture, association, or cooperative. *If the NTSB has received a contract(s) through “Pitch Day” or Small Business Innovation Program, they can be considered a NTSB. NTSB’s will only retain NTSB status for up to two (2) years.

Q: With a $950M ceiling do you intend to accomplish 80 task orders with 10 FTEs each year? Is that the plan for this requirement?

A: The government is not sure how much or how often it will be utilized. This vehicle will be utilized at a minimum by the entire Air Force to include the Space Force.

Q: What is the relationship between this contract and JEDI?

A: No relationship exists with JEDI and this contract.

Q: Any OCI concerns?

A: Depending on the access to the information you have, it may possibly lead to an OCI and exclude you from future requirements. These will be determined on a case by case basis. The Government will try and notify you up front.

Q: Any OCI with the data platform developer?

A: No, you cannot define that at the IDIQ level. Once requirement is known, an OCI determination will be made.

Q: Could you provide detail on separating data platform in vault will it cure/clean data or will tasks on vault be getting data in the right platform?

A: It will be both. It will depend on the organization and task order.

Industry Comment: 541519 NAICS suggested as more appropriate. Has a higher small business size standard $30M.

Q: Respect to NTSB, are firms that are non-US based considered?

A: No they are not.

Q: Industry Comment: Setting a price on FTEs at task order level will keep people from diving to the bottom. Will you consider that approach on this?

A: The government will consider this.

Q: Was the use of schedule 70 considered?

A: Yes, initially. Now with the magnitude of this requirement it is not feasible or manageable to process 30+ task orders for the government.

Q: Does the government intend to include an initial award with the IDIQ RFP?

A: Everyone will get a kickoff task order.

Industry Comment: Background intellectual property is important experience. Reduces tech performance risk and shortens schedule and reduces cost. Use own companies IP and branch it forward to the government. The company should keep the ownership while allowing the government to use it.

Q: Would the government consider joint data rights?

A: Yes we would consider it.

Q: Will there be a ceiling on individual task orders?

A: Yes there has to be a minimum/maximum.

Q: Can we change the response to each task order to 14 days?

A: The government is currently looking at a 30/10/3 day response. Each TO will drive the response timeline.

Industry Comment: Off-ramp companies that do not bid on set percentage of RFPs.

Industry Comment: With the magnitude of this requirement (30+ clients), may want to have options to on ramp.

Q: Is on ramp/off ramping to be done annually or as required?

A: Annually. For off ramping we are looking at no bidding, poor performance and not winning task orders.

Industry Comment: Past performance is vital. Industry cautions against lowering past performance requirements.

Q: Would a company that submits min past performance be evaluated less favorable than one submitting the max?

A: No, as of right now technical capability is most important but that is subject to change.

Q: Is the government considering classified responses to TOs?

A: No.

Q: Why isn’t the Government buying solutions instead of SMEs.

A: We don’t want vendor lock-in. Most task orders won’t need to scale.

Chat Box Dialogue

· I would concur. This approach gives the government more flexiblity when issuing task orders.

· Cost Plus should be one of the contract types considered as well.

· T&M and FFP look to be the best options , especially if a commerical contract

· Leave the USAF maximum flexibility and use a hybrid contract type for this IDIQ, which includes T&M, FFP, and CR (ODCs).

· We perform work on the commercial side as well. One item that works well is T&M with incentive based major milestones to help accelerate delivery and create more of a partnership with delivery.

· we highly recommend oral presentations to verify companies' capabilities

· We should make sure the down select process is not just based on past performance ,It should be focused on technical

· 20. When I hear commercial, I think outcome-based, performance-oriented work versus staff augmentation. The outcome-based work would also imply that the offeror have a production method to produce enterprise data science and analytic capabilities. Can you please say more about what the government is looking for?

· Key Personnel is a good element, but when more than a few are listed, creates bias. It definitely creates a hurdle for smaller/medium companies who don't have the bench that large companies do. Also, who has bench in today's world. Contingent hires, especially for constrained skills, will likely be realistic for all.

· VTC orals may enable orals to occur efficiently

· Consur that VTC orals can work very well. We've done and have some thoughts to share if youd like

· Given that there will be multiple TO's - I would recommend that Key Personnel be done at the TO level not the ID/IQ level as previously suggested.

· This PWS is almost exclusively services focused. However, many of the items listed in this draft PWS can be covered in non-developmental efforts by existing commercial capabiilties. While we understand the requirement to bring additional talent into the Air Force, how do you ensure you will bring in and evaluate commerical technologies into this process so you aren't re-building existing capability? In our experience, it's beneficial to the government, and in line with FAR Part 12, to use separate market research and solicictation to carve out items which can be fulfilled by commercial products.

· Will the Government also share the Q&A from this Webex we had issues with the audio at times.

· Can you address the Small Business Set Aside strategy

· Is commerical defined as non DOD and USAF or is commerical defined as not government?

· Regarding NAICS: From our experience, most agencies and branches use the traditional NAICS for data warehousing given it's related to data analytics

· https://www.naics.com/naics-code-description/?code=541512

· https://www.naics.com/naics-code-description/?code=541519

· Is the AF still considering the "off-ramping" strategy as communicated in the first industry day?

· At the TO level since the individuals desired are realy high level SMEs, what type of eval critieria do you imagine using so that the TOs don't end up a race to the bottom for pricing versus finding the best personnel at a "reasonable" price for those difficult to hire and retain thse senior staff?

· If I understood correctly, looking at a FAR 15.202 process. Consequently can you share the criteria for the initial evaluation? Either on this call or as an update on beta.sam, even if it is draft? The more time industry has to prepare the better our responses will be, including allowing for teaming.

· referring to the "bench" of data science professionals, as we recommended earlier, these should be identified at the IDIQ level

· Agree the sooner we have eval criteria and can comment the better

· It doesnt make sense to identify KP at the IDIQ level, unless it is the IDIQ PM. What role will these KP play at the IDIQ level?

· We are a small busines that this soliticitation is right in our wheelhouse. Would you consider a TO for a small for the innovation, development and technical leadership and another to a big for PMO, communication plan, rollout and the items that require more generic experience?

· I agree, it's not realistic that IDIQ identified Key People are available to any and all task orders, especially if TOs are running concurrent in a variety of locations.

· Small business TO lends itself to small business award pool

· I agree that key personnel should be at the task order level. It's very difficult to predict potential requirements on task orders given the broad scope nature

· Do you envision TO's to include ODC's as discussed at last Industry day or will tools be determined by the Vault platform or other USAF tools?

· Perhaps "Key Personnel" at the IDIQ level is the wrong term...the AF needs confidence that the contractor selector can deliver the level of expertise required.

· Just a note in support of firm fixed price (vs race to a bottom concern). The nature of data science is there are many paths to solve a challenge, sometimes with a dramatic improvement in speed and efficiency. We would recommend sticking with FFP task orders with clear outcomes based requirements for each TO.

· In the case of small business, I warn against milestone based payment schedules when third-party contributors (integrators, data trading partners, migrators, etc.) are involved in the critical path of achieving a milestone. Small business have to make payroll and waiting on another vendors contribution can be very risky to a small business.

· Despite being streamlined, the volume of TOs across the 30 offices could present a significant cost to the vendors in aggregate, particularly if there is a requirement vendors must bid on the vast majority of TOs. Is the AF still considering the "off-ramping" strategy as communicated in the first industry day?

· Price Banding!

· On the comment about small busiensses making payroll: The incentive based milestones contained in a T&M contract can mitigate the float for small businesses but give the government control over milestones.

· Can you change the TO turn around day to artelast 14 days, less than 7 days with limited staff is very difficult and may cause a less compehensive response.

· Will there be a ceiling on the $ value for task orders?

· Would the goernment consider joint rights

· In our RFI we also sent in a recommendation to also look at using the R&D NASIC with provides the 1500 FTE size limit for SB so that would Development in the Physical, Engineering, and Life Sciences (except Nanotechnology and Biotechnology) and/or 517110 Wired Telecommunications Carriers which would enable businesses up to 1,500 and 1,000 employees, respectively, to bid as a small business. By having both a Large Prime and a midsized Prime, that not only provides the government with Small Business credit, it also allow for the agility and innovations that come from the Small Business community. NAICS Code 541715 is applicable because data science is a research and development activity. As with any scientific research endeavor, hypotheses are created and tested. Data is collected, cleaned, transformed, curated, and then analyzed as part of a methodical research and discovery process. New algorithms are developed and tested using rigorously statistical analysis and other research tools.

· Given this is an IDIQ, a suggestion is to do a 2 phase approach for the Multi-Step Advisory Process.

· 1st phase would require:

· Only the potential prime offeror to submit one (1) case study that is in line with SOW elements that has occurred in last 3 years.

· Evaluate IDIQ labor category rates.

· Proof of TS Facility clearance for the prime offeror.

· This ensures the USAF is getting a qualified lead company with recent experience. The acquisition team can then do a down select to weed out companies that at least don’t have relevant experience and have either too low rates or too high.

· 2nd Phase would be to evaluate a technical approach to a sample task/scenario and additional past performance that would be open to prime and or their teammates.

· Our understanding is that there has been at least one LB helping SAF shape these reqs for industry for several months. Will those businesses be OCId from bidding?

· Off ramp companies that no bid a certain percentage of RFPs

· Need to be able to scale, if you already have 30 possible clients you may want to have the options to on ramp

· Is on-ramp/off-ramping do be done annually or as required? Like the as required, based on need,

· I think you have to be careful about past performance requirments...this is all new

· Past performance is vital

· What about a pricing floor, that one cannot go under?

· A minimum and maximum range would be very helpful

· For Past Performance it should not be on specific contracts which have to have some typical size, scope and completity, but possible more on work that was done on contracts which used these type of services

· PP for vehicle not for TOS

· Pricing is sensitive to the level of experience required, as well as location. It's almost impossible to ensure prices available today in San Antonio are relevant in 2 years in DC area, Los Angeles, etc.

· w/r to PP - being clear on how they will be evaluated. Size, scope, and complexity - so defining those metrics - a small TO and large could be very different in all three categories

· are you asking if govt is goint to set a competitive range?

· In many ways this will be a staffing exercise. Recommend including evaluation criteria to address hiring and retention plans.

· I recommend the gov't providing a min/max range to industry

· w/r PP - would a company that submits the min be evaluated less favorably than one submittting the max?

· Past performance at vehicle with CPARs/PPQs

· Given the broad nature of the contract, past performance could be very broad. As it relates to Data Science which is fairely new a suggestion is to use a technical scenario to for the approach to evaluate the data science element

· Data scince doesnt really care about the topic...so past perfomance should be open to broad range of capes not fovuesed on AF past performance

· Price floor established in other agencies as a way to mitigate "race to the bottom" and establlshes a price point which if you go below, your are considered non-responsible

· Since at the IDIQ level Price should not really be evaluated since the work is at the TO level. so it would be good if the Government could set the Min and Max and not have to evaluate that and then the price would be done at that TO level

· To what degree will you be interested in how we deliver capability versus just providing personnel?

· As for Technical evaluation, I have seen on some recent Cyber Proposals that the Government is asking for Senarios as part of the technical, which gives a problem and asks for a solution, vice just asking industry to respond to specific PWS reqs

· Evaluation should provide greatest weight to Understanding of the Environment, Technical Approach, followed by Oral Proposal, Past Performance, and Cost/Price. The relative importance of each factor and sub-factor is as follows: Factor 1 (Technical Approach) Factor 2 (Orals) and Factor 3 (Past Performance) are equally important but when combined are significantly more important than Factor 4 (Cost/Price).

· Recommended eval criteria: demonstrated ability to staff and perform multiple simultaneous TOs

· You might want to consider requiring vendors on the vehicle to bid on a minimum number of TOs, vice limiting number of "no bids". This would encourage vendors to submit bids when they have the right (key) people and solution, vice being forced to bid when they don't, which could lead to performance challenges...

· A greater number of awards would help with the concern of unknown customer reqs

· Since the PWS discusses work at all three classification levels, to describe Past Performance, would the government consider / allow for classified responses to TOs?

· Recommend Government Past Performance should be weighted more than Commercial since GovCloud, tool accreditation and security are vastly different

· For the singal IDIQ Prime, there was something similar that was done by Engility for an NSA contract where the prime was really just the PMO. Will see if I can get you some background on how it worked.

· The continuity across task orders can still be achieved via Multiple award IDIQ through technical standardization, guidelines, tenets etc from the CDO office.