JAIC Data T and E Draft RFP Questions Matrix 11Feb21.xlsx
XLSX spreadsheet 41 KB Posted
- Attached to
- Joint Artificial Intelligence Center Test and Evaluation Blanket Purchase Agreement Request for Proposal
- Solicitation number
- W52P1J21R0029
About this file
This solicitation requests proposals to form multiple Blanket Purchase Agreements to provide Test and Evaluation services for artificial intelligence to the Joint Artificial Intelligence Center, Department of Defense, and other federal agencies. Offerors must demonstrate experience in areas including developing and testing software and AI using a software factory or CI/CD toolchain, as well as machine learning and related fields. Proposals are due three weeks after release of the formal RFP. Awards are projected for July 2021 with a total spend ceiling of $249.5 million over the period of performance. While the solicitation is open to all businesses, set-asides for small businesses may be determined at the order level. The Performance Work Statement, Questionnaire, instructions and criteria, labor categories appendix and Department of Defense AI principles are provided as attachments to guide proposal preparation and evaluation.
View the file
Other files
Show all 13
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Q and A
| QUESTION # | DOCUMENT | SECTION | PARAGRAPH | PAGE | QUESTION/COMMENT | RESPONSE |
| 1 | Description Section on Beta.SAM | N/A | N/A | N/A | In the RFI you did not note if this was going to be full and open or a small business set aside however you did place a note there regarding the maximum size of a small business as $30 million when responding. After thinking about this I wanted to make sure I understood this notation correctly.......Can you clarify if my interpretation is correct? |
I was not able to find any data as to whether there was a procurement history as to whether this had been set aside for small business in the past as our company has an interest in responding if there is a "possibility" that it will be full and open. We just did not want to spend a lot of time and effort providing a response only to find out it really is a small business requirement, as a procurement history might indicate, as then we would be better suited to look for a SB partner. I do recognize that many times you use these RFI's as decision-makers as to whether items will go full and open or small business and if that is the case that's fine but perhaps you could provide me some background on history?
| The BPA solicitation is released as a full and open competition for all businesses to respond. Relevant history for AI T&E is not available as the scope for this requirement is dissimilar to previous AI and T&E acquisitions. Potential small business set-asides will be determined at the call order level. | ||||||
| 2 | Attachment 0002 JAIC T&E BPA PWS | 1.0 Introduction | 1.2 Scope | 3 | Will the government accept an OCI plan that allows the contractor to work other non-T&E programs within the JAIC and/or other purchasing organizations? | The PWS has been updated to include section 1.9 Organizational Conflict of Interest. |
| 3 | Attachment 0002 JAIC T&E BPA PWS | 1.0 Introduction | 1.2.2. Test and Evaluation (T&E) Services | 3 | Does the government have operational use cases developed for the Mission Initiatives to date? | Yes. Those could be shared at the call order level when the Government specifies the specific T&E need. |
| 4 | Attachment 0002 JAIC T&E BPA PWS | 2.0 Description of Services/Work Product Overview | 2.1 Key Personnel | 5 | The government states that, “The PM shall direct efforts of cross-competency teams including Contractors at multiple locations…” Will the government have a list of these locations? Do any include hazardous locations (also referenced in section 4.3. Task #2: Test and Evaluation (T&E) and Independent Verification and Validation (IV&V) Services) | No, the Government does not have a list of the locations. This is a U.S. Government wide enterprise vehicle. The location will vary based on the ordering agency and specific requirement. The location will be specified in the call order, but may include CONUS and OCONUS locations. |
| 5 | Attachment 0002 JAIC T&E BPA PWS | 2.0 Description of Services/Work Product Overview | 2.2 Labor Categories | 6 | Will contractors have the ability to add labor categories to suit requirements of call orders if necessary? | Yes. The labor categories provided by the Government are general in nature and additional labor categories can be added at the call order level. |
| 6 | Attachment 0002 JAIC T&E BPA PWS | 3.0 General Information | 3.4.5 Subcontracting Plans | 8 | Will prime contractors be allowed to add subcontractors to the team post BPA award and/or as needed per call order requirements? | Yes. An updated subcontracting plan would be required with the addition of new subs. |
| 7 | Attachment 0002 JAIC T&E BPA PWS | 4.0 Tasks | 4.2. Task #1: Artificial Intelligence (AI) Test and Evaluation (T&E) Solutions | 8 | Can the Purchasing Organization be external to the JAIC? | Yes. This will be a decentralized BPA. Any federal organization will be allowed to make orders. This will include civilian agencies, although they will not be required to use the JCF. |
| 8 | Attachment 0002 JAIC T&E BPA PWS | 4.0 Tasks | 4.2. Task #1: Artificial Intelligence (AI) Test and Evaluation (T&E) Solutions | 8 | Can the government define CONOPS ideation? | Give a Use Case and Mission Need and develop the Concept of Operations for how the product will be utilized during fielding. |
| 9 | Attachment 0002 JAIC T&E BPA PWS | 4.0 Tasks | 4.2. Task #1: Artificial Intelligence (AI) Test and Evaluation (T&E) Solutions | 8 | Does the government envision continuous T&E in support of fielded capabilities? | Potentially. This vehicle is a U.S. Government wide enterprise vehicle. Continuous T&E may be required for specific requirements or ordering agencies. |
| 10 | Attachment 0002 JAIC T&E BPA PWS | 4.0 Tasks | 4.3. Task #2: Test and Evaluation (T&E) and Independent Verification and Validation (IV&V) Services | 9 | In order to support Root Cause Analysis, the supporting contractor would be expected to have an in-depth understanding of the model’s design, data dependencies, weaknesses and strengths. Does this insight represent a potential OCI for the performing contractor that would prevent them from participating in the supplier side of JAIC AI/ML needs? | Yes. The Government does not want the T&E contractor to be involved in the provision of the AI capability for that particular requirement. The PWS has been updated to include section 1.9 Organizational Conflict of Interest. |
| 11 | Attachment 0001 - JAIC TE BPA Questionnaire | Question 1, 2, 4, 5, 6 | 1, 2, 4, 5, 6 | 1, 2, 3 | Due to the classified nature of several contracts we support for DoD and the inability of Program Offices to release their respective Statements of Work, would CDRL's be an acceptable form of documentation to describe the relevant work/Past Performance we accomplished? | The Government is only asking for unclassified data; If your company cannot provide relevant excerpts from a Statement of Work please provide an unclassified description of the work. The Questionnaire has been updated to say "Please also provide support documentation with relevant excerpts from the Statement of Work or an Unclassified Description of the Work." |
| 12 | Attachment 0001 - JAIC TE BPA Questionnaire | Question 1, 2, 4, 5, 6 | 1, 2, 4, 5, 6 | 1, 2, 3 | For Past Performance, may we utilize individual task orders from the MAC-IDIQ's we support as separate "contracts" given these task orders are varied/different efforts with a unique scope and were awarded on a separate/individual basis? | Yes. |
| 13 | Draft JAIC TE RFP | N/A | N/A | N/A | Is a ceiling value under consideration for this BPA (& associated call orders) and if so, may it be shared with industry? | Yes, the ceiling value is $249,548,124. |
| 14 | Draft JAIC TE RFP | II | c. | 2 | Are other contract types under consideration for this BPA (i.e. Cost, Cost Plus Fixed Fee, Cost Plus Incentive, etc.)? | No. The call orders can be FFP, T&M or a Hybrid of FFP+T&M. |
| 15 | Draft JAIC TE RFP | II | a. | 1 | Purpose 1 states an RFP release date of Feb 2021 but a projected award date is not included. Is a projected award date available? | The projected award date is July 2021; however, all dates are subject to change. |
| 16 | Attachment 3 - JAIC TE BPA Section L & M | Section M | 2.c. | 5 | When referencing the corporate experience review for questions 1-6, does the term "their corporate experience" include members of the prime Offerors full team? | A prime has to hold the experience. |
| 17 | Attachment 0002 JAIC T&E BPA PWS | 1 | 1.2.2 | 3 | This section mentions that project data sets must be "acquired and curated". Is the contractor also expected to generate artificial data sets? | Potentially. Generally the Government will use the Data BPA for that requirement, but the Government may ask vendors to create T&E Data. The individual call order will specify requirements. |
| 18 | Attachment 0002 JAIC T&E BPA PWS | 4 | 4 .3 | 9 | Does the government consider implementation performance evaluation to be an important characteristic when measuring and reporting T&E results? Is performance optimization within scope of T&E? | T&E serves to characterize, not optimize. The information can be used to optimize, but it is not the function of T&E to optimize. Each AI program will have its own performance metrics that will be severed. |
| 19 | Attachment 0002 JAIC T&E BPA PWS | 4 | 4.4 | 10 | Will the government give preference to particular platforms, technologies, or tools based on existing in-place architectures or expertise? | This is a U.S. Government wide enterprise contract vehicle. Each purchasing agency may have their own preferences and unique requirements. A specific call order may call for such preference for reasons justified and documented per FAR Part 13. |
| 20 | Attachment 0002 JAIC T&E BPA PWS | 1 | 1.2.1 | 3 | The PWS states, "All JAIC T&E products must align with Joint Common Foundation (JCF) and Mission Initiative infrastructure needs, i.e., cloud-based test tools that are containerized." It may be beneficial to ask Offerors what tools they may use to facilitate repeatable T&E events. | The JAIC would not want to make an evaluation decision based on tools. |
| 21 | Attachment 0002 JAIC T&E BPA PWS | 3 | 3.1 | 6 | Does the Government expect the Offeror to provide a physical location capable of handling and storing classified information? If yes, this may preclude qualified vendors who will be working remotely in a primarily cloud (i.e.. JCF) environment. | If the specific requirement is classified, then all vendors competing for that requirement would be required to be capable of handling/storing information. Unclassified work does not require vendors to be eligible for handling classified information. The JAIC wants to ensure the flexibility of the BPA to include both physical and remote T&E services, depending on the specific need as specified in each individual call order. |
| 22 | Attachment 0001 - JAIC TE BPA Questionnaire | 7 | 7 | 3 | Question 7 requires "two cubic feet" of Secret storage. Is that a requirement for all Offerors? | This question has been deleted from the Questionnaire. |
| 23 | Attachment 0002 JAIC T&E BPA PWS | 1 | 1 | 3 | Comment: The JAIC has multiple Mission Initiatives. Will Offerors be able to propose participation in a subset of the Mission Initiatives? By proposing for specific services and tools relevant to specific Mission Initiative(s), the Government will be able to receive specialized bids that will decrease the time from idea, test & evaluation, and transition to mission partners. | The Offeror does not have to specialize in all areas of the PWS in order to get a BPA. Awardees will be selected in accordance with the evaluation criteria. The government may consider specialized bids at the call order level. |
| 24 | Attachment 0003 - JAIC TE BPA Sections L and M | SECTION L: INSTRUCTIONS TO PREPARE SUBMISSIONS | 3.2 Specific Instructions by Section; Section 1; Section 3 | 3; 4 | Would the government please permit the use of a smaller font size for tables and graphics, provided the content remains legible? | Yes, tables and illustrations may use a reduced font size, no less than 8 point. The RFP Attachment 0003 has been updated accordingly. |
| 25 | Attachment 0002 JAIC T&E BPA PWS | 2.0 Description of Service/Work Product Overview | 2.1 Key Personnel | 5 | The referenced section states that "Key Personnel will be determined at the call order level, with the exception of the Program Manager," and that "The Contractor shall identify a PM by name who shall provide management, direction, administration, quality management, and leadership of the execution of the CO." |
| Can the government please provide instruction as to where in Offeror responses this key person should be identified and whether a resume is required with the Offeror's BPA proposal? | The Government is not requiring resumes for Key Personnel. A T&E BPA Manager must be identified by all Contractors who are part of the T&E BPA. This individual shall be included in the Cover Letter of the Offeror's proposal. Additionally, all call orders executed under this BPA shall have a Program Manager. Attachment 0002 PWS Section 2.1 has been updated accordingly. | ||||||
| 26 | Attachment 0001 - JAIC TE BPA Questionnaire | 1,2,4,5,6 | 1,2,4,5,6 | 1 | In the case of a teaming arrangement or subcontractor, can a prime contractor provide a subcontractor’s examples to answer questions on the BPA Questionnaire? | No. A prime has to hold the experience. | |
| 27 | Attachment 0001 - JAIC TE BPA Questionnaire | 1,2,4,5,6 | 1,2,4,5,6 | 1 | Can the respondent use a single example (i.e., previous contract) across multiple responses on the BPA questionnaire? | Yes. Respondent needs to thoroughly describe how the example answers each specific question. | |
| 28 | Attachment 0003- JAIC TE BPA Section L and M 10 DEC 20 | 3.1 | Section 3 | 3 | Is there a word or page limit for the Statement of Work portion referenced in the BPA questionnaire? | Yes. The Questionnaire has been updated to request only "relevant excerpts from the Statement of Work or an Unclassified Description of the Work." The Instructions and Evaluation Criteria has been updated to state that "Support documentation must not exceed 12 pages." | |
| 29 | Attachment 0002 JAIC T&E BPA PWS | 2.1 | 1 | 5 | In the Draft PWS, Section 2.0 references a Program Manager as key personnel; does the respondent need to name the Program Manager and provide his/her resume in response? If so, in what section of the response should that information be included? | The Government is not requiring resumes for Key Personnel. A T&E BPA Manager must be identified by all Contractors who are part of the T&E BPA. This individual shall be included in the Cover Letter of the Offeror's proposal. Additionally, all call orders executed under this BPA shall have a Program Manager. Attachment 0002 PWS Section 2.1 has been updated accordingly. | |
| 30 | Attachment 0001 - JAIC TE BPA Questionnaire | 1,2,4,5,6 | 1,2,4,5,6 | 1 | If research is used as an example on the BPA questionnaire, does the research have to have been performed under a contract or could it be company-funded (i.e., no specific client)? | Internal research and development projects can be used. If your company cannot provide contract details, it must provide proof of research such as publications, references, etc. | |
| 31 | Is the solicitation for small businesses only, or can large businesses also submit as a Prime? | The BPA solicitation is open for all businesses to respond. | |||||
| 32 | Attachment 0001 - JAIC TE BPA Questionnaire | Question 7. (A) | N/A | Page 3 | We currently have a Top Secret (TS) Facility Clearance but do not have a contract needing Secret safeguarding capability with a minimum of two (2) cubic feet. If a company can show that they have the ability to meet the requirement for Secret safeguarding capability and have a Top Secret (TS) Facility Clearance will that be acceptable? | This question has been deleted from the Questionnaire. | |
| 33 | Draft JAIC TE RFP | N/A | N/A | Page 1 | Will the solicitation be for small business only? | The BPA solicitation is open for all businesses to respond. | |
| 34 | N/A | N/A | N/A | N/A | Will Contractor Teaming Agreements (CTA) be allowed? | Yes, teaming arrangements are encouraged. | |
| 35 | Attachment 0001 - JAIC TE BPA Questionnaire | Question 7. (A) | N/A | Page 3 | If Contractor Teaming Agreements (CTA) is allowed do both companies have to be able to meet the requirement for Secret safeguarding capability in question 7. (A). | This question has been deleted from the Questionnaire. | |
| 36 | Attachment 0003- JAIC TE BPA Section L and M 10 DEC 20 | M | 2. BPA Down-Select Process | Page 5-6 | Will Primes be able to use Past Performance from Teammates or Subs in order to meet requirements for questions 1-8? | No. | |
| 37 | Attachment 0003- JAIC TE BPA Section L and M 10 DEC 20 | L | 3.1 General | Page 2 | Can a company submit as both a Prime and a Sub? | Yes. | |
| 38 | Attachment 0002 JAIC T&E BPA PWS | 1.2.2 | Last paragraph | 3 | Is the JAIC interested in solutions that could address quality issues found in data that would be used for testing as well as simply addressing them? | Potentially. This vehicle is a U.S. Government wide enterprise vehicle. Specific purchasing agencies may require vendors to explore quality issues. | |
| 39 | Attachment 0002 JAIC T&E BPA PWS | 1.2.2 | Paragraph 3 | 4 | Is the JAIC interested in using AI rather than humans to analyze the output of AI models and to create context specific and operationally relevant performance metrics? | Potentially. This vehicle is a U.S. Government wide enterprise vehicle. Specific purchasing agencies may require vendors to build a tool that uses AI to analyze AI. | |
| 40 | Draft JAIC TE RFP | II (General Information) | 1 - 2 | What will the process and criteria be for announcing, competing, and selecting Call Orders? | RFIs and RFPs will be posted to all BPA holders. It has not been determined yet if it will be via email, website, etc. | ||
| 41 | Attachment 0002 JAIC T&E BPA PWS | II (General Information) | 1.2.2 | 3 - 4 | How will T&E services be integrated / aligned with internal testing, assessment, and quality control services being performed at a Mission Initiative (MI) level? | These requirements would be detailed at the call order level. | |
| 42 | Attachment 0002 JAIC T&E BPA PWS | II (General Information) | 1.2 | 3 | Will T&E BPA Performer be eligible to bid as Prime or Subcontractor for existing or future JAIC Mission Initiative (MI) or other JAIC procurements? This question is intended to address the 1.2 Scope statement: "One consistent theme across the DoD is a Test and Evaluation organization that is independent of the project team and project management and provides an unbiased assessment of the quality and readiness of the project deliverables." | ||
| The Government does not have any restrictions for teaming arrangements. | |||||||
| 43 | Attachment 0001 – JAIC T&E BPA Questionnaire | 1, 2, 4, 5, 6 | 1 - 3 | Can the same contract be applied across multiple questions? | Yes. Respondent needs to thoroughly describe how the example answers each specific question. | ||
| 44 | Attachment 0001 - JAIC TE BPA Questionnaire | 1, 2, 4, 5, 6 | 1 - 3 | Should the Statement or Work / Description of Work reflect the statement of work provided by the customer, or should it be tailored to reflect and substantiate relevance to the question. | Yes. The Questionnaire has been updated to request only "relevant excerpts from the Statement of Work or an Unclassified Description of the Work." The Instructions and Evaluation Criteria has been updated to state that "Support documentation must not exceed 12 pages." | ||
| 45 | Attachment 0001 - JAIC TE BPA Questionnaire | 1, 2, 4, 5, 6 | 1 - 3 | Can Internal Research and Development (IRAD) projects be used in place of customer projects? | Yes. Since the company cannot provide contract details, it must provide proof of research such as Publications, References, etc. | ||
| 46 | Attachment 0002 JAIC T&E BPA PWS | 4.2 | 8 | Is it correct to interpret Task #1 to be inclusive of building/operating the actual T&E environment that hosts all the systems that will be used to test the AI models? And is the assumption that this would be all cloud based – no hybrid? | Potentially. This is a U.S. Government wide enterprise vehicle. The purchasing agency would specify if they need a system that can host/operate the T&E process. | ||
| 47 | Attachment 0002 JAIC T&E BPA PWS | 4.2 | 8 | What domain(s) would the T&E system would be located on. Will there be a need for an unclassified T&E system? Will TS/SCI T&E systems be expected to be built on C2S? Will this be MI dependent? | This is dependent on the specific requirement at the Call Order Level and the specific Purchasing Agency. T&E may occur on unclassified or classified systems. | ||
| 48 | Attachment 0002 JAIC T&E BPA PWS | 4.2, 4.3 | 8 - 10 | Will the T&E performer be expected to containerize the AI models/tools that will be evaluated? | Potentially. This is a U.S. Government wide enterprise vehicle. The purchasing agency would specify if they need the AI Algorithms containerized and prepped for T&E. | ||
| 49 | Attachment 0002 JAIC T&E BPA PWS | 4.3 | 8 | 9 - 10 | When providing test assessment/results, is there a required medium (e.g. MS Word, PPT)? Is there an expectation of other methods (e.g., dashboards)? | The specific format of each deliverable will be specified by the Purchasing Agency at the Call Order level. | |
| 50 | Attachment 0002 JAIC T&E BPA PWS | 4.3 | 4 | 9 | Will IV&V be done from requirements, design, and forward – if/when T&E performer finds defects in written requirements, etc. “Requirements and design specifications, test plans and procedures, test results, logistics training documentation, and change proposals shall be reviewed.” | Potentially. This is a U.S. Government wide enterprise wide vehicle. IV&V should be started at the beginning of the systems engineering process. However, the Purchasing Agency may incorporate IV&V at any point based on their specific needs. | |
| 51 | Attachment 0002 JAIC T&E BPA PWS | 4.3 | 9 - 10 | Will Task #2 also include testing of non-AI tools/models (e.g. other software, applications, tools developed in support of a JAIC MI)? | Yes. This BPA includes T&E support to AI Enabling Technologies. | ||
| 52 | Attachment 0002 JAIC T&E BPA PWS | 4.3 | 9 - 10 | Will T&E assessment of models include scale, speed/performance, functionality, accuracy/precision/recall, portability, technical maturity, integration readiness, etc.? | Yes. This will be specified at the Call Order Level. | ||
| 53 | Attachment 0002 JAIC T&E BPA PWS | 4.4 | 10 | Is it correct to assume that Task #3 scope is limited to T&E technologies that support T&E of AI models, as opposed to other AI models/tools? | Task 3 is scoped to cover any additional technology to help with T&E of AI. | ||
| 54 | Attachment 0003 – Appendix A: Labor Category Table | UI/UX Designer | Will UI/UX Designer LC require presentation skills and experience? Will UI/UX Designer be responsible for presentation of T&E status, results, etc.? | The role of the UI/UX designer will be dependent on the specific requirement and purchasing agency. | |||
| 55 | Attachment 0003 – Appendix A: Labor Category Table | Systems Engineer (not included) | Recommend adding Systems Engineer LC. This requirement aligns with PWS section 4.2 statement “Solutions developed under this task should follow system engineering and software development best practices”. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |||
| 56 | Attachment 0003 – Appendix A: Labor Category Table | Cloud Architect (not included) | Recommend adding Cloud Architect LC. With emphasis throughout on cloud deployments and alignment with JCF, this appears to be a critical skill set for Tasks 1 and 2. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |||
| 57 | Attachment 0003 – Appendix A: Labor Category Table | Data Engineer (not included) | Recommend adding Data Engineer LC. Data architect, data modeler, and data engineer skill sets appear to be foundational requirements to support T&E of tools/capabilities (cleansing, data set curation, ETL, etc.) | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |||
| 58 | Attachment 0003 – Appendix A: Labor Category Table | Information Assurance Engineer (not included) | Recommend adding Information Assurance (IA) / Security Engineer LC to support SSP development and ATO for the T&E environment and/or its tools, etc. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |||
| 59 | Attachment 0003 – Appendix A: Labor Category Table | DevOps Engineering (not included) | Recommend adding Configuration Management, DevOps Engineer (e.g., CI/CD), Integration Engineer (e.g. K8s), and/or Sys Admin/Network Engineer to support the deployment, O&M, and MI tool integration in the T&E (test harness) system. Alternatively, recommend explicitly adding these responsibilities and skill sets under the SW Engineer LCs. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |||
| 60 | NA | NA | NA | NA | Will the government share additional information about the expected distribution of full and open and small business awards at the BPA and call order level? | The BPA solicitation is open for all businesses to respond. Potential small business set-asides will be determined at the call order level. | |
| 61 | NA | NA | NA | NA | Does the government have a sense of scale and number of the BPA call orders in terms of total spend ceiling? | The total spend ceiling is $249,548,124. The number of BPAs is unknown at this time as it is open to the U.S. Government. | |
| 62 | NA | NA | NA | NA | Will the government share additional information about the expected distribution of full and open and small business awards at the BPA and call order level? | The BPA solicitation is open for all businesses to respond. Potential small business set-asides will be determined at the call order level. | |
| 63 | NA | NA | NA | NA | Acknowledging the government may set aside work for small or disadvantaged businesses at the call order level, can a large business who receives a BPA contract award also be a subcontractor to small businesses proposing against small business set aside opportunities? | Yes as long as 51% of the work is being done by small business. | |
| 64 | Attachment 0002 JAIC T&E BPA PWS | 4.2 | 1 | 8 | The Draft PWS states: "The Contractor shall propose, design, build, and operationalize techniques, tools, or process solutions at the direction of the Purchasing Organization to conduct T&E of AI." Is there an desired TRL of vendor technology at project kickoff, and when is a TRL-9 capability expected to be deployed operationally? Are COTS technologies being considered or only R&D? Once these tools or solutions are developed are they owned by the government, does the vendor retain ownership, or are rights shared? | There is no specific TRL level required. Both COTS and R&D are considered. Intellectual Property will be determined by the specific Purchasing Agency at the Call Order Level. In general, the Government wants to encourage that all T&E software has Government Purpose Rights. | |
| 65 | Attachment 0002 JAIC T&E BPA PWS | 1.2.1 | 2 | 3 | Will a vendor be selected as a Prime and then be expected to deliver/develop all AI technologies of the focus areas, potentially by way of sub-contractors, or will multiple contracts be awarded to include firms that specialize in as few as one focus area? | Multiple BPAs will be awarded based on their ability to respond to the questionnaire in accordance with the evaluation criteria. At the order level, only one call order will be awarded for the entire requirement of that specific call order. | |
| 66 | Attachment 0002 JAIC T&E BPA PWS | 3 | NA | NA | Is there any OCI guidance for contractors who are providing/building capabilities to the JAIC and who are also working under this BPA for the separate JAIC T&E initiative? | The PWS has been updated to include section 1.9 Organizational Conflict of Interest. | |
| 67 | Attachment 0002 JAIC T&E BPA PWS | 1.2.2. Test and Evaluation (T&E) Services | Paragraph 1, first bullet | 3-4 | Is data set development and creation included is the scope of this draft RFP, or only the T&E associated with data set quality, image annotation, and data coverage analysis for data produced by another party? | Generally, large scale data set curation and development would be funded using the Data BPA. In specific cases, it may be required to develop/curate T&E data. | |
| 68 | Attachment 0002 JAIC T&E BPA PWS | 4.0 - Tasks | In general | 8-11 | Is this draft RFP designed for SETA contractors that only focus on T&E, or are companies focused on delivering AI/ML capabilities eligible? | This vehicle is designed to identify a pool of vendors who can successfully perform T&E of AI. | |
| 69 | Attachment 0001 - JAIC TE BPA Questionnaire | Question 7 | A | 3 | Our company has a Top Secret Facility Clearance (FCL) but currently cannot store classified material onsite. Would the Government consider establishing a minimum threshold requirement of TS FCL, and if awarded, provide a reasonable amount of time to coordinate with the DCSA's Industrial Security Representative to implement a safeguarding capability? | This question has been deleted from the Questionnaire. | |
| 70 | Attachment 0001 - JAIC TE BPA Questionnaire | Questions 1,2,4,5,& 6 | We are a recently formed small 8(a) company with key individuals having decades of relevant experience. When citing "three examples of work your company has performed" will the Government take into account past performance information regarding predecessor companies, key personnel who have relevant experience, or subcontractors that will perform major or critical aspects of the requirement IAW FAR 15.305? | Performance references should not include predecessor companies. | |||
| 71 | Attachment 0002 JAIC T&E BPA PWS | 1 | 1.2.1 | 3 | This question is around the classification of the data models as well as the integrated systems. The data models may or may not be classified, but the systems they are being integrated into will most likely be. Will a contractor need a separate testing facility or will they have access to a JAIC facility to perform testing to meet the needs of the client? | Potentially. This is a US Government wide enterprise vehicle. This level of detail will be determined at the call order level | |
| 72 | Attachment 0002 JAIC T&E BPA PWS | 2 | 2.1 | 5 | Will the only KP that the Government require at the BPA level be the Program Manager? | A T&E BPA Manager must be identified by all Contractors who are part of the T&E BPA. This individual shall be included in the Cover Letter of the Offeror's proposal. Additionally, all call orders executed under this BPA shall have a Program Manager. Attachment 0002 PWS Section 2.1 has been updated accordingly. | |
| 73 | N/A | N/A | N/A | N/A | What do you anticipate as your requirement for past performance, e.g. two or three past performance references from the Offeror (prime and subcontractors), PPQs or CPARS | The Government is not requiring past performance on the questionnaire. The Government is requesting examples of work including support documentation with relevant excerpts from the Statement of Work or an Unclassified Description of the Work. | |
| 74 | N/A | N/A | N/A | N/A | We recommend the Government accept past performance from subcontracting teaming partners. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 75 | N/A | N/A | N/A | N/A | We recommend the Government require Offerors submit a CPARs in lieu of a PPQ for past performance citations. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 76 | N/A | N/A | N/A | N/A | We recommend the Government include a CMMI 3 DEV & SVC certification requirement for all Offerors. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 77 | N/A | N/A | N/A | N/A | We recommend the Government require Offerors' past performance citations be valued at a minimum of $20M for the overall value. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 78 | N/A | N/A | N/A | N/A | How many awards are anticipated for this BPA? | There is no limit. All qualified vendors will receive a BPA. | |
| 79 | N/A | N/A | N/A | N/A | We recommend the Government allow contractors to submit OCI mitigation plans in the event there is a perceived OCI | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 80 | N/A | N/A | N/A | N/A | If contractor is a joint venture, please specify any restrictions that may apply for priming this contract. | There are no restrictions for teaming arrangements. | |
| 81 | N/A | N/A | N/A | N/A | Will this solely be a small business (SB) set aside requirement or will the Government have a particular SB classification such as WOSB, SDVOSB, 8(a), or Hubzone set aside? | The BPA solicitation is open for all businesses to respond. Potential small business set-asides will be determined at the call order level. | |
| 82 | Attachment 0002 JAIC T&E BPA PWS | 1.2 | 1 | 1 | Will contractor be allowed to perform in more than one Task? i.e. Test Technology and Tools, Test and Evaluation (T&E) Services, Other Technologies or any combination thereof? | Yes. Once a company is issued a BPA it is allowed to perform on all task areas if issued a call order. | |
| 83 | Attachment 0002 JAIC T&E BPA PWS | 1.2.1 | 3 | 3 | Will JAIC release more technical/architecture details regarding the Joint Common Foundation (JCF) in the Final RFP? | Current information as it relates to the JCF is in the PWS. | |
| 84 | Attachment 0002 JAIC T&E BPA PWS | 1.2.2 | 1 | 4 | Can JAIC comment on how they envision contractor-developed test harness integrate and/or complement JAIC's own test harness development effort? | It depends. Within the JAIC, we will work to have all test tools complement one another. This is a U.S. Government wide enterprise vehicle and some of these details will be decided at the call order level. | |
| 85 | Attachment 0002 JAIC T&E BPA PWS | 1.7 | 1 | 5 | We would like to confirm that the ceiling value on each call order is $15M as per FAR 13.5c. | Yes. | |
| 86 | Attachment 0002 JAIC T&E BPA PWS | 3.5 | 1 | 8 | Will contractor perform work at a Government site(s), contractor-owned site, or a combination thereof? | Location will vary based on the ordering agency and specific requirement. Location will be specified in the call order, but may include CONUS and OCONUS locations. | |
| 87 | Section L | 3.1 | 1 | 2 | In addition to questionnaire responses and supporting documentation, will JAIC consider high level T&E product or service solution write-up from contractors? i.e. Contractor's description of how its T&E products or services will fit into JAIC's current architecture (e.g. JCF) and delivery framework. | Current information as it relates to the JCF is in the PWS. | |
| 88 | Attachment 0001 - JAIC TE BPA Questionnaire | General | General | General | Will JAIC consider adding a question to solicit contractor's input on experiences providing Government onsite support (should tasks be performed at Government sites)? | No, the BPA questionnaire does not include a question regarding on-site support. This may become an area of interest on individual call orders. | |
| 89 | Attachment 0003A: Appendix A TE LCAT Table | Machine Learning Scientist | Column B: Work Description; Column C: Skill Level; Column D: Minimum Education Level; Column E: Minimum Years | N/A | Work Description and Skill Level of "Intermediate/Mid-Level" are incongruous with requested level of education (PhD) and years of experience (10+ yrs.). We recommend changing this requirement to a Bachelor’s Degree in Computer Science, Mathematics, Statistics, Computer Engineering or related fields | Please see revisions to the LCATs document. | |
| 90 | Attachment 0003A: Appendix A TE LCAT Table | All LCATS | Column F Substitute for Minimum Education/Years | N/A | For consistency and ability to review and evaluate proposals fairly, we recommend providing the following standards for experience substitutions for degreed requirements: one year of additional education equates to 2 years of experience or 2 years of additional experience equates to one year of education. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 91 | Attachment 0001 - JAIC TE BPA Questionnaire | Questions 1,2, 4-6 | Questions 1,2, 4-6 | 1-3 | For research and industry work experience that may not have a contract reference, please confirm that the government will allow bidders to provide a POC, organization name, and scope of research and/or industry services provided only, in lieu of contract metadata. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 92 | Attachment 0001 - JAIC TE BPA Questionnaire | Question 7 | Question 7 | 3 | Can the government please confirm that a company must possess a current TS Facility Clearance at proposal submission with the ability to add Secret safeguarding capability for JAIC materials to their FCL at contract award? The requirement to have storage for JAIC materials would narrow competition to contractors already holding JAIC contracts or require a co-location memorandum of understanding between the agencies. Any storage provided for the JAIC should be done under the JAIC contract/DD-254. | This question has been deleted from the Questionnaire. | |
| 93 | Attachment 0001 - JAIC TE BPA Questionnaire | Question 7 | Question 7 | 3 | Can the government please confirm that a teammate may provide the Secret safeguarding capability? | This question has been deleted from the Questionnaire. | |
| 94 | Attachment 0002 JAIC TE BPA Section L and M | 3.2 Specific Instructions by Section | Section 3 | 3 & 4 | Will the government allow up to 2 pages per contract reference to address BPA Questionnaire questions 1-2,4-6? There is currently no stated page limit for the supporting documentation required. | The Instructions and Evaluation Criteria has been updated to state that "Support documentation must not exceed 12 pages." | |
| 95 | Attachment 0002 JAIC TE BPA PWS / Attachment 0002 JAIC TE BPA Section L and M | 2.0 Description of Services/Work Product Overview | 2.1 Key Personnel | 5 | In the PWS, the Program Manager is listed a Key Personnel. However, in the current Section L & M draft, there is no mention of a requirement to submit a resume for key personnel. We recommend that a resume be required for the PM proposed. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 96 | Attachment 0002 JAIC TE BPA Section L and M | 2. BPA Down-Select Process | a. | 5 | Please confirm that Section 1 in the below sentence should read Section 2 instead. "The Government will utilize Offeror’s completed JAIC T&E BPA Questionnaire (Section 1 of Offeror’s submission) containing questions focused on Test and Evaluation Services support to determine the most highly qualified Offerors for the BPA pool." | Yes | |
| 97 | Attachment 0001 - JAIC TE BPA Questionnaire | Questions 1, 2, 4, 5, 6 | same | 1 through 3 | We appreciate that these questions ask for examples to be demonstrated through 'previous government work, research, or work in industry.' Much testing and evaluation work - especially for AI - is included as part of a larger design/build engagement. Many of our commercial clients engage us to help them manage the full lifecycle of integrating AI into their systems and processes to achieve business results, including the test and evaluation of AI algorithms, products, and apps. We recommend that the government emphasize the need for bidders to show test and evaluation experience as a part of end to end delivery of AI capability, as such vendors can bring a contextual understanding of T&E capabilities. The government should be aware that statements of work for these kinds of engagements in industry are often sensitive and client confidential, and it will be difficult to obtain permission from commercial clients to release them. Will the government consider a description of the work performed, focusing on the relevant areas, to be sufficient evidence of experience? | The questionnaire has been updated to state that an unclassified description of work can be provided. | |
| 98 | Attachment 0001 - JAIC TE BPA Questionnaire | Questions 1, 2, 4, 5, 6 | same | 1 through 3 | We appreciate that these questions ask for examples to be demonstrated through 'previous government work, research, or work in industry.' We engage in substantial research to develop AI tools, including test and evaluation tools. This internal research and product development work does not have a contract number associated with it, but it is highly relevant and the government benefits from having access to companies that are in the business of building actual AI test tools. Will you consider allowing internal research and development programs and projects - especially those that have resulted in a commercially marketed AI test and evaluation product - to be used as evidence of experience? | Yes. If your company cannot provide contract details, it must provide proof of research such as Publications, References, etc. | |
| 99 | Attachment 3 - Section L and M | Section M Evaluation Methodology | 2.c | Page 6 | The questionnaire indicates in each question that examples can be provided through 'previous government work, research, or work in industry.' However, in Section M the evaluation criteria appears to focus solely on contracts. For example (emphasis added): "For Questions 1-6, the Government will conduct a corporate experience review based on the contract numbers submitted. The contract numbers referenced will be reviewed ..." and similarly in the following paragraph "Acceptable: Meets the requirements of the solicitation. Two, but not more than three, example contracts are provided. The focus on contracts and contract numbers is restrictive compared to the language in the questionnaire that also allows for research or work in industry to be used as evidence. Will the Government revise Section M to allow consideration of descriptions of work from these other sources to be used? | The questionnaire has been updated to state that an unclassified description of work can be provided. | |
| 100 | Attachment 3 - Section L and M | Section M Evaluation Methodology | If an Offeror receives an 'acceptable' rating on all eight questions, does that Offeror automatically receive a BPA award? | Yes. | |||
| 101 | ALL/ GENERAL QUESTION | Can the government clarify if there is an incumbent for this requirement? If yes, Please provide name and contract number. | There is no incumbent as this is a new requirement. | ||||
| 102 | Attachment 0002 JAIC T&E BPA PWS | 1.2.1 | 3 | 3 | Will the use of the JCF be required on all task orders? | Current information as it relates to the JCF is in the PWS. | |
| 103 | Attachment 0002 JAIC T&E BPA PWS | 1.2.1 | 3 | 3 | Can the government specify the JCF and infrastructure needs? | Current information as it relates to the JCF is in the PWS. | |
| 104 | Attachment 0002 JAIC T&E BPA PWS | 1.2.2 | 1 | 3 | Does the government have a means in which the vendor of the software/solution to be evaluated can supply proprietary information? | The means in which a vendor of a software/solution can supply proprietary information will be determined by specific agreements. There may be proprietary information needs at the call order level, but not at the BPA level. | |
| 105 | Attachment 0002 JAIC T&E BPA PWS | 2.1 | 1 | 5 | Can the government clarify If the cost of the key personnel Program Manager at the BPA level is at no cost to the government? | There is no cost to the Government at the BPA level. | |
| 106 | Attachment 0002 JAIC T&E BPA PWS | 4.2, 4.3 | 1, 9 | 9, 10 | Can the government confirm that the "Secretary of Defense Guidance Memo on AI Ethics" that is referenced in Section 4.2 is the document found here: https://media.defense.gov/2019/Oct/31/2002204458/-1/-1/0/DIB_AI_PRINCIPLES_PRIMARY_DOCUMENT.PDF | The Secretary of Defense Ethics memo was incorporated as Appendix B in the PWS. | |
| 107 | Attachment 0002 JAIC T&E BPA PWS | 4.3 | 1 | 9 | What programs or projects or deliverables does the government intend to conduct IV&V on? | This is a U.S. Government wide enterprise vehicle. The specific requirement for IV&V will be determined at the Call Order Level. | |
| 108 | Attachment 0003- JAIC TE BPA Section L and M 10 DEC 20 | 3.2 Section 3 | 1 | 3,4 | Can the government please clarify if there is a page limit for support documents within Section 3 of the BPA response? | The Instructions and Evaluation Criteria has been updated to state that "Support documentation must not exceed 12 pages." | |
| 109 | Attachment 4- JAIC Questions Matrix | Can the government clarify if the same 2 contracts referenced be used for more than one response in the questionnaire? | Yes. | ||||
| 110 | All | The package does not mention Organizational Conflict of Interest (OCI) restrictions. We expect there would be some restrictions given the T&E and IV&V nature of the scope. We request the Government provide specifics on OCI to help Offerors make pursuit decisions. | The PWS has been updated to include section 1.9 Organizational Conflict of Interest. | ||||
| 111 | Draft RFP & | ||||||
| Attachment 0002 PWS | II | ||||||
| 4.0 | b. Task 3 | ||||||
| 4.4 | X | ||||||
| 9 | Task #3 discussed the contractor identifying other new or relevant technologies. Are these technologies limited to T&E, or can they be across the AI domain? | New or relevant technologies related to T&E | |||||
| 112 | Attachment 0002 JAIC T&E BPA PWS | 3.0 | 3.4.4 | 8 | This paragraph discusses contractors certifying size standard for each call order. The procurement makes no reference to being a total or partial set-aside. Can the Government clarify the size standard certification requirements and when it will apply? Furthermore, we request the Government consider reserving a significant portion of the BPA awards for small businesses. As an incumbent currently providing such T&E services to the JAIC, we are proof this scope of work can be successfully delivered by small business contractors. | The BPA solicitation is open for all businesses to respond. These BPAs will be open to all agencies within the U.S. Government. Potential small business set-asides will be determined at the call order level. | |
| 113 | Attachment 0002 JAIC T&E BPA PWS | 2.0 | 2.1 | 5 - 6 | Under the required qualifications for the PM, the statement "Possess an active certification in one of the following:" is followed by 5 bullets, of which the first three are clearly certifications. The last two (on page 6) read like experience requirements as they start with "Possess a minimum of X years of experience..." Can the Government clarify in the options for certifications includes possessing one the first three bullets or one of the five bullets. | A T&E BPA Manager must be identified by all Contractors who are part of the T&E BPA. This individual shall be included in the Cover Letter of the Offeror's proposal. Additionally, all call orders executed under this BPA shall have a Program Manager. Attachment 0002 PWS Section 2.1 has been updated accordingly. | |
| 114 | Attachment 0003 Section L and M | M | 2.a, 2.b, and 2.c | 5 - 6 | The evaluation mythology for down-select seems to imply that if the Offeror responds to Questions 7(A) or 8(A) with a "No" they will not be considered for an award. While the Top Secret FCL is shown as a requirement in the PWS, the advisory board requirement is not mentioned. Is serving on a technical advisory board a requirement to receive an Acceptable rating to qualify for an award of this BPA? What is the definition of a technical advisory board in this context and can you provide an example? | Those questions have been removed from the Questionnaire. | |
| 115 | Attachment 0002 JAIC T&E BPA PWS | 3.0 | 3.2 | 6 | The PWS reads "When responding to COPR, proposed personnel shall possess the minimum security clearance, or higher required, prior to proposal submission...". What is the definition of a "COPR"? Does the Government anticipate all Call Orders will include classified work and all staff proposed must have a minimum security clearance? If so, what is the anticipated minimum personnel security clearance requirement (e.g. Secret)? | No, the Government does not anticipate that all call orders will include classified work. | |
| 116 | Attachment 0001 - JAIC TE BPA Questionnaire | Question 5 | N/A | 2 | Request the Government define Software Factory and provide an example of such in the final RFP. | Software Factory is an industry standard term. | |
| 117 | SAM.Gov Announcement | Given that much of the work done in Artificial Intelligence is, by the nature of the field, R&D in nature, would the Government consider changing the NAICS to 541715? | No - the NAICS was selected based on market research, focusing on capability delivery. | ||||
| 118 | N/A | N/A | N/A | N/A | Is this a SBSA? | The BPA solicitation is open for all businesses to respond. Potential small business set-asides will be determined at the call order level. | |
| 119 | Attachment 003A | Tab: T&E MAC LCATs | Line 5, Column D | N/A | Suggestion: Add the word "or" to the following list: "PMP, FAC P/PM Level 3, DAWIA Level III Program Management." Requiring all 3 is unrealistic and does not provide the government any additional value. | The Government reviewed and considered this recommendation from industry. Changes were not implemented into the requirements package at this time, although they will remain under consideration as the ordering guide is drafted. | |
| 120 | Attachment 003A | Tab: T&E MAC LCATs | Line 7, Column D |
This is the start of the file's text. The full file is on GovTribe.
File details come from the government source that posted it. Updated .