Atch 1 - Question-Comment Matrix 20Dec23.xlsx
XLSX spreadsheet 32 KB Posted
- Attached to
- Cockpit/Crew Resource Management and Threat & Error Management Program Courseware Development and Training Implementation Federal contract opportunity
- Solicitation number
- FA489023R0062
About this file
This document contains a question-comment matrix regarding a solicitation for Cockpit/Crew Resource Management and Threat & Error Management program courseware development and training implementation services. The Air Force is seeking proposals to provide classroom and simulator training, as well as courseware development utilizing the Instructional Systems Development model, at multiple locations for various aircraft types. Offerors are asked to submit past performance examples demonstrating experience managing complex, geographically dispersed training programs. While the technical approach will be rated on a pass/fail basis, the Air Force expresses interest in innovative proposals to modernize its CRM training approach in light of current mental health and recruitment challenges. Response date and contract award timing are not specified.
View the file
Other files for this federal contract opportunity
| File | Type | Posted |
|---|---|---|
| FA489023R0062 CRM TEM Amendment 0003 9Jan24.pdf | ||
| Atch 1 - CRM TEM Draft PWS Rev 9Jan24.pdf | ||
| Atch 2 - CRM Question-Comment Matrix 9Jan24.xlsx | XLSX spreadsheet | |
| Atch 2 - CRM TEM PPC Attendees 13Dec23.pdf | ||
| FA489023R0062 CRM TEM Amendment 0002 20Dec23.pdf | ||
| FA489023R0062 CRM TEM Amendment 0001 18Dec23.pdf | ||
| A1. CRM TEM Draft PWS 1Jun24.pdf | ||
| A2. CRM TEM Price Schedule.xlsx | XLSX spreadsheet | |
| A3. CRM TEM Draft DD 254.pdf | ||
| A4.Question-Comment Matrix.xlsx | XLSX spreadsheet | |
| A. FA489023R0062 CRM TEM Combined Synopsis Solicitation.pdf |
Show all 11
On GovTribe
Work with this file on GovTribe
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
MASTER
| Q# | Doc | Pg # | Para # | Comment/Question/Recommendation | Government Response |
| 1 | Are you able to confirm how many companies responded to the Source Selection Notice in August? | A total of 17 companies expressed interest in the requirement during market research. | |||
| 2 | If this is a fixed training-type contract – Is there a primary location for instructors to be based and an additional mobile training effort? | No to both questions. The contractor must identify the number of instructors required and where they are based out of in their Training Workload Management Plan. | |||
| 3 | Draft PWS | 28 | Is there going to be a primary place of performance? Or will instructors be required at all of the locations in Appendix 1, as you referenced in your previous email? | No primary place of performance required. The contractor must provide training at all the locations identified in Appendix 1. Where those contractors physically reside is up to the contractor. | |
| 4 | Draft PWS | 3 | 1.1 | In Section 1.1 Scope of Work, it says “These requirements will be negotiated by the winning contractor and the government as required.” Can you provide any clarification on this? | Should the government require a change to the PWS after the contract is awarded, the government will discuss and negotiate if needed, with the winning contractor. The final agreed change(s) will then be incorporated to the contract via a modification. |
| 5 | Is there a technical POC we can connect with regarding this opportunity? | Yes. However, all questions concerning this requirement should be submitted to the Contracting Office. | |||
| 6 | CRM/TEM Combined Synopis Solicitation | 22 | Table 4. Past Performance Relevancy Rating Method | Based on this being a competitive solicitation, referencing the Past Performance parameters to reach "Very Relevant", are there any other DOD 611512 NAICS code contracts in addition to the incumbent FA489018C0003 that could be used to meet the "Very Relevant" requirement? | Please refer to Table 4. The applicable NAICS code is not a criteria or part of description for relevancy. |
| 7 | CRM/TEM Combined Synopis Solicitation | 22 | Table 4. Past Performance Relevancy Rating Method | In order to promote competition, would the government consider changing the parameters of the Past Performance relevancy table to include similar contracts with large scope and magnitude in other areas, for consideration as Very Relevant? For example, change CRM to contract aircrew training (CAT); and change number of locations to "multiple locations". If the government desires to identify contractors capable of managing a large CAT/CWD contract in terms of scope and magnitude, recommend increase the value to $22M and scope to 25 contractor FTEs. | For Past Performance, the Government reduced the number of aircraft types and training locations from the current requirement to give companies other than the incumbent the opportunity to meet the "Very Relevant" rating. |
| 8 | CRM/TEM Combined Synopis Solicitation | 22 | Table 4. Past Performance Relevancy Rating Method | Please clarify that the reference to 500 FTEs in the Very Relevant criteria, 100 FTEs in Relevant, and 15 FTEs in Somewhat Relevant should read "500 students", "100 students", and "15 students" respectively. | Correct. FTEs should read: "Students". Combo Table 4 will be corrected. See Amendment 0001 issued on 18Dec23. |
| 9 | PWS | N/A | N/A | Can you define the Student to Instructor ratio for each location? | Per PWS paras 1.5.1.1, 1.5.1.3 and 1.5.1.4, class sizes shall be between 4 and 20 with larger units having bigger classes than smaller units. FTU classes with small workload numbers and instructor upgrade classes will also have smaller class sizes. |
| 10 | What LMS/TMS System is currently used? Can you confirm that it is a contractor-owned/contractor-operated system? Can you confirm that it will be transferred to the successful offeror during the transition? | The Government does not know the particular type of LMS/TMS currently being used but it is contractor owned and contractor provided. Per PWS para 1.7.2, all information and materials placed on the LMS/TMS must be transferable to the Government in a usable format at the completion of the contract. | |||
| 11 | PWS | 13 & 43 | 1.8.1 & Appendix 5: Deliverable Table | Section 1.8.1 Instructor Training and Certification Plan (ITCP) states "Contractor shall deliver the ITCP within 20 days after contract award." |
Deliverable: ITCP states Due Date: "With Contractor's Proposal" with Distribution stating "Per Solicitation Instructions"
| There is no mention of the ITCP within the combined synopsis/solicitation. Will government please confirm when the ITCP is due? | The ITCP is due within 20 days after contract award and not with the proposal. Appendix 5 will be corrected. See Amendment 0001 issued on 18Dec23 | ||
| 12 | 1.11.1 CRM/TEM Remote Training Setup | All training will be conducted using a secure method (i.e. CVR, DCS, GVS, MS Teams). |
| Can you provide information about the online learning platform that is currently being utilized for teaching students remotely? | During the COVID 19 pandemic, the incumbent contractor used a combination of CVR and MS TEAMS. | |||||
| 13 | PWS | 4 | 1.4 | The Government wants innovative ways to make CRM more than just an annual event, but does not reward innovation in the evaluation criteria. With the Tech Vol being pass/fail, how will the government evaluate and reward innovation? | Unfortunately, we cannot reward innovation in the evaluation criteria since we are using the Pass/Fail rating system. Any innovation proposed that is recognized by the Source Selection evaluation team that will improve the current CRM/TEM training programs will be incorporated into the new contract. | |
| 14 | PWS | 19 | 4.1.3 Instructor/Subject Matter Expert Qualifications | The Government will give some flexibility with this requirement in-regards to the newer Weapon Systems (e.g. F-35). |
| Can you list the aircraft types in the "Newer Weapon Systems" category? | We use F-35 as an example but could also include F-22, E-7, E-11 and EC-37. I/SME hiring qualifications will be judged on a case by case basis. | |||||
| 15 | Combo | 22 | 3.2.4 | The descriptions for the Very Relevant and Relevant Past Performance rating factors restrict those two ratings to only offerors who have managed CRM training programs. While the government has performed CRM/TEM courseware development and instruction for over two decades, restricting the Past Performance rating factor to only CRM training unnecessarily limits competition. The demonstrated ability to manage large, complex, geographically disparate USAF training program development and delivery support provides a more relevant past performance metric. Recommend the “CRM” be removed from the first sentence of each rating’s description resulting in “Effectively managed a DoD training program as prime…” for the Very Relevant rating and “Effectively managed any type of training program (DoD or commercial) as a Prime…” for the Relevant rating. | For Past Performance, the Government reduced the number of aircraft types and training locations from the current requirement to give companies other than the incumbent the opportunity to meet the "Very Relevant". See Amendment 0002 issued 20Dec23 for revised Table 4 - Past Performance Relevancy Rating Method. | |
| 16 | Combo | 22 | 3.2.4 | The total contract value requirements outlined in Table 4 for “Very Relevant”, “Relevant”, and “Somewhat Relevant” ($10M, $5M, and $5M respectively) are overly restrictive (especially for Small Business concerns) when combined with the other factors required to reach a specific rating. There are many (current and past) task orders across the DoD enterprise that are performed at disparate geographic locations with more FTE throughput accomplished at a total contract value of less than $5M. Recommend the Total Contract Value requirements be removed from the Past Performance Relevancy descriptions. We agree that the ability to manage large contracts similar in scope for USAF provides Performance Confidence, and therefore, recommend a 1 page addition to the Past Performance submission requirements to require a description of the offeror’s ability to manage complex, disparate work forces and have this management credibility added to the Performance Confidence Assessment. | For Past Performance, the total contract value requirements have been reduced from current contract value to give companies other than the incumbent the opportunity to meet the "Very Relevant" rating. Offerors will be allowed to submit 1 page addition to outline their past performance experience or ability to manage complex, disparate work forces for the Government to consider in the Performance Confidence Assessment. See Amendment 0002 issued 20Dec23. | |
| 17 | Combo | 22 | 3.2.4 | The past performance relevancy descriptions reference a minimum number of locations for providing training and courseware development. However, it is not uncommon, especially in a post-covid environment, for courseware development to be performed remotely, at a contractor’s site, or at a single location of the customers’ choosing and then flowed out to the field. Like courseware development, remote and virtual training course delivery has evolved into a standard practice with very few formal training courses requiring travel for in-person presentation. Establishing a minimum number of locations as a part of Past Performance relevancy unnecessarily shifts the focus of the criteria onto an offeror’s ability to manage / enable personnel travel to and from work locations and away from the primary objective of providing CRM/TEM training. Recommend removal of the minimum number of locations requirement in the description and adding a requirement that the past performance example require contractor travel to perform duties. | The minimum locations required for past performance evaluation has been reduced from the level required in the PWS. Even though remote CRM/TEM training will be used during the new contract, the primary means of training will be face to face with the CRM contract instructor and the aircrew members at their home base location. | |
| 18 | Combo | 15 | 8.1 | Para 8.1 states that the offeror will include any related travel cost in either the CAT or CWD CLINs IAW the PWS. Will the government please provide past actual travel costs incurred related to each MAJCOM for each year of the current task order? Without having a set “expected” travel cost to utilize during the bidding process, it puts all bidders besides the incumbent at a severe disadvantage. | The Government does not have the information for past actual travel cost as that will be determined by each bidder based on the number of trips required, the number of classes to be conducted, the number of aircrew to be trained and the number of contract instructors being utilized. | |
| 19 | PWS | 7 | 1.5.4 | Paragraph 1.5.4 describes a Contract Transition / Phase-In period. Is this period the 2-month CLIN 0101 PHASE IN period listed in A2 (the price schedule)? It appears from SOW paragraph 1.5.6 that in the first two months (CLIN 0101) the awarded contractor is taking the existing courses and redesigning them in their format (NOTE 1 from SOW 1.5.6). If this is the case, recommend the government determine a Not-To-Exceed (NTE) cost of updating the existing courses to not disadvantage all offerors except the current incumbent. The current incumbent presumably will not have to perform this activity; establishing a NTE fair value to utilize in A2 – CLIN 0101 will eliminate this possibility. If this is not the case, request the transition period requirements be discussed at the Virtual Industry Day so offerors can properly cost this activity. | The Government's intent is for the new contractor to use the 2 month phase-in period to train all instructors utilizing the existing courseware. The Table in para 1.5.6 is the timeline for updating and implementing new CRM/TEM training materials. | |
| 20 | COMBO | 22 | 3.2.4 | The requirements outlined in Table 4 Past Performance Relevancy Rating Method for Very Relevant, Relevant and Somewhat Relevant are significantly different than the 2011 and 2017 evaluation criteria and unnecessarily restricts competition. | ||
| Point A. | The total contract value requirement ($10M and $5M) is unrealistic since it appears to eliminate all but the incumbent past performance. Rationale: | |||||
| • | Across all DOD (and DHS) government buyers it appears that there is only one contract (the subject of this solicitation) that meets the criteria in Table 4. This contract has been held by the incumbent, CTI, since its inception. | |||||
| • | Several past and current Air Force aviation training development, delivery and management contracts executed across distributed locations to performers on multiple platforms are accomplished at a total contract value much less $5M (as low as $450K). | |||||
| • | Criteria outlined on page 65 of the 2011 solicitation (FA4890-11-R-0002) communicated a balanced approached to relevancy rating. The 2017 solicitation did not consider relevancy of past performance as an evaluation criterion. Since both competitions yielded the same results, it’s difficult to understand the rationale for making relevancy criteria even more restrictive. | |||||
| • | Small, medium, and large companies experienced significant reductions in contract level of effort (LOE) and or cancellations starting in 2020 because of the COVID pandemic influence on accessibility of personnel, reduction of demand due to lower operational tempos, and or redistribution of funding to cover higher fuel/resource costs. This would prevent an otherwise relevant contract from being rated highly (or at all) on the current scale. | |||||
| Point B. | The criteria “Effectively managed a DoD CRM training program as prime contractor while providing classroom and/or simulator training and courseware development utilizing the Instructional Systems Development (ISD) model” is unnecessarily restrictive. | |||||
| • | Given that the government has acceptable curriculum and instructors in place, experience managing complex, distributed training programs and a working knowledge of DOD training policy and procedures is more critical. A prime contractor with extensive experience managing large scale DOD training management support contracts can easily team with an established aviation human factors small business to update and develop new CRM TEM curriculum and exceed Government expectations. | |||||
| • | Section 1.5.5 Continuity of Services states that “In the event the incumbent is not the successful contractor for the follow-on contract award, the incumbent contractor hereby agrees to cooperate with the successor and to allow its successor to conduct interviews with its employees.” This reinforces the presence and availability of qualified ISDs and instructors to continue developing and delivering CRM TEM curriculum. | |||||
| • | The Air Force Instructional Systems Development (ISD) model as defined in AFMAN 36-2234 and AFH 36-2235 follows the industry standard Analyze, Design, Develop, Implement, and Evaluate (ADDIE) model. Experienced Commercial and/or Government/Military ISDs have the capability to update and create new courseware consistent with the ADDIE model. Management of ISD process IAW Air Force Policy and Procedures is what’s important. A prime contractor with experience managing and coordinating Air Force (CAF specifically) training across all FTU and squadron requirements would be far superior to a contractor focused solely on CRM TEM training development and delivery. |
Recommendation:
| Make Past Performance the secondary evaluation criteria and leverage the methodology for Factor 1 – Past Performance beginning on page 64 of the 2011 solicitation (FA4890-11-R-0002). Change definition of Recent to “contract performance during the past five (5) years from the date of issuance of this solicitation,” to accommodate impacts of pandemic. | Please see Amendment 0002 issued 20Dec23 for revised Table 4 - Past Performance Relevancy Rating Method. | ||||
| 21 | PWS | 3 | 1.4 and 1.5.2 | The CRM/TEM Program core curricula outlined in section 1.5.2. From its inception in the late 1970s, CRM has been defined as “utilizing all available resources to safely accomplish the mission.” Over the decades, the program has evolved through several “generations” but remains mostly focused on crew interactions and traditional human factors, to include Threat and Error Management (TEM). |
The solicitation evaluation of technical capability (acceptable or unacceptable) focuses on maintaining the status quo from a value-added perspective. Based on the requirement outlined in the PWS, the CRM training has not evolved to adequately address modern aviation cockpit/crew challenges. Since the publication of Advisory Circular 120-51E in 2004, a great deal has changed in aviation.
Concerns:
| • | The post-COVID mental health crisis is impacting tens of thousands of aviation and non-aviation professionals, adding distractions, stress, and impacting decisions that can serious safety implications. | ||||
| • | Recruitment and retention challenges have led military services into a system-wide service retreat with a significant shortage in qualified pilots and enlisted aircrew, further exasperating individual and crew ability to cope with the same or even more demand. | ||||
| • | Vast leaps in technology have made the automated flight systems a “silent crewmember” that we must communicate with effectively. | ||||
| Recommendation: Using evaluation methodology beginning on page 67 Factor 2 – Mission Capability Technical of the 2011 solicitation (FA4890-11-R-0002), change the Technical Approach to the primary evaluation criteria. This better aligns with the defined purpose of the contract outlined in Section 1.4 of the solicitation, “the contractor is required to develop innovative ways to make CRM/TEM more than simply an annual training event—to continue to integrate CRM so that it is a conscious part of the way aircrews do business.” | Please see Amendment 0002 issued 20Dec23 for revised Table 4 - Past Performance Relevancy Rating Method. | ||||
| 22 | COMBO | 22 | 3.2.4 | The requirements outlined in Table 4 Past Performance Relevancy Rating Method for Very Relevant and Relevant establish “at least five different aircraft weapon systems and at least 10 locations” or “two or more different aircraft weapon systems and at three or more locations” during each period of performance as a criterion for relevancy. |
This appears to be a method to establish the complexity/scale of a given contract. However, it unnecessarily restricts competition since a contract could easily focus on the same number of roles across one or to weapons platforms.
Recommendation:
Since people are the focus of CRM TEM training, recommend the government uses air crew and support roles (e.g. Air Force Specialty Code (AFSC) as the basis for establish scale or complexity. Use the methodology for Factor 1 – Past Performance beginning on page 64 of the 2011 solicitation (FA4890-11-R-0002) in lieu of the current methodology. Change definition of Recent to “contract performance during the past five (5) years from the date of issuance of this solicitation,” to accommodate impacts of pandemic. The Government prefers using aircraft types versus Air Force Specialty Codes in regards to past performance evaluation. Please refer to Combo Paragraph 7.3.2 Recency Determination for recency definition.
&"Arial,Bold"SOURCE SELECTION INFORMATION - See FAR 2.101 and 3.104
&"Arial,Bold"SOURCE SELECTION INFORMATION - See FAR 2.101 and 3.104 Page &P of &N
File details come from the government source that posted it. Updated .