AF-WIN RFP FA300221R0017 Questions_Answers.pdf

Ques #

Answer

This is a new requirement.

Do you know if there are any potential prime contractors that may be interested in subcontracting a portion of this contract out? This is a new requirement.

Is there currently a system of record being used? (i.e. database, computer vision algorithm, etc)

Can you describe the current access points and data systems used to store data?

What time period does the current data set cover?

Is the data segemented in any way to address societal changes, profiency changes in required skills, etc?

What is the current system and process used?

If the requirement to run on a standalone machine without networking capabilities is flexible, we recommend exploring cloud architecture to further automate this process and interface directly with Air Force systems. For example, a future end-state might be cloud components that enable AF-WIN to be updated automatically each time the AFECD is updated.

Is there currently an approved Institutional Review Board (IRB) plan that can be used to continue the ongoing work? Or will award of this contract require a new IRB review?

The RFI states that historically, 160 man-hours per year are expended on this effort. It is expected that more frequent AFSC changes will be propagated to AF-WIN as a result of efficiency gains from this effort? Is the current biannual tempo (mentioned p 3,4,6) the optimum?

Multiple types of input documents are mentioned (AFECD MS Word and PDFs, TTMS-JM spreadsheets, GSD&M spreadsheets, etc).

Could you provide a rough estimate of how frequently each is used, e.g. are most input documents expected to one or two particular types?

Database specialists are mentioned; can you provide some clarity around what kind of databases are expected to be used in light of the non-networked requirement? Will some limited connectivity to database servers or shared drives be available, or will all databases be local?

Performance objectives 2 and 3 require 80% reduction in manual data processing. Can you provide more information about how this will be measured?

The Price spreadsheet contains 1 Clin (line item), is one price for entire project acceptable or is a breakdown individual labor categories required?

Is an analysis of the efficiency/accuracy of the existing AF-WIN application required?

Vendor and staff, Dr. Stephen E. Watson, has experience in building like systems; is this a valid methodology for estimating cost.

How will the Government provide required data?

Q&A for AF-WIN RFP #FA300221R0017

Could you please tell me if this is a new requirement, or if there is a current or previous contractor who has performed similar services? If available, can you please send along the incumbent contract or task order number?

Question

Is there interest in using algorithmic, automated approaches for image coding and selection to mitigate bias and accelerate this process as well? Frameworks for responsible machine learning, AI, and data science are available and beginning to be adopted across the field.

Can we assume end-users responsible for modifying syntax will be able to directly edit SAS or R code, or is a standalone graphical user interface for non-technical users desired? Would there be interest in deploying something like a Shiny application as a lightweight interface to user-modifiable values?

Because of the diverse requirements beyond just code, we recommend a more general NAICS, such as 541690 Can we assume end-users responsible for using the tool are comfortable executing the software from the command line or R/SDS integrated development environments (IDE)? Or is a graphical user interface for using the tool desired?

Do you have a minimum / maximum page requirement for the proposal?

If CPARS evaluations are not available can separate evaluations be sent to current/past contract Cos/CORs?

Taken into consideration but preponderance of work is NAICS 541511.

A labor breakdown is now required which shall be included on the Price Proposal Spreadsheet. You may alter the spreadsheet by adding the labor breakdown. Please also provide a breakdown of the cost for each individual task delineated in the Combined Synopsis Solicitation. Include this information in the body of your proposal, not on the Price Proposal Spreadsheet.

Do you require two separate applications/programs as specified in 1.1.2.2.1 and 1.1.3.2.1 or is a single program to both functions acceptable? a. Are the any GUI requirements? b. Are the programs completely standalone or are they required to send log/user summaries to a central server? c. Is providing source code sufficient for the editing requirement? If not what specific editing capabilities are required?

There is no minimum or maximum page requirement.

Evalutions will not be allowed to be sent separately from the proposals. However, if CPARS are unavailble evaluations from COs/CORs may be included in the proposal.

Past, relevant performance and experience of the contractor should be included in the proposal. Individual resumes are not required but may be provided to support that the proposed personnel meet the expertise outlined in the PWS.

First, be aware that solutions will not be able to interact with any central server and/or external database. However, solutions can generate log/user summaries for record-keeping and validation purposes, but they will only be retained locally. We do not have any hard requirement for a one versus two program solution, only that the end result provides an acceptable solution for both task 2 and task 3 requirements (post-survey filters vs. occupational snapshots). There will likely be significant overlap in portions of the two programs where data extraction takes place, so efficiencies may be realized by combining the two solutions into a single program. On the other hand, it may be wise to keep the programs separate to reduce potential cascading issues in one requirement impacting another requirement. There is no requirement for a GUI and the ability to edit program syntax is sufficient, but if the solution is able to develop a lightweight, flexible GUI that reduces monotonous syntax-level editing, then that is acceptable. Syntax editing capabilities should permit end-users to make edits to the solution in response to Air Force requirement changes such as adding/removing a new filter, adding/removing AFSCs, etc. Please note that all solutions, whether syntax or GUI-based, need to be in SAS or R.

No.

With the current timeline, we do not expect the proposed solution to be fully operational by the 1-Nov AFECD update. Realistic deliverables by that time would be a firm understanding of the current AF-WIN content updating process as well as a well-defined proposed solution in the process of being realized/developed. Hopes are to be able to implement the proposed solution in whole or in part by the 1-May AFECD update. However, don't be bogged down by the 1-Nov or 1-May date, as those are not deadlines for the contractor, just AFECD publishing dates. If the proposed solution is completed 1-Jun we can still use database extracts and documents from the 1-Nov or 1-May AECD update to assess the solution. In fact, it may be preferable that the solution is applied to an AFECD where the previous manual process has been completed to identify gains in efficiency/accuracy compared to the manual process. AFPC/DSYX would have significantly more confidence in the proposed solution if we can compare/contrast to a past AFECD update before applying it to a future AFECD update where a manual process was not already completed for comparison.

Processing time for initial recruit entry data is to be improved by 80%; how is the current processing time measured and documented?

Is AF-WIN a centralized system? As required update programs are standalone, non-networked different recruiting stations could have different filters is that this intent?

With incremental development what is expected to be delivered for the 1 Nov system update.

Will any onsite meetings be required?

Should resumes be included in the proposal?

The AF-WIN survey itself is not a standalone system - it is hosted on Airforce.com through Salesforce by an existing Air Force contractor.

The non-networked requirement here is in relation to how post-survey content updates are pulled from local file extracts of known information sources (databases, PDFs, etc.). That is, we are asking for a solution that, once database extracts are collected and loaded onto a non-networked computer, we can run the solution to reliably and accurate extract changes to these documents and provide an updated, validated set of documents to the contractor who maintains the AF-WIN. The current and future process will still involve providing the AF- WIN contractor a set of files to update the AF-WIN on Salesforce, and we do not anticipate the capability to interact directly with Salesforce or the AF-WIN at this time. We are not interested in developing different filters for non-networked recruiting stations, and the assumption by Air Force Recruiting Service is that all recruiting stations are networked or the recruit has personal access to the internet to take the AF- WIN on Airforce.com.

See response to Q11.

The government will provide required data to the contractor via DoD SAFE. Data provided will be in excel, CSV, Word, PDF, and SAS file formats. The government will also provide SAS syntax used in the current, manual process as a starting point for the contractor to provide a more streamlined solution.

The AF-WIN is heavily based on the Navy JOIN, which was originally developed in part by Dr. Stephen Watson. However, what is being asked here is not to develop a vocational interest tool or vocational interest tool content, but to streamline the updating process of post-survey filter and occupational snapshot content that informs/impacts how the user interacts with survey results AFTER the survey has been completed and a fit score has been calculated. If Dr. Watson has completed similar work, please refer to it when estimating cost with the caveat that we are asking for a low-tech, non-networked solution here that can extract AF-WIN post-survey content from local database extracts in excel, word, or PDF format.

An analysis quantifying efficiency/accuracy gains of the provided solution versus the current, manual process would be beneficial. This could be achieved by applying the provided solution to a past AF-WIN update and comparing the solution's results with the historical manual process results. Differences in new/old process could be highlighted and a manual check would determine which approach resulted in correct vs. incorrect values.

These documents are used once per AF-WIN content update cycle. The current process involves extracting relevant columns/data/text from each document and integrating it into an AF-WIN post-survey filter content and occupational snapshot content spreadsheet. Input documents will all be in the following formats: Word, PDF, CSV, Excel, or SAS.

No connectivity to database servers or shared drives will be available, and all databases will be local extracts. Databases will be in flat, CSV, Excel, or SAS/SQL format, and the requirement for a database specialist is more so related to their skills in accurately and reliably manipulating and processing data while ensuring a high-level of data integrity.

Most end-users will be comfortable with editing syntax in SAS and/or R. However, if a low-impact, flexible graphical user interface can be developed to reduce need to edit syntax, then we would be open to that capability.

End-users will be able/responsible for modifying syntax and directly editing SAS or R code based on changing needs/requirements.

However, if the implementation of an R-based Shiny application can be used to reduce direct editing while still providing flexibility in response to changing needs/requirements, we would be open to that capability.

Not at this time. We are looking for a low-tech solution that provides two "sets" of images for each AF-WIN item so that images don't need to be collected again for the next 5-7 years.

The AF-WIN is focused entirely on assessment of vocational interest, and as a result, does not contain information with regards to proficiency changes in required skills, etc. However, task 1.1.1. (the survey images task) does take into account gender/racial/ethnic minority representation within AF-WIN images. This means that images collected should not stereotype any gender/racial/ethnic categories (e.g., healthcare images with only females), and adequate group representation should be present for each AF-WIN item. Because each AF- WIN item contains three images, this translates to no fewer than 1 of 3 images for each item having a minority group as the main focal point.

Informational sources referenced in the RFP are updated semi-annually with the publishing of a new AFECD making each previous iteration obsolete. However, while each update results in an entirely new operational document, not all content is 100% updated. There are some AFSCs where AFS entry requirements have not changed in 5-10 years, and others where requirements have changed within the last iteration of the AFECD. The AFECD contains an appendix that summarizes which AFSCs changed in the most recent AFECD document. Solutions should focus on being able to extract post-survey filter and occupational snapshot information for all AFSCs and then in subsequent updates identify which AFSCs entry requirements have been changed/revised based on updates to informational sources.

Currently all data are gathered manually from sources described in the RFP (e.g., AFECD, TTMS-JM, Salesforce) as local database extracts and AF-WIN content is manually updated/revised as AFS entry requirements change within a basic MS Excel spreadsheet format. Changes to entry requirements are manually updated and a new version of the excel sheet is saved and provided to the contractor that updates the AF-WIN survey tool. We have no present or future capability to interface with AFS entry requirement information sources, so solutions should focus on automating extraction of standalone text/data in a non-networked computer after data extracts are manually gathered from information sources.

Currently no official systems of record are being used. All AF-WIN content is stored within basic excel documents and windows folders.

When AF-WIN content is updated, an updated excel spreadsheet is provided to the contractor that manages the Salesforce database(s) that host the AF-WIN, but this database is not a system of record as defined by the Air Force. AFPC/DSYX does not directly interact with the hosted databases and we do not anticipate the future ability to interact/interface with any systems of record or Salesforce databases.

Solutions should focus on automating/streamlining development of updated excel spreadsheets on a local non-networked computer that are then manually sent to the contractor that maintains the AF-WIN.

This will be measured by comparing the amount of manual data processing required by the proposed solution compared to amount of manual data processing required by the current process. For example, if 80 hours of manual data processing/checking is estimated/required by the current process, the solution should result in no more than approximately 16 hours of manual data processing/checking. This will be quantified during the application of the contractor's solution during a semi-annual AFECD update cycle. The areas in which manual data processing can be reduced the most include 1) reducing the need to manually read the AFECD by extracting changes related to updated AFSCs, 2) reliably extracting/processing data from PDF and database extracts (e.g., TTMS-JM, etc.) and 3) verification of extracted AFS entry requirement information. For task 3 this also includes the above while also minimizing the need to individually interact with CFMs when they update occupational snapshot information.

No. AFS entry requirement changes via the AFECD are on an established semi-annual update schedule and AFPC/DSYX has no control over that tempo (it is set by policy). Since updates to the AF-WIN are a direct response to updates to the AFECD and AFS entry requirements, AF- WIN updates are/will always also be semi-annual unless the tempo of AFECD updates increases or decreases. In short, the AF-WIN always updates content reactively in response to updates to the AFECD.

The work being completed here falls under programmatic evaluation and process improvement of an operational Air Force selection/classification program. Therefore it does not require approval from an institutional review board (IRB).

The requirement to run on a standalone machine without networking capabilities is not flexible, and all proposed solutions should keep this in mind. It may be a distant future goal to further automate this process with automatic updates to the cloud, but currently we do not have that capability, and we do not anticipate having that capability in the foreseeable future.

For all tasks (1.1.1. - 1.1.3.) the process is 100% manual, meaning information sources (e.g., local database extracts) are manually reviewed by the individual updating AF-WIN content, and changes are keyed into relevant spreadsheets to update them. The spreadsheets are then provided to the contractor that maintains the AF-WIN and they make the needed changes to the AF-WIN site. Solutions should focus on automating/streamlining the extraction of data from information sources once standalone extracts are obtained. Specifically, we want to automate/streamline the current manual process of reviewing each information source extract and updating AF-WIN content spreadsheets based on observed changes. We will not have the capability to automatically interact with information source databases, nor will we be able to automatically upload changes directly to the AF-WIN website. The desired solution is to create a set of standalone programs that can process information source extracts reliably and update AF-WIN content spreadsheets quickly so we can send them to the contractor.

Sheet1