Amendment 1 - Questions 6.10.xlsx
XLSX spreadsheet 29 KB Posted
- Attached to
- NCSER Crowdsourcing Campaign Support
- Solicitation number
- 919900-24-1327
About this file
This document is an Amendment 1 containing questions and responses related to a federal contract opportunity for a "NCSER Crowdsourcing Campaign Support" solicitation from the Department of Education Contracts and Acquisition Management.
The key details are:
- The contract supports the Institute of Education Sciences (IES) National Center for Special Education Research (NCSER) in conducting a crowdsourcing campaign to obtain information from stakeholders on how individuals with disabilities and their families have experienced special education.
- The contractor will be required to design and implement a challenge through Challenge.gov for analyzing the crowdsourced data using either qualitative methods or natural language processing.
- The contractor may also be required to raise the profile of NCSER's nearly 20 years of funded research.
- The government provided responses to 56 technical and contractual questions from potential offerors covering topics such as the campaign's goals, data collection and analysis, communication and outreach, project management, judges and evaluation criteria, optional tasks, security and data privacy, and budget.
View the file
Other files
| File | Type | Posted |
|---|---|---|
| Amendment 2 - Request for Proposal NCSER Crowdsourcing.docx | DOCX document | |
| Amendment 1 - NCSER Crowdsourcing Support PWS.docx | DOCX document | |
| Steps in the ICR Process Fact Sheet.docx | DOCX document | |
| Amendment 1 - Provisions and Clauses.docx | DOCX document | |
| Amendment 1 Attachment A NCSER Crowdsourcing Support PWS 5.17.docx | DOCX document | |
| Attachment B Price Template.xlsx | XLSX spreadsheet | |
| Attachment A NCSER Crowdsourcing Support PWS.docx | DOCX document | |
| Request for Proposal NCSER Crowdsourcing.docx | DOCX document | |
| Attachment C Provisions and Clauses.docx | DOCX document |
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Sheet1
| Question Number | Reference | Question Category | Question | Government Response | CO Comments | CS Response/notes |
| 1 | Technical | Goals and Expected Outcomes: Can you provide more specific details on the goals and expected outcomes of the crowdsourcing campaign beyond those stated in the PWS? | Nate Jones as the new Commissioner of NCSER is developing a strategic plan for NCSER and future funding opportunities. He and the staff of NCSER would like to hear from learners with disabilities and their families about their experiences in special education to better understand what is working and where the needs are. We want to hear from as broad as possible of a group of learners and their families representative of this nation in terms of ages and types of disabilities, regions of the country, urbanicity. |
The PWS governs the specific contract requirements.
| 2 | Technical | Survey and Data Collection: What specific topics or questions are expected to be included in the survey for individuals with disabilities and their families that are not detailed in the documents? | NCSER is interested in the lived experiences of people with disabilities and their families especially related to special education in schools today to understand the pressing problems facing special education today. We are currently doing some other need-sensing activities that will inform the questions and survey design. NCSER leadership and program officers intend to be involved in the creation of the survey questions along with others from across IES. | ||
| 3 | Technical | Technical and Methodological Approach: Are there any preferred methodologies for analyzing the collected data (qualitative methods, NLP, etc.) that you would recommend beyond what is mentioned? | The Optional Task 2 Challenge will have a prize for use of qualitative methods to analyze the data and a prize for the use of Natural Language Processing methods. There are no preferred methods. We are awarding prizes to both since both methodologies can provide valuable information to NCSER. The challengers are the ones conducting the analysis not the Contractor. The Contractor is managing the challenge which includes subtask 2.4 Recruiting and Vetting Judges who should include people with expertise in Qualitative methods and people with expertise in NLP | ||
| 4 | Technical | Technical and Methodological Approach: Can you provide examples of previous successful crowdsourcing campaigns similar to this one? | We are not aware of any crowdsourcing campaigns directly related to this one but we are aware of other organizations using Challenge.gov for crowdsourcing such as: Time is Now: Advancing Equity in Science & Technology https://www.challenge.gov/?challenge=ostp-time-is-now REACH U Challenge https://www.challenge.gov/?challenge=reachu-challenge | ||
| 5 | Technical | Communication and Outreach: What are the key performance indicators (KPIs) for the communications plan and outreach strategy, particularly those not detailed in the PWS? | The requested information is outlined in the PWS. | ||
| 6 | Technical | Communication and Outreach: Are there any specific stakeholder groups or organizations that should be prioritized in the landscape analysis and outreach efforts beyond those listed? | Outlined in the PWS, subtask 1.5 the contractor will be given "information IES/NCSER has already collected on stakeholders and amplifiers in the special education space" that can reach the desired target of individuals with disabilities and their families, but this list may not be complete. We are seeking contractor support to find other ways to reach the target audience as described in the PWS. If there is any priority it would be on organizations that reach the largest number of individuals with disabilities and their families but also is comprehensive so that we are reaching all disability types. So the landscape analysis will include multiple organizations so that it is broad enough to not leave any group out. | " | |
| 7 | Technical | Project Management and Timeline: Is there flexibility in the proposed timeline, especially regarding the optional tasks? | The applicants may provide recommendations and justifications for adjustments to the timeline in their proposal for both the main task and the optional tasks given the understanding that NCSER is under pressure to learn from these stakeholders in as short of a time as possible to inform future work of the Center. | ||
| 8 | Technical | Project Management and Timeline: Can you elaborate on the process and criteria for evaluating the success of the campaign and the analysis challenge? | The Success of the Task 1 Crowdsourcing campaign would be evidenced by responses that demonstrate representativeness in terms of geographics (states, urbanicity), ages/grades, types of disabilities, types of schools and special education programs, and participation by both individuals with disabilities and their families. Success would also be evidence by the richness of the information collected and insights it provides to the future work of NCSER. | ||
| 9 | Technical | Judges and Evaluation Criteria: What qualifications are required for judges in the analysis challenge, and what conflict of interest policies are in place beyond the basic criteria provided? | IES will provide the contractor with Non Disclosure Agreements and Conflict Of Interest templates to use with the judges who are selected that we usually use for this purpose. The Contractor provides a list of recommended judges that capture people with expertise in special education, disability types, and the analysis methodologies to be used for the challenge (qualitative methodologists and natural language processing methodologist). The list could also include experts in families, since families of students with disabilities will be responding. The experts could also include practitioners, such as special education administrators, and other leaders who have an understanding of the special education system more broadly including programs in juvenile justice systems, related service providers (e.g., Speech and Language Pathologists, Occupational Therapists), and Assistive and Augmented Communication Technologies. We will discuss this further with the awarded Contractor once we decide to activate Option 2. | ||
| 10 | Technical | Optional Tasks: Under what conditions would Task 2 (analysis challenge) and Task 3 (raising NCSER's profile) be activated beyond what is described in the documents? | If limited responses are received that are manageable for NCSER's program officers to review then we would not activate Option 2. We are hoping that we receive a large number of detailed responses from the Crowdsourcing that would necessitate the Option 2 task. Option 3 task is important to NCSER and a need at this time; however, whether we activate it will be based on the government need and availability of funds for Option 3 at the time Option 1 is completed. | ||
| 11 | Technical | Optional Tasks: Are there any additional deliverables or expectations for these optional tasks beyond what is outlined in the PWS? | Not that we are aware of at this time. | ||
| 12 | Security and Data Privacy: Are there any additional data privacy protocols or compliance measures we should be aware of beyond those mentioned in the documents? | The Department periodically updates the guidance and policies around data privacy and security and compliance. At the time of the award the contractor should receive the most updated guidance and policies related to privacy and security. It is the contractor's responsible to be aware of and comply with the departments privacy and compliance measures. | |||
| 13 | Budget and Funding: Can you provide more detail on the expected budget for this project, including funding for prizes and honoraria beyond what is mentioned? | For honoraria, NCSER has typically paid subject matter experts up to the GS15 Step 10 rate which is listed in subtask 2.4, but the contractor can propose the honoraria rate in the proposal they feel is appropriate. We have not determined the prizes yet because it will depend on the amount of data to be analyzed from Task 1. At that point we will determine the prizes. No other budget information can be provided. | |||
| 14 | Submission Requirements: Are there any formatting or submission guidelines beyond those already provided, especially for the Excel and PDF documents? | No. | |||
| 15 | Contract | Submission Requirements: Can you confirm the process for submitting questions and receiving amendments to the RFQ beyond what is already described? | Please see RFP document for Q&A process. | ||
| 16 | RFP | Technical | Can any of the marketing or communication for Task 1 be done outside of IES channels? | Task 1 subtask 1.6 includes use of local media engagement strategies and media outlets in addition to the IES channels. The contractor may use information gained from the landscape analysis that could also provide additional channels. | |
| 17 | RFP | Technical | For the key staff role with moderate security clearance, does this need to be at the prime contractor level, or can the requirement be met through a subcontractor? | Yes, the contractor personnel called a Security Liaison to IES for the contract must be at the prime contractor level. Work for Task 1 can begin while the contractor personnel is going through the clearance process. The person will need to be cleared by the time Option 2 is exercised, since they collect personally identifiable information and information that needs to be kept private and secure for processing of the honorarium. | |
| 18 | PWS, E. Scope of Work & POP, Base tasks | Technical | There are multiple references to the Base Task - Crowdsourcing Campaign being announced via the Federal Register (timeframe table page 4, sub-task 1.7 page 10). The note in sub-task 1.6 (page 10) says that "the Crowdsourcing Campaign will be announced through Challenge.gov and not the Federal Register". Can you please clarify if announcing via the Federal Register is needed for this task? | IES will determine whether there is a need to announce in the Federal Register and if so, IES/NCSER will be the one responsible for any Federal Register announcement. The Contractor is responsible only for the Challenge.gov posting and announcement. | |
| 19 | PWS, E. Scope of Work & POP, Optional Task 2 | Technical | There are multiple references to the Optional Task 2: Challenge being announced via the Federal Register (timeframe table page 4, sub-task 2.7 page 16). The note in sub-task 2.5 (page 10) says that "the Crowdsourcing Campaign will be announced through Challenge.gov and not the Federal Register". Can you please clarify if announcing via the Federal Register is needed for this task? | IES will determine whether there is a need to announce in the Federal Register and if so, IES/NCSER will be the one responsible for any Federal Register announcement. The Contractor is responsible only for the Challenge.gov posting and announcement. | |
| 20 | PWS | Technical | Can IES clarify their goals for Task 1? | See response to Question 1 | |
| 21 | PWS | Technical | Can IES clarify their goals for Task 2? | We are hoping that we will receive a large number of responses to Task 1 that would be too time consuming for NCSER staff to analyze. Therefore we would like some proposals for ways to apply either qualitative data analysis methods or natural language processing to learn from the responses. We think that both approaches are valuable and might result in different insights. We would like to learn from the information received and feel that the challenge is one way to do that that would be less time consuming and burdensome on staff and be able to be done by experts in these two methodologies | |
| 22 | PWS, pg. 3 | Technical | What have been past barriers to collecting data about how individuals with disabilities and their families have experienced special education? | The impetus for this competition is that NCSER would like to make a more concerted effort to ground our research interests in the needs of the communities we serve. We could use traditional approaches like a request for information through the federal register, but the concern is that such an approach would not reach sufficient families. We typically reach our "usual networks" of people who are familiar with IES and NCSER. Our goal here is to get past these networks to a much larger pool of respondents. We would like to see if use of contractor support and the Crowdsourcing idea will reach and engage more families. | |
| 23 | PWS, pg. 3 | Technical | How will the data collected through the crowdsourcing campaign be used beyond Task 2? How will the data be used if there are not sufficient responses to warrant conducting the challenge in Task 2? | The data will be used by IES/NCSER staff to inform future funding opportunities and a new strategic plan for the Center. The data will be analyzed by NCSER program officers if it is a small amount and not sufficient to warrant activating Option 2. | |
| 24 | PWS, pg. 3 | Technical | What volume of responses is IES seeking in Task 1? Is IES seeking any specific representation in the sample? | We are seeking representativeness in terms of geographic diversity (states, urbanicity), ages/grades, types of disabilities, types of schools and special education programs, and participation by both individuals with disabilities and their families. Success would be evidence by the richness of the information collected and insights it provides to the future work of NCSER rather than volume of responses. We are hoping to receive sufficient information to warrant activating Option 2. | |
| 25 | PWS, pg. 5 | Technical | Given there are no monetary prizes, what incentives will be offered for families to participate? | We do not plan to offer any incentives to families to participate. See this example Crowdsourcing campaign on Challenge.gov that did not offer any prizes or incentives to people to respond. https://www.challenge.gov/?challenge=ostp-time-is-now&tab=overview | |
| 26 | PWS, pg. 6 | Technical | What amount does IES anticipate allocating to prizes for the Task 2 challenge? | We have not established an amount at this time. It will be based on the volume of data in terms of number of responses and length of narrative information (characters/words) received in Task 1. While the cost may not change as much for Natural Language Processing methodologies, cost in terms of time for qualitative methods may change based on volume of response text. We will take that into consideration when determining prizes after Task 1 is completed. | |
| 27 | PWS, pg. 8 | Technical | The PWS refers to OMB approval. Will the crowdsourcing campaign survey require PRA clearance? | We have revised the PWS timeline to allow more time for clearance. | |
| 28 | PWS, pg. 8 | Technical | Given this is not a competition, what is the rationale for using Challenge.gov to manage this campaign? | Other federal agencies we have spoken to have found Challenge.gov successful for Crowdsourcing so we are interested in trying this approach. There are barriers to having contractors do similar work by setting up their own website to crowdsource the information, because often an ATO is required which can take up to a year for approval. We are on a faster timeline than would allow for an ATO. We don't have the option currently to use our own website for this information in a way that has the features that Challenge.gov provides. | |
| 29 | PWS, pg. 8 | Technical | Would IES be open to using another FedRAMP tool for this survey? Challenge.gov has limited functionality compared to other platforms, and is not designed for surveys. The user experience (including account requirements) may also limit the response rate. | We recommend applicants propose to use Challenge.gov but then also make a recommendation for another approach and provide a feature/cost comparison. | |
| 30 | PWS, pg. 11 | Technical | What is the rationale for structuring Optional Task 2 as a challenge? | We are looking for innovative approaches to this data analysis task by experts in these methodologies and we are looking at getting responses faster than we can get using our typical approaches (use of grants and contracts) given how long the procurement and grant-making cycle takes. While up to two teams will win a prize for each analysis type in the end we will have the analysis from all of the teams that apply, so we feel it can provide a wealth of valuable information, since some teams may having findings that others do not. | |
| 31 | PWS, pg. 11 | Technical | Will Task 2 teams be submitting approaches to the analysis (i.e. qualitative or NLP methodologies) or the output of the analysis (i.e. a summary of themes using Task 1 data)? | The Optional Task 2 teams will determine their approach for summarizing the data. Some might use a mix of narrative findings with exemplar text samples as well as data visualizations, summary tables of themes using descriptive statistics. It will be up to them to provide a strong report that describes their data analysis approach and the findings. The judges will work with IES on the scoring rubric and it can include factors that relate to the analysis plan, visualizations and summary information provided in the findings. We will provide some guidance on Challenge.gov as to the judging criteria and also submission directions that can guide their submission. IES/NCSER will work with the contractor on the submission guidelines. | |
| 32 | PWS, pg. 11 | Technical | Will the data collected through Task 1 be shared directly with entrants? | Yes, IES can provide support for data sharing using an External Sharing Site we have. | |
| 33 | PWS, pg. 11 | Technical | Would IES consider a multi-phase structure for this challenge? | The applicants may provide recommendations and justifications for adjustments to the challenge by adding a multi-phase approach for Optional Task 2. However, an attempt should be made to address the current timeline. It is possible that a multi-phase approach could be beneficial. For example, releasing a percentage of the data to teams and then judges from this first phase would select some to move to Phase 2 where they would get the full set of data? It would be important to justify how this would add value rather than just cost and time to the plan and any tradeoffs. We would prefer to constrain cost and time given the need to understand this information to inform our work soon. | |
| 34 | Attachment A - PWS - Section E. Scope - Anticipated Prize Schedule | Technical | Does the government have a rough order of magnitude/range estimates for prize amounts? | No. See response to Question 26 | |
| 35 | Attachment A - PWS | Technical | What is the disposition (roles, size) of the IES/NCSER team that the contractor will be engaging? | You will primarily engage with the COR. Some meetings will include a program officer from NCSER called the IES Lead. Other times the meeting will also include the Commissioner of NCSER. So at most 3 people but more frequently one person. | |
| 36 | Attachment A - PWS, Sub-Task 1.6 | Technical | What data/research and current outreach materials does IES/NCSER have to support the communications plan outlined in Sub-Task 1.6? Will this be made available to the contractor upon award? | Outlined in the PWS, subtask 1.5 the contractor will be given "information IES/NCSER has already collected on stakeholders and amplifiers in the special education space" at the time of award that can be used to reach the desired target of individuals with disabilities and their families, but this list may not be complete which is the purpose of the landscape analysis. | |
| 37 | Is there a known budget or not-to-exceed price for this work? | No. | |||
| 38 | Is this a follow-on or new requirement? If follow-on, please provide incumbent award information. | New | |||
| 39 | Technical | Has NCSER determined a prize purse for the challenge? | No. See response to Question 26 | ||
| 40 | Technical | How many key personnel are required and what labor categories are considered key personnel? | The PWS only requires the contractor to propose a single key personnel for PII management | ||
| 41 | Is the primary place of performance expected to be at the contractor’s site? | Yes | |||
| 42 | Will travel be required for this effort? | No | |||
| 43 | Technical | Can the government provide an example of the type of data that will be analyzed in the data analysis challenge? | Like most surveys it can include multiple choice, true/false, short answer and long response (paragraph) answers. The narrative responses in the short and long response answers will be what will need special attention by qualitative methodologists and/or natural language processing methodologits to better understand the experiences shared by learners with disabilities and their families. | ||
| 44 | Will the contractor be required to build an online survey using an existing COTS platforms? | No. See this other Challenge.gov Crowdsourcing campaign to see how this information can be collected through Challenge.gov. https://www.challenge.gov/?challenge=ostp-time-is-now&tab=overview The Applicant can also recommend another approach, but should include the plan to use Challenge.gov. | |||
| 45 | Technical | Approximately how many responses would need to be collected to warrant the optional data analysis challenge? | It is not the number of responses but the volume of narrative to be analyzed based on open ended responses. If it is more than NCSER program officers can easily analyze on top of their existing responsibilities then Option 2 will be activated. That will depend on the current number of NCSER staff and responsibilities at the time Task 1 has ended. | ||
| 46 | Technical | Can NCSER please elaborate on the data distribution strategy for the data analysis challenge? Will entrants be provided a data sample to analyze or simply provide an analysis approach for the data? | See response to question 31 | ||
| 47 | Could Govt. specify the expected deliverables from the use of machine learning techniques such as natural language processing? Are there any preferred tools or platforms for conducting the NLP analysis? | There are no preferred tools or platforms for conducting the NLP Analysis. There is no deliverable for the machine learning technique. It is not a contractor task it is part of the challenge. So there will be submission guidelines that IES will work on with the contractor to specify what is required in the submission. The guidelines will be posted to Challlenge.gov. The goal is to use NLP to understand themes in the survey feedback similar to use of qualitative methods to identify themes. | |||
| 48 | What is the approval process for changes in key personnel after contract award, and are there specific qualifications or clearance levels required for replacement personnel? | Please see key personnel clause. | |||
| 49 | Can the Government provide further clarification on the specific outcomes expected from the crowdsourcing campaign? | Please see answer #1. | |||
| 50 | Are there any specific platforms or tools besides Challenge.gov that the agency prefers for implementing the crowdsourcing campaign? | No we would prefer that Challenge.gov be used. | |||
| 51 | Could the Government elaborate on the types of qualitative and natural language processing methods preferred for analyzing the crowdsourced data? | No, it is the job of the challengers to propose the best approach given the dataset. | |||
| 52 | What is the estimated budget for this project, and how is it segmented across the different tasks? | This information is not provided. | |||
| 53 | What is the anticipated start date and execution timeline for the project? | See Timeline in Revised PWS amendment. | |||
| 54 | Can the Government provide incumbent details such as Contractor Name, Contract Value, and Contract Number? | There is no incumbent. This work is new. | |||
| 55 | Are incumbents eligible to re-bid on this task order? | Not applicable, no incumbent. | |||
| 56 | What is the maximum ceiling value of the project? | This information is not provided. |
File details come from the government source that posted it. Updated .