Attachment I- Supplemental Instructions to Offerors.pdf

Supplemental Proposal Development Instructions to Offerors

ED-IES-12-R-0021

Throughout each component offerors shall explain how each activity described in the statement of work will be performed in relation to each operational assessment, field test, and special study. In addition, each offeror shall describe in detail the tasks to be carried out and the deliverables to be provided under each activity. Beyond that the statement of work has specified particular things that are listed below that should receive extra attention in preparing the proposal. In most cases, these directions are directly related to evaluation criteria. These more specific directions are presented component by component.

Procurement Overview

The eight components shall be bid as an alliance, and proposals must be in the form of a team bid that covers the eight contracts and the general activities specified in Chapter 9, with offerors working together to submit a unified group of proposals that will be accepted or rejected in their entirety. Further, the first three and the fourth and fifth components are grouped together because NCES believes that a single firm shall bid on each of these components.

The Alliance team shall include separate technical and cost proposals for each of the eight components (chapters 1 – 8) , all activities specified in Chapter 9, and the options identified in Chapter 9. The Alliance team, in each of the eight proposals, shall address the full schedule of activities specified in Exhibit D. Each proposal shall describe the respective offerors relationships with other offerors. Each proposal shall identify the hand-offs to and received from other offerors and specify a schedule and procedures for ensuring the quality and timeliness of the hand-offs. The proposals shall be coordinated so that the government can clearly understand the proposed relationships between and among the offerors. The Alliance team shall present an overview of their plan and shall include a graphic depiction(s) of the various hand-offs and the schedule.

In any configuration of bids, each team proposal shall cover all parts of the PWS and create a team approach that optimizes the strengths of each partner organization.

Because the flow of work across contractors is crucial to the successful implementation of NAEP, the offerors shall ensure that their proposals document how transitions will be accomplished. The clarity and comprehensiveness of the approach to flow across contractors will be an important factor in the selection of a team of contractors.

Each offeror shall be familiar with the elements of all nine chapters in the PWS and their proposals shall be consistent with and complementary to all other proposals. The

Government will look for a fully coordinated effort.

Each offeror shall work to maximize the utilization of small and minority-owned businesses, with the ideal goal of achieving approximately 43.5% of the subcontracted amount award going to small businesses and small business plans shall accompany each component proposal.

As part of their respective overall technical and cost proposal, offerors shall include a separate OPTIONS section of their respective proposals for each of the following major options under consideration—see further details in Chapter 9. Proposals to conduct these options will be evaluated as part of the overall proposal rating for technical quality and cost.

Further, as part of their technical proposals, offerors shall include a “Global

Assumptions” workbook with separate tabs for each year (e.g., 2013, 2014, etc. and the major options). For example, these assumptions would include (by subject and grade), number of items, number of blocks, sample sizes including both schools and students, estimated numbers of SD/ELL students, estimated number of special sessions needed during each NAEP testing window, etc.

Each member must certify that they understand and agree with the global assumptions and have based their technical and cost proposals accordingly.

1 TUDA OPTIONS FOR READING AND MATH

1A 2015 NAEP Reading/Math marginal cost per additional TUDA

1B 2017 NAEP Reading/Math marginal cost per additional TUDA

2 HIGH SCHOOL TRANSCRIPT STUDY OPTIONS

2A 2013 HSTS small-scale pilot study for state-level transcript data collection in

2B 2015 High School Transcript Study--10 states, plus additional cost per state

2C 2015 High School Transcript Study--Math and Science textbook special study

3 US HISTORY, CIVICS, GEOGRAPHY

3A 2014 History, and Civics -- 10-state pilot, plus additional price per state

3B 2014 History, Civics, and Geography -- 10-state pilot, plus additional price per state

4 2015 NATIONAL INDIAN EDUCATION STUDY The National Indian Education Study (NIES) is a two-part study designed to describe the condition of education for American Indian and Alaska Native students in the United States. The study is sponsored by the Office of Indian Education (OIE) and conducted by the National Center for Education Statistics for the U.S. Department of Education.

(see http://nces.ed.gov/nationsreportcard/nies/ for additional details. For budgeting purposes, offerors should assume a design similar to the 2011 study)

5 2015 NAEP ASSESSMENT OPTIONS

5A 2015 Reading and Math 12th grade 15-state pilot, plus additional price per state

5B 2015 Science ICTs and HOTs administered at state level http://www.ed.gov/about/offices/list/oese/oie/index.html http://nces.ed.gov/nationsreportcard/nies/

5C 2015 Science to include 21 TUDAs, plus a per TUDA cost to add Science

6 LINKING STUDIES WITH INTERNATIONAL ASSESSMENTS

6A 2015 NAEP/TIMSS Grade 4 Special Study

6B 2015 NAEP/TIMSS Grade 8 Special Study

6C 2015 NAEP/PISA [statistical moderation and up to 6 state validation]

6D 2015 NAEP/PISA

7 2016 NAEP Arts Assessment [assuming no new Arts Item Development]

8 2017 WRITING ASSESSMENT

8A 2017 Writing at Grade 4 pilot only

8B 2017 Writing at Grade 4 national only [excluding pilot]

8C 2017 Writing at Grade 4 national and state level [excluding pilot]

8D 2017 Writing at Grade 4 national, State, and 10 TUDAs, plus per TUDA cost for up to 15 additional TUDAs

9 SPECIAL STUDY OPTIONS—see NAEP bidders’ page for the 2013 sample design related to special studies

9A 2013 HSLS/NAEP Special Study [administer NAEP to students in HSLS:09 cohort in three states TN, MI, FL; merge data files from HSLS and NAEP:2013]

9B 2013 NAEP Reading -- Special Study--

9C One large, one medium, and one small study [Up to 3 of these options may be exercised during period of performance]

9D One large study [Up to 3 of these options may be exercised during period of performance]up to $1M each

9E One medium study [Up to 3 of these options may be exercised during period of performance]up to $649,000 each

9F One small study [Up to 3 of these options may be exercised during period of performance]up to $99,000 each

9G 2015 NAEP/ECLSK:11 Special Study [ 2011 Kindergarten cohort will be 4th graders in 2015]

9H 2016 NAEP/LTT Special Study—to examine what a specified score difference means; for example, what does 5 points or 10 points mean on the NAEP scale?

In preparing your technical and cost proposals, offerors should consider that a significant amount of work related to the 2013, 2014, 2015, and 2017 assessments has occurred or will occur under the existing contracts by the time of contract award. This includes:

Item Development

2013 Assessments:

Grade 12 2013 Operational Math and Reading cognitive and questionnaires

(student, teacher & school)

Grade 8 TEL 2014 pilot Assessments (cognitive and survey questionnaires (student, teacher & School))

Note: No teacher questionnaires for grade 12, social studies or TEL

2014 Assessments

Grade 4/8 2015 Math and Reading pilot cognitive and questionnaires (student, teacher & school)

Grade 4/8/12 Science paper/pencil pilot, pre-ADC clearance

Grade 4/8/12 Civics, Geography & US History no new work – re-administer the

2010 items.

2017 Assessments

Grade 4 2017 Writing CBA pilot, pre-ADC clearance

Sampling and Data Collection

Contracts for hotel space for training field staff in Nov-December for 2013 assessment--Hotel contracts will transfer to new contractors;

Final 2013 sample design memo --for each assessment, test, and special study including selection of replacement schools, and contingencies for nonparticipating jurisdictions and students.

Schedule of Data Collection Activities--from development of training and outreach materials to administering the assessment for each assessment, test, and special study.

Copies of all materials sent to schools to encourage school participation and the inclusion of all students capable of taking the assessment.

Schedule for receipt and distribution of assessment materials.

Plan for transmitting collected data to MDPS and DAR contractors.

Materials, Distribution, Processing, and Scoring

2013 Printing

Mathematics 4, 8, and 12

Reading 4, 8, and 12

Science 4, 8, and 12

Mathematics pilot 4 and 8

Reading pilot 4 and 8

Any materials needed for 2013 special studies

2013 Ancillaries—including Math manipulatives

1 Alliance Planning and Coordination

1.1 Conduct strategic and operational planning

1.1.1 Conduct strategic planning

Identify and explain an industry-recognized model of strategic planning suitable for accomplishing the strategic planning work outlined in task 1.1.1, and present a rationale for the model identified.

Include a rationale for the model of strategic planning proposed , and show how it is particularly suited to the work outlined in Task 1.1.1.

Present evidence of successful experience identifying and interpreting industry trends and emerging issues, particularly with regard to the large-scale assessment industry, student behavior, and education resources in general.

Outline a process for identifying opportunities and threats that support or detract from efforts to advance NAEP as the leader in large scale assessment, and to promote currency in assessment practices.

Include means for keeping the Associate Commissioner of the Assessment

Division(AD) and NCES managers abreast of developments and issues within the assessment industry, and in student behavior, and education resources in general.

Propose a staffing plan for accomplishing the work outlined in 1.1.1.

Propose an approach to ensuring that the Executive Steering Committee is continually apprised of developments in the broad NAEP context, and that

Committee input is aligned with broad NAEP goals.

1.1.2 Conduct operational planning towards meeting NAEP’s primary goal

Propose a process for coordinating NAEP’s progress towards meeting its primary goal of measuring student progress over time.

Propose a plan for achieving, and evaluating progress towards achieving

NAEP’s performance objectives aimed at meeting the goal of measuring student progress over time, while maintaining the lead in large-scale assessment.

Include targets for gauging progress towards meeting objectives, and shall state clearly how the contractor shall coordinate the work of the Alliance towards achievement of these objectives.

1.1.3 Conduct operational planning towards achievement of NAEP-related organizational performance targets

Propose a process for ensuring achievement of NAEP-related organizational performance targets as defined by the Government Performance and Results

Act of 1993 (GPRA).

1.2 Coordinate NAEP assessment activities across multiple contractors and NCES

1.2.1 Manage the implementation of tasks across multiple-contractors

Propose an assessment coordination plan for the delivery of a high quality, timely assessment process that includes reports, special studies, and research and development activities.

Include a process for ensuring excellent communication among contractors and NCES, and coordination across all phases of the assessments.

Include a process for ensuring that tasks across the Alliance are synchronized, either directly or indirectly.

Delineate a process for interconnecting and coordinating with non-Alliance

NCES contractors as appropriate.

1.2.2 Coordinate planning and review meetings among NAEP contractors and

NCES

Propose a process for coordinating planning and review meetings among

NAEP contractors and NCES.

Address the potential of using technology to substitute for face-to-face meetings where appropriate.

1.2.3 Coordinate timely responses to quick turnaround requests

The offeror shall propose a process for ensuring timely responses to quick turnaround requests.

1.3 Coordinate NAEP R&D

Propose a process for integrating the management of all subtasks of 1.3 with overall planning and coordination activities described in this procurement, to define a framework for conducting NAEP research and development according to NAEP priorities and programmatic goals and objectives.

1.3.1 Assist NCES in developing an R&D agenda

Propose a process for developing the NAEP R&D agenda.

1.3.2 Coordinate NAEP special studies

Propose a process for ensuring that NCES and NAEP contractors are kept abreast of developments in NAEP R&D within and outside the Alliance.

1.3.3 Coordinate development of information collection requests (ICR) for OMB approval

Propose a project plan for NAEP information clearance requests.

Include a schedule of ICR development activities and submittals, based on the current projected assessment cycle.

Include a process for updating the plan on an ongoing basis, to include NAEP studies and development work requiring OMB clearance.

Include a process for integrating schedules for all anticipated interactions with relevant contractors, consultants and NCES that must occur prior to submitting a completed ICR package for OMB approval.

1.3.4 Manage production of technical documentation on the Web

Propose a plan for managing the development of NAEP Web-based Technical

Documentation to ensure that the completed work is on schedule, and of high quality, in keeping with NCES technical standards.

2 Design, Analysis and Reporting

The offeror shall explain how each activity will be performed in relation to each assessment, test, and special study. In addition, the offeror shall explain separately how each of the overarching activities such as technical documentation shall be performed in conjunction with the Planning and Coordination contractor and other Alliance contractors. The tasks to be carried out and the deliverables to be provided under each of these activities should be described in detail.

2.1 Design

Offerors shall demonstrate their understanding of the relevant provisions of the legislation (http://www.ed.gov/policy/elsec/leg/esea02/pg97.html#sec602)

The offeror shall describe an analysis plan with a sample size that meets the

NCES statistical standards for subgroup reporting on important variables, and that takes into consideration design features of the assessment, such as the different types of items used (multiple choice, and short and extended constructed response). The plan should include descriptions of the statistical methodology that will be used to analyze the data. Analysis plans will specify how the data will be analyzed, e.g., whether variables will be combined in an index of a particular indicator such as socioeconomic status or educational resources in the home.

The offeror shall describe how special case assessments for Puerto Rico should be designed, and their capabilities for implementing this assessment.

The offeror shall describe how the assessments will be created in the Spanish language, including translation of replacement items and secure items from previous assessments. Plans for item analysis, scaling, and evaluation of comparability with performance data for other jurisdictions should be addressed. The offeror shall describe how the results of the Puerto Rico assessments shall be integrated with the results for the nation and the states.

The offeror shall describe plans for implementing the operational mathematics assessment for Puerto Rico in 2013 as described in the 2013 design document on the Bidders page, the 2013 field test for the 2015 assessment, and the operational assessment in 2015.

Offerors shall demonstrate the capacity to design development processes for survey instruments (e.g., background questions) that can be closely coordinated with the cognitive assessment design.

The offeror shall describe how computer adaptive testing might be implemented for NAEP.

Offerors shall demonstrate the capacity to design cognitive assessments using the evidence-centered design method and develop scaling procedures for interactive computer tasks in science. [this description should also be included in the ID proposal]

Offerors shall describe how subjects can be spiraled together within testing sessions to allow multiple assessments to be conducted simultaneously. Items shall be configured such that reliable data may be collected on a sample basis and burden on respondents shall be minimized. Offerors shall demonstrate they have the capability to construct a large-scale assessment using matrix-sampling principles.

http://www.ed.gov/policy/elsec/leg/esea02/pg97.html#sec602

The offeror shall demonstrate understanding of the booklet design process and describe their proposed process.

The offeror shall describe their general plan for pilot testing NAEP assessments with details on their proposed plan for coordinating these activities with the ID pilot testing activities.

The offeror shall propose designs for field tests of all assessments, including required sample sizes for the proposed analyses of data. The offeror shall describe the methodology for developing reporting scales and other analyses being proposed.

The offeror shall describe their plans for coordinating with the MDPS and ID contractors on setting overall standards for inter-rater reliability; monitoring scoring for consistency with prior scoring procedures and cross-assessment calibration; and providing feedback to MDPS for revision of scoring rubrics, with special emphasis on the hand-offs between the three contractors.

The offeror shall describe their proposed process for finalizing the development of the final booklets.

2.2 Analyses leading to Reporting Results

The offeror shall propose psychometric analyses that are compatible with previous NAEP methodology and the subject framework, as well as with matrix sampling methods. If offerors propose analytic methods other than those used for assessments conducted between 1990 and 2011, they should demonstrate that it could fulfill the requirement of maintenance of trend.

The offeror shall propose an appropriate approach to interpreting the results of all DIF analyses, and propose an appropriate mechanism for deciding whether questionable items should be eliminated from scaling the assessment. The offeror is encouraged to propose methods for improving item discrimination and more precise detection of bias.

The offeror shall describe the scaling model and methodology they shall be using. Offerors shall describe how they will use item response theory scaling techniques to produce scales consistent with those used in previous NAEP assessments. The offeror shall demonstrate that proposed model fit procedures are appropriate for the scales to be reported, and how they will produce marginal maximum likelihood (MML) estimates or alternative score data from which means, percentages, achievement level categories, standard errors, and significance tests can be computed (the Analysis section of the

NAEP Technical Documentation on the Web, http://nces.ed.gov/nationsreportcard/tdw/, contains a description of the scaling methodology, including computing MML estimates, used in operational assessments). The offeror shall describe how they plan to develop performance scales that will measure general performance, as opposed to item-by-item performance, that take into account the varying difficulty of individual items, and that discriminate between students of different ability levels.

The offeror shall propose a methodology for linking within content scales sufficiently consistent with the approach used in prior assessments so that new scale scores can be transformed and placed on the content scales of earlier assessments to maintain short-term trends wherever possible. For assessments in which new frameworks were introduced (reading and science in 2009, writing in 2011), the offeror shall propose methods for linking results from these assessments to those conducted in prior years to determine whether trends can be continued.

The offeror shall propose a methodology for testing the validity of the cross-grade scale in Main NAEP Reading, and the cross-age scales in Long-term

Trend NAEP.

As part of their technical proposal for the 2016 NAEP assessments, the offerors shall select and describe an appropriate approach to developing linking transformation constants and placing the data from the 2016 assessment on the long-term trend scales.

The offeror shall propose procedures to estimate key population characteristics that provide results consistent with past assessment estimates.

This does not preclude the offeror from suggesting improvements to past procedures.

The offeror shall describe statistical procedures appropriate to estimating variances due to sampling of summary statistics that describe the distributions of all population and subpopulation proficiencies and other survey characteristics. NAEP proficiency estimates are based on test questions administered in many different cognitive assessment booklets each containing an incomplete set of cognitive questions, resulting in many test question responses being missing by design. The offeror shall describe statistical procedures appropriate to estimating the variances due to measurement error of summary statistics that describe the distributions of all population and subpopulation proficiencies. The offeror shall also describe statistical procedures for estimating standard errors of summary statistics that take into account both measurement and sampling errors.

The offeror shall describe a linking study design to relate NAEP to an external assessment such as PISA that is economical and valid.

The offeror shall propose a methodology for summarizing the data to meet the needs of the various assessments and for producing separate data summaries for the subscales in each content area and for the composite scores.

Since new insights into handling complex issues in design and analysis will be sought, the contractor shall staff the assessment development effort with experienced senior staff members who have demonstrated a high level of creativity and innovation in previous work. The offeror shall include resumes of staff indicating sufficient expertise and experience to conduct these types of analyses.

The offeror shall describe their understanding of the topics listed in the statement of work related to assessment research and development. The offeror shall propose two projects for consideration under the Task Order

Component funded under the Planning and Coordination contract and indicate how they would lead to improvement in the NAEP program.

2.3 Reporting Assessment Results

Offerors should propose how they might analyze NAEP 12 th

-grade results to provide measures of readiness for postsecondary education or the world of work.

The offeror shall demonstrate their understanding of the required processes and their capacity to meet the reporting quality and timing requirements. The offeror shall describe in detail their procedures for preparing NAEP reports to meet NAEP’s high quality and timing standards and should include specific strategies for the 2013 reports.

It is essential that the contractor carefully edits and implements strict quality control procedures for all publications and other materials for reporting. The offeror shall describe in the proposal how the quality of drafts and finished products shall be ensured.

The offeror shall demonstrate their understanding of the state and urban district reporting process and their capacity to meet the reporting quality and timing requirements. The offeror shall describe in detail their procedures for preparing NAEP data and report templates to meet NAEP’s quality standards for state reports.

Offerors shall describe the content they propose to place on the NAEP public

Web Site, and how they would either enhance the current NAEP Question

Tool or make a transition to a new question tool.

2.4 General Activities

The offeror shall propose the membership on the DAC either including those on the current DAC or offering new membership of equal or greater expertise and status in the assessment community. The offeror shall describe how they would use the DAC in their work. The offeror is encouraged to propose how the DAC could function efficiently to take advantage of their expertise in advising NAEP on technical matters.

The offeror shall describe their process for acquiring the current tool and their process for coordinating with the Web Contractor and meeting the web standards administered by the Web Contractor. The offeror is encouraged to offer enhancements to the current tool including greater flexibility for analysis, greater use of database technology, and enhanced functionality.

The offeror shall submit the names and resumes of the proposed key staff. The offeror shall describe the staffing plan and provide a guarantee that persons identified as key staff shall work on the project for at least 6 months.

3 Item Development

3.1 Cognitive Item and Scoring Rubric Development

3.1.1 Develop cognitive item development plans

Describe of the underlying principles and methods that will guide the development of high quality cognitive items that are valid and effective measures of content across all levels of cognitive difficulty, aligned with the framework, free of construct irrelevant variance, bias and sensitivity issues, and meet the tenets of universal design.

Include an annotated outline of an item development plan that illustrates understanding of what a NAEP design plan would include. The plan shall incorporate the proposed procedures for collaborating with the DAR, MDPS, and

Web Technology contractors in developing the plan.

Include how the IDTS will be used to conduct an inventory of all available items in the IDTS to complete the task.

Include recent projects similar in scope and complexity where the offeror developed similar planning documents.

3.1.1.1 OPTIONAL TASK: Implementation plan for assessments with new or updated frameworks

Describe the role of the ID contractor and the procedures for contributing to the development of an Alliance implementation plan.

3.1.2 Design and develop cognitive items and scoring rubrics

Describe the underlying design principles and processes for developing cognitive assessment items and rubrics in detail. Include rationales for development processes proposed to ensure that there are a sufficient number of high quality items for subjects after the pilot.

Delineate and describe procedures for ensuring the quality of the items.

Include an annotated outline for designing and developing cognitive items and rubrics that demonstrates your understanding of how items and rubrics should be developed.

Describe cost effective and efficient procedures for developing a large number of high quality and complex items. All software proposed for generating NAEP items shall use open source codes.

Train cognitive item writers

Describe the underlying principles used to guide the training of item writers in developing items that cover a wide range of difficulty, and the procedures for carrying out training. Include examples of the training agenda, qualifications of the item trainers and writers, evaluation and monitoring procedures, training schedule, and item security procedures as part of the proposal. Include recent projects similar in scope and complexity where similar planning documents were developed.

Include recent projects similar in scope and complexity where cognitive item writer training was conducted.

Collaborate with MDPS/SDC and provide scoring support to MDPS

Describe in detail the process that will be used to collaboratively develop scoring rubrics and training materials so that after the scoring of pilot items the MDPS contractor can take full charge of training trainers and scoring.

Describe the processes and standards used for creating training sets.

Include example outlines of scoring guides and training materials (e.g., training handbooks for each assessment and grade) representing knowledge and experience in developing these documents.

Delineate methods for streamlining and improving scorer training, and use of scoring using technologies, including some form of geographically distributed scoring.

Describe the processes that will be used to collaboratively develop Frequently

Asked Questions with the SDC contractor.

Acquire copyright licenses

Describe plans for obtaining new copyright licenses for new passages and images, and tracking, updating and maintaining current copyrights of passages and images.

Describe implications of copyright licenses for use on technology-based tasks/items.

Describe plans for maintaining a comprehensive and current/updated record of copyright permission agreements for passages, graphic images, etc.

Conduct cognitive interviews and/or small-scale pilot testing

Describe the methods for conducting cognitive interviews and/or small-scale pilot testing, e.g., cognitive labs, question tryouts, etc., the rationale for conducting specific types of cognitive interviews, proposed item types for cognitive interviews, recruitment plans.

Include evidence of having successfully conducted cognitive interviews and/or small-scale testing previously.

Conduct bias and sensitivity reviews

Describe underlying principles and processes for ensuring that passages, items, and audio/visual stimuli are free of bias, sensitive, and accessible. Include qualifications of reviewers, the criteria that will be applied, and frequency of performing the reviews.

3.1.3 Configure cognitive items into assessment blocks

Describe the general assumptions and principles that will guide assembly of items into blocks for both pilot and operational booklets.

Describe the principles and processes to be used in determining which items will be included in the operational assessment and the procedures for working with

DAR.

3.1.4 Prepare and certify assessment booklets

Describe procedures for ensuring that the assessment blocks are free of error.

3.1.5 Translate cognitive items into Spanish

Describe translation approaches that assure the quality of the translation.

Describe underlying principles and approaches that will be used to evaluate the equivalence of translated items with the parent English items.

Include descriptions of experiences in translating and developing bilingual assessments.

3.1.6 Post-data collection item quality reviews for both pilot and operational assessment

Describe the procedures and assessment data that will be used to evaluate the quality of the assessment items in the item pool. Include procedures for working with the DAR contractor.

Include a schedule of plans for conducting in-person cognitive item quality review meetings with NCES.

Include a description of a summary report that specifies design, development, psychometric properties, and item statistics.

Determine cognitive items to be released

Describe in detail the rationales, procedures and plans for determining the cognitive items proposed for release. Include a schedule.

Describe in detail the procedures for working with the Web Technology contractor to ensure efficient and timely uploading of the released items in the

NAEP Questions Tool.

3.2 Develop questions for the survey questionnaires

3.2.1 Evaluate existing question pool

Describe approaches for conducting an initial plan of the periodic review of all questionnaires, including core items, subject-specific items and other contextual/policy items over the entire five-year period of the contract.

Delineate the criteria that will be used to evaluate the quality of existing background variables, the types of analyses to be performed, and a review schedule.

3.2.2 Prepare long-range core and subject specific development plans

Include an annotated outline demonstrating knowledge of the processes for developing an overall questionnaire variable plan.

Describe the underlying principles, rationales, and methods that will be employed for developing issues papers. Include procedures for working with Alliance members on the identification of other sources of data.

Coordinate core student data collected from all sources

Describe the procedures and plans that will be used for coordinating core student data collected from all sources as part of the proposal. Include a schedule for coordinating the core student data collected from all sources.

3.2.3 Develop new survey questions

Describe the principles and procedures that will guide the review for developing new or revising existing student core, subject-specific and home/school contextual variables and associated survey questions. Include a schedule for conducting the review of all the survey questions.

Include underlying principles and procedures for developing new survey questions, and implications for transforming paper-pencil questionnaires into technology-based questionnaires.

Conduct cognitive interviews

Describe the methods and underlying principles used for conducting cognitive interviews to develop survey questions, the recruitment plans including a description for ensuring a diversity of the sample of participants.

Describe experiences in conducting cognitive interviews.

3.2.4 Prepare and certify the questionnaires

Include a plan for the procedures that will be used to prepare both paper-pencil and technology-based questionnaires as part of the technical proposal.

3.2.5 Translate background questions

Describe the underlying principles of the approach for translating survey questions

Describe approaches for evaluating the equivalence of translated questions with the parent English questions as part of the technical proposal.

Describe experiences in translating and developing bilingual survey questions.

3.2.6 Post-pilot and -operational questionnaire question quality reviews

Describe the procedures and survey question data that will be used to evaluate the quality of the survey questions in the survey question pool. Include a schedule of plans for conducting question survey quality review meetings with NCES.

Describe the plans for preparing a survey questionnaire technical documentation report to include the types of information that will be included.

3.3 Item/survey question reviews

Conduct cognitive item reviews

Describe a review plan that includes standing committee reviews and accessibility, bias, sensitivity reviews prior to submitting passages, and topics/task concepts.

Include a schedule for conducting the reviews.

Describe how members of standing committee or expert panels will be identified and how item review meetings will be convened.

Describe the underlying principles and procedures for training facilitators to conduct small and large subject-specific review meetings.

Describe effective and cost efficient approaches for conducting reviews of cognitive items/survey questions. Include rationales for proposing alternative item review approaches.

Describe the approaches with rationales for preparing review materials, conducting meetings, recording the meeting comments, and addressing all comments from the subject-area standing committee meetings and state and urban district item reviews.

Describe experiences conducting small and large panels to review cognitive items/survey questions.

Prepare review materials and address comments

Describe approaches for the use of both paper and technology-based review books and procedures for ensuring the quality of the review materials. Include alternative approaches for the producing review materials.

Describe processes for consolidating reviewer comments and submitting responses to NCES. Include methods for recording and documenting all reviewer comments received from the various committees, NCES, and resolutions of the comments.

3.4 Develop and maintain an Item Development Tracking System (IDTS) on the

NAEP web system

Describe how the IDTS will be used in all ID processes described in Tasks 3.1-

3.3. This shall include delineation of the steps for how all the data from the existing system might be integrated into the IDTS, rationales for the steps and a schedule of development activities.

Describe collaboration with the DAR/MDPS/Web Technology contractor to develop the IDTS. Include discussion of how existing systems may be integrated into the IDTS or how the IDTS may be integrated into the existing systems.

Describe how the ID contractor will work with the Web Technology and MDPS contractors to maintain the integrity of the ITDS.

Include recent projects where an item banking system was developed or collaborated with another contractor to develop one.

4 Sampling and Data Collection

4.1 Design and Select School Samples

The offeror shall propose a sampling plan for each national and state assessment, pilot/field test and special study including a plan and schedule for the long-term trend (i.e. the fall, winter and spring) data collections. The offeror shall propose sampling plans which:

Randomly select individual students, rather than intact classes for assessment sessions in schools [For situations where all 4 th grade students are sampled for a NAEP administration that spirals the cognitive subject matter across students, students may remain in their intact classes.];

Wherever resources permit, conduct national-only assessments in as many different sampled schools as is feasible to reduce design effects due to clustered sampling;

Wherever possible, minimize the joint probability of selection of the same schools in adjacent assessment cycles; and

Specify the optimal sample design for combining state samples into a nationally representative sample for NCLB required and non-required subject area assessments

The offeror shall propose contingency plans for generating a nationally representative sample if participation problems, which threaten the representativeness of the combined sample, occur at the state level. Similarly, in addition to their sampling plan for the mandatory state assessments in reading and mathematics, the offeror shall propose contingency plans for generating representative state samples if some school districts that do not participate in Title I elect not to participate in their state’s assessments. The offeror should also propose a contingency plan for the prompt selection of a replacement schools for any school declining to participate.

Offerors planning to use a sampling frame other than the CCD for public schools shall provide a detailed discussion of the frame’s benefits to the

NAEP program. As part of their proposal, the offeror shall describe the sampling frame they would use for selecting private schools.

The offeror shall demonstrate their understanding of potential threats to maintaining NAEP trends and should propose sampling strategies that may ameliorate the effects of these potentially threats.

The offeror is strongly encouraged to suggest enhancements or improvements to existing practice that would improve the efficiency and effectiveness of

NAEP. Proposals to reduce the burden on schools, districts, and states are especially encouraged, as are proposals that suggest new uses of technology to improve quality, efficiency, and cost savings.

The offeror shall propose new technology to streamline the electronic submission of school rosters. In addition, the offeror shall propose sampling plans that randomly select individual students, rather than intact classes for assessment sessions in schools in most cases.

4.2 Collect Data

The offeror shall propose a plan to secure the participation of public and private schools selected for all NAEP operational assessments, pilot/field tests, and special studies. The offeror’s plan for securing public school participation should include coordination of contractor recruitment efforts with those of the NAEP State Coordinator and should describe the training and support services that will be made available through the State Service

Center. The offeror shall propose a data collection schedule including, but not limited to, contacting states, districts and public and private schools;

contacting principals and requesting updated lists of students; and conducting each assessment. The offeror’s plan should address differences in approach for the mandated state assessments in reading and mathematics, and for the other NAEP assessments and activities. The offeror’s plan shall describe what kind of outreach activities will be carried out and what type of materials will be developed to encourage school and student participation and inform school staff about NAEP. The plan should include a timeframe for carrying out activities and describe how materials will be disseminated.

The offeror shall develop a plan for working with the NAEP State

Coordinators and participating schools to ensure parental notification letters are distributed and for documenting parental refusals.

The offeror shall describe strategies for working with NAEP State

Coordinator to identify nonparticipating schools and the process for selecting substitute school for nonparticipating schools, when appropriate. The plan should include proactive strategies for increasing participation at grade 12 and private schools and converting non-responding or nonparticipating private schools.

The offeror shall describe the process of selecting and preparing all field staff from recruitment, conducting security checks to ensure that no one with a questionable personal background, which may place students at risk, is sent into America’s schools representing NAEP, and employment of adequate and appropriate staff. The offeror shall provide a detailed description of their plan for deploying field staff across the country and throughout the various assessment cycles. In addition, offeror shall describe a staffing plan that minimizes the amount of time spent in schools and maximizes the number of field staff available to administer regular and accommodation sessions at the same time.

The offeror shall describe the training for the data collection staff including development of materials, uses of technology with an emphasis on distance learning strategies and should propose improvements to existing procedures.

The offer shall present a plan for ensuring school staff are adequately prepared to administer the assessment without requiring school staff to attend in-person training. The proposal should also describe the plan for monitoring teachers.

The offeror shall propose a general schedule of activities associated with each assessment cycle to illustrate key activities (e.g. material development, training, outreach activities) for the entire contract period.

The offeror shall describe a plan for administering the assessments from securing and managing of assessment materials to administering the assessment within the schools to conducting quality control checks prior to returning the materials to the MDPS contractor. The offeror shall describe data collection procedures that are consistent with the data collection activities of the current NAEP including hands-on tasks as well as address the need to collect data in the future using a computer-based platform to present interactive computer task as reflected in 2009 science framework and online writing task specified in the 2011 writing assessment framework. NCES would consider options that could provide efficiencies and/or burden reduction while maintaining quality and timeliness. In order to maintain trend, the assessments shall be administered using the same procedures as in the

2005 and 2006 NAEP administration unless newly recommended procedures can be shown to improve the assessment while maintaining the validity and reliability of trend lines bridging back to previous assessments. The offeror shall develop procedures for providing the accommodations for students with disabilities or limited English proficiency that were identified in collaboration with the cognizant school staff and maintain a log of all activities related to the provision of accommodations. The offeror shall develop procedures for ensuring that the data remain secure throughout the data collection process.

The procedures shall cover all project personnel and all project-related activities. The offeror’s proposal should include a plan for meeting the

Department’s security requirements and NAEP’s strict confidentiality requirements. The offeror shall describe the process for receiving and processing complaints and substantive comments.

For the purposes of this proposal, the offeror shall assume that the participation rate will be 90 percent for grades 4, 8, and 12 th grade assessments. The offeror shall describe plans for determining if make-up sessions are required and strategies for gaining cooperation of students to participate in make-up sessions especially at 12 th grade.

The offeror shall describe procedures for transmitting completed test booklets and questionnaires to MDPS contractor.

4.3 Generate Sampling Weights

The offeror shall describe the process for the development of sampling weights for all appropriate entities in all operational assessments, pilot/field tests, and special studies. The plan should describe method used to meet 6 month and 12-month reporting requirements including quality control checks.

The offeror shall describe a plan for working with DAR to ensure sampling error estimates are appropriate.

4.4 Conduct High School Transcript Study

The offeror shall describe a plan for conducting the transcript study that includes the following elements: securing participation of schools in the transcript study; developing templates for collecting student transcripts;

collecting relevant school-specific information, such as classroom instructional materials, course catalogs and graduation requirements;

scheduling the collection of transcripts; and providing for data security. The offeror shall propose procedures consistent with those used for the 2000 and http://nces.ed.gov/nationsreportcard/about/naep_admin.asp http://nces.ed.gov/nationsreportcard/about/naep_admin.asp

2005 HSTS

; however, offerors are encouraged to propose improved data collection and analysis procedures whenever possible.

The offeror shall propose a methodology for summarizing the data and schedule for development of the HSTS reports.

The offeror shall describe the procedures for linking the HSTS and NAEP data files, conducting statistical analyses, and preparing reports.

The offeror shall demonstrate their understanding of the required processes and their capacity to meeting the reporting quality and timing requirements.

The offeror shall describe in detail their procedures for preparing the HSTS reports to meet NCES’s high quality and timing standards. The offeror shall describe how the quality of the drafts and finished reports shall be ensured.

The offeror shall demonstrate their understanding of the required processes and their capacity to meet the reporting quality and timing requirements. The offeror shall describe in detail their procedures for preparing HSTS reports to meet NCES’s high quality and timing standards and should include specific strategies for the 2009 report.

The offeror shall describe in the proposal how the quality of drafts and finished products shall be ensured. The quality control procedures shall ensure that drafts and final reports are free of typographical errors, data in text and tables are correct, and that tables and charts conform to the standards evidenced in recent NAEP publications. NCES publication standards shall serve as a guide, but are subject to alteration in the Report Cards to achieve certain design considerations.

The offeror shall describe the process for coordinating with the DAR and web contractor to populate and maintain this tool and to meet web standards administered by the Web Contractor.

5 The NAEP State Service Center

The offeror shall provide details for participation in the continuing activities of the State

Service Center. The offeror shall provide evidence that the transition from the current contractor will be made seamlessly and with no disruption in service.

5.1 Orientation and Training

The offeror shall provide for one full-time equivalent TUDA Coordinator on the Center staff to serve as a liaison with the TUDA Coordinators and to coordinate and support their activities.

http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2004455;

http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007467 http://nces.ed.gov/statprog/2002/std1_2.asp http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2004455 http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2007467 http://nces.ed.gov/statprog/2002/std1_2.asp

The offeror is encouraged to consider various electronic media to support and sustain communications, support, and training. NCES is particularly interested in exploring the use of IPODs or similar technology as a means of making recorded podcasts available to Coordinators.

In designing the four types of training described in the SOW, the offeror shall propose multiple training strategies with a rich menu of training approaches and a proposed course of study from the overlap year (i.e., October 1, 2007—

September 30, 2008) thru July 2012. This may include video content, as well.

The offeror shall provide evidence of their ability to provide individualized instruction and support for the range of expertise that can be expected from

State Coordinators and TUDA Coordinators. A one-size-fits-all approach will be considered non-responsive. The offeror shall provide examples of similar training programs conducted with state education staff.

The offeror shall describe their vision for the training component of the

Center's work and for meeting the mission described above and a plan for achieving the vision. The offeror shall provide evidence of their capacity to conduct the training and examples of previous training…