ED-NAG-16-R-0003_-_RFP_Attachments_A-H.pdf

Request for Proposals: ED-NAG-16-R-0003

List of Attachments

Attachment A: Statement of Work

Attachment B: Instructions to Offerors (Technical Proposal and Past Performance)

Attachment C: Quality Assurance Surveillance Plan (QASP)

Attachment D: Instructions to Offerors (Business Proposal)

Attachment E: Sample Budget Submission Format and Invoicing

Attachment F: Technical Evaluation Criteria

Attachment G: Past Performance Form

Attachment H: Small Business Subcontracting Plan

REQUEST FOR PROPOSALS: ED-NAG-16-R-0003

ATTACHMENT A: WORK STATEMENT

DEVELOPING ACHIEVEMENT LEVELS

FOR THE 2017 NATIONAL ASSESSMENT OF EDUCATIONAL

PROGRESS GRADE 4 WRITING ASSESSMENT

March 31, 2016

CONTENTS

1. INTRODUCTION

2. BACKGROUND

2.1 HISTORY OF NAEP

2.3 NATIONAL ASSESSMENT GOVERNING BOARD POLICY ON ACHIEVEMENT LEVELS

2.4 ACHIEVEMENT LEVELS ACTIVITIES 1992 THROUGH 2014, AND 2017 PLANS

2.5 OVERVIEW OF TECHNICAL AND PROCEDURAL ISSUES

2.5.1 Standard Setting Methodology

2.5.2 Prior Standard Setting Methodologies

2.5.3 Use of Feedback during the ALS Process

2.5.4 Training and Instructions for Panelists

2.5.5 Composition, Size, and Timing of Panels

2.5.6 Judgments Using Empirical Data

2.5.7 Describing the Achievement Levels

2.5.8 Policy Guidance for Developing the Achievement Levels

2.6 TECHNICAL DIRECTION OF THE CONTRACT

3. SCOPE OF WORK

3.1 TASK 1: PROJECT MANAGEMENT

3.1.1 Subtask 1.1: Kick-off Meeting

3.1.2 Subtask 1.2: Weekly Meetings with COR – phone calls and/or webinars

3.1.3 Subtask 1.3: Monthly Meetings with COR and NCES staff/contractors

3.1.4 Subtask 1.4: Monthly Progress Reports

3.1.5 Subtask 1.5: Plans for Item/Data Security

3.1.6 Subtask 1.6: Secure Project Workspace

3.2 TASK 2: DEVELOP PLANNING DOCUMENT

3.3 TASK 3: PRODUCE A DESIGN DOCUMENT TO GUIDE THE ACHIEVEMENT LEVELS-

SETTING WORK

3.4 TASK 4: CONDUCT FIELD TRIAL

3.5 TASK 5: CONDUCT PILOT STUDY

3.6 TASK 6: CONDUCT ACHIEVEMENT LEVELS-SETTING PROCESS

3.7 TASK 7: CONDUCT VALIDATION STUDIES

3.8 TASK 8: TECHNICAL ADVISORY COMMITTEE ON STANDARD SETTING

3.9 TASK 9: GOVERNING BOARD AND OTHER NAEP MEETINGS

3.10 TASK 10: DEVELOP FINAL REPORT

4. SCHEDULE OF PROJECT DELIVERABLES

1. INTRODUCTION

The National Assessment Governing Board (hereafter referred to as the Governing Board or the Board) has a requirement to develop and implement a process to produce a set of recommendations to assist the Governing Board in developing achievement levels for the 2017 National Assessment of Educational Progress (NAEP) writing assessment for grade 4, in accordance with the Board’s policies on standard setting. As part of the achievement levels setting process, the contractor shall also design and conduct related validation studies.

The achievement levels shall be established by the Board to define what students should know and be able to do in writing at grade 4. The NAEP writing assessment is scheduled to be administered in 2017. The achievement levels will be used for reporting NAEP results to the American public. In arriving at a policy decision on writing achievement levels, the Governing Board will be informed by the recommendations for the achievement levels produced under this contract.

2. BACKGROUND

2.1 History of NAEP

For over four decades, NAEP has provided information on the achievement and performance of students in the United States. The National Center for Education Statistics (NCES), within the Department of Education, has responsibility for administering NAEP. The Governing Board is responsible for setting policy for NAEP.

NAEP, known as “The Nation’s Report Card,” is the most comprehensive and only continuing, valid source of information on what U.S. students know and can do in various subject areas.

For most assessments, nationally representative samples of students (from about 10,000 to 190,000) are drawn from three grade levels for assessments in various subject areas. In addition to national data, assessments in mathematics, reading, science, and writing1 provide state-level information on school performance based on comparable, representative samples of students at grades 4 and 8. The 2015 Trial Urban District Assessment (TUDA) included 21 large urban districts in the assessments of reading and mathematics.

The resulting data on student knowledge and performance have been accompanied by descriptive contextual information allowing analyses of a variety of student experiences and factors related to student achievement. Tracking student performance has permitted the nation to assess U.S. students’ academic progress over time.

1 The most recent state-level administration of writing occurred in 2007. The NAEP assessment schedule can be found at: https://www.nagb.org/naep/assessment-schedule.html . The 2017 writing assessment will be a national sample of students in grades 4 and 8.

https://www.nagb.org/naep/assessment-schedule.html

2.2 Enabling Legislation for NAEP Achievement Levels

Public Law 107-279— signed on November 5, 2002, is the authorizing legislation for NAEP and the Governing Board. The following are among the specific responsibilities assigned to the Governing Board in the legislation:

Select the subject areas to be assessed;

Develop appropriate student achievement levels;

Develop guidelines for reporting and disseminating results;

Develop standards and procedures for regional and national comparisons.

Section 303(e) states as follows:

a. STUDENT ACHIEVEMENT LEVELS-

(1) ACHIEVEMENT LEVELS-The National Assessment Governing Board shall develop appropriate student achievement levels for each grade or age in each subject area to be tested under assessments authorized under this section, except the trend assessment described in subsection (b)(2)(F).

(2) DETERMINATION OF LEVELS-

A. IN GENERAL- Such levels shall--

(i) be determined by--

(I) identifying the knowledge that can be measured and verified objectively using widely accepted professional assessment standards; and

(II) developing achievement levels that are consistent with relevant widely accepted professional assessment standards and based on the appropriate level of subject matter knowledge for grade levels to be assessed, or the age of the students, as the case may be.

B. NATIONAL CONSENSUS APPROACH-After the determinations described in subparagraph (A), devising a national consensus approach.

C. TRIAL BASIS-The achievement levels shall be used on a trial basis until the Commissioner for Education Statistics determines, as a result of an evaluation under subsection (f), that such levels are reasonable, valid, and informative to the public.

D. STATUS-The Commissioner for Education Statistics and the Assessment Board shall ensure that reports using such levels on a trial basis do so in a manner that makes clear the status of such levels.

E. UPDATES-Such levels shall be updated as appropriate by the National Assessment Governing Board in consultation with the Commissioner for Education Statistics.

The legislation requires that the Board develop achievement levels that are consistent with relevant, widely accepted, professional assessment standards and are based on the appropriate level of subject matter knowledge for grade levels to be assessed.

2.3 National Assessment Governing Board Policy on Achievement Levels

To accomplish its legislated responsibilities, the Board has established a policy entitled Developing Student Performance Levels on the National Assessment of Educational Progress (amended March 4, 19952).

The policy specifies:

The purpose for developing student performance levels on NAEP is to clarify for all readers and users of NAEP data that these are expectations which stipulate what students should know and should be able to do at each grade level and in each content area measured by NAEP.

The policy specifies three achievement levels, as follows:

The levels-setting process shall produce three threshold points for each content area and at each grade level assessed, demarcating entry into three categories:

basic, proficient, and advanced. These levels are defined as:

Proficient This level represents solid academic performance for each grade assessed.

Students reaching this level have demonstrated competency over challenging subject matter, including subject-matter knowledge, application of such knowledge to real world situations, and analytical skills appropriate to the subject matter.

Basic This level denotes partial mastery of prerequisite knowledge and skills that are fundamental for proficient work at each grade.

Advanced This level signifies superior performance beyond proficient.

The policy also specifies:

Developing achievement levels shall be a widely inclusive activity of the Board, utilizing a national consensus approach, and providing for the active participation of teachers, other educators (including curriculum specialists and school administrators at the local and state levels), and non-educators including parents, members of the general public, and specialists in the particular content area;

and:

2 http://nagb.org/content/nagb/assets/documents/policies/developing-student-performance.pdf http://nagb.org/content/nagb/assets/documents/policies/developing-student-performance.pdf

Proposed achievement levels shall be reviewed by a broad constituency, including consumers of NAEP data, including policymakers, professional groups, the states and territories. In carrying out its responsibilities, the Board will ordinarily engage the services of a contractor who will prepare recommendations for the Board’s consideration on the levels, the descriptions, and the exemplar exercises.

The Board’s Achievement Levels policy and the guidelines regarding implementation of the policy are provided in http://www.nagb.org/content/nagb/assets/documents/policies/developing-student-performance.pdf.

2.4 Achievement Levels Activities 1992 through 2014, and 2017 plans

The Governing Board has completed comprehensive processes to set achievement levels in the following NAEP subjects and years:

(a) 1992 assessments in mathematics and reading;

(b) 1994 assessments in U.S. history and geography;

(c) 1996 assessment in science;

(d) 1998 assessments in civics and writing;

(e) 2005 assessment in grade 12 mathematics;

(f) 2006 assessment in grade 12 economics;

(g) 2009 assessment in science;

(h) 2011 assessment in grades 8 and 12 writing; and

(i) 2014 assessment in grade 8 technology and engineering literacy.

Achievement levels established in these subjects have been used in reporting NAEP assessment results in The Nations Report Card (www.nationsreportcard.gov).

The achievement levels-setting procedures to date were accomplished under the direction of the Board through contracts with ACT Inc. (Formerly American College Testing), Measured Progress, and Pearson.3 This work was accomplished with the assistance of a number of technical teams; input from a broad spectrum of educational, assessment, business, and policy groups and the general public; involvement of local, state, and federal individuals and organizations; and evaluations by various technical teams and advisory panels. The achievement levels-setting process was implemented in accordance with the policy framework, definitions, and technical procedures outlined in the Board’s Achievement Levels policy. Additional technical and procedural issues arising from previous levels-setting efforts are presented in the next section.

Student performance results reported by achievement levels are widely used by NAEP stakeholders such as educators and policy makers at national, state, and large urban district levels. The achievement levels provide useful means of understanding and interpreting NAEP scores, and have been accepted as good indicators of student

3 Achievement level reports are available on the Governing Board website at:

http://www.nagb.org/publications/achievement.html http://www.nagb.org/content/nagb/assets/documents/policies/developing-student-performance.pdf http://www.nationsreportcard.gov/ http://www.nagb.org/publications/achievement.html performance. For example, most recently, the Final Report on the Evaluation of NAEP describes the strengths of NAEP achievement levels as follows:

“As a policy decision, achievement levels can be set with consideration of multiple factors that inform the final decision. In education, the primary source of evidence comes from studies that involve educators’ judgments about student performance based on a policy definition…Many of the procedures for setting achievement levels for NAEP are consistent with professional testing standards….The use and interpretation of achievement levels are also components of a program’s validity framework.”4

Recommendation #3 in the final report states:

“NAEP should continue to explore methodologies for setting achievement levels.

Stakeholders continue to use achievement levels as one means of interpreting NAEP results…”

Recommendation #4 states:

“The validity evidence collected by NAEP from internal and procedural sources suggest that the methodology was implemented as intended and that panelists had a positive experience with the process…the internal and procedural evidence supports the validity of the process…”

The final report encouraged NAEP to prioritize gathering external validity evidence that supports the uses and interpretations of its achievement levels.

2.4.1 Plans for Setting the 2017 Grade 4 Writing Achievement Levels

The Board is committed to reporting achievement level data for the subjects and grades tested in the National Assessment. Previous achievement levels activities have provided an excellent foundation upon which to base future levels-setting efforts. The contractor shall consider the findings of these major levels-setting activities, in particular the 2011 writing achievement-levels settings at grades 8 and 12, in the design of achievement levels-setting procedures for the writing assessment at grade 4.

Notwithstanding the benefits of past experience, it should be noted that the 2017 NAEP writing assessment at grade 4 is the first operational administration of a new Writing Framework and it is the first time that the NAEP writing assessment will be administered on tablets with detachable keyboards. The NAEP writing assessment is wholly computerized; students compose and construct their responses using word processing software on a computer5, with the option to use commonly available tools.

4 U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, Evaluation of the National Assessment of Educational Progress, Final Report, Washington DC 2009. Pages 69, 76-77, and 95.

5 The 2017 NAEP Writing assessment will be delivered using customized software on Microsoft Surface Pro 3 or 4 tablets with the MS Windows 10 operating system.

One must also note that the NAEP writing assessment is a fully direct assessment of writing, i.e., the exercises are entirely constructed response and are holistically scored on a multi-point scale. The contractor shall address these assessment characteristics in developing the methods and procedures for the achievement levels-setting (ALS) process.

The contractor shall follow the basic tenets and procedures in the Board’s policy on achievement levels throughout the study.

The Governing Board adopted a new framework beginning with the 2011 writing assessment and was unique in that it was the first wholly computerized NAEP assessment ever implemented. The 2011 NAEP writing assessment was administered at grades 8 and 12; the 2011 writing report card can be found at:

http://www.nationsreportcard.gov/writing_2011/. The NAEP Writing Computer-Based Assessment Tutorial can be found at: https://www.youtube.com/watch?v=NEhS2x2pvB4.

The contract to develop achievement levels for the 2011 NAEP writing assessments at grades 8 and 12 was intended to include achievement levels setting work for the grade 4 assessment that had been planned for 2013. However, the Governing Board postponed the administration of the grade 4 assessment from 2013 to 2017 due to budgetary constraints. Consequently, the achievement levels setting contract for the 2011 NAEP writing assessments at grades 8 and 12 was amended to remove the work for grade 4.

This contract was awarded to Measured Progress; final reports on this work can be found at https://nagb.gov/publications/achievement.html.

The 2017 NAEP Writing Framework and the Test Specifications, adopted by the Governing Board, are available on the Governing Board’s website at https://nagb.gov/publications/frameworks/writing/2017-writing-framework.html.

The framework and test specifications for the 2017 writing assessment focus on developing a comprehensive and challenging assessment. The writing content measured by NAEP was informed by international assessments, state standards and assessments in writing, the craft of writing and technology, and best practices. The NAEP Writing Framework was widely distributed for public review and evaluation, and the comments were very positive. The writing NAEP has included direct writing tasks for students since the early 1980s.

The current framework and test specifications call for expanding the contexts for writing in the 21st century, and they are designed to support the assessment of writing as a purposeful act of thinking and expression used to accomplish many different goals.

Although NAEP cannot assess all contexts for student writing, the results of the 2017 NAEP writing assessment will offer new opportunities to understand students’ ability to make effective choices in relation to a specified purpose and audience for their writing in an “on-demand” writing situation. In addition, the assessment results will provide important information about the role and impact of new technologies on writing in K-12 education.

2.5 Overview of Technical and Procedural Issues

During the process of setting achievement levels for NAEP, several technical and procedural issues (described below) have been raised and discussed by the Board. The http://www.nationsreportcard.gov/writing_2011/ https://www.youtube.com/watch?v=NEhS2x2pvB4 https://nagb.gov/publications/frameworks/writing/2017-writing-framework.html contractor shall address these issues and others as it sees fit, in its technical approach for setting achievement levels on the 2017 NAEP writing assessment at grade 4.

2.5.1 Standard Setting Methodology

During the last 25 years that NAEP has been developing achievement levels, much standard-setting research has been conducted that has expanded the repertoire of methodologies available. Many methods involve ratings of test items and the associated scoring rubrics; several involve ratings of students and student work; yet others involve ratings of score profiles; and a few are classified as compromise methods.6 The contractor’s work shall further the information available to the Governing Board regarding appropriate and valid standard-setting methodologies for a wholly-computerized assessment that consists of exercises entirely constructed response and scored on a multi-point scale. The 2017 grade 4 writing assessment consists of 22 prompts; each student will receive two prompts and will have 30 minutes to respond to each prompt.

Although no specific standard-setting methodology is required, the contractor shall follow these principles in its study design and execution of the deliverables:

(1) be consistent with the 2017 NAEP writing scoring, scaling, and analysis technology;

(2) employ sound statistical procedures for computing cutscores and mapping judgments to the NAEP scale;

(3) maximize the use of available data and minimize the loss of data;

(4) allow participants to consider more complex aspects of NAEP such as rubric-based scoring of constructed responses;

(5) incorporate the use of computers in the ALS process to the extent feasible;

(6) identify all policy decisions in advance, for Board resolution;

(7) be feasible, clear, concise, simple to explain and implement; and

(8) collect evidence to support the reasonableness of the recommended levels, including alignment to existing writing achievement levels at grades 8 and 12.

The proposed methodology shall include ways of enhancing the procedural validity of the process and provide evidence of the external validity of the outcomes.

The contractor shall use sound standard-setting methodologies for the 2017 writing achievement levels-setting (ALS) activities. The methods employed shall have a solid research base and shall exhibit all the technical characteristics necessary to meet the current psychometric standards in the field.

2.5.2 Prior Standard Setting Methodologies

Since 1992 a variety of methodologies has been used to develop the achievement levels

6 An excellent summary of each of these methods can be found in Hambleton, R.K. & Pitoniak, M.J.

(2006). Setting performance standards. In R. Brennan (Ed.), Educational measurement, 4th ed. (pp. 433- 470). Washington DC: American Council on Education and Westport CT: Praeger Publishers.

in various content areas. The Governing Board has extensive experience with the modified Angoff procedure for NAEP achievement levels setting. The final version of the evolution of the methodology for NAEP is represented by the 1998 ALS procedures for civics and writing.

Modifications of the bookmark process were used for setting achievement levels for the 2005 grade 12 mathematics NAEP7, for the 2006 grade 12 economics NAEP, for the 2009 science NAEP, and for the 2014 technology and engineering literacy NAEP.

The Mapmark procedure used for NAEP incorporates item maps to show the location of each item with respect to the score scale, given a mapping criterion of RP=.67 (a .67 probability of correct response). The Board has approved the use of this mapping criterion in previous achievement levels-setting procedures. Because the actual cut score is determined by the RP criterion, the Board continues to remain interested in more information regarding achievement levels-setting panelists’ understanding of this criterion and how it impacts the outcomes of the achievement levels-setting process.

A computerized modified bookmark procedure, using the Computer-Aided Bookmarking (CAB) software8, was implemented for the 2009 Judgmental Standard Setting Studies that were conducted to identify the NAEP grade 12 Reading and Mathematics scores that represented the knowledge and skills needed for entry-level credit-bearing college courses and for job training programs in five selected occupations.

A computerized item mapping procedure, using Microsoft Excel workbooks to display items and record information, was implemented for setting achievement levels for the 2014 grade 8 Technology and Engineering Literacy NAEP.

The 2011 writing assessment in grades 8 and 12 consisted solely of constructed response exercises, scored on a multi-point holistic scale; it was also the first wholly computer-administered NAEP assessment. The Body of Work method was implemented for the 2011 writing assessment ALS, using the Body of Work Technical Integration and Enhancements (BoWTIE) software9. The ALS process was fully computerized.

2.5.3 Use of Feedback during the ALS Process

The Board is interested in informing the judgments of panelists efficiently and effectively through the provision of feedback. The contractor must provide detailed information regarding the feedback to be provided to inform panelists about their judgments. The contractor shall demonstrate plans to inform the panelists about the relationships of their

7 Ibid, see Chapter 2 of the Final Report which provides a detailed review of the 2005 mathematics achievement levels which concluded that the achievement levels were sound and valid for NAEP reporting.

8 As part of the contract closeout activities for the NAEP grade 12 preparedness research project judgmental standard setting (JSS) studies (contract #ED-NAG-10-C-0004), user manuals and technical documentation were developed for Computer-Aided Bookmarking (CAB). The software, user manual, and technical documentation are available on the Governing Board’s website at:

https://www.nagb.org/publications/achievement/software/cab.html.

9 As part of the contract closeout activities for the 2011 NAEP writing achievement levels setting at grades 8 and 12 (contract #ED-NAG-10-C-0003), user manuals and technical documentation were developed for BoWTIE. The software, user manual, and technical documentation are intended are available on the Governing Board’s website at: https://www.nagb.org/publications/achievement/software/bowtie.html.

https://www.nagb.org/publications/achievement/software/cab.html https://www.nagb.org/publications/achievement/software/bowtie.html judgments to student performance and the achievement levels definitions, and provide a rationale for the feedback information, including the timing and frequency of such feedback. The technical proposal shall discuss plans to provide sample feedback, including numeric, graphic, and interactive formats, and provide a rationale for the feedback information, including the timing and frequency of such feedback. Types of feedback shall include, but not be limited to, the following:

(1) p-value data for each item, overall and conditional on each rubric score point;

(2) inter-rater consistency data;

(3) intra-rater consistency data;

(4) impact or consequences data (provided at least once during the standard-setting process and prior to recommendations of final cutscores).

2.5.4 Training and Instructions for Panelists

The largest investment in standard setting is in the training procedures that are used to prepare panelists to make their judgments and provide the Board with well-informed recommendations. It is, therefore, important that the contractor develop training as standardized as possible across the subgroups, if independent groups are proposed. All panelists shall view the same training materials and receive consistent instructions and guidance from the lead trainer. Consequently, plenary sessions for all participants shall be implemented at several points throughout the standard-setting sessions, including, but not limited to, the following:

(1) overview of the entire process and the agenda, specific activities, and time allotted;

(2) context of the standard-setting study, including the purpose of the assessment, how it was developed, content coverage, why writing achievement levels are being developed, role of panelists, and role of Board;

(3) instruction in the 2017 Writing Framework, assessment and item specifications, and scoring rubrics;

(4) instruction in understanding and internalizing the achievement level descriptions and borderline performance;

(5) instruction in the standard-setting methodology being used;

(6) instruction in specific process tasks (e.g., rating tasks) panelists will complete;

(7) instruction in providing feedback;

(8) review of cutscores, cut score variability, and impact data; and

(9) instruction on the use of new information when first introduced for implementation in the process.

The Standards for educational and psychological testing10 emphasize the need for providing thorough training for participants: “Care must be taken to assure that [judges] understand what they are to do and that their judgments are as thoughtful and objective as possible. The process must be such that well-qualified participants can apply their

10 AERA, APA, NCME. (2014). Standards for educational and psychological testing. Washington, DC:

AERA.

knowledge and experience to reach meaningful and relevant judgments that accurately reflect their understandings and intentions” (p. 101).

The contractor shall structure the training process so that the panelists are introduced to the “big” picture in the plenary sessions to provide an overview of the tasks they will be completing. They then must be given more detailed instruction and training in the implementation of the instructions, with ample opportunity for asking questions and discussion with their peers in the breakout groups, under the direction of process and content facilitators. The training modules should build on one another so that there is appropriate repetition and reinforcement throughout the process.

Facilitators provided by the contractor shall include a lead trainer who will assume responsibility for the plenary sessions and ensure standardization in the process. Content and process facilitators will assume responsibility for each group (if independent groups are proposed). Depending on the role of computers in the training proposed, there may be a need for one or more technical/computer facilitators as well. The contractor shall provide credentials of the lead trainer and proposed facilitators to demonstrate their technical and content expertise. Facilitators must be knowledgeable in the subject area and experienced, capable trainers in a large group setting. Presentations shall be prepared and rehearsed prior to each implementation.

The contractor shall provide information and some initial training materials in advance of convening the panelists to maximize efficient and effective use of time for the ALS process. The contractor shall use the most current, innovative and cost-effective methods for training panelists. The contractor may propose the use of interactive web-based training and other innovative modes for delivery of training, along with a clear rationale for the timing and materials to be provided via each mode of training.

2.5.5 Composition, Size, and Timing of Panels

The composition of the achievement levels panels—two-thirds educators and one-third non-educators—has been deemed appropriate by the Board since the beginning of its standard setting initiative in 1992. According to Board policy, the panelists must represent a balance of demographic characteristics such as gender, race/ethnicity, and geographic region. Issues relating to the panel size and timing of the panel meetings will depend on the overall design proposed by the offeror. Sufficient time is essential for the panelists to receive training and perform the ALS tasks appropriately. In addition, the scheduling of meetings should provide for the maximum time possible for all components of the achievement levels process, including Board direction and decision-making, as well as the preparation of reports of the NAEP results in terms of the achievement levels.

2.5.6 Judgments Using Empirical Data

Consistent with the Board’s policy, the achievement levels panelists should be presented with extensive information regarding the assessment. At an appropriate time during the levels-setting activities, the contractor shall present participants with actual student performance on the assessment items and with information regarding performance over all items relative to the proposed cut scores. The rationale for providing panelists with the specific empirical data, the timing of the introduction of such information within the process, and the impact on the panelists’ ratings must be clearly specified and documented. The offeror shall recommend data, both to inform the panelists’ judgments during the levels-setting activities and to inform the Board’s decision regarding the achievement levels.

2.5.7 Describing the Achievement Levels

The Governing Board went through a process of engaging content experts, outside the scope of this procurement, to develop final achievement levels descriptions from the preliminary descriptions for each assessment area that appeared in the 2011 NAEP Writing Framework. The final recommended descriptions were adopted by the Governing Board in 2012 and will be provided to the contractor for use in the ALS process. These descriptions are the fundamental component of NAEP achievement levels, and they shall be central to the judgmental process.

2.5.8 Policy Guidance for Developing the Achievement Levels

As noted earlier, the Governing Board has established policy guidelines and procedures for developing achievement levels. These documents are available at http://www.nagb.org/content/nagb/assets/documents/policies/developing-student-performance.pdf.

In addition to the policy highlights and definitions summarized above, technical proposals submitted in response to the Request for Proposals shall also reflect the entire scope of the Board policy in discussing issues and design features of the writing levels-setting activity for grade 4. The composition of the achievement level panels described above must adhere to the specific requirements stated in the policy guidance.

2.6 Technical Direction of the Contract

The process by which the National Assessment Governing Board reaches decisions on the achievement levels for the writing assessment will be informed by the technical work of this contract. For achievement levels-setting initiatives, the Board has assigned the Committee on Standards, Design and Methodology (COSDAM) the responsibility of monitoring and overseeing the achievement levels-setting process. The performance of this work shall be subject to the direction of the Governing Board Contracting Officer (CO), guided by the Contracting Officer’s Representative’s (COR’s) technical direction.

The ability of the contractor’s staff to communicate effectively and productively with the COR is critical. The COR will closely monitor the contractor’s work to ensure performance of activities as specified in the contract and to guarantee technical quality, policy relevance, and objectivity.

The technical direction shall consist of, but not be limited to the following activities:

Monitoring the contractor’s performance to ensure compliance with the technical requirements of the contract to include the project schedule and deliverable submissions;

Providing direction to the contractor to ensure compliance with appropriate Board policies and specifications as contained in the appendices to this statement of work and as communicated through regularly scheduled reviews of work-in-progress by COSDAM and the Board; and

Recommending to the CO final acceptance or rejection of all deliverables based on the review and disposition of deliverables by COSDAM and the Board.

The contractor shall work closely with the COR in all aspects of the work and communicate by teleconference at least once per week to report on the project status. All contract-related communications or proposed changes in the scope of work will be directed to the Contracting Officer. All deliverables shall be sent to the CO, with a copy to the COR.

It is necessary to ensure that the contractor adheres to quality control, cost controls and meets all contract requirements efficiently and effectively. Attachment C to the RFP provides the requirements for a detailed Quality Assurance Surveillance Plan (QASP) to ensure that the following key objectives are met:

Adhere to project timelines as stated in the Planning Document;

Produce a Design Document that reflects all key requirements in the RFP with concrete plans for implementation and continuous monitoring;

Ensure high quality and accurate deliverable products;

Assure data security and controls; and Adhere to the contract budget through cost controls.

The offeror shall expand or propose additional metrics to the QASP and submit the proposed QASP with the technical proposal. A final QASP shall be submitted to the Governing Board five working days following the kick-off meeting.

3. SCOPE OF WORK

In planning for and setting achievement levels for the 2017 writing assessment, the contractor shall carry out the tasks outlined below and shall be expected to develop and implement creative, effective, and cost-efficient approaches to accomplish the goal of setting achievement levels on the 2017 writing assessment.

3.1 Task 1: Project Management

The project management plan must provide for effective and efficient project control.

The offeror shall demonstrate the ability to manage and support the project overall and on a day-to-day basis. Effective management requires the ability to manage multiple tasks simultaneously, as well as the ability to maintain flexibility in the implementation of tasks requiring time-sensitive data and other necessary resources. The management plan must ensure effective and efficient administration of the project, including appropriate oversight of subcontractors if proposed. Proposed staff, together, must have both managerial and technical expertise, knowledge, and experience needed to successfully carry out the project. Detailed requirements for project management and staffing are provided in the Instructions to Offerors – Technical Proposal and Past Performance (Attachment B).

Project management includes the following subtasks:

3.1.1 Subtask 1.1: Kick-off Meeting

Within 10 working days following the award of the contract, the contractor’s key management staff will meet with the CO, COR and other key Governing Board staff in Washington, D.C. to review key elements of the contract requirements, discuss work plans, and review the proposed project schedule and QASP.

Contract communications and key staff roles and responsibilities will also be discussed. A summary of the kick-off meeting minutes and a final QASP shall be submitted electronically to the CO with a copy to the COR, within one week of the meeting. Feedback shall be provided by the Governing Board within five working days, with the final deliverables submitted no later than five working days after Board feedback.

3.1.2 Subtask 1.2: Weekly Meetings with COR – phone calls and/or webinars

The contractor shall arrange weekly meetings (typically 1 hour) with the COR at a mutually convenient time, to occur by telephone and/or webinar. The contractor shall submit an agenda to the COR no later than one business day prior to each call and shall submit brief written minutes within two business days after each call. The minutes are intended to capture the major discussion and decision points from each call.

3.1.3 Subtask 1.3: Monthly Meetings with COR and NCES staff/contractors

In order to carry out the tasks outlined in this statement of work, the contractor will need to request data and other materials produced as part of NAEP operations via the COR. The contractor is responsible for providing all numerical and other data necessary to calculate the final recommended achievement level cut scores and to identify prompts/responses selected to illustrate each achievement level.

The operational work of the NAEP program is carried out by the National Center for Education Statistics (NCES), via several different contract vehicles. Monthly meetings (via telephone and/or webinar) shall be scheduled for technical reviews of past and future activities and requirements for achievement levels-setting work.

These meetings will include the Project Director, the Governing Board COR, and a representative from the National Center for Education Statistics. Additional staff and contractors may be included at the discretion of the COR and NCES liaison. These meetings will be conducted via teleconference or web-based conferencing.

The contractor shall submit brief written minutes within two business days after each call.

The minutes are intended to capture the major discussion and decision points from each call.

3.1.4 Subtask 1.4: Monthly Progress Reports

The contractor shall submit monthly progress reports, summarizing contractual activities and financial expenditures to the Board for the duration of the project.

Monthly progress reports shall describe contractual activities, including major tasks and accomplishments, problems and suggested solutions, significant activities and events, any decisions that may be needed from the Board, and plans for the next reporting period. These progress reports shall be submitted by the fifteenth day of each month (or on a specified date agreed upon by the CO and the contractor). The CO, through the COR, will respond within five calendar days to each monthly progress report with any technical direction or re-direction that may be warranted in order to carry out Board policy and direction.

In addition to monthly progress reports, the contractor shall provide monthly reports on financial expenditures through invoice submissions. Each invoice shall include a summary of the overall project costs broken down by the ten task areas described in Section 3 Procurement Tasks. This will include task/contract funding; tasks costs by current reporting period; cumulative costs for each task;

and a balance of funds for the overall contract.

Following this summary, the contractor shall provide, by task, a breakdown of labor and other direct costs. Detail for other direct costs shall be summarized by line item description and cost. Budgeted versus actual costs for each line item will be provided. Sufficient detail for each cost, such as lodging and per diem, supplies, etc. shall be provided to substantiate billed costs.

3.1.5 Subtask 1.5: Plans for Item/Data Security

The Governing Board is especially sensitive to the fact that many groups and individuals will have access to various parts of the secure NAEP item pool during the achievement levels-setting process, and they will have information on results prior to the official release of data. The contractor shall ensure that secure materials and data are securely controlled as well as confidentiality maintained by employees at all times. Security is a serious concern and it is the contractor’s responsibility to guarantee security for NAEP materials once in the contractor’s control. It is a felony to disclose confidential NAEP data or materials.

It is the responsibility of the contractor to develop effective data security plans and demonstrate how security procedures will be employed and monitored at all times for the duration of the contract. The contractor shall specifically address security procedures for (1) item distribution, (2) item review, (3) data review, (4) storage of computers/tablets containing secure materials, (5) server security and avoidance of distributed denial of service (DDoS), if applicable, and (6) hotel and other staff security maintenance. The contractor’s data security plans will be incorporated in a Design Document to be produced as specified in Task 3 in the Scope of Work, described in detail in Section 3.3 below. These plans must be approved in writing by the CO before implementation.

NCES requires that any person(s) who will be reviewing or using secure data or materials sign compliance forms during the course of this contract.11 These forms will be supplied by the Governing Board COR for the contractor to complete. The contractor shall forward all compliance forms to the COR, who will forward a copy to NCES.

The contractor will be responsible for storing and shipping NAEP tablets that contain a secure form of the operational writing assessment, in addition to the computers provided by the achievement levels setting contractor to conduct the standard setting process.

3.1.6 Subtask 1.6: Secure Project Workspace

Throughout the project, the contractor will need to share secure documents with the COR, the Technical Advisory Committee on Standard Setting (TACSS), the Board’s Committee on Standards, Design, and Methodology (COSDAM), and other relevant groups and individuals. The contractor shall set up a secure project workspace when such documents need to be shared for project phone calls or meetings.

3.2 Task 2: Develop Planning Document

Based on the schedule of activities discussed at the kick-off meeting and the agreements reached at this initial meeting, the contractor shall prepare and provide to the CO, with a copy to the COR, a Planning Document which, like the technical proposal, further outlines by task, as well as chronologically for the contract period, each activity to be performed under this procurement in a level of detail that allows long-range planning for the Board and Board staff. A Gantt chart or similar chart depicting project schedules and resources shall be included in the Planning Document. This document shall also serve as a monitoring document for keeping the contract on schedule. The Planning Document shall be submitted within thirty calendar days of the award of the contract. Upon request by the COR, the contractor shall provide an updated version of the Planning Document to reflect changes as needed. The contractor shall submit the final Planning Document in electronic format in Microsoft Word as well as in PDF accessible formats.

3.3 Task 3: Produce a Design Document to Guide the Achievement Levels- Setting Work

The offeror is required to design a procedure for implementing a technically sound methodology for setting writing achievement levels at grade 4 that reflects the principles outlined in Section 2.5. The process must conform to the policy framework, definitions, and technical procedures outlined in the Board’s policy on setting achievement levels (see http://www.nagb.org/content/nagb/assets/documents/policies/developing-student-performance.pdf).This policy states that the Board will define three levels of achievement

11 See http://nces.ed.gov/statprog/instruct_access_faq.asp

— basic, proficient, and advanced — for each grade and subject tested by the National Assessment.

The comprehensive design document will provide the foundation for all achievement levels-setting activities. (See the table in Section 4 for a list of key project deliverables.)

In addition to other factors, the design rationale must clearly reflect consideration of the following:

(1) Outcomes and issues related to the activities of the Board regarding the previous establishment of achievement levels, in particular the 2011 Writing achievement levels at grades 8 and 12;

(2) The unique design features of the 2017 Writing assessment at grade 4 (Please see the Writing Framework and Test Specifications at https://www.nagb.org/publications/frameworks.html#writing for details);

(3) The design, implementation, and results of the achievement levels-setting at grade 4 from the 2017 NAEP writing assessment (Please see the writing 2011 achievement levels-setting process and technical reports at https://www.nagb.org/publications/achievement.html for an example); and

(4) Relevant research and advances in the field of standard setting.

The contractor shall make the designs effective in the descriptions of panelists’ tasks; the presentations of training, instructional, and informational materials; the displays of performance results and other feedback; the use of technology to collect, analyze and report panelists’ recommended levels; the collection of procedural, internal, and external validity evidence, and other components of the achievement levels activities. The Governing Board is particularly interested in ways to increase the efficiency and effectiveness of the achievement levels-setting process through the use of computers in the process. However, a distributed design (where panelists participate in the achievement levels setting activities virtually) will not be considered due to concerns about test security.

A draft of the Design Document shall be shared with COSDAM for discussion during the November 17-19, 2016 Governing Board meeting. The contractor shall submit electronically the final Design Document specified in this section to the National Assessment Governing Board through the CO for approval no later than December 16, 2016.

The design document for the writing grade 4 ALS process shall include, but not be limited to, discussions of the following features of the process:

A. Achievement Levels-Setting Methodology – The Governing Board has experience with a variety of methodologies for setting achievement levels on assessments in various content areas including writing, as noted in Section 2.5.2.

The 2017 writing assessment in grade 4 will be wholly computerized and will be administered by tablet with detachable keyboard. It will consist solely of constructed response exercises, scored on a multi-point holistic scale. The contractor shall explore https://www.nagb.org/publications/frameworks.html%23writing https://www.nagb.org/publications/achievement.html various alternative designs for the standard setting process, and recommend a procedure for setting achievement levels for writing at grade 4.

The panelists will need to use tablets to take the NAEP assessment and to understand the process that students experienced. Tablets used in the NAEP writing administration will be made available to the contractor for use by the panelists (to take a sample form of the NAEP assessment) throughout the writing achievement levels-setting process.

Further, the use of computers shall be incorporated into the ALS process to the extent feasible. The contractor may choose to use or adapt the Body of Work Technological Integration and Enhancements (BoWTIE) software package12. If computers are proposed to be used for the achievement levels setting process, detailed specifications and cost estimates shall be provided. If computers are not proposed for use in the achievement levels setting process, the offeror shall describe alternative approaches that will achieve the intended objectives.

The recommended methodology must meet all requirements for NAEP achievement levels setting. The contractor must provide a rationale for this methodology and explain the reasons why this procedure is being recommended. Details of the proposed methodology shall include analysis and quality control procedures. The contractor must review both the pros and cons of various methods that were considered as reasonable alternatives for achievement levels setting on the writing NAEP, along with the contractor’s final recommendation proposed for this work.

B. Field Trial – The contractor shall design a small-scale field trial for the purpose of testing the logistics associated with the use of computers during the ALS process. The field trial panel does not need to be a nationally representative sample of panelists but should consist of individuals with backgrounds similar to the participants of the pilot and operational studies.

C. Pilot Study – The contractor shall design a pilot study to implement the…