Appendix_A_MPR_TechProposal-Redacted.pdf

Promoting College‐ and Career‐Ready Standards in Adult Basic Education

TECHNICAL PROPOSAL

In response to ED-VAE-11-R-0043

OFFEROR

MPR Associates, Inc.

2150 Shattuck Avenue, Suite 800 Berkeley, CA 94704 Phone: 510-849-4942 Fax: 510-849-0794

AUTHORIZED NEGOTIATOR

Susan Choy, President schoy@mprinc.com

SIGNATURE

EXTENT OF AGREEMENT

MPR Associates, Inc. stipulates that it will comply with all of the terms, conditions, and provisions included in the Government’s solicitation. MPR Associates, Inc. fur-ther agrees that it will provide the services offered in its proposal in accordance with the pricing structure submitted in its proposal.

May 19, 2011

This proposal includes data that shall not be disclosed outside the Government and shall not be duplicated, used, or disclosed—in whole or in part—for any purpose other than to evaluate this proposal or quotation. If, however, a contract is awarded to this offeror as a result of—or in connection with—the submission of this data, the Gov-ernment shall have the right to duplicate, use, or disclose the data to the extent pro-vided in the resulting contract. This restriction does not limit the Government’s right to use information contained in this data if it is obtained from another source without restriction. The entire proposal is subject to this restriction.

APPENDIX A

MPR TECHNICAL PROPOSAL

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION ii

1. Contents 1.Contents ............................................................................................................ii

2. Introduction Authors: P. Joyce, K. Chernus, MPR

3. Technical Approach Task 1: Project Management

Authors: P. Joyce, MPR Task 2: Validate the Common Core State Standards for Adult Basic Education

Authors: S. Pimentel, S. Pimentel, Inc.

Task 3: Review the Alignment of the CCSS and the NRS Core Outcome Measures for Educational Gain

Authors: L. Condelli, AIR Task 4: Update and Assess the Resources in the AECSW Website

Authors: S. Pimentel, S. Pimentel, Inc.; R. Perkins, Design Perspec-tives; K. Chernus, MPR

Task 5: Convene Sessions at the National Meetings of Directors of Adult Education to Promote College- and Career-Ready Standards in ABE

Authors: S. Pimentel, S. Pimentel, Inc.

4. Management Plan Authors: P. Joyce, K. Chernus, MPR

Management Approach Staff Attrition Plan Experience of Proposed Key Personnel

5. Organizational Capacity/Experience/Past Performance Authors: K. Chernus, MPR; S. Pimentel, S. Pimentel, Inc.; R. Perkins, Design Perspectives; L. Condelli, AIR; L. Delaney, Windwalker

6. Small Business Participation Authors: P. Joyce, MPR

Appendix A. Resumes

Appendix B. Subject Matter Experts Letters of Commitment

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 1

2. Introduction

Considerable attention has focused recently on the role of content standards in improving the quality of adult education. Adult basic education (ABE) content standards can form a framework for aligning curriculum, instruction, and professional development to help students improve their basic skills. The Office of Vocational and Adult Education (OVAE), U.S. Department of Educa-tion (ED), has supported standards-based reform over the past eight years by offering key re-sources for state administrators and policymakers, local program directors, and instructors. These include an online standards repository, the Adult Education Content Standards Warehouse

(AECSW); a guide for implementing a standards-based system; and a series of professional de-velopment materials and training sessions for the Standards-in-Action (SIA) initiative. The Pro-moting College- and Career-Ready Standards in Adult Basic Education project will expand this work to include a focus on college and career readiness by helping states prepare to adopt or align ABE content standards with the knowledge and skills needed by adults to enter postsecon-dary education and careers.

MPR Associates, Inc. (MPR) has joined with the American Institutes for Research (AIR), and three small businesses—Design Perspectives (DP), Susan Pimentel, Inc., and Windwalker

Corporation (WWC)—to form a team with all the knowledge and expertise to successfully meet the goals and requirements of the Promoting College- and Career-Ready Standards project. We will work collaboratively with ED in support of their efforts to encourage states to develop and use standards specifying the knowledge that adult students need to enroll in credit-bearing col-lege courses and job training. Resulting products, including reports describing the implications of aligning the CCSS with adult education instruction and the NRS measures and updated AECSW, will provide states with the necessary tools for helping students prepare for careers and college.

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 2

As this proposal will show, our team offers:

1) Successful past participation in numerous OVAE standards-based initiatives, including the

AECSW (AIR and Susan Pimentel, Inc.), Support for State-Level Adult Education Content Stan-dards (MPR and DP), SIA (MPR and Susan Pimentel, Inc.), and National Diffusion of the SIA

Innovations (NDSIAI) (MPR and Susan Pimentel, Inc.).

2) Extensive content expertise in key areas related to this proiect, including: ABE, ASE, and

ELL programs and research; standards-based reform in K-12 and adult education; adult educa-tion content standards; Common Core State Standards (CCSS) for English language arts and lite-racy (ELA/Literacy) and mathematics; sustainability processes; the NRS; standards validation and alignment processes; transitions to postsecondary education and the workforce; designing and delivering professional development to adult educators; materials development and produc-tion; website design, development, and evaluation.

3) Many years of experience successfully managing large-scale projects for OVAE and nu-merous other offices in ED.

4) Letters of commitment from more than 25 potential reviewers (see Appendix B) with rele-vant knowledge in key areas, including: CCSS development; CCSS implementation in adult edu-cation; Equipped for the Future (EFF) national standards; state and local implementation of SIA processes and materials; NRS implementation; professional development for ABE/ASE staff;

literacy, mathematics, and ELL instruction for adults; and LINCS regional centers and resource collection.

5) Experience partnering on other ED-funded projects.

6) Demonstrated ability to successfully organize and manage the logistics for large meetings, including arranging travel and accommodations for large numbers of out-of-town participants.

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 3

3. Technical Approach

MPR, AIR, DP, Susan Pimentel, Inc., and WWC will work in partnership with ED to achieve the objectives of Promoting College- and Career-Ready Standards in Adult Basic Education.

MPR will lead Tasks 1 and 4 and contribute to all of the tasks; Susan Pimentel, Inc. will lead

Tasks 2, 4.a, and 5; AIR will manage Task 3; DP will lead Task 4.b; and WWC will handle meeting logistics and travel for the expert reviewers in Tasks 2 and 3.

Task 1: Project Management

MPR will manage the project and communicate regularly with ED staff to ensure that project work progresses on schedule and within budget and meets ED’s expectations for quality.

Task 1.a. Participate in Post‐Award Meeting

Within 5 business days of contract award, MPR will meet with ED staff to discuss contract requirements and project goals, tasks, deliverables, activities, timeline, and other pertinent issues at a location and time determined by ED. The meeting will be conducted according to an agenda developed by ED. Within 2 business days of the meeting, MPR will submit a written summary of the meeting and a revised schedule of deliverables with calendar dates. Within 2 business days of receiving feedback from ED, MPR will make requested revisions and submit a final summary.

Task 1.b. Confirm Plans for Tasks 2 and 3

Within 5 business days after the post-award meeting, MPR’s project director and Task 2 and

3 directors will meet with ED to address questions, clarify expectations, and determine needed refinements to the validity and alignment plans. ED will create the agenda for the meeting. After the meeting, MPR will develop schedule(s) of milestones that include additional materials to be developed to accomplish the work of Tasks 2 and 3, with sufficient time included for ED to re-view deliverables. Within 5 business days of the meeting, MPR will submit a summary of the

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 4

meeting and the milestones schedule(s). Within 3 business days of receiving feedback from ED, MPR will make requested revisions and submit a final summary and schedule(s).

Subtask 1.c. Conduct Biweekly Status Meetings

MPR’s project director and other key staff will meet biweekly with ED via teleconference to share project status updates. Within 2 weeks of the kick-off meeting, MPR and ED will schedule a regular day and time for these 90-minute meetings. MPR and ED will develop meeting agen-das. MPR will send the agendas to meeting participants within 1 day of the teleconferences.

Within 2 days after each call, MPR will submit a written summary to ED. MPR will revise meet-ing summaries as necessary, based on ED’s feedback.

Subtask 1.d. Provide Quarterly Briefings on Tasks 2 and 3

During years 1 and 2, MPR will conduct 8 quarterly status briefings on Tasks 2 and 3 by te-leconference for ED staff. Within 10 business days of each briefing, MPR will submit a 1-page briefing document with pertinent information for the meeting. Within 3 days of receiving ED’s feedback, MPR will revise and submit final briefing documents.

Task 2: Validate the Common Core State Standards for Adult Basic Education

The CCSS Initiative was a state-led effort coordinated by the National Governors Associa-tion Center for Best Practices and the Council of Chief State School Officers to develop educa-tional standards for ELA and math for voluntary state adoption. Standards were developed in collaboration with teachers, school administrators, and content experts, to provide a clear, consis-tent framework for preparing students for college and work. To date, 44 states have adopted the

CCSS and continue to work on common assessments and other implementation efforts.

The purpose of validating the CCSS for ABE is to aid ED’s efforts to raise awareness and understanding of the critical skills and knowledge expected by colleges, universities, and em-

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 5

ployers and to help ABE, ASE, and ELL programs better prepare adults for success in postse-condary education and the workforce. Our proposed validity review will answer the following questions about adult education and the CCSS: 1) Do the CCSS target appropriate content in the areas of ELA/Literacy and mathematics? 2) Which CCSS are most important for ABE? 3) To what extent are ABE programs likely to be able to implement the CCSS?

We propose to build on the process used to develop the CCSS, adapting it to adult students.

The process will focus on identifying the content adults need for college and career readiness and omit content that, while desirable within the context and longer timeframe of K-12 education, is not essential for adult learners. This process will be completed within 7 months of the contract start date.

Key steps in the proposed validation process are described below and include: identifying re-viewers; synthesizing information about ABE and college and career readiness skills; facilitating in-person and electronic discussions with review teams about the relevance of the CCSS for adult education; validating review team findings with key stakeholders; and synthesizing results of the validation process in a final report.

Preparatory Work. Experts with knowledge of college and career readiness skills will be identified from adult education, postsecondary education, and workforce development and play a critical role in determining the applicability of the CCSS to ABE. Within 5 weeks of the contract start date, MPR will submit a draft list of possible reviewers, including their names, affiliations, and relevant experience.

To ensure broad representation, account for any reviewers’ potential unavailability, and al-low for ED’s input, MPR will submit at least 27 names, assuming that the final review team will include 20 participants (10 for ELA/Literacy and 10 for mathematics). MPR will identify poten-

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 6

tial reviewers from the following categories: adult education policy makers; state and local adult education program administrators; adult education instructors; and ELA/Literacy and math con-tent experts. We also will seek to include former adult learners who transitioned to postsecon-dary training or work, faculty from developmental education and freshmen courses, and employ-ers. This broad representation will ensure that the standards are validated and accepted by key stakeholders at national, state, and local levels. Based on ED’s feedback, MPR will finalize the list of recommended reviewers by September 28, 2011. A list of potential experts is included on

p. 20.

Information on college and career readiness and its implications for adult education will be collected and synthesized to prepare for reviewer discussions. This information will be presented in an accessible format and may be drawn from: selected developmental and credit-bearing freshmen college course syllabi; academic research; college admissions and placement tests; and content from the new GED test, for example. Where possible, information will be summarized in side-by-side charts showing its relationship to specific CCSS content. The results of this analysis will be provided to review team members before the first meeting, so that in-person discussions can focus on areas of alignment with the CCSS.

Convening Expert Reviewers. MPR will convene two teams of selected reviewers (one fo-cused on ELA and one on mathematics) in 2 two-day face-to-face meetings and 3 virtual work sessions per review team. The ELA team will be led by Susan Pimentel with assistance from

MPR; the math team will be led by Dr. Francis (Skip) Fennell, the L. Stanley Bowlsbey Profes-sor of Education and Graduate and Professional Studies at McDaniel College and CCSS math work team member (Dr. Fennell will be assisted by CCSS math consultant Ms. Melanie Alkire).

Ms. Pimentel, in collaboration with Dr. Fennell, MPR, and ED will develop agendas for each

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 7

meeting. All meetings will be conducted within 6 months of the contract start, with the precise timeline to be determined with ED. Proposed content for the meetings for each team includes:

• Meeting 1. The goals of the first meetings are to discuss the implications of the information collected in Step 2 and how it aligns with the CCSS. At this meeting, team members will assess each CCSS standard for its relevance to adult education and the degree to which the standards could be taught in ABE programs. Ms. Pimentel will develop materials to help guide this review, including a rubric for coding the general applicability of a standard to adult education and its specific relevance to the GED exam, postsecondary education, and employment, with space for reviewers to describe their rationale.

• Meeting 2. The goals of the second meetings—held several months after the first meet-ings—are to review the decisions made in Meeting 1 and identify any important adult edu-cation content missing from the CCSS. The teams also will discuss the implications of their findings for ABE teaching and learning, including recommendations for guidance to sup-port state efforts in the adoption of college and career ready standards across ABE systems.

The meetings will be held at ED’s offices in Washington, DC. WWC will handle all meeting logistics, including travel and per diem costs for participants according to amounts listed in the

Federal Travel Regulations.

Following each face-to-face meeting, Ms. Pimentel and MPR will synthesize additional in-formation, as requested by the teams, and present drafts of the teams’ decisions and recommen-dations for review and refinement. Conference calls will be held with teams between meetings to verify and refine the contents of the draft documents. Project staff will document all key content decisions made at each meeting (face-to-face and virtual) and supporting evidence for inclusion in the final report. By the end of these meetings, project teams will have agreed on initial areas of

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 8

alignment with the CCSS, identified gaps or missing content, and highlighted implications for

ABE instruction, as summarized in the draft documents that will feed into the task report.

We anticipate that the most challenging aspect of the project will be to connect the outcomes of Task 2 with Task 3. To address this challenge, Dr. Larry Condelli, Task 3 director, will partic-ipate in the validation review meetings in Task 2 and Ms. Pimentel will participate in the Task 3

CCSS-NRS alignment activities.

Sharing Results. Project staff will coordinate a process for sharing results from the teams’ discussions with a larger group of stakeholders to help ensure field-ownership of the teams’ work. For this step, team members will share draft documents with their colleagues, such as their peers in other jurisdictions or in their programs (e.g., other state or program directors and instruc-tors or college faculty), and collect additional feedback to inform team discussions. Susan Pi-mentel will collaborate with ED to develop materials to guide this review. In addition, three

CCSS authors will be asked to review the draft work of the team to ensure that key concepts of the CCSS K-12 standards are represented (Drs. Coleman, McCallum, and Zimba have agreed to serve in this capacity).

Analyzing Findings. Ms. Pimentel, with support from MPR, will compile and summarize findings on the validity of the CCSS for ABE, including results of team discussions and feedback from the field. Specifically, they will describe the content of highest importance for ABE, as identified by review teams, and highlight any content omitted from the CCSS in preparation for writing the task report. Ms. Pimentel and the project director will present initial findings during a quarterly briefing, as described in Task 1.

Validation Report. The task report will include a description of the validation review process, as well as key content decisions made by the teams and evidence supporting those deci-

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 9

sions. It also will outline implications for ABE in the future. In particular, key messages will be developed to help educate the field and promote adoption of career- and college-ready standards.

To develop the report, MPR will draft an outline based on discussions with ED during a biweek-ly status meeting and submit the outline, with the meeting summary, within 8 months of the con-tract award date. MPR will finalize the outline based on ED’s feedback within 5 business days of receiving ED’s comments. MPR will submit a draft report within 10 months of the contract award date and a final report, based on ED’s review of the draft report, within 12 months of the contract award date.

Task 3: Review the Alignment of the CCSS and the NRS Core Outcome Measures for Educational Gain

Content standards define expectations for what learners should know, thereby offering guid-ance on instruction, professional development, and assessment. The work of Task 3 addresses the latter by evaluating the degree to which skills acquired by learners through content-based in-struction can be assessed according to the NRS framework. This alignment of the CCSS with measures of educational gain, as described by NRS educational functioning levels (EFL), is criti-cal to the successful implementation of the standards because it demonstrates to states, programs, and instructors that outcomes match assessments and align with accountability requirements.

Further, when teaching, outcomes, and assessments are aligned, teachers will obtain feedback on their students that can be used to improve instruction.

Dr. Larry Condelli of AIR, project director of the NRS support project, will lead this task and bring his extensive knowledge of the EFLs and NRS requirements. He will be assisted by other

NRS project staff and Dr. Terry Salinger of AIR, an expert in content standards alignment.

Alignment Model. Over the last 15 years, researchers have developed several models for aligning standards, frameworks, and assessments. The most widely used model, appropriate for

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 10

this task, was developed by Norman Webb (1997).1 His model uses standardized procedures and criteria for conducting alignment analysis studies through the use of qualitative expert judg-ments, quantified coding, and analysis of standards and assessments. The result of the analysis is a set of quantitative and qualitative measures for each standard showing the degree of alignment between content and assessments. Although the model was developed primarily for aligning tests to standards, its procedures can be adapted for aligning the NRS EFLs and the CCSS.

Webb offers 4 criteria for judging alignment between content standards and assessments or assessment frameworks, in this case the EFLs: (1) categorical congruence, the extent to which the standards and assessments address the same content; (2) depth of knowledge consistency, the extent to which the standards and assessment framework require the same complexity of know-ledge, e.g., whether what is expected from learners in an assessment is as cognitively demanding as what is stated in the standards; (3) range of knowledge correspondence, the extent to which the breadth of knowledge represented by the standards is comparable to what is required in as-sessment; and (4) balance of representation, whether the knowledge represented in both stan-dards and assessment is equally mapped across all areas. Webb also describes 24 subtasks to conduct an alignment study, including preparatory work (tasks 1-4), work done during an expert panel meeting (tasks 5 to 22), and post-meeting analysis and reporting activities (tasks 23 and

24). The project team will adapt this approach to conduct the alignment process for the CCSS and NRS educational gain outcomes.

Preparatory Work. First steps in the process are selecting expert reviewers, confirming a date and location for the meeting, and preparing background analyses and materials for review-ers. The PWS specifies 8 reviewers participating in the 2 review panels. Webb notes, however, 1 Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education (Research Monograph No. 8). Madison, WI: National Institute for Science Education, University of Wisconsin–Madison.

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 11

that 5 panel members are optimal. MPR and AIR will discuss the panel composition with ED, but plan to recruit 8-10 experts across the two panels for ELA and mathematics (4-5 for each panel). MPR will nominate experts with knowledge of the CCSS, adult education content stan-dards, the NRS, and the two content areas. We expect candidate experts to include state adult education staff with experience developing content standards, state NRS staff, local adult educa-tion program directors, teachers, and adult education researchers in reading, ELL, and mathemat-ics. Project staff also will consider community college staff and staff from employment programs that work with adult education and have experience with NRS accountability requirements. Sug-gestions for possible experts are included on p. 20. MPR will submit a draft list of prospective panelists 5 weeks after contract award and a final list by September 28, 2011.

The work completed under this task will support the work of Task 2. For this reason, signifi-cant work will not begin until Task 2 is near completion (6-7 months after contract award), espe-cially because there may be some overlap between reviewers for Tasks 2 and 3. Project staff will provide Task 3 experts with all relevant documents developed under Task 2 to assist them in preparing for the alignment process. AIR will compile materials and prepare a preliminary analy-sis of the NRS EFL framework and content standards before the meeting. Materials may include:

the NRS Guidelines on assessment and core outcomes; EFL tables; summaries of NRS-approved

ABE assessments; summary of NRS assessment policy requirements; and descriptions of the

CCSS. Project staff will develop the preliminary analysis of the EFLs by creating two separate summaries, one combining ABE reading and writing and functional descriptors and another dis-playing numeracy/mathematics descriptors. Each set of descriptors will be organized by level, highlighting the relevant skills in each set of descriptors. The relevant CCSS identified in Task 2 and other CCSS, as appropriate, with the descriptors will be arrayed side-by-side. These docu-

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 12

ments will prepare panelists for the review and allow them to focus on the alignment task at the review meeting.

AIR will develop a coding protocol for panelists to assist in the alignment review. For each of Webb’s 4 criteria, codes will be created for reviewers to indicate whether concepts are in con-gruence, or coincide exactly; in alignment, indicating that concepts are in agreement; in contra-diction, indicating that standards and NRS EFLs contain conflicting text; or showing a gap in content, indicating that one text lacks information contained in the other. We will develop addi-tional codes as necessary after contract award.

Conducting the Review Meeting. AIR, with support from MPR, will hold separate 2-day meetings to conduct the alignment study after the 9th month following contract award (one meet-ing on ELA and the other on math). Using Webb’s suggested process, the meeting will begin with an overview of the alignment process. Staff will explain the Webb model and review NRS educational gain indicators and assessments, the CCSS, relevant alignment work done under

Task 2, and the coding scheme. The remainder of the meeting will consist of 4 activities: (1) in-dependent work, the main activity of the meeting, where panelists code the documents indepen-dently. As they finish each NRS level, meeting facilitators will collect the work and (2) analyze results and determine congruence among panelists. These analyses provide the basis for (3) ad-judication of work through discussion, when panel members discuss their coding and reach con-sensus. The adjudication also provides an opportunity to determine whether retraining is needed.

At the completion of each major activity and at the conclusion of the meeting on the second day, panelists (4) debrief and provide feedback on the process.

The feedback process consists of a debriefing questionnaire and discussion designed to elicit more detailed information about the alignment. This discussion is essential for addressing uncer-

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 13

tainties and unresolved issues. Topics may include whether the most important elements of as-sessment for the CCSS have been identified (and if not, what is missing), a general overall eval-uation of the alignment, how fully the NRS and CCSS align in each area, and other general comments.

The meetings will be held at ED’s offices in Washington, DC. WWC will handle all meeting logistics, including travel and per diem costs for participants according to the amounts listed in the Federal Travel Regulations.

Alignment Report. AIR will complete the NRS alignment review within 13 months after contract award. Project staff will prepare a report summarizing the alignment process, after first providing an oral briefing to ED as part of a project status meeting, submitting a draft outline in month 15, and a final outline within 10 days of receiving ED’s comments.

MPR will submit the draft final report in month 18 and revised final report in month 21 after receiving ED’s feedback. The report will discuss the preliminary work done for the reviewers and describe the alignment review process. For each NRS level in each content area (ELA and mathematics), the report will present results of the reviewers’ coding, specifically identifying areas of congruence, alignment, contradiction, and gaps. The report will discuss the implications of the findings for the NRS EFLs and outcome measures and suggest changes needed to improve alignment. It also will discuss findings as they relate to Task 2 and present recommendations for next steps for the CCSS and the NRS to promote successful implementation of the standards.

Task 4: Update and Assess the Resources in the AECSW Website

MPR and our partners Susan Pimentel, Inc., and Design Perspectives will update the SIA sec-tion of the AECSW website and assess the value, relevance, and placement of the warehouse and its resources.

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 14

Task 4.a. Update the SIA Section of the Warehouse

A key product of the NDSIAI project (2009-2011) is a set of sustainability materials de-signed to support statewide implementation of adult education content standards. Developed by

MPR in collaboration with Susan Pimentel, Inc., these materials address 4 stages of sustainabili-ty: 1) assessing the context; 2) mobilizing stakeholder support; 3) conducting infrastructure and policy analysis; and 4) making changes operational.

Using a process similar to that used in developing Standards-in-Action: Innovations for

Standards-Based Education, MPR and Ms. Pimentel will work with ED to transform these sepa-rate documents into a coherent guide for state adult education programs. The guide will: 1) de-scribe the importance of addressing sustainability at the beginning of any education reform in-itiative; 2) explain the background and purpose of the sustainability materials; 3) provide an overview of the guide contents; 4) detail each sustainability stage and its components; and 5) of-fer step-by-step instructions on implementing each stage. It also will include research on sustai-nability and lessons learned from the field.

MPR will submit a draft outline of the guide within 4 months after the project begins; a final outline will be submitted within 8 days of receiving ED’s comments. To ensure that the outline is consistent with ED’s vision, project staff will discuss the proposed approach to the guide during a biweekly status meeting. MPR will submit the draft guide within 7 months and a final guide responsive to ED’s feedback within 9 months of the contract start date.

MPR also will design a revised SIA main page to accommodate both the new sustainability materials as well as the SIA materials that are currently on the AECSW website. Within 8 months of the contract start date, MPR will deliver this design in an idea paper that includes conceptual drawings and a narrative that provides a rationale for the design choices as well as development

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 15

considerations. MPR will revise the conceptual drawings based on ED's feedback and consult with the website developer to implement the new design.

Task 4.b. Plan and Conduct an Assessment of the Warehouse Resources

DP, in collaboration with MPR, will conduct an assessment of the AECSW to determine its future utility by collecting data on the value, relevance, and placement of the warehouse and its resources. Ron Perkins, DP principal and owner, conducted a usability evaluation of the AECSW for ED’s Support for State-Level Adult Education Content Standards project. MPR will convene content users and Web design and development experts to conduct the following assessment ac-tivities: 1) Usability tests and resource reviews by state and local ABE program administrators who use content standards, standards experts, instructors with experience implementing stan-dards, and professional development staff, through usability tests, resource reviews, and facili-tated group discussions about the usefulness, visibility, and organization of resources on the site;

and 2) An assessment of Website interface by Web designers, Web content developers, and

LINCS and ED staff to consider the interface between the AECSW and the consolidated LINCS

Resource Collection and ED agency Websites. Feedback from these activities will be analyzed and described in a report summarizing outcomes of the assessment activities.

MPR and DP will submit a draft operational plan within 25 months of the contract start date.

The plan will describe the assessment activities, approaches, materials, expected outcomes, and proposed list of participants (some suggestions are included in the list of prospective experts on

p. 20). The plan also will include a timeline specifying the sequence of activities, materials de-velopment, ED review schedule, and contractor completion dates. MPR and DP will discuss de-velopment of the draft plan with ED during a biweekly status meeting. After receiving ED’s feedback, MPR will submit the final operational plan 27 months after the start of the contract.

The following outlines some initial ideas to be incorporated into the operational plan.

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 16

Usability Tests and Resource Reviews. MPR and ED will identify 9 adult educators to partic-ipate in the Website and resource assessment activities (another 6 Web experts will be recruited to participate in the interface review along with ED staff; 30 percent more reviewers than needed will be nominated). Examples of possible nominees are included on p. 20.

The first assessment activity will be a formative usability testing procedure to evaluate user ability to access and retrieve content and the value of information to the user. Visibility of the

Website also will be assessed. This testing procedure is well known and documented in the hu-man factors literature.2 Research shows that usability problems can be identified with a relatively small number of test participants. Small samples of behavior, independently observed, yield reli-able data on usability and understanding for software and Websites.3 Over 3 consecutive days, 3 members of each user group per day will participate in one-on-one interviews with a DP test moderator who will lead them through a series of predefined tasks (to be described in the opera-tional plan). Participants’ performance on these tasks will be observed by Mr. Perkins and pro-vide data on the quality, format, and utility of the website and help identify obsolete or missing resources, such as those related to college- and career-readiness standards. Held at MPR’s offices in Washington D.C., the test sessions will be observed by Mr. Perkins from a separate room us-ing a closed-circuit TV system.

Each session will be recorded using a picture-in-picture format that includes a close-up of the computer screen and an insert of the participant’s face, with audio. During interviews, partici-pants will identify 1-2 resources according to their interests for evaluation. Following the inter-views, participants will conduct structured reviews of the resources using a rubric to guide their

2 This general approach to Usability Engineering is thoroughly documented in Deborah Mayhew (1999), The Usability Engineer-ing Lifecycle, Morgan Kaufmann Publishers.

3 Robert A. Virzi (1990), Proceedings of the Human Factors Society 34th Annual Meeting, v.1 pp. 291-294; and Joseph S. Du-mas and Janice C. Redish (1993), A Practical Guide to Usability Testing , p. 412, Norwood, NJ: Able Publishing Co.

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 17

assessment of the quality and usefulness of the resource. MPR developed reviewer guidelines and rubrics for similar resource reviews for the National Institute for Literacy’s Technical Assis-tance for Establishing Peer Review Panel and the U.S. Department of Homeland Security Office of Citizenship’s U.S. Civics and Citizenship Online Resource Center. For these projects, review-ers rated publications and online resources on a variety of criteria including: appropriateness of the topic for the target audience; clarity of writing and accessibility to intended audiences; quali-ty of evidence; connections between evidence and conclusions; accuracy of findings; relevance to existing policy priorities; and representation of current research and issues.

At the end of each day, after interviews and resource reviews have been completed with all user group members, participants will take part in a facilitated group discussion about what con-tent, if any, should be added to the website, especially college- and career-ready standards. Po-tential questions to elicit the types of resources and the rationale for adding the resources will be included in the operational plan.

Website Interface Assessment. During a 1-day session, a working group of Web designers and developers will convene to review the AECSW and provide input on the interface between the AECSW and consolidated LINCS Resource Collection and the ED Website and the interface among the resources included on the 3 sites. Before this session, project staff will need details on how data is uploaded and saved in all 3 Website databases and access to key technical personnel.

Using this information and content reviewers’ feedback, the working group will determine the feasibility and technical processes required to consolidate the resources and Websites and con-sider design options for making the content easy to upload/update and seamless from the end-user’s point of view.

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 18

Every effort will be made to schedule the Website interface assessment session during the same week as the usability and resource reviews to minimize staff travel costs. MPR will handle the logistics and cover travel and per diem costs for participants attending the face-to-face ses-sions. Based on prior experience conducting similar evaluations, MPR does not anticipate that

OMB clearance will be required for the proposed Website and resource assessment activities.

DP will produce a report based on the analysis of the data gathered through these activities. It will include recommendations about the future use of the AECSW and the value, relevance, and placement of its resources. Within 32 months of the contract start date, MPR and DP will discuss the contents, structure, and length of the draft report with ED during a biweekly meeting. MPR will submit the draft report within 34 months of the contract start date. After receiving ED’s feedback, MPR will submit the final report within 36 months of the contract start date.

Task 5: Convene Sessions at the National Meetings of Directors of Adult Education to Promote College‐ and Career‐Ready Standards in ABE

The project team will provide opportunities for states to learn about college- and career-ready standards and outcomes of Tasks 2, 3, and 4.a by conducting concurrent sessions and a mini-institute at the national meetings of State Directors of Adult Education in 2012, 2013, and 2014.

While project staff will work with ED to determine format and content, initial ideas for these sessions include:

2012: Understanding College‐ and Career‐Ready Standards for ABE. MPR, Susan Pi-mentel, Inc., and AIR will offer a 90-minute session (repeated twice) to introduce state staff to college- and career-ready standards for ABE and the project activities used to va-lidate the CCSS for ABE and align them with the NRS EFLs. During the session, the project team, led by Ms. Pimentel, will engage participants in a discussion about the

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 19

priority content and proficiencies contained within the CCSS ELA/Literacy and mathe-matics standards and how they compare to ABE program content.

2013: An Approach to Developing College‐ and Career‐Ready Standards. This mini-institute will address ways to design and implement standards, beginning with developing processes to validate the CCSS standards for use in participants’ programs. Using an ap-proach similar to the Task 2 review process, participants will learn how to design a process to identify relevant college- and career-readiness content that can serve as the foundation on which to develop content in ABE, ASE, and ELL programs.

2014: Implementing College‐ and Career‐Ready Standards in ABE. A second mini-institute will focus on implementing the CCSS, with particular attention to how to bring about the necessary shifts in practice. This institute will outline a strategy to focus on a few key action steps that can have a positive impact on student preparation for college and careers.

The project team will develop print and electronic materials to support state implementation of college- and career-ready standards. The project director, task director, and other key staff will meet with ED via teleconference 12 weeks before each state directors meeting to discuss the agenda, activities, and materials for the session(s). MPR will submit a meeting summary to ED 4 business days after each teleconference. MPR will submit a concept paper describing the purpose of the session and intended outcomes for each session 9 weeks before each session. After receiv-ing ED’s feedback, project staff will revise the papers as necessary. MPR will submit final pa-pers within 4 business days of receiving ED’s comments.

Prospective Reviewers. In collaboration with ED, MPR will identify ABE, ELL, CCSS, EFF, employer, students, Web, and other experts for Tasks 2, 3, and 4 through our networks and

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 20

research. The following potential nominees have indicated their interest in participating (letters of support are included in Appendix B).

Name Position/Title Relevant Expertise Melanie Alkire Consultant, Alkire Consulting and Design CCSS, ABE standards alignment Kaye Beall Co-Director, LINCS Region I Regional Resource

Center, WEI

LINCS

Meryl Becker- Prozocki

Senior Associate, Professional Development and In-structional Support, KY

SIA, ABE math standards imple-mentation, CCSS

Margaret Bowles State Director, MT SIA, ABE math standards imple-mentation, K-12 content standards development

Gayle Box Associate, Professional Development and Instruction-al Support, KY

SIA, ABE math standards imple-mentation, CCSS, AECSW, NRS

Miriam Burt Manager, Adult ELL Projects, Center for Applied Linguistics (CAL)

ELL, assessment alignment with NRS, workplace readiness

Lynne Cage Professional Development Coordinator, GA SIA, Reading standards implemen-tation, NRS, transitions

Debby Cargill Lead for ESOL/ Program Development, Prince Wil-liam County Public Schools, VA

SIA, ELL standards implementa-tion, state standards

David Coleman President, Student Achievement Partners CCSS development Tanya Conover EL Civics Grant Coordinator/ PW Workforce Center

Liaison/ ELL Instructor, Prince William County Pub-lic Schools, VA

SIA, Adult and K-12 ELL stan-dards implementation

Bobby Creech State Regional Area Coordinator, GA SIA, Reading standards implemen-tation, curriculum development

Cris Crowley Director, Adult Education, Madison Community Col-lege, Kentucky

SIA, ABE math standards imple-mentation, NRS, transitions

Donna Curry Training Specialist, CLS SIA, ABE math standards, CCSS alignment to EFF, NRS

Francis (Skip) Fennell

Professor, Education Department, McDaniel College CCSS, K-12 Math

Diane Gardner Assistant Director, CLS EFF standards, ABE math and workforce preparation, NRS

William McCallum Head, Department of Mathematics, University of Ari-zona/Achieve

CCSS development

Peggy McGuire Senior Research Associate and Training Specialist, CLS

EFF Standards, ABE standards-based education, NRS

Pam Meader Instructor, Math Department Chair, Portland Adult Education, ME

ABE math standards and instruc-tion

Sarah Moore Adult ELL Specialist, CAL ELL standards, instruction, and assessment

Andy Nash Professional Development Specialist, WEI SIA, EFF standards, standards de-velopment, CCSS

Tamika Phillips Coordinator of Instruction/Instructor, Muscogee County Adult Education, GA

ABE reading instruction, standards implementation

Beth Ponder Director, LINCS Regional Resource Center; Coordi-nator, LINCS Region II Resource Collections, CLS

LINCS, NRS

Carey Reid Curriculum and Assessment Specialist, WEI ABE teacher preparation, curricu-lum and assessment, NRS

Raye Nell Spillman Senior Coordinator, Work Ready U—Adult Education and GED Services, LA

SIA, ABE standards development and implementation

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 21

Name Position/Title Relevant Expertise Reecie Stagnolia State Director of Adult Education, KY ABE math standards implementa-tion, CCSS Randall Stamper Director, Office of Adult Education and Literacy, VA SIA, GED and ELL standards im-plementation John Strucker Project Manager and Senior Researcher, WEI ABE standards, assessment, read-ing, ELL Byrdeen Warwood Coordinator, Adult and Community Education, Bozeman Public Schools, MT SIA, ABE math standards imple-mentation, CCSS, transitions

Cynthia Zaftt Senior Advisor, National College Transition Network, WEI

CCSS, EFF, college-and career-readiness, transitions

Jason Zimba Professor of Mathematics and Physics, Bennington College/Co-Founder, Student Achievement Partners

CCSS development

4. Management Plan

Management Approach. Peter Joyce, director of MPR’s Washington, DC Office, will serve as project director, with overall responsibility for planning, management, and quality con-trol. He will work collaboratively with ED and MPR’s partners to ensure that all work conforms to ED’s expectations and is completed in a timely manner. Dr. Joyce will be MPR’s primary liai-son with ED, conduct biweekly status meetings, provide quarterly briefings, and communicate with ED between meetings by telephone and e-mail. He also will coordinate the work of task and sub-task directors, subcontractors, and consultants. To ensure the quality of all deliverables, Dr.

Joyce will conduct an internal review process that includes the project staff and lead editor.

To monitor performance, the project director will track the progress of work, deliverables, expenditures, and labor hours at the contract, task, and subtask levels and compare them against the planned schedule, budget, and staff labor allocations.4

Staff Attrition Plan. While MPR does not anticipate any changes in staff before project completion, should staff attrition occur, MPR will take the following steps to ensure successful

4 To ensure accurate cost reporting, MPR uses the DELTEK Government Contractor’s Accounting and Management System and has extensive experience using this software to administer federal contracts.

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 22

completion of project tasks: identify staff with appropriate knowledge and expertise and brief new staff on project history, processes, expectations, timeline, progress, Tasks, and deliverables.

The Staff Loading Chart (table 1) contains the list of Tasks, key personnel, and staff hours by

Task. Table 2 contains staff time commitments to Federal and Non-Federal projects. The Project

Management Chart (table 3) includes Tasks, key staff, start and end dates, other benchmarks and milestones, deliverables, and due dates.

Experience of Proposed Key Personnel. The personnel described below have the neces-sary knowledge, experience, and skills to carry out project work successfully. For additional in-formation, see their resumes in Appendix A.

Peter Joyce (Ed.D. Harvard University), director of MPR’s Washington DC office, will serve as project director, with overall responsibility for planning, management, and quality con-trol. He brings more than 30 years of leadership and project management experience in education and workforce development in both public and private sectors. Dr. Joyce joined MPR in January

2011, after 11 years at Cisco Systems. At Cisco, he designed and supported the implementation of the Professional Career Accelerator, a career development program for workers that includes job task analyses, pre/post assessments of technical and other job skills, individualized career development plans, and customized training. He also played a lead role in developing the Cisco

Networking Academy Program, an e-learning initiative designed to help youth and adults devel-op the technical, problem-solving, and teamwork skills necessary to secure careers in IT and networking.

Kathy Chernus (M.F.A., University of Maryland), director of adult education at MPR, will serve as principal investigator and provide senior-level support to the project. Ms. Chernus over-sees MPR’s adult education work and currently manages ED’s Strengthening America’s

PROMOTING COLLEGE‐ AND CAAREER‐READY STA

MPR TEC

NDARDS IN ADULT

CHNICAL PROPOSA

T BASIC EDUCATIO

AL

ON 23

PROMOTING COLLEGE‐ AND CAAREER‐READY STA

MPR TEC

NDARDS IN ADULT

CHNICAL PROPOSA

T BASIC EDUCATIO

AL

ON 24

PROMOTING COLLEGE‐ AND CAREER‐READY STANDARDS IN ADULT BASIC EDUCATION 25

Competitiveness Through Adult Math Instruction (the Adult Numeracy Instruction [ANI]) initia-tive. She convened a group of math and adult numeracy experts to determine the applicability to adult education of the National Mathematics Advisory Panel’s (NMAP) recommendations for K-

12 math…