16-233-SOL-00310.pdf

Solicitation 16-233-SOL-00310 Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native Hawaiian Programs (OAA Title VI)

1 SECTION 1 - SERVICES AND PRICES

1.1 BRIEF DESCRIPTION OF SERVICES

The purpose of this solicitation is to award a contract to conduct an evaluation of the Title VI program to determine the outcomes of the program for tribal elders, tribal communities, and staff implementing the programs under Title VI grants.

1.2 TYPE OF ORDER

In consideration of complete and timely delivery of the supplies described in Section 2, the Contractor shall be compensated on a Time and Materials basis.

1.3 SEVERABILITY OF SERVICES

The services acquired under this contract are non-severable services. The services specified in each contract line item (CLIN) have been determined to be no-severable services - a specific undertaking or entire job with a defined end product of value to the Government.

1.4 CEILING PRICE

The hours and total price per labor category are for budgeting purposes only. The number of hours actually used for each category may exceed or be less than the estimate by category as long as the ceiling price for the contract/order period is not exceeded. The ceiling for each contract/order period is identified in Section 1.5. The Government shall not be obligated to pay any amount in excess of this ceiling price nor is the Contractor obligated to continue performance if to do so would exceed this ceiling price. In the event that the Government chooses to exercise optional contract/order periods, unused funds from a previous contract/order period may not be used to fund any services taking place in a subsequent contract/order period. The ceiling price may be modified by the Government if it is determined that the estimated hours are insufficient for the work required.

1.5 CONSIDERATION AND PAYMENT (T&M)

In consideration of satisfactory performance of the work as described throughout this contract/order, the Contractor shall be paid in accordance with the fixed loaded hourly rates as shown herein for the specified categories of labor based on the number of actual hours incurred in the performance of the work specified. Fully loaded rates shall be all inclusive of direct labor, fringe benefits, applicable indirect costs, and any profit for each labor category. Additionally, the contractor shall be paid for materials and travel directly chargeable to this contract/order. The base period is as follows:

and Native Hawaiian Programs (OAA Title VI)

Base Period (54 Months)

Labor Category Number of Hours

Loaded Hourly Rate Total $

Materials (item description) Quantity Rate Total $

Travel Total $

Total Labor, Materials, & Travel

2 SECTION 2 - DESCRIPTION/SPECIFICATIONS

2.1 BACKGROUND

2.1.1 The Department of Health and Human Services Administration for Community Living Strategic Priorities

The overall vision of the Administration for Community Living (ACL) is that all people, regardless of age and disability, live with dignity, make their own choices, and participate fully in society. To achieve this ACL, works to develop a comprehensive, coordinated and cost-effective system of long-term care that helps older adults and individuals with disabilities to maintain their dignity in their homes and communities. As part of the Administration for Community Living, the Administration on Aging (AoA) is the principal agency of the U.S Department of Health and Human Services designated to carry out the provisions of the Older Americans Act of 1965 (OAA). The strategic priorities of AoA are to empower older people and their families to make informed decisions about, and easily access, health and long-term care options; enable seniors to remain in their own homes through the provision of home and community-based services. Central to these priorities is the pursuit of consistent and effective approaches to support older adults in their own homes and communities, and to coordinate the provision of supportive services to seniors and their caregivers in an integrated system of long-term care. These themes resonate with the DHHS’s dedication to encouraging the development of healthy and supportive families and communities and to promoting economic independence and social well-being across the lifespan.

2.1.2 ACL and Program Evaluation

Consistent with requirements of the Government Performance Results Modernization Act (GPRMA), ACL/AoA integrates its strategic priorities and plans with performance measurement criteria. ACL/AoA has three major performance measures: improve program efficiency, improve client outcomes, and improve effective targeting of vulnerable elders. However, simply measuring performance at a macro level does not provide the level of detail required to understand emerging trends, systems issues and program innovation at the community level. Through program evaluations, ACL/AoA seeks a better understanding of key programs, such as the programs under Title VI of the Older Americans Act for American Indians, Alaska Natives, and Native Hawaiians.

ACL is interested in documenting the value of the Title VI programs for individuals, families, communities and Tribes/Tribal Organizations. This contract is for the conduct of an evaluation of the Title VI program to determine the extent to which it is achieving the objectives of the Title VI program as stated in the OAA and grantee guidance materials.

2.2 TECHNICAL HISTORY

Title VI of the Older Americans Act (OAA) establishes grants to Native Americans for nutrition services, supportive services, and family caregiver support services. The purpose of Title VI is “to promote the delivery of supportive services, including nutrition services, to American Indians, Alaskan Natives, and Native Hawaiians that are comparable to services provided under Title III.” (42 U.S.C. 3057) In addition, Congress stated that “older individuals who are Indians, older individuals who are Alaskan Natives, and older individuals who are Native Hawaiians are a vital resource entitled to all benefits and services available and that such services and benefits should be provided in a manner that preserves and restores their respective dignity, self-respect, and cultural identities.” (42 U.S.C. 3057a). Title VI of the Older Americans Act is comprised of three sections: Part A is the Indian Program; Part B is the Native Hawaiian Program; and Part C is the Native American Caregiver Support Program. Each section identifies the eligibility criteria for the section and the specific services that are authorized by that section. After meeting program requirements, Tribal organizations have flexibility to allocate resources among the various activities funded by each program. Tribes may also decide the age at which a member is considered an elder and, thus, eligible for services.

http://www.aoa.acl.gov/AoA_Programs/OAA/index.aspx and Native Hawaiian Programs (OAA Title VI)

The authorizing legislation for the data collection is found in Title II of the OAA. The requirements stipulated under section 206(a, c) directs ACL/AoA to “…measure and evaluate the impact of all programs authorized by this Act, their effectiveness in achieving stated goals in general, and in relation to their cost, their impact on related programs, their effectiveness in targeting for services under this Act unserved older individuals with greatest economic need (including low-income minority individuals and older individuals residing in rural areas) and unserved older individuals with greatest social need (including low-income minority individuals and older individuals residing in rural areas), and their structure and mechanisms for delivery of services, including, where appropriate, comparisons with appropriate control groups composed of persons who have not participated in such programs. Evaluations shall be conducted by persons not immediately involved in the administration of the program or project evaluated.”

In FY 2014 3-year grants for Nutrition and Supportive Services were awarded to 263 Tribes and Tribal organizations (representing over 400 Tribes) and one organization serving Native Hawaiian elders and 231 3-year grants for the Native American Caregiver Support Program. These programs represent an important part of the communities’ comprehensive services.

To obtain more information about the service provided to Native elders and their caregivers, visit the AGing Integrated Database (AGID), an on-line query system based on AoA-related data files and surveys, which also includes population data from the Census Bureau for comparison purposes.

Contractual History: This contract builds on an Evaluability Assessment conducted under a previous contract. That contract resulted in a logic model (Figure 1) and medicine wheel model (Figure 2) that define the categories of program inputs, outputs, and outcomes that should be included in the evaluation conducted under this contract. The Evaluability Assessment process also resulted in the formation of an Advisory Committee that continues to be used by ACL and should have a substantive role in the evaluation conducted under this contract. Further, the basic approach recommended as a result of the Evaluability Assessment, and defined as a participatory evaluation framework, is the approach that should be used for the evaluation conducted under this contract. The report generated under the Evaluability Assessment contract is available at: http://www.aoa.acl.gov/Program_Results/docs/EA-of- TitleVI-v2.pdf

2.3 SCOPE OF WORK

The overarching purpose of this solicitation is to contract for a comprehensive evaluation of the OAA Title VI program that examines the effects of the program on:

1. Older Indians, their families and caregivers

2. Tribal communities

3. Intergenerational connections in Tribal communities

4. Management of the Title VI program

Prospective contractors should be familiar with the variety of issues that promote and inhibit the delivery of supportive services to American Indians, Alaska Natives, and Native Hawaiians in a manner that preserves and restores their respective dignity, self-respect, and cultural identities. In addition, prospective contractors should be aware of, and have experience with the varied cultures and traditions among Native Americans. Researchers studying racial and ethnic minority populations are often faced with unique challenges such as language barriers and cultural sensitivities. Native American tribes and communities are distinguished from other ethnic and cultural groups in part by the magnitude of historical trauma that they have faced and in part by the diversity across tribal communities (Fisher and Ball 2003).

Conducting research within tribal communities is complex and a series of methodological issues must be addressed during the process of conceptualizing the evaluation, including the consideration of cultural and social norms in the design, the impact of cultural differences on program operations and outcomes, community experience with evaluation, including historical mistrust of researchers, and lack of culturally appropriate research methods. The work under this contract must be conducted within a Participatory http://www.aoa.gov/AoARoot/Site_Utilities/Standard_External_Disclaimer.aspx?redirection=http://www.agidnet.org http://www.aoa.gov/AoARoot/Site_Utilities/Standard_External_Disclaimer.aspx?redirection=http://www.agidnet.org http://www.aoa.acl.gov/Program_Results/docs/EA-of-TitleVI-v2.pdf http://www.aoa.acl.gov/Program_Results/docs/EA-of-TitleVI-v2.pdf http://www.aoa.gov/AoAroot/Site_Utilities/Disclaimer.aspx http://www.aoa.gov/AoAroot/Site_Utilities/Disclaimer.aspx� and Native Hawaiian Programs (OAA Title VI)

Action Research (PAR) framework that both values and accurately reflects the cultures of the individuals, communities, and programs under study.

Quoters are encouraged to propose modifications that will improve the efficiency and rigor of the project as long as those modifications ensure the cultural sensitivity, utility, feasibility, propriety and accuracy of work conducted.

Objective 1: Project Planning, Coordination, and Communication

The success of the project will require effective planning, coordination, and ongoing project management.

In order to ensure that this evaluation effort is successfully progressing, the contractor will need to develop key project management materials and maintain regular contact with the Contracting Officer’s Representative (COR) including regular status meetings; submitting monthly progress reports, and making necessary adjustments to the Project Work Plan on an as need basis and in consultation with the

COR.

Task 1.1 Project Kickoff Meeting

Within 10 business days of contract award, the Contractor shall develop a meeting agenda and conduct a kickoff meeting with ACL. The meeting shall initiate the communication process between all parties and may be held at ACL’s offices in Washington DC or may be conducted by conference call/webinar upon mutual agreement by both parties. The Kick-off Meeting is the first meeting with the project team members to discuss the project and the work that will be completed. This meeting introduces the members of the project team and provides the opportunity to discuss the role of each team member in the project work. Discussion topics may include, but are not limited to the following:

• Review of project purpose, objectives, research questions and expected products

• A discussion of the evaluation approach

• An overview of existing program issues

• Development of a plan for stakeholder involvement and Tribal recruitment

• Development of a detailed evaluation plan

• Refinement of the project work plan to guide the overall direction of the project

• A discussion of the format and timetable for deliverables including any draft and proposed interim reports and proposed trainings

• A review of the Contractor’s project governance strategies and processes

• Any other questions or issues identified by the Contractor

Task 1.2 Final Project Work Plan

The contractor shall manage the work to be accomplished under this contract through a Final Project Work Plan that is a formal written deliverable, due within 5 business days of the kick off meeting. The Final Project Work Plan should be a refinement of the work plan presented in the contractor’s Technical Proposal that addresses any initial comments or questions arising from ACL’s initial review of the technical proposal, feedback from the initial kick-off meeting described in Task 1.1, and ACL review of subsequent drafts of the Project Work Plan. In preparing this and all other contract deliverables, the contractor shall build in adequate time for receiving and incorporating comments from the COR. The Work Plan shall clearly delineate the contractor’s plan for maintaining regular contact with the COR, plans for documenting key project decisions, approach for ensuring adequate ACL/AoA feedback on key project deliverables, plan for key project milestones, due dates, and personnel assignments. This plan is considered a formal deliverable.

Task 1.3 Ensure Internal and External Stakeholder Support For Study

The contractor shall maintain close communication with the COR and program staff at ACL as well as with the Title VI Grant Program stakeholders identified by ACL This is an ongoing task and does not and Native Hawaiian Programs (OAA Title VI) include a formal deliverable. ACL anticipates that the contractor will continue contact with the Advisory Committee created under the Evaluability Assessment and develop and an Evaluation Working Group comprised of 2-3 individuals proposed by each Title VI grantee participating in the evaluation (more information about the Advisory Committee is provided under Task 2.4 and about the working group under Objective 3). For the purposes of this contract, the Advisory Committee is the committee of Tribal Resources Centers staff, Tribal researchers, Federal Stakeholders, and Title VI grantees developed under the Evaluability Assessment Contract. The Evaluation Working Group will be the 2-3 people identified by each Tribe selected to participate in the evaluation.

Task 1.4 Continue Project Management Activities

The contractor shall continue performing the project management activities articulated in the Final Project Work Plan delivered to ACL in Task 1.2 and continue to maintain regular contact with the ACL COR. This shall include at least monthly project status reports and bi-weekly project status conference calls. These calls will be scheduled for a mutually agreeable time. The Contractor is responsible for preparing a brief agenda to be distributed to the ACL COR at least one business day prior to the meeting and creating brief summaries of each meeting. Under this task the Contractor is also responsible for making any necessary revisions to the Project Work Plan on an as needed basis in consultation with the ACL COR. ACL/AoA considers this to be an informal deliverable that will be provided to ACL on an ongoing basis throughout the course of the contract.

Objective 2: Refine the Evaluation Approach and Develop Data Collection Tools

The contractor shall develop a detailed evaluation plan and tools needed for the collection of evaluation data. The contractor shall also support ACL through the process of OMB PRA clearance.

Task 2.1: Develop a Detailed Evaluation Plan

Based on the recommendations from the Evaluability Assessment, the contractor shall develop an evaluation plan that details the approach for recruiting Tribes, designing the data collection tools, collecting and analyzing Tribal data, and producing interim and final evaluation reports. As identified in the Evaluability Assessment, ACL is interested in a Participatory Action Research (PAR) approach which is an evaluation approach that has been successful with Native populations as it integrates communities and their members as full and equal partners in all phases of the research process. PAR is an approach to research in communities that emphasizes participation, action, and collective inquiry grounded in experience and social history. The quoter should consider how and whether gifts or incentives may be a culturally respectful and appropriate gesture and if they are to be included as part of the contract explain what they would be, when they would be distributed, and account for their cost. This plan is a formal deliverable due 45 days after award (approximately November 4, 20161).

Task 2.2: Review Evaluability Assessment and Logic Models

The data collected should be significantly informed by the work conducted through the Title VI Evaluability Assessment and, in particular, the logic model and medicine wheel models developed (Figures 1 and 2 below). Each input, output, and outcome should be considered for inclusion in the evaluation. ACL understands that it may not be feasible to collect data for all outputs and outcomes listed in those models.

In those cases, the contractor must provide an explicit explanation for their exclusion. ACL also understands that multiple questions or perspectives may be required to accurately measure selected outputs and outcomes. This should be reflected in the proposed data collection tools and detailed evaluation plan.

Task 2.3: Review other surveys

Where possible the contractor should use existing data collection tools that have been tested with similar populations and which will allow for comparisons between the data collected under this contract and other AoA, Tribal, or national populations. These include, but are not limited to:

1 Where approximate dates are provided, they assume a September 1 start date. If the start date is delayed, all dates will be adjusted accordingly

• Data collection tools used for ACL’s national evaluations of its Title III-C Nutrition Services Program and its Title III-E National Family Caregiver Support Program. These evaluation tools will be made available to the contractor upon award.

• The Tribal assessment that Tribes are encouraged to complete before applying for a Title VI grant entitled “Identifying Our Needs: A Survey of Elders VI” and available on the website of the National Resource Center on Native American Aging at https://www.nrcnaa.org/needs-assessment. Current guidance for Tribes regarding the assessment is available in Attachment 1.

Task 2.4: Consult with the Advisory Committee and the Evaluation Working Group

As part of the Title VI Evaluability Assessment, ACL created an Advisory Committee to inform the evaluation design. This group should continue to be consulted as needed throughout the evaluation to help ensure that the approach, tools, analyses, and reports for this project are feasible for, and accurately reflect the experiences of Title VI grantees. The Advisory Committee consists of approximately 20 individuals and typically meet by phone with additional communication through e-mail and, when possible, at the annual Older Indians Conference held by ACL’s Title VI Program. In the past the Advisory Committee has been consulted approximately 4 times per year. In addition the Evaluation working Group, which will be formed under Objective 3, shall be consulted regularly throughout the evaluation.

https://www.nrcnaa.org/needs-assessment

Figure 1: Title VI Program Logic Model

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native Hawaiian Programs (OAA Title VI)

Figure 2: Title VI Program Medicine Wheel Model

Task 2.5: Submit Draft and Final Data Collection Tools

The contractor shall submit a full set of draft tools for ACL review and upon receiving comments revise the tools and resubmit as the final tools that will be submitted to OMB as part of the Paperwork Reduction Act clearance process (Task 2.6). Tools should cover the collection of data about:

• How Nutrition, Caregiver, and supportive services are provided at the start of the 2017 Title VI grant period and at the end of the 2017 Title VI grant period (Process data).

• How the Title VI grant programs are operated at the Tribal level (including program quality and adherence to Title VI requirements) at the start of the 2017 Title VI grant period and at the end of the 2017 Title VI grant period (Process data).

• Consumer expectations for the program at the start of the 2017 Title VI grant period and consumer retrospective ratings of the effects of the program for themselves as individuals, for their families, for the community, and for the Tribe at the end of the 2017 Title VI grant period (Outcome data).

• Community capacity to meet Tribal members’ needs for nutrition, caregiver, and supportive services as documented through the Tribes’ Community Needs assessments (Identifying Our Needs: A Survey of Elders VI), as well as the Tribes’ ability to provide intergenerational interaction and opportunities for cultural expression at the start of the 2017 Title VI grant period and at the end of the 2017 Title VI grant period (Outcome data)

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native

NOTE: The 2017 Title VI grant period will be April 1, 2017 and March 31, 2020.

While the categories of inputs, outputs, and outcomes for the evaluation were defined under ACL’s Evaluability Assessment, and the Contractor will draft initial versions of the data collection tools, the tools will be revised with input from the evaluation working group. This includes input into non-substantive features like questions order, word choice, data collection type (telephone, in-person, etc.), and the time required for data collection. Draft tools should be submitted by 60 days after contract award (approximately November 29, 2016) and the final tools submitted for use with the OMB PRA package 75 days after contract award (approximately December 20, 2016).

Task 2.6: Provide Support to ACL Regarding Obtaining OMB Paperwork Reduction Act (PRA) Clearance

The contractor shall help ACL draft the PRA clearance package and respond to OMB questions as needed. The contractor shall also make any needed revisions to the data collections tools identified as a result of the OMB PRA process. The OMB package is considered a formal deliverable due within 85 days of contract award (approximately January 5, 2017)

Objective 3: Develop Tribal Sample and Evaluation Working Group

ACL anticipates that Tribes will be included in the evaluation based on at least the following factors:

1. An interest in and willingness to participate;

2. Having conducted their Tribal Assessment using the “Identifying Our Needs: A Survey of Elders VI” tool (available on the website of the National Resource Center on Native American Aging at https://www.nrcnaa.org/needs-assessment); and

3. Having submitted a Tribal Resolution as part of their 2017 grant application that allows the Tribe to share the data from their assessment conducted for the 2017 and 2020 awards (grants have a three year period of performance and the majority of grantees apply each cycle)

Task 3.1 Create an Evaluation Working Group Work Plan The contractor shall develop a plan with a timeline to describe the expected purpose, goals, and outcomes for the Evaluation Working Group (the 2-3 individuals identified by Tribes that participate in the evaluation who will participate in all aspects of the evaluation and provide guidance regarding the approach, data methods, data to be collected, trainings conducted, and reports produced). It should outline potential topics for trainings and meetings that will be held with the working group members. This document shall also include both outcomes for the members of the group in terms of knowledge and skills they will be expected to gain and outcomes for ACL in terms of the quality of the evaluation, as well as methods for measuring whether the outcomes were achieved. This document is considered a formal deliverable due within 105 days of contract award (approximately February 3, 2017).

Task 3.2 Develop the Grant Funding Opportunity Announcement (FOA) to allow Tribes to apply to participate in the evaluation The contractor will develop recruitment materials (examples of materials developed for other ACL evaluations are available on the ACL website (http://www.aoa.acl.gov/Program_Results/Program_Evaluation.aspx) and a simple FOA to award up to $15,000 per Tribe to as many as 30 Tribes to participate in the evaluation. The contractor shall work closely with ACL and the Advisory Committee on the FOA which will include language about the participatory nature of the work, the requirement of having completed a Tribal Assessment using the “Identifying Our Needs: A Survey of Elders VI” tool and having a signed Tribal resolution releasing to ACL their Tribal assessment data from the 2017 grant period and the 2020 grant application period if they apply for a grant in 2020, and nomination of 2-3 staff or Tribal members to participate on the Evaluation Working Group. The contractor will help ACL weigh the merits of either allowing all interested Tribes to apply, selecting a sample of tribes to balance factors such as geographic location and size and then inviting them to apply, or a combination of the two. The FOA will be distributed based on a joint decision between the contractor and ACL regarding sampling. It may also be necessary for the contractor to work with ACL to help recruit Tribes if there are not enough applicants. The FOA is considered a formal http://www.aoa.acl.gov/Program_Results/Program_Evaluation.aspx

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native deliverable due within 165 days of contract award (approximately May 2, 2017). NOTE: All recruitment materials developed to be posted on the ACL website must be 508 compliant.

Task 3.2: Review Applications and Make Grant Awards The contractor will receive the grant applications and review them according to criteria developed jointly with ACL. Upon final approval by ACL, the contractor will make the grant awards of approximately $15,000 per Tribe for up to 30 Tribes. Assuming 4 weeks for Tribes to respond to the grant opportunity, the final award documents are considered a formal deliverable due within 203 days of contract award (approximately June 26, 2017).

Task 3.3: Kick off meeting with Tribal Evaluation Grantees The contractor will host one, or possibly 2 meetings to accommodate the time zones of the grantees, to welcome grantees to the project. The meeting should, at a minimum, cover the transfer and uses of funds as well as the roles and responsibilities of ACL, the Contractor, and the Tribal grantees. The kick off meetings are considered a formal deliverable due 212 days after contract award (approximately July 10, 2017)

Task 3.4: Compose Evaluation Working Group Starting with the lists of individuals submitted with the grant applications, the contractor will confirm and keep track of the members of the Evaluation Working Group, welcome them through individual communication and discussion of expectations, and set up a ‘kick off’ meeting for the group as part of the training scheduled for the 2017 Older Indians Meeting (Task 5.1) that will take place in July/August 2017 (approximately 230 days after contract award). Under later tasks under Objective 5, the contractor will provide training for Evaluation Working Group members and engage them in decision-making related to the evaluation.

Objective 4: Data Collection and Analysis

The contractor shall develop an IT security plan, and collect and analyze data needed for the evaluation.

Because of the participatory nature of this evaluation the contractor shall coordinate the data collection (e.g., finalizing the data collection tools, determining the data collection methods that are best suited for each Tribe, agreeing upon the degree of involvement by the contractor vs. the Tribal members) with the evaluation grantees and their representatives on the Evaluation Working Group, the Advisory Committee, and ACL.

Task 4.1: Develop and IT Security Plan The Contractor shall develop an IT Security Plan and conduct related security assessments in accordance with the Federal Information Security Management Act (FISMA). The IT Security Plan must ensure the integrity, confidentiality, when appropriate, and availability of all data collected on behalf of the Federal government. All records that are the property of the Federal government must be maintained in accordance with HHS policies and procedures, and National Archives and Records Administration (NARA) disposition schedules. The Contractor shall provide a certification statement concerning the proper maintenance of all records to the COR at the beginning of the contract. The Contractor shall discuss the disposition of records with the COR and obtain COR approval before any records are disposed. The Contractor shall notify the COR within 24 hours concerning any loss of data integrity, any unauthorized disclosure of data, or any misuse of data. The IT security plan is considered a formal deliverable due within 144 days of contract award (approximately March 31, 2017).

Task 4.2: Create a Data Analysis Plan. The Contractor shall submit a data analysis plan that provides a detailed description of the techniques and statistical analysis that will be conducted to answer the evaluation questions. This is considered a formal deliverable due 154 days after contract award (approximately April 14, 2017).

Task 4.3: Conduct/Support On-Site Data Collection

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native

The contractor will collect a range of process and outcome evaluation data across at least two time periods. Below is a possible outline for the data collection:

1. Time 1

a. Process Evaluation Data

i. Nutrition (from program staff and consumers)

ii. Caregiver (from program staff and consumers)

iii. Supportive Services (from program staff and consumers)

iv. Program Management (from program staff and consumers)

b. Outcome Evaluation Data (baseline)

i. Tribal Assessment for 2017 award (tribal level consumer data)

ii. Consumer data-Program expectations for Title VI services

iii. Community capacity to meet Title VI goals of providing needed nutrition services, caregiver support services, and a range of other supportive services, help to individuals maintain their health and remain independent in their communities “in a manner that preserves and restores their respective dignity, self-respect, and cultural identities.”

2. Time 2

a. Process Evaluation Follow-up

i. Have there been any changes to process since Time 1 with a particular focus on program management changes, increased use of data for decision-making, strengthening program quality, and program changes to better meet community need

b. Outcome Evaluation Data

i. Tribal Assessment for 2020 award (tribal level consumer data)

ii. Consumer data-Retrospective rating the effect of the program on quality of life

(cultural, spiritual, and emotional), empowerment, health (physical and mental), and independence

iii. Community capacity data on the effect of the program on local capacity to meet needs of elders

It is expected that most of the data collection will occur on-site. But, final determinations about data collection methods will be made jointly between the Evaluation Working Group, ACL, and the contractor.

In some cases the contractor will provide guidance for data collection conducted by Tribal Members and in others the contractor staff will collect the data. The time 1 data collection should be completed by approximately 415 days after contract award which is April 30, 2018 (10 months after the award of the evaluation funds to Tribes). The time 2 data collection should be completed by the end of the 2017 Title VI grants which will be March 31, 2020 (896 days after contract award).

Task 4.4: Analyze Data The data shall be analyzed using appropriate data analysis techniques. When analyzing quantitative data, the Contractor will produce frequencies and conduct other relevant analysis to identify themes and patterns within and across sites, consistent with standard research practices. This at a minimum shall include:

• frequency tables for each measure with the minimum, maximum, range, median and mode,

• correlations, and

• cross-tabs

When analyzing qualitative data the contractor shall develop a coherent method for identifying, examining, and interpreting patterns and themes in the data collected to be able to answer the following questions:

1. What pattern/common themes emerge around specific items in the data? How do these patterns (or lack thereof) help shed light on the broader study questions?

2. Are there any deviations from these patterns? If, yes, what factors could explain these atypical responses?

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native

3. What interesting stories emerge from the data? How can these stories help to shed light on the broader study question?

4. Do any of the patterns/emergent themes suggest that additional data needs to be collected? Do any of the study questions need to be revised?

5. Do the patterns that emerge support the findings of corresponding quantitative analyses that have been conducted?

6. What does this information tell us about the ultimate effectiveness of the Title VI Program at the Tribal/Community-level, the regional-level (or amongst like Tribes), and the Federal-level?

This is an ongoing task that should occur as needed to allow for timely delivery of interim and final reports under Objective 7.

Objective 5: Training and Technical Assistance

One goal for this contract is to build evaluation capacity among Title VI Tribal grantees and to “give back” to participating grantees in recognition of their contribution to the evaluation.

Task 5.1: In-Person Evaluation Working Group Meetings (2017, 2018, 2019)

In conjunction with the annual “Older Indians” conferences to be hosted by ACL in 2017, 2018, and 2019, the contractor shall conduct three, two-day trainings (one training per year). The training shall be added either in the two days before or the two days after the conference to realize time and cost efficiencies as many of the Evaluation Working Group members will fund travel to the Older Indians meeting under their Title VI grant. Therefore, it is estimated that this contract will cover the expenses for travel for approximately 37 people (32 Evaluation Working Group members and 5 contractor staff) and lodging and per diem for approximately 82 people (77 Evaluation Working Group members and 5 contractor staff).

The contractor shall also be responsible for other training costs such as meeting rooms and supplies (flip charts, handouts, etc.). Locations for these meetings have not yet been determined.

The contractor shall be responsible for developing and producing meeting and training materials as needed (e.g., agendas, handouts).These trainings will include (1) decision-making sessions to come to agreement on important evaluation approaches and activities, (2) time for participants to talk about their challenges and successes with the evaluation, and (3) skills building and/or training sessions for which attendees will receive certificates of participation.

Task 5.2: Coordinate with TA providers

The contractor will work with the three National Resource Centers that support the Title VI program:

• National Resource Center on Native American Aging (https://www.nrcnaa.org)

• Ha Kupuna: National Resource Center for Native Hawaiian Elders (http://manoa.hawaii.edu/hakupuna/)

• National Resource Center for American Indian, Alaska Native, and Native Hawaiian Elders (https://www.uaa.alaska.edu/elders/)

The purpose of the coordination is to develop good dissemination pathways for information, and to supplement the evaluation resources offered by the Centers without duplicating information or effort. To the extent possible, although the trainings conducted under this contract will be specific to the needs of Evaluation Working Group members and their Tribes, the trainings should be recorded and made available, through the resource centers, to others interested in conducting evaluations. The contractor will also provide other TA as appropriate and jointly approved by ACL, the Evaluation Working Group, and the contractor.

https://www.nrcnaa.org/ http://manoa.hawaii.edu/hakupuna/ https://www.uaa.alaska.edu/elders/

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native

Task 5.3: Provide Skills Training for Evaluation Working Group Members

In addition to providing in person training to Evaluation Working Group members under Task 5.1, the contractor shall provide one (1) additional training per year remotely in conjunction with the existing technical assistance providers for the Title VI grants program. The contractor will develop a list of evaluation training topics and work with the Evaluation Working Group, ACL, and the resource centers to make a final determination about which trainings will be offered and the schedule for training. The trainings are considered a formal deliverable.

Objective 6: Tribal Data Analysis and Support

Another benefit to be offered to the Tribes participating in the evaluation will be Tribe-specific data analysis and support.

Task 6.1: Work With Tribes to Identify Data Analysis and Reporting Needs

The contractor will work with each Tribe to assess their available data, data analysis and reporting needs, and the level of data analytic capacity. This assessment could take the form of focus groups, staff interviews, site visits in conjunction with Time 1 on-site data collection, or other approaches jointly determined by the Tribe, the contractor and ACL. The timing of this task will be determined in conjunction with each Tribe participating in the evaluation.

Task 6.2: Develop and Implement Plans for Providing Tribal Data Analysis and Support

After Tribes’ analytic needs are identified, the contractor will develop a Tribe specific analysis support plan that takes into account the needs of each tribe and the contract resources (time and funding) available for this task. Upon approval of each plan the contractor shall provide analysis and support according to the plan. While the purpose of this task is to provide individualized support to each tribe, when reasonable and acceptable to the Tribes, support can be conducted for groups of Tribes simultaneously. The plans are considered a formal deliverable and are due within 4 weeks of the identification of each Tribe’s analytic needs.

Task 6.3: Get Feedback from the Evaluation Working Group, Tribal leadership, and Advisory Committee members

To ensure that the analytic support is meeting the needs of the Tribes, the contractor will ask for feedback from the Tribes regarding the process and products delivered at three points:

1. At the end of the needs identification process,

2. Once the support plan is developed, and

3. After the support has been provided.

The contractor should develop a memo summarizing the feedback received and include recommendations for future evaluation capacity building efforts. The memo is considered a formal deliverable and is due 855 days after contract award (approximately January 31, 2020)

Objective 7: Reporting and Data Documentation

The contactor shall develop annual interim reports and a final comprehensive report that describes the methods and findings of the evaluation and makes recommendations for program improvement. The contractor shall also conduct final project briefings. NOTE: All products developed to be posted on the ACL website must be 508 compliant.

Task 7.1: Interim Project Reports

The contractor will work with ACL, the Advisory Committee, and the Evaluation Working Group to develop a list of interim reports to be delivered throughout the contract period. While the final list and timing of reports will be developed after the award of the grants to tribes for the evaluation, ACL anticipates that there will be one interim report per contract year of up to 50 pages in length. For example, there may be a report in year one that describes the participatory evaluation method and presents grantee

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native characteristics, a report in year two that analyzes and interprets the process evaluation and community level data gathered as part of the time 1 data collection, a report in year 3 that reports on retrospective consumer data about the effect of the Title VI program on the community, and a report in year 4 that compares the time one Tribal needs assessment data with the time two needs assessment data and documents changes in program management over the Title VI grant period. These reports are considered formal deliverables with due dates to be determined once the Evaluation Working Group has been created.

Task 7.2 Draft Final Report

The contractor shall draft a final report that provides the:

1. Introduction and overview of the project;

2. Evaluability methodology

3. Results and findings

4. Conclusions and recommendations

The draft report is a formal deliverable due 1,025 days after contract award (approximately October 1, 2020).

Task 7.3: Conduct Briefings Title VI Stakeholders

The contractor shall conduct up to 4 briefings with Title VI stakeholders identified by ACL for the purposes of informing stakeholders about the evaluation findings, encouraging discussion about data interpretation, possible next steps for ACL, and for getting feedback on the draft final report. Briefings may be conducted in person or by webinar or other method that will be mutually agreed upon by the contractor and ACL. At least one briefing should be planned in conjunction with the 2020 Older Indians Conference. Briefings should be conducted within 1,064 days after contract award of contract award (approximately the end of November 2020)

Task 7.4 Deliver Data and Documentation to ACL

The Contractor will create data documentation files (electronic) which will include but are not limited to a code book with variable definitions, data frequencies, data collection instruments and instructions. The data documentation files shall be submitted with the data. The Contractor will deliver al project data to ACL/AoA and/or an entity ACL/AoA designates as a repository for the data set(s) in a form approved of by ACL/AoA along with all code books and data documentation. The Contractor shall deliver draft materials as needed so that the final product is completed by contract end.

Task 7.5: Final Project Report

The report should include revisions based on comments about the draft report from ACL and feedback received through the briefing(s). This is a formal deliverable and should be completed by the end of the contract period.

2.4 DISCLOSURE OF INFORMATION

Information made available to the Contractor by the Government for the performance or administration of this effort shall be used only for those purposes and shall not be used in any other way without the written agreement of the Contracting Officer.

The Contractor agrees to assume responsibility for protecting the confidentiality of Government records, which are not public information. Each contractor or employee of the Contractor to whom information may be made available or disclosed shall be notified in writing by the Contractor that such information may be disclosed only for a purpose and to the extent authorized herein.

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native

2.5 LIMITED USE OF DATA

Performance of this effort may require the Contractor to access and use data and information proprietary to a Government agency or Government contractor which is of such a nature that its dissemination or use, other than in performance of this effort, would be adverse to the interests of the Government and/or others.

Contractor and/or contractor personnel shall not divulge or release data or information developed or obtained in performance of this effort, until made public by the Government, except to authorize Government personnel or upon written approval of the Contracting Officer (CO). The Contractor shall not use, disclose, or reproduce proprietary data that bears a restrictive legend, other than as required in the performance of this effort. Nothing herein shall preclude the use of any data independently acquired by the Contractor without such limitations or prohibit an agreement at no cost to the Government between the Contractor and the data owner which provides for greater rights to the Contractor.

2.6 HHS SECTION 508 ACCESSIBILITY STANDARDS NOTICE (SEPTEMBER 2009)

This contract is subject to Section 508 of the Rehabilitation Act (the Act) of 1973 (29 U.S.C. 794d), as amended by the Workforce Investment Act of 1998, and the Architectural and Transportation Barriers Compliance Board (Access Board) Electronic and Information Accessibility Provisions (36 CFR Part 1194). Section 508 of the Act requires that, unless an exception applies, all communications products and services that require a contractor or consultant to produce content in any format that is specifically intended for publication on, or delivery via, a federally owned or federally funded website permit the following:

(1) Federal employees with disabilities to have access to and use information and data that is comparable to the access and use of information and data by federal employees who are not individuals with disabilities.

(2) Members of the public with disabilities seeking information or services from a federal agency to have access to and use of information and data that is comparable to the access and use of information and data by members of the public who are not individuals with disabilities.

(Note: Information about Section 508 of the Act is available at http://www.section508.gov/. The complete text of Section 508 can be accessed at http://www.access-board.gov/sec508/provisions.htm.)

Accordingly, regardless of format, all web content or communications materials specifically produced for publication on, or delivery via, HHS websites, including text, audio, or video, under this contract shall conform to applicable Section 508 accessibility standards. Remediation of any materials that do not comply with the applicable accessibility standards of 36 CFR Part 1194 as set forth herein shall be the responsibility of the Contractor.

The following Section 508 accessibility standards apply to the content or communications material identified in this SOW or PWS:

• Web-based Intranet and Internet Information and Applications (1194.22)

• Telecommunications Products (1194.23)

• Video or Multimedia Products (1194.24)

• Information, Documentation, and Support (1194.41)

2.7 QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

The Government intends to utilize a Quality Assurance Surveillance Plan (QASP) to monitor the quality of the Contractor’s performance. The oversight provided for in the contract and in the QASP will help to ensure that service levels reach and maintain the required levels throughout the contract term. Further, the QASP provides the COR with a proactive way to avoid unacceptable or deficient performance, and provides verifiable input for the Contractor Performance Assessment Reporting System (CPARS). The QASP may be updated by modification to the contract.

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native

3 SECTION 3 – PACKAGING AND MARKING

All information and deliverables shall be delivered to the Contracting Officer’s Representative and shall be marked as follows:

1. Name and address of the Contractor;

2. Contract Number;

3. Description of item contained therein; and

4. Consignee's name and address.

Evaluation of The Administration for Community Living’s (ACL) American Indian, Alaska Natives and Native

4 SECTION 4 – INSPECTION AND…