TPP23 Tier2 Rigorous Eval QA 04062023_final.pdf

Updated April 6, 2023

HHS Office of Population Affairs Web: opa.hhs.gov | Email: opa@hhs.gov Twitter: @HHSPopAffairs | YouTube: HHSOfficeofPopulationAffairs

Teen Pregnancy Prevention Program (TPP) Tier 2 Rigorous Evaluation Cooperative Agreement

AH-TP2-23-001

QUESTIONS & ANSWERS

file://ct-files/g-drive/Fertility%20and%20Family%20Structure/OPA%20Communications/Option%20Year%202/1-%20Strategic%20Communications/1.8%20Graphic%20Design/opa.hhs.gov

Contents General Programmatic Budget

General

1. Can additional items beyond those listed on page 59 section H.2, be included in the appendix section as long as all items do not exceed the page limit? Those listed include Work Plan, Logic Model, Memoranda of Agreement (MOAs) and/or Letters of Commitment (LOCs), Organizational Chart, Curriculum Vitae/Resumes/Biosketches for Key Project Personnel, and References Cited.

Yes. The NOFO does not prohibit the submission of additional items in the appendix, as long as the application page limit is not exceeded.

2. On page 19, section D.2.a., the instructions state that tables must use 12-point font and may be single spaced. Is this true of graphics as well?

Under the appendices section on page 19 it states that documents such as resumes/curricula vitae, organizational charts, tables, or letters of commitment may use formatting common to those documents, but the pages must be easy to read. The same would apply to graphics. Additionally, as a reminder, any files uploaded or attached to the Grants.gov application must be Adobe PDF, Microsoft Word, or image formats (JPG, GIF, TIFF, or BMP only) and must contain a valid file format extension in the filename.

We will not accept Microsoft Excel files. The use of compressed file formats such as ZIP, RAR, or Adobe Portfolio will not be accepted. (p. 41)

3. Can multiple cooperative agreements be awarded to the same Principal

Investigator (PI) under the same funding opportunity?

An organization may submit more than one proposal for this funding opportunity if each of the proposals submitted by the organization is for a unique project. The NOFO does not restrict an organization or Principal Investigator (PI) to a single award, therefore, a PI may be awarded multiple cooperative agreements under this NOFO if the multiple proposals submitted are successfully awarded.

However, the NOFO also states that an application will be disqualified, that is, we will not review it and we will give it no further consideration, if you successfully submit multiple applications from the same organization for the same project. In this instance, OASH will only review the last application received by the deadline. (p.17).

4. Page 7 of the NOFO says "The NOFO supplemental materials (Section 1) include a complete list of interventions that are not eligible without substantial modifications." I don't see this list. Is the list in a separate document? Thank you!

You are correct, Section I - Supplementary Materials does not include a list of interventions. Please disregard that sentence. However, the NOFO does include web links on page 7 which provide all the TPP Interventions rigorously evaluated previously under federal funding.

As a reminder, these interventions are not appropriate for this funding unless significant changes have been made to the intervention to address findings from the previous evaluation results. Interventions identified as effective by the HHS Teen Pregnancy Prevention Evidence Review (TPPER) are also not appropriate for this funding (https://tppevidencereview.youth.gov/FindAProgram.aspx), unless significant adaptations are proposed to the intervention, ultimately creating a new intervention (p.7).

5. The NOFO does not discuss the Authorizing Agent as mentioned in one of the webinar slides. Is the Authorizing Agent separate from PD/PI or is this one and the same? Can the same staff member act as AA and PD/PI?

The NOFO Page 39, section 5a states that you must register an authorizing official for your organization. OASH does not determine your organization’s authorizing official;

your organization makes that designation.

The Authorized Official and Project Director/Principal Investigator (PD/PI) may be the same person. The NOFO does not indicate otherwise. As a reminder, only one PD/PI will be named on any resulting award. You should clearly identify the individual in that role in your application (p. 18).

6. When are applications due for Opportunity Number: AH-TP2-23-001?

Applications are due on Wednesday, May 17, 2023, by 6pm ET

7. Can the same organization submit applications to the Rigorous Evaluation NOFO, TPP Tier 1 Advancing Equity in Adolescent Health through Replication of Evidence-Based TPP and/ or TPP Tier 2 Adolescent Sexual Health Innovation Hubs?

Yes, an organization can submit an application to more than one NOFO. Each NOFO is different, and organizations should check the eligibility and expectations of each individual NOFO to determine whether or not to apply. HHS/OASH/GAM will deem each application submitted by an entity eligible according to the eligibility information included in the specific NOFO.

8. I am wondering if this is a single submission opportunity. Can we submit more than one application for this grant if the applications are for two separate projects? I am seeking to clarify the language on page 17 of the application that reads as follows, “If you successfully submit multiple applications from the same organization for the same project, we will only review the last application received by the deadline.”

Yes. You may submit more than one application if each is for a different project. As you noted, page 17 of the NOFO states, “If you successfully submit multiple applications from the same organization for the same project, we will only review the last application received by the deadline.”

9. Can an individual submit a grant application?

Grants are awarded to organizations rather than individuals. An application may be submitted by an individual authorized to act/sign for an organization and to assume the obligations imposed by the grant and any additional conditions of the grant. However, the award will not go directly to an individual but to the organization which the individual represents.

10. What is page limit for the application?

Your total application (i.e., the Project Narrative plus Appendices) must not exceed 100 pages. The following items do not count toward the Project Narrative page limit: all required forms, including SF-424, SF-424A, SF-LLL, Project Abstract Summary, and Budget Narrative (including budget tables) (Section C.4) (p.17).

11. What is page limit for the project narrative?

Your Project Narrative must not exceed 50 pages. The following items do not count toward the Project Narrative page limit: all required forms, including SF-424, SF-424A, SF-LLL, Project Abstract Summary, and Budget Narrative (including budget tables) (Section C.4) (p.17).

12. Do the appendices count toward the 50 pages allowed for the project narrative?

No, but the Appendices do count toward the page limit for your total application. Your total application (i.e., the Project Narrative plus Appendices) must not exceed 100 pages. The following items do not count toward the Project Narrative page limit: all required forms, including SF-424, SF-424A, SF-LLL, Project Abstract Summary, and Budget Narrative (including budget tables) (Section C.4) (p.17).

13. Will OPA advise or consult with our organization about potential ideas for my proposal before submission of the application?

No. Since this is a competitive funding announcement, OPA cannot provide input regarding the content of individual grant applications and is therefore unable to answer questions specific to the content of an application.

Applicants are encouraged to carefully review the NOFO and should put forward the best proposal the organization can create based on application and evaluation criteria published in the announcement.

14. What organizations are eligible to apply for the funding?

Applicants eligible to apply for this notice of funding opportunity are any public or private (profit or nonprofit) entity located in a State (which includes one of the 50 United States, District of Columbia, Commonwealth of Puerto Rico, U.S. Virgin Islands, Commonwealth of the Northern Mariana Islands, American Samoa, Guam, Republic of Palau, Federated States of Micronesia, and the Republic of the Marshall Islands).

Faith-based organizations and American Indian/Alaskan Native/Native American (AI/AN/NA) organizations that are public or private entities are eligible to apply. Public or private community-based organizations are eligible to apply (p.15-16).

Please reference the Eligibility Information in the NOFO beginning on page 16 for additional examples of eligible Organizations.

15. Any FTE expectation or requirement for the PI/PD position?

There is no expectation or requirement regarding the FTE level of the PI/PD position.

The NOFO does note that only one Project Director/Principal Investigator (PD/PI) will be named on any resulting award. You should clearly identify the individual in that role in your application. This individual should be the person who will be responsible for the programmatic aspects of the project if an award is made (p. 18).

See Sections D.3.a.4. Organizational Capability, Partnerships, and Staffing (p.26) and D.3.a.5. Project Management and Work Plan (p.27) for more information on what to include in your application as it relates to the PI/PD role. Section D.3.b - Budget Narrative Content (p.28-34) also provides guidance on what information to include as it relates to personnel and staffing.

16. What's allowable for images/graphics in the narrative?

Any files uploaded or attached to the Grants.gov application must be Adobe PDF, Microsoft Word, or image formats (JPG, GIF, TIFF, or BMP only) and must contain a valid file format extension in the filename. We will not accept Microsoft Excel files.

In addition, the use of compressed file formats such as ZIP, RAR, or Adobe Portfolio will not be accepted. We will not contact you for resubmission of uncompressed versions of files. Compressed files in the application will not be forwarded to the independent merit review panel for consideration.

We strongly recommend that electronic applications be uploaded as Adobe PDF. If you convert to PDF prior to submission, you may prevent any unintentional formatting that might occur with submission of an editable document. Although Grants.gov allows you to attach any file format as part of your application, we restrict this practice and only accept the file formats identified above for compatibility with our other systems. Any file submitted as part of the Grants.gov application that is not in a file format listed above will not be accepted for processing and will be excluded from the application during the review process (p.41).

17. Do you have to be a OPA previously funded grantee to apply for this funding?

No. Eligibility is not based on previous OPA funding. Any public or private entity, including faith-based, community-based, and Tribal organizations, is eligible to apply for an award under this NOFO. (p. 15-16).

18. Should the applicant organization be the evaluator or the program implementer or the intervention developer?

Any eligible organization may apply for funding under this NOFO (p.15-16). Either the evaluator’s organization, the program implementer’s organization, or the intervention developer’s organization may apply as the lead applicant, provided that the organization is an eligible entity. The NOFO does not specify what types of organizations should be the lead agency or which ones should be the partners. Information regarding eligibility pertains only to applicants. We expect recipients to use partnerships to fulfill capacity needs and successfully complete the project. Recipients and key partners should have the collective expertise necessary to successfully accomplish the goals and objectives for the overall project (p. 11). For information on partnerships, please review Sections D.3.e - Collaborate with Partners and Maintain Organizational Capacity (p.11) and D.3.a.4. Organizational Capability, Partnerships, and Staffing (p.26-27) for information on what should be included in your application to describe your partnership efforts and those with whom you may want to partner. We recommend you also review Section E.1 - Criteria. There you will find the criteria that will be used by federal staff and an independent review panel to assess all eligible applications (p.42-45).

19. I am attempting to apply to the NOFO (AH-TP2-23-001) - on Grants.gov. However, when I log into my account - the "Apply" button is not enabled. I will also need access to create a Workspace - as I am the Executive Director for the 501c3 agency applying.

Please contact Grants.gov directly for any issues with their system. Below is the contact information for Grants.gov:

• Email: support@grants.gov

• Telephone: 800-518-4726

Programmatic

1. How is the Application Content used to score applications?

Applicants are encouraged to submit all information requested in the Application Content section of the NOFO (Section D.3). An independent review panel will evaluate applications that are not disqualified and meet the responsiveness criteria (Section C.3). These reviewers are experts in their fields, and are drawn from academic institutions, non-profit organizations, state and local government, and Federal government agencies. Based on the Application Review Criteria as outlined under Section E.1, the reviewers will comment on and rate the applications, focusing their comments and ratings on the identified criteria. In addition to the independent review panel, Federal staff will review each application for programmatic, budgetary, and grants management compliance. (p.45)

OPA recommends thoroughly reviewing Section E.1 – Criteria which specifically outlines the criteria that will be used by federal staff and an independent review panel to assess all eligible applications (p.42-45).

mailto:support@grants.gov

2. On page 44, under the section titled “Implementation of the Intervention and project Improvement,” there is a typo in the 4th bullet. Can you confirm the correct sentence?

The sentence in Section E.1. of the NOFO page 44 should read, “The applicant demonstrates capacity through sufficient staffing assignments; staff with the necessary and relevant experience, qualifications, and training; previous experience managing rigorous evaluation studies similar in size and scope to the proposed project; and examples of successful rigorous evaluation studies in similar settings and contexts and with similar populations as the proposed study.”

3. On page 21 of the NOFO it states, “Provide a thorough description of your impact evaluation design in your application; an optional sample template is available under the Related Documents tab on grants.gov for this opportunity.” We do not see on grants.gov in the Related Documents tab where this optional sample template is located. Is this sentence intended to reference the optional template that is provided on pages 74-76 of the NOFO?

Yes, the optional evaluation template is only in the NOFO, Section I. Supplementary Materials (p.74- 77). The reference to the material being posted on the Related Documents tab on grants.gov is an error.

4. On page 20 “You may reference a logic model included in your appendices (Section D.3.c) to support your narrative; an optional sample template is available under the Related Documents tab on grants.gov for this opportunity”. The reference to the related material tab on grants.gov does not appear to be on grants.gov as stated.

Please advise.

The logic model template may be found on the OPA website: https://opa.hhs.gov/grant-programs/funding/current-funding-opportunities#TPP. The reference to the material being posted on the Related Documents tab on grants.gov is an error.

5. Can you please confirm that intimate partner violence measures can be included as either primary or secondary outcome measures in their proposed rigorous evaluation plan? Does OPA have any guidance on the mandatory reporting requirements (or voluntary reporting) that might be triggered if IPV is identified?

Yes, the NOFO states that recipients may propose additional research questions that address risk factors for unintended pregnancy, such as … intimate partner violence (p.22). The additional research questions may be either primary or secondary outcomes, as the NOFO does not place restrictions on what questions must be primary or secondary based on topic. OPA expects recipients to collect and analyze outcomes that are relevant to the intervention’s logic model and proposed research questions (p.9).

Regarding mandatory (or voluntary) reporting, recipients should follow their state’s laws and policies, as applicable, and consult with their Institutional Review Board (IRB).

6. Is it acceptable for evaluation staff to both enroll study participants and provide the intervention to individuals assigned to the treatment condition if they do not have access to any outcome data nor are they responsible to data analysis and reporting?

https://opa.hhs.gov/grant-programs/funding/current-funding-opportunities#TPP https://opa.hhs.gov/grant-programs/funding/current-funding-opportunities#TPP

Federal staff and an independent review panel will assess all eligible applications in part on the “extent to which the applicant clearly describes how they would conduct the study in a credible and neutral manner, including at a minimum, evaluation staff who do not have perceived or actual conflict of interest and can remain independent of the intervention,” as well as on the “extent to which the applicant clearly discusses study limitations and biases and presents a sound plan that is likely to address them” (p.44).

Since this is a competitive funding announcement, OPA cannot provide input regarding the content of individual grant applications and is therefore unable to answer questions specific to the content of an application.

Applicants are encouraged to carefully review the NOFO and should put forward the best proposal the organization can create based on application and evaluation criteria published in the announcement.

7. Regarding the timing of data collection, are there specific definitions for what OPA considers to be short and long-term data collection time points?

The NOFO suggests broad ranges for the timing of data collection: “[w]e anticipate that recipients will collect survey data at baseline, at one short term follow-up (e.g., between 0 months-1-year post-intervention), and at one long-term follow-up (e.g., 9 months or more post-intervention)” (p. 9). In addition, we expect recipients to use the final 6-9 months of the project award period for data analysis and reporting (p. 9).

The applicant should determine the specific time frame within the suggested ranges that would be most appropriate for the intervention under evaluation, and realistic for a 5-year project period. Federal staff and an independent review panel will assess all eligible applications on the extent to which “timelines for data collection are realistic for a 5-year project and are consistent with the project work plan” (p.44).

Please be sure to review Section E.1 - Criteria. There you will find the criteria that will be used by federal staff and an independent review panel to assess all eligible applications (p.42-45).

8. Would scale measures be considered appropriate for primary outcome measures? Or should we follow the specifications outlined in the TPP evidence review protocol version 6.0 and avoid “black-box type measures” for primary outcomes?

The NOFO states that “when feasible, we expect the evaluation design to meet the criteria for the quality of an evaluation study per the criteria in the HHS Teen Pregnancy Prevention Evidence Review (TPPER), version 6.0.

(https://tppevidencereview.youth.gov/ReviewProtocol.aspx)” (p. 8). You should adhere to the study quality criteria in TPPER, version 6.0 to the extent feasible for your proposed project and provide justification for any proposed deviation from the TPPER criteria in your application.

https://tppevidencereview.youth.gov/ReviewProtocol.aspx)

9. Who can do observations?

Section I – Supplementary Materials of the NOFO includes the TPP2020 Performance Measures. In this section, the NOFO states that "An observer ideally should be independent from the implementation, familiar with the program model, and may be an internal or external evaluator, supervisor (program director, program coordinator), or a program partner" (p.73). Please note that this information is subject to change. Recipients will be provided the final performance measures with accompanying guidance within the first six months of funding.

10. Do we need to have preliminary evidence such as from a pilot evaluation?

Yes, you are expected to have preliminary evidence; such evidence may come from a pilot evaluation or other previous research such as formative evaluation, implementation evaluation, pre-post tests, or other types of evaluations (p. 21). “We expect any intervention proposed for evaluation under this initiative to already have compelling, positive preliminary evidence from previous formative evaluation and prior research, including documented effects on intended outcomes” (p. 6).

11. Am I understanding that we don’t have to work with or measure youth outcomes? We can instead measure parent or caregiver outcomes?

No. All recipients should measure at least one youth outcome related to a sexual risk behavior. You may collect parent or caregiver outcomes in addition to the youth sexual risk behavioral outcome described above if the proposed parent/caregiver outcomes are relevant to your intervention’s logic model (p. 22). You should provide supporting information that clearly links all proposed research questions and outcomes to the reduction of unintended teen pregnancy and clearly demonstrates how the outcomes you are testing would address the needs of the community and population of focus. You should also describe how answering these questions will improve programs, outcomes, and/or policies for youth (p. 22).

In addition, Federal staff and an independent review panel will assess all eligible applications on the following “At least one research question assesses the intervention’s impact on a sexual risk behavior associated with unintended teen pregnancy” (p. 43).

12. Does our preliminary evidence have to be from this specific curriculum or can similar findings from the field be used?

The intent of the NOFO is that preliminary evidence would be from the specific curriculum/intervention proposed for evaluation under the opportunity. "We expect any intervention proposed for evaluation under this initiative to already have compelling, positive preliminary evidence from previous formative evaluation and prior research, including documented effects on intended outcomes (p. 6)." Applicants should “summarize all previous testing, research, and evaluation done on the intervention . . . [and] specify the type of evaluation(s) done such as formative evaluation, implementation evaluation, or pre-post tests, or other types” (p. 21).

13. When working with younger adolescents (e.g., 5th and 6th graders), can we ask preliminary sex behaviors, such as kissing or touching, rather than sexual behaviors?

Or perhaps ask behaviors such as alcohol use instead of sexual behaviors?

You may collect pre-sexual behaviors and alcohol use in addition to a sexual risk behavior, but the pre-sexual behavior should not be counted as a sexual risk behavior.

Section A.2.b. states that “Recipients may collect additional outcomes beyond those recognized currently by the TPPER, as long as at least one of the study outcomes is a sexual risk behavior, and the other outcomes collected are ‘associated risk factors’ clearly associated with unintended teen pregnancy” (p. 8).

Please also refer to Section D.3.1.2.b. (p.22), which describes what you should include in your application to justify your proposed research questions and outcomes. In addition, Federal staff and an independent review panel will assess all eligible applications on the following “At least one research question assesses the intervention’s impact on a sexual risk behavior associated with unintended teen pregnancy.” (p. 43)

Since this is a competitive funding announcement, OPA cannot provide input regarding the content of individual grant applications and is therefore unable to answer questions specific to the content of an application. Applicants are encouraged to carefully review the NOFO and should put forward the best proposal the organization can create based on application and evaluation criteria published in the announcement.

14. Is there a way to search current and past TPP Tier 2 funded programs by target populations, to ensure that we are not duplicating previous work?

Summaries of past TPP Tier 2 funded programs are not searchable by target or priority population. To view current and previous TPP Tier 2 funded grantees, please reference the following links, which can be found in Section A.1. of the NOFO (p. 7):

• Phase 2 Rigorous Evaluation of Promising TPP Interventions (TPP20 Tier 2 Phase

2) – (https://opa.hhs.gov/grant-programs/teen-pregnancy-prevention-program-tpp/tpp-grantees/current-tpp-grantees ),

• Rigorously Evaluating New TPP Approaches (Tier 2b) – 2015-2020 (https://opa.hhs.gov/grant-programs/teen-pregnancy-prevention-program-tpp/tpp-grantees/past-tpp-grantees-2015-2020),

• TPP FY2010 Tier 2 Research and Demonstration (https://opa.hhs.gov/sites/default/files/2020-07/summary-researchdemonstration.pdf), and

• Personal Responsibility Education Program Innovative Strategies (PREIS) funding (FY2010, FY2015, FY2019) (https://www.acf.hhs.gov/fysb/grant-funding/personal-responsibility-education-innovative-strategies-preis-grantee-profiles)

15. Is changing an existing EBP (and past OPA funded) program's delivery method from in person to virtual considered a "major change to delivery mechanism" that would be deemed appropriate for rigorous evaluation under this Tier 2 NOFO?

Since this is a competitive funding announcement, OPA cannot provide input regarding the content of individual grant applications and is therefore unable to answer questions specific to the content of an application.

Applicants are encouraged to carefully review the NOFO and should put forward the best proposal the organization can create based on application and evaluation criteria published in the announcement.

16. Can you define what you consider "sexual risk behaviors"?

Sexual risk behaviors may include but are not limited to “initiation of sexual activity, frequency of sex, number of sexual partners, contraceptive use, STIs, pregnancies, births, etc.” (Section D.3.a.2.a, p. 22). You may also refer to the HHS Teen Pregnancy Prevention Evidence Review (TPPER), version 6.0.

(https://tppevidencereview.youth.gov/ReviewProtocol.aspx) which outlines the measures that studies must demonstrate impacts on as it relates to sexual risk behaviors or its health consequence.

17. Does the intervention need to be selected before applying, or can we decide on that during the planning period?

Yes, the intervention will need to be selected before applying. Section D.3.a.1, Significance of the Proposed Intervention states, “Identify your intervention by name and clearly and succinctly describe the who, what, when, where, why, and how of your intervention” (p. 20).

18. Would the youth-level outcomes we collect in the study have to be self-reported measures?

It is not required that youth-level outcomes be self-reported. The youth-level outcomes may be based on a self-reported measures or the applicant may use secondary or administrative data for the analysis. Please see Data Collection Plan and Outcomes, Section D.3.a.2..e. (p.

23) for more information. Please refer to the Section E.1.c (p. 43) for the relevant scoring criteria that reviewers will use to assess your application’s evaluation design methodology.

19. With regard to positive preliminary evidence from a previous formative evaluation, would this include qualitative data collected during a formative evaluation?

https://tppevidencereview.youth.gov/ReviewProtocol.aspx

Yes, the positive preliminary evidence may include but is not limited to qualitative data collected during a formative evaluation. “We expect recipients to implement interventions that have compelling, positive preliminary evidence from previous formative evaluation and prior research, including documented effects on intended outcomes” (p. 8).

Please be sure to review Section E.1 - Criteria. There you will find the criteria that will be used by federal staff and an independent review panel to assess all eligible applications (p.42-45). Applicants are encouraged to carefully review the NOFO and should put forward the best proposal the organization can create based on application and evaluation criteria published in the announcement.

20. Is there any requirement around independence of the evaluation or lead evaluator?

We expect recipients to “conduct their studies through a credible and neutral process, which includes, at a minimum, evaluation staff who do not have perceived or actual conflict of interest and can remain independent of the intervention. We strongly encourage use of an independent, external evaluator” (Section A.2.b., p. 9).

21. Does OPA conduct a separate medical accuracy review in addition to the grantee-led review by subject-matter experts?

Yes. We expect recipients to demonstrate that subject matter experts have reviewed the materials, in order to ensure that all materials delivered to study participants within the funded project are medically accurate, age appropriate, culturally, and linguistically appropriate, trauma-informed, user-centered, and inclusive of all youth. We also expect recipients to submit all program materials to OPA for a medical accuracy review. Recipients may not begin implementation with the materials until after OPA has approved the use of the materials. See Section A.2.d – Complete OPA Material Review (p.10-11) for more information.

22. Please confirm the ages of youth eligible for an intervention.

Primary participants receiving the intervention evaluated under an award should be adolescents and youth, their caregivers, or youth-serving professionals (p.6). For purposes of this NOFO, OPA is defining adolescents and youth to be individuals between the ages of 10-24.

23. How many research questions should an applicant propose?

There is no specified minimum or maximum number of research questions mentioned within the NOFO. Federal staff and an independent review panel will assess all eligible applications on the extent to which (p.43):

The proposed research questions are well-reasoned, focused, well-aligned with the intervention’s theory of change, and clearly address the reduction of unintended teen pregnancy, behavioral risk factors underlying unintended teen pregnancy, or other associated risk factors for unintended teen pregnancy. At least one research question assesses the intervention’s impact on a sexual risk behavior associated with unintended teen pregnancy. At least one research question assesses the intervention’s impact on a sexual risk behavior associated with unintended teen pregnancy. Behavioral risk factors may include sexual activity, number of sexual partners, contraceptive use, sexually transmitted infections (STIs), etc. Other risk factors may include mental health, educational attainment, healthy relationships, intimate partner violence, substance use, etc.

Since this is a competitive funding announcement, OPA cannot provide input regarding the content of individual grant applications and is therefore unable to answer questions specific to the content of an application. Applicants are encouraged to carefully review the NOFO

24. What does Cooperative Agreement mean?

Awards made under this NOFO will be cooperative agreements. A cooperative agreement is a form of assistance that allows for substantial involvement by OPA. Substantial involvement is in addition to the usual monitoring and technical assistance provided under a grant.

Section B.3 - Federal Agency Substantial Involvement (p.15) outlines what may be included as “substantial programmatic involvement.”

25. To be considered a “robust design”, how large does sample size need to be?

There are no specified sample sizes; your team should propose a sample based on power calculations and justify it within the evaluation design section of your application. The design should be feasible for the intervention in the setting and with the intended population.

Smaller sample sizes may be acceptable if the proposed design and analysis has sufficient rigor and is capable of producing evidence of effectiveness. Please see Section D.3.a.2.h - Impact Evaluation Design - Sample Size (p.24) for more information. When feasible, we expect the evaluation design to meet the criteria for the quality of an evaluation study per the criteria established in the HHS Teen Pregnancy Prevention Evidence Review (TPPER), version 6.0. (https://tppevidencereview.youth.gov/ReviewProtocol.aspx) (p. 8).

Since this is a competitive funding announcement, OPA cannot provide input regarding the content of individual grant applications and is therefore unable to answer questions specific to the content of an application. Applicants are encouraged to carefully review the NOFO

26. What is an OPA-directed Federal evaluation?

An OPA-directed Federal evaluation is an evaluation conducted by an OPA-funded contractor. Examples of previous federal studies conducted by OPA include a cost study, a meta-analysis, effectiveness studies, and implementation studies. As a condition of the award, we expect recipients to participate in in an OPA-directed Federal evaluation, if selected and funding for such an evaluation becomes available (p.9).

27. Do we need or are we expected to have Institutional Review Board (IRB) approval at the start of the grant?

You are not required to have IRB approval at the start of the grant. However, you are expected to have approval from an IRB by the end of the planning period (p.10).

https://tppevidencereview.youth.gov/ReviewProtocol.aspx

In your application, include a proposed timeline for acquiring the approval of the IRB for the proposed activities. Any award based on your application does not supersede the need for IRB review, approval, or determination of an exemption. Do not include in your application the package prepared for your IRB review (p.23).

28. What does OPA mean by expecting “rigorous impact evaluations to have the most robust possible design that is feasible for the intervention in the setting and with the intended population”?

OPA defines rigorous impact evaluation as “either a randomized control trial (RCT) or a quasi-experimental design (QED) with counterfactual condition; the most robust possible design that is feasible for the intervention (p.66).” Please see the Glossary of Selected Terms and Definitions (starting on p.62) for more information on how OPA defines randomized control trial (RCT) and quasi-experimental design (QED). The definition for QED also includes examples.

You should propose an appropriate study and justify the rigor by describing your processes for forming study groups, collecting data, etc. You are encouraged to incorporate analyses for small sample sizes such as Bayesian methods. Please refer to the evaluation design section of the NOFO for more information (Section D.3.a.2 – Impact Evaluation Design, p.21-25).

29. What sorts of interventions are eligible to be evaluated under this funding opportunity?

We are seeking evaluations of interventions that address key adolescent health disparities and have the potential to make a significant contribution to preventing unintended teen pregnancy. We anticipate funding a wide range of adolescent health innovations under this funding announcement; the eligible interventions may include, but are not limited to, programs, curricula, clinical interventions, technological interventions, systems-level interventions, community-level approaches, and others. Interventions may work directly with adolescents, their caregivers, and/or with youth-serving professionals to address adolescent health behaviors relevant to teen pregnancy prevention. We expect any intervention proposed for evaluation under this initiative to already have compelling, positive preliminary evidence from previous formative evaluation and prior research, including documented effects on intended outcomes. We expect the intervention to have demonstrated support from previous participants and communities. The intervention should fit the population and setting proposed for the application. Interventions should have a clearly described theory of change and logic model (See Section 2.a.) (p.6).

30. What interventions are not appropriate to be evaluated under this funding?

TPP Interventions rigorously evaluated previously under federal funding are not appropriate for this funding unless significant changes have been made to the intervention to address findings from the previous evaluation results. See page 7 of the NOFO for examples of previous federal funding.

Interventions identified as effective by the HHS Teen Pregnancy Prevention Evidence Review (TPPER) are also not appropriate for this funding

(https://tppevidencereview.youth.gov/FindAProgram.aspx), unless significant adaptations are proposed to the intervention, ultimately creating a new intervention (p.7).

31. Do we need to include all Memoranda of Agreement (MOAs) in our application?

If available at the time of submission, signed Memoranda of Agreement (MOAs) or signed Letters of Commitment (LOCs) may be submitted for each partner (or one signed MOA with all partners) and include specific roles, responsibilities, resources, and contributions of partner(s) to the project. Submit a signed MOA or LOC from the intervention developer/purveyor/copyright holder indicating that you may use the materials as described within the application in the project (p.35).

The signed LOCs must detail the specific role and resources that will be provided, or activities that will be undertaken, in support of the applicant. The organization’s expertise, experience, and access to the targeted population(s) should also be described in the LOC.

Fully-executed MOAs may be required within the first 30 days following the start of the project of any award made under this announcement (p.36).

Please note that Federal staff and an independent review panel will assess all eligible applications on the extent to which “Signed Memoranda of Agreement (MOAs) or Letters of Commitment (LOCs) within the appendices support the partnership and demonstrate the level of commitment needed for the project to be successful.” (p.44)

Budget

1. What percent of the budget should go to evaluation?

There is no specified percentage of the budget that should go to evaluation. Please refer to page 28 of the NOFO which states, "Your budget narrative should justify the overall cost of the project as well as the proposed cost per activity, service delivered, and/or product. For example, the budget narrative should define the amount of work you have planned and expect to perform, what it will cost, and an explanation of how the result is cost effective."

2. Can we request additional funding to cover rising costs in out-years, even if it exceeds the maximum allowable amount?

The award ceiling is $1,000,000 per budget period. Recipients will be required to submit a non-competing continuation application for each budget period after the first. Funding for all approved budget periods beyond the first is generally level with the initial award amount and is contingent upon the availability of funds, satisfactory progress of the project, adequate stewardship of Federal funds, and the best interests of the Government. (See p. 14).

3. Should our current approved indirect rate be used for all years of the budget?

You must state the method you are selecting for your indirect cost rate (p. 28). You should enclose a copy of the current approved rate agreement in your Budget Narrative file. If you have multiple approved rates, indicate which rate as described in your approved agreement https://tppevidencereview.youth.gov/FindAProgram.aspx is being applied and why that rate is being used. The de minimis rate method only applies if you have never received an approved negotiated indirect cost rate from HHS or another cognizant federal agency. If you are waiting for approval of an indirect cost rate, you may request the 10% de minimis rate. If you choose this method, costs included in the indirect cost pool must not be charged as direct costs to the award (p. 32). Please note that the approved rate has to be used in all years of the project.

4. Are there caps on any areas of the budget? That is, within the total budget, other than indirect costs, are there caps on areas such as evaluation, program, or technology?

The only specific limitations noted regarding the total budget are stated on:

• Page 32 of the NOFO - “Indirect costs on Federal awards for training are limited to a fixed rate of eight percent of Modified Total Direct Costs (MTDC) exclusive of tuition and related fees, direct expenditures for equipment, and subawards in excess of $25,000 (45 C.F.R. § 75.414 (c)(1)(i))” and

• Page 40, item B Salary Rate Limitation - “You should not budget award funds to pay the salary of an individual at a rate in excess of Federal Executive Pay Scale Executive Level II. As of January 2023, the Executive Level II salary is $212,100.

This amount reflects an individual’s base salary exclusive of fringe benefits and any income that an individual working on the award project may be permitted to earn outside of the duties to the applicant organization. This salary rate limitation also applies to subawards/subcontracts under an HHS/OASH award.”

However, your budget narrative should describe how the categorical costs are derived and discuss the necessity, reasonableness, and allocation of the proposed costs (p.28).

5. How do I obtain a negotiated indirect cost rate with the Federal Government?

Contact the U.S. Department of Health and Human Services Cost Allocation Services regional office that is applicable to your state. Find out more about this process by visiting:

https://www.hhs.gov/about/agencies/asa/psc/indirect-cost-negotiations/index.html

6. Are renovations allowed under this funding opportunity?

No.

7. The NOFO lists an award floor of $500,000 and an award ceiling of $1,000,000 are these totals referring to the total grant or annual amounts?

Those totals are annual amounts for each year which is a budget period.

8. Is there a maximum percentage of the annual budget that is allowable for evaluation expenses?

No. The NOFO does not state a minimum or maximum for the evaluation expenses.

However, recipients will be required to provide a summary narrative and line-item budget for https://www.hhs.gov/about/agencies/asa/psc/indirect-cost-negotiations/index.html each year beyond the first. For categories or items that differ significantly from the first budget year, provide a detailed justification explaining these changes (p.28).

9. Where can I find information regarding unallowable costs?

Please be sure to carefully review Section D.7 Funding Restrictions (beginning on p.39) for specific information regarding allowable, unallowable, and restricted costs. You can also look under the guidance within GENERAL PROVISIONS FOR SELECTED ITEMS OF COST and through the following link: https://www.ecfr.gov/current/title-45/subtitle- A/subchapter-A/part-75/subpart-E/subject-group-ECFR5d90ba314caea08

10. Can an applicant include in their budget request incentives for participants to participate in the evaluation (whether control or intervention)?

Yes, participants can be provided incentives for evaluation participation. The program participation cost and or activities must be in support of the goals and objectives of the NOFO. Please see 45 CFR §75.456 - Participant support costs. Participant support costs are defined as direct costs for items such as stipends or subsistence allowances, travel allowances, and registration fees paid to or on behalf of participants or trainees (but not employees) in connection with conferences, or training projects.

https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-A/part-75/subpart-E/subject-group-ECFR5d90ba314caea08/section-75.456

If including incentives as part of your budget request, you must include a plan for oversight of federal award funds. The plan must describe for any program incentives proposed, the specific internal controls that will be used to ensure only qualified participants will receive them and how they will be tracked.

11. Can youth who participate on an advisory board for the grant project or who serve as advisors for the grant project be provided with stipends?

Stipends for participants of an advisory board/council (such as a youth advisory board or community advisory board) and other costs associated with the advisory board/council may be allocable and charged directly to the TPP award if the aim of the advisory board/council is to meet the expectations of the NOFO. This includes providing guidance on the design, implementation, and monitoring of the project. The scope of work and related budget narrative must describe the youth advisory board and/or community advisory board’s membership, functions, and costs and explain why the advisory board is necessary to carry out the TPP funded project.

12. Does the application require cost sharing or match funds?

No, it is not a requirement (p.16).

13. Is Budget and Budget Narrative required for each year or just Year 1?

Provide a budget justification, which includes explanatory text and line-item detail, for the entire first year of the proposed project. For subsequent budget years in an anticipated multi-year project, provide a summary narrative and line-item budget for each year beyond https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-A/part-75/subpart-E/subject-group-ECFR5d90ba314caea08 https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-A/part-75/subpart-E/subject-group-ECFR5d90ba314caea08 https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-A/part-75/subpart-E/subject-group-ECFR5d90ba314caea08/section-75.456 https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-A/part-75/subpart-E/subject-group-ECFR5d90ba314caea08/section-75.456 the first. For categories or items that differ significantly from the first budget year, provide a detailed justification explaining these changes (p.28).

Funding for all approved budget periods beyond the first is generally level with the initial award amount and is contingent upon the availability of funds, satisfactory progress of the project, adequate stewardship of Federal funds, and the best interests of the Government (p.14).

14. Clarifying that the 100-page limit does not include budget and budget narrative. Only narrative and appendices.

Your total application (i.e., the Project Narrative plus Appendices) must not exceed 100 pages. The following items do not count toward the Project Narrative page limit:

• All required forms, including SF-424, SF-424A, SF-LLL

• Project Abstract Summary

• Budget Narrative (including budget tables)(Section D.2.a) (p.17).

15. Are indirect costs limited to 8% of MTDC for all applicants even if the applicant’s organization has an approved rate that is higher?

Page 32 of the NOFO indicates that the indirect cost category has one of two methods that you may select. You may only select one and must clearly identify that selection in your submitted budget. These methods are as follows:

• Your organization currently has an indirect cost rate approved by the Department of Health and Human Services (HHS) or another cognizant federal agency. You should enclose a copy of the current approved rate agreement in your Budget Narrative file. If you request a rate that is less than allowed, your authorized representative must submit a signed acknowledgement that the organization is accepting a lower rate than allowed.

• Per 45 C.F.R. § 75.414 (f) Indirect (F&A) costs, “any non-Federal entity [i.e., applicant] that has never received a negotiated indirect cost rate, … may elect to charge a de minimis rate of 10% of modified total direct costs (MTDC) which may be used indefinitely. As described in § 75.403, costs must be consistently charged as either indirect or direct costs but may not be double charged or inconsistently charged as both. If chosen, this methodology once elected must be used consistently for all Federal awards until such time as a non-Federal entity chooses to negotiate for a rate, which the non-Federal entity may apply to do at any time.”

Note that the de minimis rate method only applies if you have never received an approved negotiated indirect cost rate from HHS or another cognizant federal agency. If you are waiting for approval of an indirect cost rate, you may request the 10% de minimis rate. If you choose this method, costs included in the indirect cost pool must not be charged as direct costs to the award. Indirect costs on Federal awards for training are limited to a fixed rate of eight percent of MTDC exclusive of tuition and related fees, direct expenditures for equipment, and subawards in excess of $25,000 (45 C.F.R. § 75.414 (c)(1)(i)).

Page 32 also explains how applicants should provide justification for and the calculation of their indirect costs total.