SSb_Ekwueme et al .pdf

This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and education use, including for instruction at the authors institution and sharing with colleagues.

Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited.

In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier’s archiving and manuscript policies are encouraged to visit:

http://www.elsevier.com/copyright http://www.elsevier.com/copyright

Author's personal copy

Evaluation and Program Planning 31 (2008) 136–144

Identifying and controlling for program-level differences in comparative cost analysis: Lessons from the economic evaluation of the National

Breast and Cervical Cancer Early Detection Program$

Sujha Subramaniana,�, Donatus U. Ekwuemeb, James G. Gardnerb, Bela Bapata, Caren Kramera aRTI International 1440 Main Street-Suite 310 Waltham, MA 02451-1623, USA bDivision of Cancer Prevention and Control Centers for Disease Control and Prevention (CDC), 4770 Buford Hwy, NE Atlanta, GA 30341-3724, USA

Received 17 May 2007; received in revised form 7 November 2007; accepted 1 February 2008

Abstract

Performing economic evaluations of established health care programs is essential to identify and control for underlying program-level variations and to make valid comparisons. At a time when the need for such evaluations is growing, health care professionals have limited information on the methodological challenges of performing these evaluations. In this study, we used the National Breast and

Cervical Cancer Early Detection Program to illustrate these potential underlying variations. We performed site visits to four grantees and collected activity-based cost data from nine additional representative programs. We identified five specific types of cost factors that should be considered when evaluating and comparing health care programs: clinical services, service mix, in-kind contributions, indirect costs, and year-to-year expenditures of specific activities. A key lesson is that case studies and pilot testing should be performed before initiating cost analysis to identify underlying variation and to test appropriate methods to adequately control for these differences.

r 2008 Elsevier Ltd. All rights reserved.

Keywords: Economic evaluation; Cancer screening; Breast cancer; Cervical cancer; Cost of cancer screening

1. Introduction

Economic evaluation methods can provide a framework that helps policymakers and stakeholders make informed decisions about health care programs (Drummond, Scul-pher, Torrance, O’Brien, & Stoddart, 2005). The methods consist of cost analysis, cost–benefits analysis, cost-effec-tiveness analysis, and cost–utility analysis (Drummond et al., 2005; Gold, Siegel, Russell, & Weinstein, 1996;

Haddix, Teutsch, & Corso, 2003). These economic evalua-tion methods can be performed using a variety of study designs that include randomized clinical trials, prospective interventions embedded in the real-world setting, and retrospective assessments. In randomized trials, observed and unobserved differences between health care programs can be eliminated, or at least significantly reduced, and in prospective intervention studies specific processes can be adopted to minimize the impact of any such differences in the study design. When performing retrospective assess-ments, the underlying variations in program components that may impact cost or effectiveness must be accounted for in order to perform valid comparisons (Benson & Hartz, 2000; Concato, Shah, & Horwitz, 2000; ).

In the past decade, the need to evaluate established health care programs has grown. With the introduction of the Program Assessment Rating Tool (PART) in 2004, emphasis has been placed on assessing previously funded federal programs. PART systematically evaluates and ranks federally funded programs to assess whether a program has procedures in place to measure the

ARTICLE IN PRESS

www.elsevier.com/locate/evalprogplan

0149-7189/$ - see front matter r 2008 Elsevier Ltd. All rights reserved.

doi:10.1016/j.evalprogplan.2008.02.002

$The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease

Control and Prevention (CDC).

�Corresponding author. Tel.: +1781 434 1749; fax: +1 781 434 1701.

E-mail address: ssubramanian@rti.org (S. Subramanian).

cost-effectiveness of the program and to achieve efficiencies (PART, 2006). However, assessing established federal programs poses special challenges. Systematic variations in underlying factors between programs that offer the same services may be present because the program components were designed in the past and because the intervention or service delivery is ongoing. In these assessments, investi-gators do not have the opportunity to prospectively design the programs to be evaluated. Rather, they have to use appropriate methods to derive cost estimates that provide valid comparisons across programs. Economic evaluation is the comparative analysis of alternative courses of action in terms of both their cost and consequences. Therefore, for valid comparisons of health care programs, cost measures must be comparable. A critical and essential step in assessing the true cost-effectiveness of health care programs is to identify and control for systematic differences in program components to derive comparable cost estimates.

To date, no studies have systematically assessed the impact of underlying program-level components or factors on the operating costs of health care programs and the methodological challenges posed by these factors. The methodology to account for differences in patient-level factors such as age and sex is well established in health eco-nomics and health services research literature (Diez-Roux, 2000; Greenland & Morgenstern, 2001; Winship & Morgan, 1999). For example, differences in patient-level factors are controlled for by using risk-adjustment scores (Iezzoni, 2004; Keenan, Buntin, McGuire, & Newhouse, 2001). In addition, instrumental variables and propensity scores can be used to control for underlying differences between the study and control groups. A few program-level factors have also been traditionally accounted for in economic evaluations. For instance, the underlying differ-ences in unit cost among programs attributable to geographic location have been generally controlled for using the Consumer Price Index (CPI) (Drummond et al., 2005). Because ongoing health care programs often have complex structures and service delivery mechanisms, other factors that are not traditionally considered in economic evaluations need to be identified and controlled for so as to generate cost estimates that allow for valid comparisons across programs.

The objective of this study is twofold—to identify the underlying program-level factors that need to be controlled for in comparative assessments of established health care programs, so as to generate comparable cost estimates and to develop methods to minimize the impact of these factors. We used the National Breast and Cervical Cancer Early Detection Program (NBCCEDP) to illustrate the types and the magnitude of these potential factors. The NBCCEDP is federally funded and provides free cancer screening to low-income, underinsured women throughout the United States. The complete details of the NBCCEDP have been described elsewhere (Benard et al., 2004;

Eheman et al., 2006; Ryerson, Benard, & Major, 2005).

Briefly, the NBCCEDP was established in 1990; it provides clinical breast examinations, mammograms, and Pap tests to eligible women and diagnostic testing for those women whose screening outcome is abnormal. Through coopera-tive agreements, the Centers for Disease Control and Prevention (CDC) fund 68 locally administered screening programs that have diverse service delivery structures.

Most of these programs have been in existence for more than 10 years. Thus, an evaluation of these 68 programs offers a unique opportunity to perform an in-depth exploration of the methodological issues that need to be considered in performing valid cost comparisons.

2. Methods

We performed both a quantitative and qualitative assessment to identify potential factors that need to be considered in comparing the costs of the 68 NBCCEDP programs. We visited four different programs to undertake an in-depth case study of each program. We selected programs that differed in their screening delivery structure (centralized, decentralized, or mixed structure), in the volume of women screened, and in geographic location.

Centralized programs offer clinical services in satellite offices, but all other activities are performed centrally, such as tracking and case management services. Decentralized programs, on the other hand, contract with local and regional health departments, private hospitals, or other health care facilities to fully manage and provide screening and diagnostic services and other activities. Programs with mixed service delivery structures have both centralized and decentralized features.

The key objective of the site visits was to gather information to describe each program’s service delivery structure, identify potential factors that could impact the cost per woman screened, and to assess appropriate methods to collect costs related to the NBCCEDP. We used a detailed protocol and a site visit questionnaire to guide our interviews at each program site to ensure consistency across the four sites. The site visit question-naire contained open-ended questions on the program’s infrastructure, screening delivery structure, outreach activ-ities, data management systems and fiscal databases, funding sources, type of staff employed and activities performed, non-labor expenditure, and overhead cost allocation. We met the program director and financial officer of each program, and visits were performed in March and April of 2004. We compiled detailed transcrip-tions of the interviews that were conducted during each of the visits and systematically compared the responses from all four programs for each question in the interview guide.

Qualitative data on each question were then compiled by section to gain a better understanding of the similarities and differences between the four programs.

We used two sources to obtain the necessary quantitative data. First, we developed a cost estimation question-naire to collect activity-based costs from the programs.

ARTICLE IN PRESS

S. Subramanian et al. / Evaluation and Program Planning 31 (2008) 136–144 137

Activity-based costing assigns cost to specific program activities. We derived activity-based cost estimates by allocating total cost to specific activities or components of the NBCCEDP. Deriving activity-based costs is important as they provide more in-depth information than the total cost to compare across programs and understand the distribution of cost across the key NBCCEDP activities. The cost estimation questionnaire was based on well-established methods for collecting cost data for health care program evaluation (French, Dennis, McDougal, Karuntzos, & Hubbard, 1992; French, Roebuck, & McLellan, 2004; French, Dunlap, Zarkin, McGeary, & McLellan, 1997; Salome, French, Miller, & McLellan, 2003). Using the information that was collected during the visits, we developed sections specific to the NBCCEDP.

For example, we learned that for decentralized programs we needed to collect details on outreach activities both at the central location and at satellite offices, and therefore we developed a specific section that addressed this need. The cost estimation questionnaire collected expenditure data related to personnel, consultants, cancer screening and diagnosis, case management, contracts, and administrative or overhead activities, such as telephone and rent. For these expenditure categories, we allocated program costs to the following NBCCEDP activities: management, screen-ing, case management, tracking and follow-up, public education and outreach, professional education, coalition and partnership, quality assurance and improvement, and surveillance and evaluation. The four programs provided details on the distribution of both labor and non-labor costs to the activities performed in order to appropriately allocate the expenditure. For example, each staff member provided details on the proportion of time spent on NBCCEDP activities. We also obtained payment data for each screening and diagnostic procedure performed by the programs.

The questionnaire was administered to nine NBCCEDP programs. We used a combination of random selection and purposive sampling to identify a representative group of nine programs. The program selection process is outlined in Fig. 1. We began with the 51 programs in each of the states and the District of Columbia. We excluded programs administered by tribes and territories because they are generally the smallest programs and also operate through unique service provision structures. From the 51 programs, we excluded the 4 site visit programs. The remaining 47 programs were stratified into decentralized, mixed, and centralized programs; we then randomly selected six programs from each stratum. From each group of six programs, identified for each type of delivery structure, we manually selected three programs to ensure diversity across all the programs in geographic location, size of the area served, number of women screened, service mix (the proportion of breast and cervical cancer screenings), and type of funding source. Overall, we selected nine programs; of these, three were centralized, three decen-tralized, and three mixed.

The questionnaire was administered in Microsoft Excel to the nine programs, and the cost data were collected retrospectively for the 12-month period from July 2003 to June 2004. We emailed the survey instrument to the nine programs and organized a conference call to introduce the

ARTICLE IN PRESS

Stratified by delivery mechanism:

decentralized, mixed, centralized.

51 programs*

6 mixed programs

The 4 programs that participated in site visits were excluded

Selected 3 programs from each service delivery type, ensuring optimal variation among the final choices.

Final 9 programs (3 decentralized, 3 mixed, 3 centralized)

Randomly selected 6 from each group

20 decentralized programs

16 mixed programs

11 centralized programs

47 programs remaining

6 decentralized programs

6 centralized programs

Fig. 1. Program selection flowchart. *Programs administered by tribes and territories were not included.

S. Subramanian et al. / Evaluation and Program Planning 31 (2008) 136–144138 questionnaire and answer any questions. The questionnaire collected data on the following: funding, in-kind contribu-tions, labor expenditure, consultants, contacts, equipment/ supplies, and indirect cost. We also provided each program with a detailed user’s guide to assist them in completing the questions. All nine programs completed the questionnaire simultaneously and on average required 22 h. The data manager at each site served as the key liaison for completing the questionnaire and was assisted by the fiscal staff and the program director as required. The cost data provided by the programs were compared with the year-end fiscal status report to verify total expenditure for the annual period. Overall, the programs were able to allocate more than 90% of the total cost to specific NBCCEDP activities. Upon completion of the analysis, we held a final conference call with the participants to share the findings from the cost data collection and solicit feedback on the questionnaire content and format.

Using the information collected from the nine programs, we calculated total program cost (mean, range, and 95% confidence intervals), the cost of clinical services for mammogram and Pap tests, and in-kind contributions to the programs. From these calculations, we report both the in-kind contributions as reported by the programs and the standardized in-kind contributions based strictly on the market value of resources donated to each program (opportunity cost). The programs provided details on labor and non-labor in-kind contributions, their estimated value in dollars, and the method used to estimate the value of these resources. To derive consistent estimates across the programs, we used the same valuation approach. In addition, we excluded contributions that did not reflect true opportunity cost from the list of in-kind contributions reported by the programs. For example, some programs claimed the differential between physician charge and reimbursement as an in-kind contribution. Because physi-cians are generally not paid the amount charged, this differential is not an opportunity cost and we excluded these contributions from the standardized estimates. We also report the percentage of total cost spent on profes-sional education and on public education and outreach.

The purpose of reporting costs spent on professional and public education and outreach is to illustrate the variation in cost among the programs.

The second source of quantitative data was the minimum data elements maintained by CDC. In this database, each of the 68 NBCCEDP programs provides data on the women, who are enrolled in their program, semi-annually.

The database contains information on demographics, types of screening and diagnostic services, and final diagnosis and outcomes of the women who are served by the NBCCEDP. From this database, we abstracted information on the number of women screened and the proportion of women requiring follow-up testing. This information was used to derive estimates of the service mix for the programs. In this study, we report the proportion of breast cancer and cervical cancer screenings, the mean, the range, and 95% confidence intervals for each of the nine programs that were involved in pilot testing the questionnaire.

We also reviewed relevant literature in the fields of health economics, technology assessment, and program evaluation and consulted experts in these fields to identify appropriate methods to control for variations. We inter-viewed individuals with expertise in performing economic evaluation to identify potential approaches that can be used to control for differences among the NBCCEDP programs. Whenever possible, we evaluated the use of the methods identified to control for differences. For example, we assessed the use of the regional CPI to control for unit price differences in the cost of mammograms among the programs.

3. Results

Our analysis of the quantitative and qualitative data gathered from the nine program sites revealed five specific types of cost factors that should be accounted for to perform valid cost comparisons across the programs:

differences in the cost of clinical services, in the service mix, in the magnitude of in-kind contributions, in the allocation of indirect costs as fixed overhead versus direct charges, and in the differences of expenditures on specific activities. Each of these factors is discussed in detail below.

3.1. Cost of clinical services

As shown in Table 1, we found differences in the average cost (i.e. amount paid by a program to providers) of a mammogram among the nine programs; the cost per mammogram ranged from $73.25 to $90.76. Screening costs related to mammograms are generally a large proportion of the total program cost, and thus any differential in reimbursement rates for screening proce-dures should be fully assessed.

The maximum reimbursement rate for the services covered by the NBCCEDP is the rate allowed by Medicare for the screening procedures in a given locality. The Medicare reimbursement rate varies by locality, because the payment amounts are adjusted to reflect the variation in costs in a given area. Thus, we expected variations in the payment rates to reflect the underlying cost of living in a given locality. Although a large proportion of the providers are reimbursed at the Medicare rate, some programs negotiate and pay rates lower than the Medicare reimbur-sement. During our site visits and subsequent conversa-tions with the programs, we learned that several factors can affect the ability of a particular program to negotiate lower rates, including the availability or supply of mammography centers, prior relationship of the program with the provider, and the volume of women who receive screening services. Another factor that influences whether a parti-cular program negotiates for a lower rate is the differential

S. Subramanian et al. / Evaluation and Program Planning 31 (2008) 136–144 139 lag time in updating Medicare reimbursement rates.

Therefore, for a specific annual period, program variations may exist beyond the anticipated differences in Medicare cost adjustments.

The variation in input prices in economic assessments is usually controlled for by using the regional CPI. Because Medicare rates are generally better approximated by the overall CPI, researchers have generally used the overall, or ‘‘all item’’, CPI rather than the medical care component of the CPI (Baicker & Chandra, 2004; Mansley, Dunet, May, Chattopadhaya, & McKenna, 2002). For completeness, we compared the results using both the overall CPI and the medical CPI and did not find significant differences in the adjusted cost. In Table 1, we present the cost of mammograms adjusted for regional cost differences using the overall CPI. The adjusted cost ranged from $69.00 to $85.49; this range is slightly narrower than the unadjusted cost that ranged from $73.25 to $90.76. Overall, after applying the regional CPI, we found that the variation remained in the average cost of a mammogram among the nine programs. This finding may suggest that significant differences do exist between the programs in reimburse-ment rates or that the regional CPI does not adequately adjust for program-level differences. In either case, a variation of $16 can lead to significant cost differences based on the number of women screened. For example, screening 10,000 women would result in a total cost difference of $160,000. A potential solution to address this difference in unit cost is to use state-specific cost-of-living adjustments (Leonard & Walder, 2000) rather than regional CPI, because these adjustments may be more sensitive in controlling for cost of living variations. In addition, sensitivity analysis can be performed by applying the national Medicare reimbursement rate to all programs to assess the impact of reimbursement differences on the cost of the NBCCEDP.

3.2. Differences in service mix

The NBCCEDP provides both cervical (Pap test) and breast (mammography) cancer screening, and the propor-tion of these services, or service mix, varied among the programs. Because a mammogram is significantly more expensive than a Pap test, those programs with a higher proportion of breast cancer screenings will have higher overall costs. The average cost of a mammogram was $80.95 and $14.83 for a Pap test. To explore the impact of this differential on total and average costs, we com-pared the service mix across the programs. As reported in Table 1, the proportion of breast and cervical cancer screenings ranged from 39.2% to 61.5%. This finding indicates that some programs provided fewer mammo-grams than Pap tests, and others had the opposite service mix. Thus, any comparative analysis of the programs should control for this mix of screening tests. In multi-variate analysis, for instance, the proportion of breast cancer screens can be included as an independent variable to account for such differences.

Besides the mix of breast and cervical screenings, we also learned during our site visits that programs may enroll women with an abnormal screening result who had already had a mammogram or Pap test outside the NBCCEDP.

These women require a diagnostic examination and expensive follow-up tests, including diagnostic mammo-grams and colposcopies that add significantly to the overall program costs. Thus, an NBCCEDP program that enrolls a large number of women who require diagnostic examinations but were actually screened outside of the NBCCEDP incur higher costs for serving fewer women.

As a result, these programs appear to have a higher average cost per woman served than do those programs that do not enroll women with abnormal test results. This is another difference in service mix that needs to be controlled for

ARTICLE IN PRESS

Table 1

Cost of clinical services and service mix

Program Average cost of a mammogram ($)

Cost of mammogram:

adjusted for regional cost differencesa ($)

Average cost of a Pap smear ($)

Proportion of breast screens compared to cervical screens (%)

1 90.76 85.49 14.76 53.3

2 80.94 80.59 14.76 61.5

3 78.48 78.48 14.76 60.0

4 74.04 74.04 14.76 39.2

5 73.25 69.00 15.40 61.2

6 82.77 82.77 14.76 57.0

7 76.69 76.35 14.76 54.8

8 82.05 74.51 14.76 57.5

9 89.53 84.33 14.76 57.5

Range 73.25–90.76 69.00–85.49 14.76–15.40 39.2–61.5

Mean 80.95 78.40 14.83 55.8

95% CI 76.91–84.98 74.85–81.95 14.69–14.97 51.3–60.2

Note: CI ¼ confidence interval.

aAll regions adjusted to the South using a ratio of the medical component of the consumer price index (CPI) for 2004.

S. Subramanian et al. / Evaluation and Program Planning 31 (2008) 136–144140 when performing comparative assessments between pro-grams. Because these women will not receive screening examinations during the year of enrollment, they will not be included in the total number of women who were ‘‘screened’’ by the program. However, these women will be included in the count of women ‘‘served’’ by the program.

The ratio of women served versus the ratio of women screened can be included in multivariate models to control for this difference among programs.

3.3. Variation in magnitude of in-kind contributions

In-kind contributions reported by the nine programs ranged from $432,507 to $3,433,153 (Table 2). These contributions include both labor and nonlabor services to the programs. Examples of in-kind contributions are time volunteered by physicians for activities related to the Clinical Advisory Board that is convened by each of the 68 NBCCEDP programs to advice on clinical issues and donations of print materials for outreach activities. Over-all, these contributions were quite large and accounted for 33.6–59.7% of the total program cost. When adjusted to reflect true opportunity cost, in-kind contributions declined significantly for some programs, but the average in-kind contributions were still $950,027 and ranged from 3.9% to 54.5% of the total program cost. Overall, in-kind contributions play a significant role in the operations of many programs and are a key source of variation between the programs. Therefore, these in-kind contributions should also be included in comparative assessments across programs. At a minimum, the cost per women screened should be assessed both including and excluding in-kind contributions to understand variations across the programs.

3.4. Differences in allocation of indirect costs as fixed

overhead versus direct charges

Indirect or overhead costs, in contrast with direct costs, are expenses that have been incurred for purposes common to all NBCCEDP activities but cannot be identified and allocated directly to a specific activity. These expenses include utilities, building maintenance and renovation, accounting, and payroll. We found large variations in how indirect costs were quantified by the parent organizations (example, state health department) within which the programs are embedded, and also in the types of costs that were included. Indirect costs were calculated by three different methods: as a percentage of salary, as a percentage of direct costs, and as a predetermined fixed amount. In many cases, in addition to a specific allocation, a program also paid directly for network connection, phone services, and rent. In some cases, all potential overhead items and services were bundled together into one single payment, and in others several separate costs were combined. To accurately account for all overhead costs, health care program evaluators must obtain details on how overhead costs are calculated and a description of what each payment represents. Because these indirect costs may sometimes be arbitrarily assigned and vary from year to year, comparisons between programs s hould also be performed both including and excluding these costs to assess the impact of indirect cost on overall program costs.

3.5. Variation in cost of specific program components

NBCCEDP programs perform a large number of activities besides offering cancer-screening tests. These

ARTICLE IN PRESS

Table 2

In-kind contributions to the programs

Program As reported by programs Based on opportunity Costa

In-kind contribution ($)* Total program cost ($) Percentage of total program cost (%)

In-kind contribution ($) Percentage of total program cost (%)

1 433,731 974,810 44.5 433,731 44.5

2 1,632,993 3,267,851 50.0 1,632,993 50.0

3 1,667,734 4,455,695 37.4 415,790 9.3

4 525,000 1,358,981 38.6 525,000 38.6

5 2,088,860 3,862,332 54.1 1,419,100 36.7

6 3,433,153 6,312,989 54.4 3,433,153 54.4

7 906,808 1,519,066 59.7 415,330 27.3

8 432,507 1,285,450 33.6 49,543 3.9

9 2,077,226 5,403,829 38.4 225,600 4.2

Range 432,507–3,433,153 974,810–6,312,989 33.6–59.7 49,543–3,433,153 3.9–54.4

Mean 1,466,446 3,160,111 45.6 950,027 29.9

95% CI 811,832–2,121,059 1,866,083–$4,454,140 39.6–51.7 249,168–1,650,885 17.0–42.8

Note: CI ¼ confidence interval.

aOnly contributions that strictly represent opportunity cost are included.

S. Subramanian et al. / Evaluation and Program Planning 31 (2008) 136–144 141 activities include case management, tracking and follow-up, outreach, professional development, and public educa-tion. We found wide variations in the cost of these activities across programs, particularly for professional education and outreach activities. In Fig. 2, we present the proportion of total cost dedicated to professional education and outreach. The expenditure for professional education varied from 1% to 16% of total costs. For public education and outreach, the variation is even larger and ranged from 2% to 28%. For various reasons, the cost of these activities may be high one year and low another year. Thus, those analyses that use one year of data may have skewed findings that favor programs with low expenditures for that specific period of analysis. Therefore, to minimize the impact of year-to-year variation in activity costs, several years of cost data should be analyzed. For example, an analysis of data from three consecutive years may improve comparability across programs.

4. Lessons learned

In this study, we used both qualitative and quantitative data sources to assess underlying program-level factors that can impact program cost and that need to be controlled for when comparing costs across health care programs. Our assessment of the data from nine NBCCEDP programs identified several program-level factors and potential approaches to derive valid cost estimates to compare across programs. First, the use of regional CPI may not adequately control for differences in the cost of clinical services. A potential solution is to use state-specific cost-of-living adjustments. Second, service mix differences, whether related to screening mix or screening and diagnostic mix, should be identified and controlled for using appropriate weights for each program.

Third, in-kind contributions can often be a significant part of a program’s operations, and these costs should be included in any comparison. Fourth, indirect or overhead costs may have both a lump-sum amount and individual charges for specific items. Thus, the data collection process should be able to identify and collect both types of indirect costs. Lastly, using only one year of data may provide a skewed comparison of program costs. Using multiple years of data would provide more reliable estimates that are not influenced by year-to-year variations.

Our assessment confirmed some well-known sources of program-level cost variations, such as cost of living differences, and identified additional ones such as the role of in-kind contributions. Our analysis also revealed that traditional methods to control for differences between programs, such as regional CPI, may not be adequate, particularly for a large and complex program such as the NBCCEDP. Therefore, an important lesson from this assessment is that comparative cost analyses of health care programs are complex and that significant effort should be placed on understanding the underlying differ-ences between these programs. These factors should be systematically identified before initiating comparative cost analyses across programs. Both qualitative and quantitative assessment should be performed whenever possible because both approaches yield valuable informa-tion. Case studies are particularly useful in identifying underlying factors that may impact cost in the early stages of any program evaluation. Such activities require time

ARTICLE IN PRESS

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

Prog ram

Program

Pe rc en t o f T ot al C os t

Prog ram

Prog ram

Prog ram

Prog ram

Prog ram

Prog ram

Prog ram

Prog ram

Professional Education Public Education and Outreach

Fig. 2. Percentage of total cost for professional and public education.

S. Subramanian et al. / Evaluation and Program Planning 31 (2008) 136–144142 and resources, but are well worth the effort as the final product is a higher quality assessment. Even when resources are limited and economic evaluations are performed as a secondary objective and have to rely on retrospective data, performing a few interviews with key program staff can offer useful insights for the economic assessment.

Another key lesson we learned through our site visits and the pilot testing of the cost-collection questionnaire is that stakeholders, such as the individual NBCCEDP programs, should be consulted early in the evaluation process, because they often provide critical feedback and are most familiar with the program operations. Involving these individuals ensures that the issues that are important to their programs are included in the cost analysis.

In addition, involving stakeholders in the process facili-tates implementation of the findings from the economic evaluation.

Overall, our analyses highlight that identifying under-lying factors that may impact program costs is a necessary and important step in performing economic assessments of health care programs. The findings from this study will be incorporated into the next phase of the economic assessment of the NBCCEDP. The empirical results from this assessment also provide benchmarks to under-stand the impact of underlying program variations when performing cost comparisons of health care programs in general. The approach, methods, and lessons identified in this study can assist other researchers in designing and performing high-quality economic evaluation of health care programs.

Acknowledgments

We sincerely thank the selected states that participated in this phase of the project. In addition, the authors thank Ms. Amber Moore for her assistance in the development of the site visit questionnaire.

References

Baicker, K., & Chandra, A. (2004). Medicare spending, the physician workforce, and beneficiaries quality of care. Health Affairs, W4-

184–W4-197.

Benard, V. B., Eheman, C. R., Lawson, H. W., Blackman, D. K., Anderson, C., Helsel, W., et al. (2004). Cervical screening in the

National Breast and Cervical Cancer Early Detection Program, 1995–2001. Obstetrics & Gynecology, 103(3), 564–571.

Benson, K., & Hartz, A. J. (2000). A comparison of observation studies and randomized, controlled trials. New England Journal of Medicine, 342, 1878–1886.

Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observation studies, and the hierarchy of research designs. New

England Journal of Medicine, 342, 1887–1892.

Diez-Roux, A. V. (2000). Multilevel analysis in public health research.

Annual Review of Public Health, 21, 171–192.

Drummond, M. F., Sculpher, M. J., Torrance, G. W., O’Brien, B. J., &

Stoddart, G. J. (2005). In Methods for the economic evaluation of health care programme (3rd ed.). Oxford, England: Oxford University

Press.

Eheman, C. R., Benard, V. B., Blackman, D., Lawson, H. W., Anderson, C., Helsel, W., et al. (2006). Breast cancer screening among low-income or uninsured women: Results from the National Breast and Cervical

Cancer Early Detection Program. Cancer Causes and Control, 17(1), 29–38.

French, M. T., Dennis, M. L., McDougal, G. L., Karuntzos, G. T., &

Hubbard, R. L. (1992). Training and employment programs in methadone treatment: Client needs and desires. Journal of Substance

Abuse Treatment, 9(12), 293–303.

French, M. T., Dunlap, L. J., Zarkin, G. A., McGeary, K. A., &

McLellan, A. T. (1997). A structured instrument for estimating the economic cost of drug abuse treatment. The Drug Abuse Treatment

Cost Analysis Program (DATCAP). Journal of Substance Abuse

Treatment, 5, 445–455.

French, M. T., Roebuck, M. C., & McLellan, A. T. (2004). Cost estimation when time and resources are limited: The brief DATCAP.

Journal of Substance Abuse Treatment, 3, 187–193.

Gold, M. R., Siegel, J. E., Russell, L. B., & Weinstein, M. C. (1996).

Cost-effectiveness in health and medicine. New York, NY: Oxford

University Press.

Greenland, G., & Morgenstern, H. (2001). Confounding in health research. Annual Review of Public Health, 22, 189–212.

Haddix, A. C., Teutsch, S. M., & Corso, P. A. (2003). Prevention effectiveness: A guide to decision analysis and economic evaluation

(2nd ed.). New York, NY: Oxford University Press.

Iezzoni, L. I. (2004). Risk adjusting rehabilitation outcomes: An overview of methodologic issues. American Journal of Physical Medicine and

Rehabilitation, 83(4), 316–326.

Keenan, P. S., Buntin, M. B., McGuire, T. G., & Newhouse, J. P. (2001).

The prevalence of formal risk adjustment in health plan purchasing.

Inquiry, 38(3), 245–259.

Leonard, H. B., & Walder, J. H. (2000). The federal budget and the states:

Fiscal year 1999, 15 December 2000. Retrieved November 17, 2006 from/http://www.ksg.harvard.edu/taubmancenter/pdfs/fisc/complete_

99.pdfS.

Mansley, E. C., Dunet, D. O., May, D. S., Chattopadhaya, S. K., &

McKenna, M. T. (2002). Variations in average costs among federally sponsored state-organized cancer detection programs: Economics of scale? Medical Decision Making, 22, S67–S79.

PART (2006). Program Assessment Rating Tool (PART). Retrieved

November 17, 2006 from /http://www.whitehouse.gov/omb/part/ fy2006/2006_guidance_final.pdfS.

Ryerson, A. B., Benard, V. B., & Major, A. C. (2005). National Breast and Cervical Cancer Early Detection Program: 1991–2002 national report. Atlanta, GA: U.S: Department of Health and Human

Services, Centers for Disease Control and Prevention (Retrieved

January 10, 2007 from) /http://www.cdc.gov/cancer/nbccedp/Reports/

NationalReports/index.htmS.

Salome, H. J., French, M. T., Miller, M., & McLellan, A. T. (2003).

Estimating the client costs of addiction treatment: First findings from the Client Drug Abuse Treatment Cost Analysis Program (Client

DATCAP). Drug and Alcohol Dependency, 71(2), 195–206.

Winship, C., & Morgan, S. L. (1999). The estimation of causal effects from observational data. Annual Review of Sociology, 25, 659–706.

Sujha Subramanian is a Health Senior Economist at RTI International.

Her primary research interests are in the fields of cancer economics and health services research. She is currently directing the economic evaluation of the National Breast and Cervical Cancer Screening Program

(NBCCEDP) and the National Program of Cancer Registries (NPCR).

Donatus U. Ekwueme is a Senior Economist at the Division of Cancer

Prevention and Control, Centers for Disease Control and Prevention

(CDC). He is the lead CDC economist directing the economic evaluation of the National Breast and Cervical Cancer Screening Program

(NBCCEDP). He is also currently leading other projects focused on

S. Subramanian et al. / Evaluation and Program Planning 31 (2008) 136–144 143 cancer economics, including an assessment of the cost of performing colorectal cancer screening.

James G. Gardner is a program consultant with the Centers for

Disease Control and Prevention (CDC). He works with the National

Breast and Cervical Cancer Screening (NBCCEDP) grantees to provide technical assistance and has developed several reporting tools for the program.

Bela Bapat is a research analyst at RTI International. She is currently working on several program evaluations and has developed Excel and Web-based data collection tools to collect cost and resource use data from programs.

Caren Kramer was a research analyst at RTI International and participated in the economic analysis of the National Breast and Cervical

Cancer Screening Program (NBCCEDP). She has experience performing cost data analysis and systematic literature reviews.

S. Subramanian et al. / Evaluation and Program Planning 31 (2008) 136–144144