Southeast_Cluster_Initiative_Program_Evaluation_-_USFS_Comment.pdf

I. UACED Technical Approach – Southeast Alaska Development Program – Evaluation The University of Alaska Center for Economic Development (UACED) is pleased to respond to your Request for Southeast Alaska Cluster evaluation services. We have a team with the necessary expertise to help you accomplish your program evaluation goals.

Proper program evaluation provides economic development officials with the ability to assess the return on their investment. Since 2010, the US Forest Service has invested over $1,000,000 to support diversification of the economy of Southeast Alaska, as part of the transition plan for the Tongass National Forest. This funding went toward a number of initiatives, including cluster development efforts and seed capital for several revolving loan funds in the region.

Economic development program evaluation is often a difficult task to successfully monitor. Program evaluations that focus exclusively on quantitative impacts (unemployment rate, average wages, etc.) can often miss the causal relationships behind these impacts. These quantitative measures usually fluctuate significantly, and often depend on a multitude of factors above and beyond the efforts of program managers. By a similar token, qualitative measures can just as easily paint a misleading picture. They may suffer from small sample sizes and a response bias from participants.

Making matters worse, without prior evaluation methods and metrics in place, performing a comprehensive and reliable ex post program evaluation can be an arduous task.

With these factors in mind, UACED favors a multi-faceted approach to program evaluation, one which focuses on a combination of quantitative and qualitative measures. These measures can then be viewed holistically to make an overall judgement of the relative effectiveness of a particular investment or program. This knowledge will then assist both economic development practitioners and funders alike. Our analysis is designed to not only look backward at the success of a program, but also to look forward by recommending approaches that can be adopted for future program implementation and ongoing evaluation purposes.

Our Approach

Our methodology to evaluate the Southeast Alaska Cluster Initiative will be to apply a multi-faceted evaluation approach to the US Forest Service’s investments in Southeast Alaska. As part of this process, UACED will develop a logic model demonstrating the causal relationships of each stage of the process, from funding to outcomes. From there, the various components of the Southeast Alaska Cluster Initiative that will be evaluated include:

Assessment of each revolving loan fund established through US Forest Service funding Identification of accomplishments and challenges of Southeast Alaska cluster initiatives:

o Ocean Products o Visitor Products o Renewable Energy o Mining Services and Supply o Research o Arts and Culture

Linkages between cluster development initiatives and regional economic development strategies and outcomes

Southeast Alaska Cluster Initiative Program Evaluation Methodology

Comment [UFS1]: See emailed comments.

I’m careful to only consider the Southeast Cluster Initiative, as designed and planned from beginning. I suppose it is a development program, but it has specific parameters.

Comment [UFS2]: I understand the use of ROI in this paragraph, but also feel it is overplayed and underestimates.

Comment [UFS3]: Where did revolving loan fund come from? While JEDC operates a RLF, it is not part of the Southeast Cluster Initiative. It is also not something the USFS funds or supports in any other fashion. You can remove all reference to RLF.

Comment [UFS4]: Yes, we should state the challenges up front. The most important being, programs are often launched without any tools in place to measure performance over time. Let’s make sure a limitations section is included any final deliverable.

Comment [UFS5]: Feel free to cut out all reference to “look forward” due to the small nature of this contract. There should be some general recommendations for improving the “effectiveness” of the program, but it doesn’t need to be lengthy or fully comprehensive.

Comment [UFS6]: Do you have an example analysis that I can review? I’m curious to see how this plays out on paper.

Comment [UFS7]: Again, you can eliminate all reference to revolving loan fund.

Comment [UFS8]: While investments have been made in up to 6 clusters, only four have had any significant traction. And, the USFS has specifically only invested in four clusters.

I’m just making the point to narrow the scope of work a bit. It would be good to discuss all the clusters at some level, but narrow focus to only those that have remained active.

Comment [UFS9]: Very important and difficult task. I used your “approach” to further define objectives in my email statement. I think we are on target here.

From my perspective, there are two approaches to evaluating outcome/effectiveness/impact: cluster initiative outcomes and business/community/regional impacts.

Comment [UFS10]: I appreciate the model to the left, but am concerned that you can’t jam all this into a small contract. Let’s discuss objectives, feasibility, and practicality.

Focus Groups and Key Informant Interviews

Two essential qualitative components in the Southeast Alaska program evaluation will be the convening of applicable focus groups and key informant interviews. Utilizing focus groups will allow program evaluators the opportunity to gather detailed information about individual perceptions and opinions on the effectiveness of each initiative and its various components. Focus groups also provide participants with the opportunity to build off of the ideas of other attendees, allowing for meaningful dialog and information exchange. Key informant interviews will serve to inform focus group queries as well as allow for more confidential and targeted feedback.

UACED will convene focus groups representing various key stakeholder groups that have participated in the Southeast Alaska Cluster Initiative, or those that have interacted with project leaders. Depending on participant interest and budget constraints, UACED anticipates convening focus groups covering one or more of the following stakeholder segments:

Cluster initiative working groups Innovation Summit attendees Private sector businesses who received money from revolving loan funds established through the Southeast

Alaska development project Economic development and other similar professionals involved in overseeing the delivery of project funds

Where it is not practical to organize a focus group, UACED will reach out to relevant stakeholders through key informant interviews. These interviews will allow UACED to gain a greater understanding of the overall effectiveness of the Southeast Alaska Cluster Initiative. UACED will also intentionally incorporate key informant interviews with USFS-identified critical stakeholders and others that may not be available to participate in a focus group.

Questionnaires

Questionnaires may be distributed to relevant stakeholders via an online platform, which will serve both qualitative and quantitative functions. Questionnaires will focus on identifying key success metrics for the Southeast Alaska Cluster Initiative, and will be be tailored to each relevant stakeholder group. Targeted groups include the economic development community, cluster initiative working groups, and the business community. While some questions will likely overlap, others will be tailored based on the group being approached. UACED will utilize an adaptive surveying tool, one in which the previous answers of a respondent determine the next questions that they will be asked, allowing UACED to gather data with greater specificity than through a traditional questionnaire.

Questionnaires will be designed to gauge individuals’ perceptions of the overall effectiveness of the Southeast Alaska Cluster Initiative. Questions will cover the following topics:

Effective Program

Evaluation

Focus Groups and Key

Informant Interviews

Questionnaires

Third-Party Data Evaluation

Logic Model Formation

Holistic Cluster Asset Mapping

Comment [UFS11]: A full inventory of past and present cluster group participants is an important base. I’ve been working on this a bit in my free time. From my perspective, feedback from participants via focus groups, interviews, or surveys is the most important part of this overall project…. Alongside an analysis of type of participation and continuity over time. It will be equally important to reach out to those members that dropped off the radar – especially the private sector.

I understand cluster groups is a cost effective means of collecting feedback and offers other advantages, however, I also worry about diversity in opinion, openness, and confidentiality. It would be ideal to carry out a mix of focus groups (open), key informant interviews (chairs and champion), and quantitative survey instrument. This is where I can provide some labor if needed and meet with at least the chairs.

Ultimately I would like feedback collected in a systematic manner (survey), which I can cite for multiple purposes.

Comment [UFS12]: See above comment.

Where not practical or not appropriate.

Anyway, need to give some thought to our outreach method, line of questioning, and the best available tool.

Comment [UFS13]: I understand the ease of Online tools, but my experience has been HORRIBLE with response rate. Let’s talk about this tool.

Comment [UFS14]: Interesting to think about this per “stakeholder group”. I’m most interested in cluster groups participants (especially private), regional economic development leaders, and perhaps a few community leaders.

Comment [UFS15]: The most important part of this collecting feedback. We should carefully think through line of questioning and how best to collect feedback. I’m also interested to know if cluster participants fully understand the cluster development process or rather participate in these groups only as a means of reducing barriers to economic development. Sort of semantics here, but we are investing in a strategic approach to economic development – just facilitating volunteer groups focused only on their individual business challenges.

Attitudes regarding cluster development since the Southeast Alaska Cluster Initiative began to determine if improved networking or alliances of firms and businesses now exist Opinions regarding the effectiveness of the Southeast Alaska Cluster Initiative – have foundational needs been addressed adequately (human resource training, access to technical assistance, improved access to capital, etc.)

Knowledge of key developments or improved industry competitiveness arising due to the Southeast Alaska

Cluster Initiative Viewpoints regarding the effectiveness (strengths and weaknesses) of revolving loan programs in Southeast

Alaska

Businesses who received funding from the Southeast Alaska revolving loan programs will be asked additional questions aimed at generating quantitative data, including:

Number of jobs created as a result of the funding Number of new businesses created as a result of the funding Private and public sector additional funding leveraged as a result of the funding Growth in sales and profits (where applicable) as a result of the funding Rating of how effective the funding has been for their business

The aim of the questionnaires will be to determine the extent to which the economic development and business communities believe that the Southeast Alaska development project has been effective. Questionnaires will be designed to identify specific impacts resulting from the program. Questionnaires will also seek to identify areas for improvement specific to the program.

Third-Party Data Evaluation

As part of its analysis, UACED will also incorporate third-party quantitative data into its evaluation. This data will be aimed at identifying the overall effectiveness of the program in a broader sense. Many different quantitative metrics and datasets will be utilized throughout this process. Amongst other items, UACED’s third party data analysis will include the following:

Unemployment trends within the region, including those within targeted cluster sectors Income trends within the region, including those within cluster sectors Tourism product development and trends within the region Economic Census data and trends specific to target industry clusters

A core component of this third-party data evaluation process will be the formation of a Location Quotient (LQ) analysis for the region’s major industry clusters. The LQ analysis focuses on regional employment concentration and can be used as a baseline for cluster identification and trends analysis. A high LQ in a given industry indicates an area of specialization for a region. This LQ analysis will allow staff to clearly identify major industry clusters in the Southeast region alongside regional trends. UACED staff will compare current LQ results with LQ data spanning the previous decade to identify major regional trends, including emerging, stable, and declining clusters. UACED will evaluate the cluster trends in the years leading up to the Southeast Alaska Cluster Initiative, and will compare them with the rate of growth/decline in the years following the project. This will help to identify whether clusters have accelerated their growth, or reduced their decline, in the years following the project’s introduction. Clusters will be compared not only to national employment averages, but statewide averages as well.

Logic Model

Comment [UFS16]: Yep, delete reference to RLF program. However, the line of questioning is important for the private businesses that participate. Has participation in cluster working groups reduced barriers to business development, created jobs, created opportunity, impacted growth/sales/profit, or any other notion of overall effectiveness. We could ask the question at the business level and follow-up by asking for perceptions of overall impact for the industry, community, and region.

With only a five-year timeframe, it is nearly impossible to measure overall regional impact other than through the perceptions of those that have been impacted.

Also, need to collect qualitative stories of success and maybe challenge, too.

Comment [UFS17]: Yes, yes, and YES! See prior comments. Final summary analysis will only be as good as the project design to collect feedback – we’ll need to have a plan.

Comment [UFS18]: Isn’t this just a fancy way of noting secondary data analysis? Or, alternatively, regional socioeconomic conditions? I concur that it would be good to look at overall effectiveness within a broader context, however, we’ll need to acknowledge the short time frame and difficulty in linking cluster activities to overall regional conditions.

Additionally, it would be good to use what JEDC has already developed from the onset of the Southeast Cluster Initiative. They looked at LQs, asset mapping, and cluster strength. If you go the route of secondary data analysis, it will need to align with JEDC’s baseline work.

Comment [UFS19]: The use of LQs has always bothered me as it is the application of tool without acknowledgement of geography or transportation. However, maybe there are certain aspects we can pick out an compare pre- and post-cluster work investment.

Comment [UFS20]: This is a repeat of JEDC work. Let’s take a look at their original asset mapping work and see if there is a way we can measure post five years of work.

Comment [UFS21]: Interesting approach.

I’m really curious as to how this will play out with the Southeast Cluster Initiative.

How do we define and outcome or impact?

There is where we started – with grandiose goals in the original planning document of years past. And, then, there is what we have today. Perhaps goals were not completely achieved, but there have been unanticipated benefits along the way.

UACED will incorporate the development and use of a logic model into its program evaluation. Logic models are tools used by program evaluators that display a graphical representation of the logical relationships between resources, activities, outputs, and outcomes of a program. They serve as a way to quickly and easily identify the “if-then” relationships between different components of a program. One of the benefits of developing a logic model is that it makes it easier to define and measure the resources being put into a program. Another benefit is that performance measures can be developed across all steps of the process. Logic models can also be used to clearly see the benefits that have taken place, and trace them back to their root sources. Logic models can also help to identify best practices for future evaluation methods, allowing economic developers to plan for their future program evaluation needs.

UACED Logic Model Template

1. Inputs What resources go into a program (money, staff time, supplies, equipment)?

2. Activities What activities are undertaken by the program (development of materials, training programs, etc.)?

3. Outputs What is produced through those activities (workshops held, people trained, etc.)?

4. outcomes/Impacts Benefits that result from the program (increased skills, knowledge, employment, etc)?

I. UACED Technical Approach – Southeast Alaska Development Program – Evaluation