Attachment 2 - Report on SPPA Planning Meeting 10.21.05.pdf

Discussion Summary

SPPA Advisory Committee Meeting

October 21, 2005

Prepared by Timothy Triplett and Daniel Swenson

Urban Institute

Table of Contents:

Table of Contents:

Meeting Information Section 1: Introductions and Overview

1.1 Introductions

1.2 Background

Section 2: Alternatives for conducting the 2007 survey

2.1 Overview of where we are now

2.2 Key points made during the discussion

2.3 Summary

Section 3: Potential Changes to the Questionnaire

3.1 Key discussion points on broadening some of the questions

3.2 Key discussion points on getting greater geographic and demographic detail ... 13

3.3 Key discussion points on adding new items or replacing items

3.4 Concluding Remarks

3.5 Summary

Appendix A: Data collection alternatives for the 2007 SPPA Appendix B: Recent Research Using SPPA Data Appendix C: SPPA Historical Overview

Meeting Information

Date of meeting: Friday October 21, 2005

Time: 9:30am to 3:30pm

Place: National Endowment for the Arts (NEA)

OPOB, conference room (M-07)

Washington, DC

Participants:

Kelly J. Barsdate, Director of Research, Policy and Evaluation, NASAA

Tom Bradshaw, Senior Research Officer, NEA

Paul DiMaggio, Professor of Sociology, Princeton University

Mary McIntosh, Principal and President, Princeton Survey Research Associates

Bonnie Nichols, Research Analyst, NEA

Francie Ostrower, Senior Research Associate, Urban Institute

Ken Prewitt, Professor of Public Affairs, Columbia University

John Robinson, Professor of Sociology, University of Maryland

Marc Scorca, President and CEO, OPERA America

Peter Hildick-Smith, President, Codex-Group LLC

Daniel Swenson, Research Associate, Urban Institute

Steven Tepper, Associate Director, Curb Center at Vanderbilt University

Timothy Triplett, Survey Associate, Urban Institute

Section 1: Introductions and Overview

1.1 Introductions

Overall it was an excellent mix of experts in the room and most of the participants mentioned that they use the Survey for Public Participation in the Arts (SPPA) data. In addition, several participants were experts in survey methodology including the former director of the census bureau, Ken Prewitt. Furthermore, John Robinson who designed the very first (1982) SPPA was among the participants.

During the introductions participants expressed their reasons for wanting to help the

National Endowment plan for the next SPPA. A common theme was that SPPA provides a reliable benchmark of the estimated number of Americans participating in various arts activities. Race and gender analysis was also specifically mentioned. With multiple decades of data now available, trend analysis was also noted as an important component of the SPPA data.

1.2 Background

Since the first SPPA, repetition of the survey has brought greater attention. The front-page article in the New York Times (2/12/96) about the “graying” of arts audiences cited

SPPA data and brought further attention. Decade to decade comparisons have proven particularly useful. The SPPA needs a large sample size because relatively few people participate in certain arts activities, such as opera and ballet. The 1982, 1992, and 2002 decade comparisons have been the most important. (The SPPA 1997 results were anomalously higher due, at least in part, to a different mode of data collection.)

The SPPA is funded entirely by the National Endowment for the Arts (NEA). The

Endowment has previously put out feelers with arts service organizations, federal cultural agencies, and foundations that fund the arts to see if they were interest in collaborating, or maybe helping fund the SPPA. Lacking strong interest in collaborative funding, the

NEA continues to be the sole funder of the survey.

There has been interest in having the SPPA explore additional activities and/or additional reasons, motivations, barriers to participation. Historically, the SPPA looks at a relatively narrow set of participation activities. There is also interest in expanding the core questions, and broadening definitions to include additional forms of participation.

Section 2: Alternatives for conducting the next SPPA

2.1 Overview of where we are now

Surveys of the general population are increasingly hard to do given lower response rates.

Westat, when contacted, said it would now likely cost twice as much to field the survey than it did in 1997, and even then they anticipate that the response rate using the same random digit dial (RDD) sampling design would only be a little higher than 30%. It used to be RDD sampling frames included 95% of all households which was much better than mail sample frame coverage rates, but that’s no longer the case. Some studies that rely on telephone surveys are even going back to doing mail and in-person surveys. In effect, an

RDD telephone survey is not an option. Sampling frame, costs and coverage rates rule out going back to using RDD. Also, the Endowment can’t afford to do in-person surveys as a stand-alone. As such, the options have been narrowed down to 3 possibilities.

The first option is to again (as in 2002) have Census conduct the SPPA as a supplement to the Current Population Survey (CPS) or conduct it (as in 1982, 1985, and 1992) as a supplement to the National Crime Victimization Survey (NCVS). Unfortunately, the

Census is not taking supplements to these two surveys in 2007 due to a software changeover. The earliest possible date that the SPPA could be added as a supplement to the CPS is May of 2008. This is the same for the crime victimization survey (NCVS), with the additional problem of having to possibly collect some data in 2009, since a single month of the NCVS does not yield a large enough sample.

A second option is the American Time Use Survey (ATUS). Like the CPS this is a BLS study conducted by the Census Bureau. This survey uses CPS panel respondents but only after they have completed their full CPS rotation. As such the response rates are lower than for the actual CPS. The CPS has about a 90% response rate compared with about

60% for the ATUS, and also supplements generally will get a slightly lower response rate. However, ATUS dovetails topically with the SPPA. The ATUS also would not rely on any proxy interviews, nor would the NCVS. Proxy interviews are sometimes necessary in order to complete the SPPA questionnaire following the CPS interview, and this can be problematic.

A third option is using an internet survey firm, such as Knowledge Networks, who could conduct the SPPA as an Internet survey using randomly recruited panel respondents.

Internet survey firms aim to compose their panels in such a way as to yield nationally representative samples. If the SPPA were to be fielded on the Internet, it would be the first time it was self-administered.

John Robinson pointed out that some Knowledge Networks surveys are close to measures found in the GSS, but others are not. On the other hand, the presence of the interviewer might skew the GSS, and the GSS may not be the gold standard. Participants felt that the

SPPA will inevitably be fielded on the Internet, but that now might not be the time to make the switch. One participant asked if we might do a pilot Internet survey this year alongside a more traditional mode of data collection? Another participant asked whether

Internet surveys have the problem of the wrong person filling out the survey (i.e. not the intended panel member).

2.2 Key discussion points on options for conducting the SPPA

What do we want the next SPPA to do for us? It seems like the 1997 survey never existed. Either 2007/2008 should give us comparable data, or it should be used to test other modes of collection. What we want the data to be is a big question.

There are advantages and disadvantages to an Internet based survey. The ability to show images and video is a powerful tool. And compared with a NORC telephone (RDD) survey, the Knowledge Networks survey tracked estimates closely. The downside is that it’s not a true random sample. Rather, they “train” people to be respondents. A few years ago the National Science Foundation was fielding a survey of perceptions of science. A similar discussion to this current discussion was then being held as to whether or not the survey should be fielded as an Internet survey. The decision then was to not field the survey online.

Well, maybe the Internet will take over. Or maybe they’ll “solve the cell phone problem.”

It would be good to do a pilot Internet survey, and use that to determine how/when the

SPPA eventually goes fully to Internet. Another problem: are respondents crafting fiction? I wouldn’t be comfortable with a wholesale shift to Internet. An Internet pilot might be a good way to attract additional funding.

If we wait, will more people have broadband? Has anyone studied the effect of broadband versus dial-up on Internet surveys? It seems not to matter what type of connection and Internet surveys can have a few “bells and whistles.” But in general they should be simple.

A major design change isn’t the solution since response rates are falling regardless.

“What if we give a census and no one comes?” This government is going to collect data regardless. In the next 5 – 10 years, there will likely be a move away from surveys and instead more reliance on using administrative data sources.

Everything suggested requires resources, and we’re under a resource ceiling. Testing mixed modes could be positioned less as research and development, and more as being good stewards of the data and keeping up to speed with the changing nature of survey research. I think there’s a way to make that case.

Another consideration that we need to discuss is whether our “gold standard” (CPS supplement) census might not be available until 2008—is it worthwhile to wait?

Yes, because there is a “census effect.” The Census always gets lower estimates.

We also need to keep in mind the Census is under tight budget constraints, for instance the American Communities Survey could lose their funding. And they’re throwing around the idea of not having a long form for the 2010 census. Budgets are low—it’s not a good environment for funding. No guarantees. The current mindset in the Congress is that it’s better not to have a long form.

While methodologically, there are reasons to wait, it is important to the credibility of arts organizations to report on themselves, which necessitates 5 and 10 year intervals. The field needs to present itself as professional, credible.

But then what does that say about the 1997 survey? Well, the opera numbers looked good in 1997, so the data was useful to us.

Do we face a “Bowling Alone,” Putnam question? 30 years from now, we may have lower participation estimates, but maybe that’s because people are doing different things.

We have to make sure to measure what people are actually doing. Which argues for playing around with questions on off years, or look to other surveys and other research.

How good is our reservation on the CPS for 2008? They probably would push for us to have a shorter survey. Probably they would charge $550 – 600K. With the CPS the

SPPA would be on 2 outgoing rotations, the survey would take 1 month to field, and we’d get approximately 17,000 respondents. The NCVS is spread out over several months but there may not be much seasonal effect on arts participation.

Since the American Time Use Survey is newer, they’re not undergoing the software upgrade. With this survey, data would be collected throughout the year. However, the latest American Time Use Survey itself fell just short of achieving a 60% response rate.

So, with the Census the two options are either tack on to the CPS or the American Time

Use Survey and if the Time Use Survey takes a year to administer, would we get the data any earlier with that than with the CPS? Probably not—the CPS and Time Use Survey would both get us the data in around the same time frame.

If we do the time use survey, the data might not be comparable. Though, the move from

NCVS to CPS didn’t change the data much. The Westat study in 1997 is the only incomparable year.

The time use survey depends on people who thought they were done with their panel rotation. However, unless we understand biases with response rates, we shouldn’t say that 70% is necessarily better than 60% or 40%.

With RDD, it’s clear that there’s some self-selection. Sampling frame and response rates do matter. Well, not so much the response rate—it’s the house effect.

If the estimates were abnormally low, that could be bad for the field. Which was not the problem with the 1997 data. It’s important that the estimates were abnormally high rather than abnormally low. Higher estimates brought more attention from the press. We were quieter in 2002, because we didn’t want to talk about the decline.

The “Reading at Risk” research used only data from 1982, 1992, and 2002.

It is likely that the CPS, NCVS, or American Time Use Survey—any will give us decent data, and regardless the data will be a bit late. CPS is probably the best, but maybe we should have a contingency plan. I think we can say that we are waiting for the ‘gold standard,’ the best, the Census. It’s better to wait, than for arts organizations to have to explain different methodology, and possibly unexpected estimates.

One other data source is the GSS. GSS is the best alternative to the Census; it’s done by

NORC. Do they use incentive payments? Is that a problem for the NEA? Another problem with GSS us that it is a 90-minute interview, with many detailed questions.

Who’s going to answer a 90-minute survey? There’s got to be some self-selection. That could explain the different numbers (higher estimates for similar SPPA items).

Is there a way to use consumer expenditure data to see what arts activities people are spending money on? People keep a diary of their expenditures. Other relevant data is an important conversation, but no substitute for the SPPA.

Certainly, the Census Bureau signals credibility. Census will make heavier use of the

Internet in the future. But the question is really whether data from voluntary response surveys will be used in the future. People are holding on, but they are holding on with incentive payments. We’ve spent 100 years on improving the quality of survey data.

We’ve spent no time on administrative data. Over the next 25 years that will happen—at a certain time you have to shift. CPANDA could be a leader on this. It would be hard for the NEA to lead on this. People are starting to bring in administrative data. The problem, though, is that it’s confidential. That’s true of both government and commercial enterprises (e.g. FBI, bank records). NEA itself has to coattail, not lead. SPPA shouldn’t have to be on the cutting edge of survey technology. But maybe we should take little steps toward bigger steps in the future. Administrative data is important, but much of what we’re interested is not transactional in nature. There will always be, in the arts, a need to have people tell us their stories.

2.3 Summary

It seemed clear that most people prefer that the SPPA remain a supplement to the CPS even if that means a delay in the data collection. However, given the current Census bureau climate and their schedule, an alternative or some contingency planning should be further explored. Were that planning to include a major design change toward Internet data collection, some additional pilot work and benchmark testing would need to be done.

Since an SPPA supplement to the American Time use Survey would likely take a full year of data collection in order to complete enough interviews, there doesn’t seem to be any major advantage of switching from CPS to the ATUS.

Section 3: Potential Changes to the Questionnaire

3.1 Key discussion points on broadening some of the questions

Regarding the questionnaire, there’s interest in broadening some questions. There continues to be some bias (toward “high” art), as NEA is the sponsor. Perhaps we should expand to ask about “popular” music. Even If we go with the CPS, we could probably split the sample. Maybe splitting the sample for continuity is needed. Yes, and some cognitive testing to make sure people understand the terms.

It is important to ask about symphony, chamber, choral, and other music. Maybe get rid of “classical”, since it is slightly inaccurate and too grouped. Yes, good to disentangle classical. But then there are concerns about comparability. Also, concern that respondents understand what, for example, “chamber music” is. Well there is only some much we can do, for instance we don’t predetermine what a book “is.” Another realm where this sort participation happens is church. There is some conflation there, without asking specifically about the two.

Probably the biggest way people participate is through their children’s schools. This is a blind spot. Should ask first about participation excluding elementary and secondary school, and then “did you go to any arts events at a school.” Otherwise, we reinforce the notion of arts as “elite.” Related, questions about venues. We should bring back questions about where participation occurs—this is relevant for funding decisions. Looking at venues is important for participation data. Asking about high schools, etc., is easier than asking about venues more broadly. If you get into venues, it’s difficult unless it’s tied to a particular art form. That doesn’t hold for schools, however. Schools are less problematic.

Broadening definitions will result in problems for the interviewers and raise questions such as; “What’s this?” “Who’s Joan Baez?” “What style of music?” The more detailed you get, in trying to pin things down often leads to a reduction in the reliability of the data. It gets complicated. Agreed, the place to look at venues is in local area studies.

Then you can say, “Did you go to a specific place?” It is problematic to look at venues in a national study. “What did you mean when you said that?”

A recent study asked a bunch of questions about the last 12 months. Then further questions about the most recent event. Asked only after you branch. This revealed problems with recall. In 1982 and 1985 we asked only about the last month. The past months sum to greater overall annual estimates. So there is going to be measurement error and inconsistency in the estimates depending on the recall period. So, we probably want to avoid changes in the recall period.

There is always going to be some definitional fuzziness on some level. We need to be comfortable, whether or not we see it, as jazz etc., because what is important is that the respondent sees it as such. We should be as precise as we can be, but on some level there will be ambiguity. Yes, categories are problematic. Keep in mind that we’re as much interested in trends, as we are definitional exactness.

Exploding “classical,” then expanding to pop, etc. Same with reading, ask about other types of reading. There is a manual for interviewers. They are trained to do some back and forth with respondents. The issue of including other kinds of music is important. A lot of the current work is looking at that. It is particularly interesting when looking at age.

It’s likely the younger kids are participating in different ways. “Some other type of live music performance, such as…” If you add that, participation estimates will go up. Is there use for that data within the NEA? It’s good to have the respondent buy-in to the questionnaire. Broadening music categories is a logical extension, but we would probably want to do it with a split sample. Then, see if the data is different and whether new questions have impacted the core of the survey.

If you want respondent buy-in, might want to talk about what they like in the beginning.

May affect data. Another way to get “yes” answers is to ask about participation in elementary and secondary schools. But, then there are comparability issues.

For media participation, add iPod, new digital devices. Maybe add reading preference questions as with music? Problem with adding is our overall concerns about the length of the questionnaire. Looking at “Reading at Risk,” this is a very rarified definition of reading. SPPA did get a little at that by asking, “did you read any book?” We certainly we want to look at nonfiction.

What about Amateur participation, e.g. “Do you play an instrument?” This is a huge gap.

We also know that knitting is gaining popularity. Not convinced that people are excluding knitting from “sewing, crocheting.”

SPPA is supposed to be a measure of cultural health. Is there some other way to get at that? Maybe ask about contentment with offerings, e.g. Amateur artmaking ties into that. Is there a model to determine cultural health instead? (i.e. develop an index of cultural health using existing data). With amateur artmaking, maybe ask how many times in the last 12 months.

There is a real need for insight into public perceptions. But I’m not convinced the SPPA is the venue. There may not be room for SPPA to address that in a comprehensive way.

Give thought to amateur, informal, not necessarily public participation. I would sacrifice comparability to get a better idea about informal participation. Top of my cut list is

“what would you like to do more?”

Music genres change over time. Also interested in a vocational participation. It’s problematic to ask these questions under the rubric of leisure. Maybe alienate the more professional artists. I have the opposite perspective—people include all kinds of things under the rubric of “leisure.” There are other ways to be as broad without using the word “leisure.” Also, if you want start with “yes” answers, you could move leisure activities up, before media participation.

If we’re extending/expanding questions and disciplines, do we do likewise for media participation—that is for all artistic disciplines, ask about live attendance, and also about watching via media. Be careful with new technologies. Seeing the things can be done on television, on the computer or even on your cell phone. Also, listening to a CD is not what it once was. May need broader, more inclusive categories.

Instead of asking about specific media, say “non-live” or “not face-to-face.” Maybe ask about ownership of types of media. Do you have an iPod, TV, CD player, etc. What about people using other people’s computers? Well, it’s messy because each person in the household may own/use different equipment. Some questions may not be answerable if we allow proxy interviews. Maybe we need a shorter survey for proxy interviews.

“Play a composition in a public performance,” should be expanded to include posting songs online. The same goes for writing (blogging, etc.) Well, roughly 10% who do art actually do public performances. Not a large number of respondents.

Typically, SPPA has updated the music categories. There has been lots of research using music preferences. Plus, we get a “yes.” Everyone has a music preference. I’d like to see

1A (go more often than you do now) cut. Only problem is that 1A has some political value. We can say, “Look, people want more.” If we expand categories, we may find that people, who go to country/rock, actually would like to go to more classical.

3.2 Key discussion points on getting greater geographic and demographic detail

One advantage of working with Westat is that we have zip code data. This is problematic with the Census. The Census is wary of geographic information as an identifier. A limitation of the SPPA is the inability to distinguish between urban and rural. Maybe get the Census to give outside MSA vs. inside MSA or low-income area vs. high-income area. With Census, you can’t get a zip code because of confidentiality.

With so many respondents, we should at least get metro vs. non-metro. Census has done that by running special tabulations for us, but won’t give us geo-codes. What about 4-digit rather than 5-digit zip codes. Or even 3-digit combined with metro vs. non-metro.

Would it be possible for them to connect zip code with an income variable like the percent below poverty? The Census could do the matching in house with another data set.

Right now there are only 10 states that love the SPPA. For the 40 others, they see it as less relevant. CPS gives ability to look at some states but with most States you run into problems of having too small of a sample.

Would it be useful if the SPPA information could say something about under-served areas, e.g. rural; people under the poverty line; race/ethnicity, language groups, immigrants; seniors, especially in rural and/or impoverished areas? Need to check with

Census to gauge their willingness to do additional tabulations. Maybe find out the density of arts organizations and have the census do the matching. There is a constituency that wants state-level data, but from an individual perspective, it may be less important.

Coding areas with high arts density sounds like a backwards way of getting at the complicated venue question. Venue is mostly relevant to a subset of participation activities, although it’s also relevant to the issue of supply. The venue question does scream for local detail. However, there are all kinds of problems with getting special access to the Census geographic data, though in theory it’s possible.

3.3 Key discussion points on adding new items or replacing items

With “what do you want more of?” maybe “rate your access.” It’s in our nation’s interest to see if people can’t do what they want to do. Better to address the access question separately with lots of qualitative data. People want to do more of everything. But it is incumbent on the NEA to look at the demand side. 1A adds some useful dimension to the demand equation. I wouldn’t sacrifice it unless we had to to get better information on participation in elementary and secondary schools.

We might also want to ask, “Would you like to have done this survey on the Internet?”

Basically what the SPPA measures is human behavior. With barriers and preferences it’s

“a little of this a little of that.” Unless we expand the barriers and preferences section broadly, it doesn’t make sense to change it. We can’t get too far into barriers and motivation without a major redesign of the SPPA.

Association of Rock with Heavy Metal is a problem. Also, we do get differences in what people want to do more of. Theatre, for example, has dropped relative to music.

On the book side…unpack. Where is there interest or not? If we are indeed reading less, what is driving it? From purchase data, we just look at people who buy books, so what’s missing is a representative sample. We only look at 15% of the population. We miss people who read but don’t buy. Are audio books big enough to make a dent? With online downloads, audio books are increasing. Also, how about adding a book analog to

2A (music preferences). Reading preferences? Is there a small set of genres we could ask about? Could squeeze categories to maybe 12. On the retailer level, they use about

20. What’s the longest short-list SPPA could accommodate? Even with 12 categories it could be a costly question as it takes a long time to get through all the different choices.

How about getting Census to put more of their data (basic CPS) on the SPPA file, including questions that were asked of the same person in earlier CPS survey months.

We can currently link to earlier months, but it is messy. Maybe we would have to pay for earlier months of the CPS survey.

Music preference is a popular question. But how important is it to ask every time. Maybe should rotate preference questions—with reading, maybe others? Or break into modules.

The Census doesn’t like to use modules. The likely maximum number of modules they would allow is two. Plus, modules result in reduced sample sizes.

One domain that’s unattended to is film [or the Media arts in general]. NEA funds art films, media centers. But it’s hard to get at/define, to distinguish between “art” and

“movies.” Maybe as a leisure activity, “did you create a film or video?” Or, “did you attend a film festival?”

Distinctive to the music preferences question is that it’s one of the only places where you can get at cultural/ethnic issues. This question often use when looking at race differences.

Also, the question of whether tastes are increasingly omnivorous. Would not want to skip a year with this question. Plus, reading and music are arguably the most ubiquitous forms of arts participation.

Trying to get people to recall when they had lessons is hard. Maybe rethink the questions.

Have cohorts, and what you did under 18 goes down as people age. People forget. Ask parents about children’s arts education? Yes, maybe that line of questioning is better than asking about people’s own arts education experience. Both are important—even if people forget, you can still compare. There’s a study that found that arts education is much more consistent in school than at home. Maybe in school vs. out-of-school is the more important distinction. Maybe drop the age detail.

Keep arts learning section, keeping the touchstones. It’s policy relevant, makes some difference in participation patterns. Also, if you want to make the case that arts education leads to meaningful gains. Some would holler if you take out the age gradations, but maybe you could simplify them. Maybe explode “elsewhere” in terms of “in-school” vs.

“elsewhere.” More detail about where arts education is happening—places of worship, community center. Maybe “take any lessons in the past year?” Venues for contemporary training would be more relevant than earlier training. College as a predictor, in college, it’s less of a policy issue, since students are self-directed.

I’d like to see questions about parents taking children to arts events. The “elsewhere” question is a surrogate for parental involvement. Compare the impacts of different kinds of socialization. “Were you taken as a child to…?” Chop age granularity into only under

18, and over 18. Ask, “what did you get as a kid?” and “what do you do for your kids.”

To do that you may want to use scales rather than a simple yes/no. Do you also have to separate by art form? In 1982, there were 4 questions that get at this. These were pretty good. Also, ask how many kids are in the household and their ages?

I’d give up the “listening to it on the radio” question. On arts learning there are 3 domains—schools, parents, and other. The family should be the unit of analysis rather than the individual. Asking about the Family should be easier than asking about experiences as a child. Parents’ education is so valuable—it correlates with so much.

The percentage of people taking lessons is so low that maybe that question could be cut.

What about cutting Travel/trips or the Internet questions? Internet use is important. GSS shows a correlation between Internet use and arts participation. Is that Internet usage or amount of time on the Internet? Is their differentiation between uploading to the Internet

vs. downloading (active vs. passive participation)? Maybe Internet should be included in leisure activities. Perhaps include Internet in media consumption.

Serious “cultural tourism” is more important than travel/trips. Tourism is a huge market, and people are making specific trips in order to have arts experiences. We need one question, better worded, to get at travel. And mileage is a better measure of magnitude than time spent in transit. It’s important that the art event is the purpose of the travel.

People in the field though, might want a vaguer wording, to produce a higher estimate.

Most of the time the arts event is not the primary motivation to travel. So, there will be some dissatisfaction in the field when the numbers go down. Having arts as the primary purpose of the travel is important.

What about skills—can you play music, or draw? Maybe try to measure the respondents’ ability? Do you play for people? Maybe we already pick that up with the amateur art making questions—e.g. “do you play an instrument?” If we do opt for getting on the

CPS for 2008, we would have more time for testing questions and wordings.

Maybe we could test questions via the Internet, which would allow us to test questions in the context of a longer survey. Is there some cognitive component to the testing? Yes, there is a qualitative component. After the test, the interviewer asks, “can we contact you?” Do we have a backup plan in place? Testing questions on the Internet could test the utility of the backup plan—assuming the backup plan is to use an internet survey firm, such as Knowledge Networks. OMB might not look kindly on a survey that uses a different mode to test the questions than to field the survey.

3.4 Concluding Remarks

Geographic detail was discussed earlier. Demographic detail consists of what the Census gives us. I want the same demographic information as in 1982—even if they have to pull from other CPS survey rotations.

Some next steps…we are going to query users that aren’t necessarily here today…foundations, service organizations, other researchers, and others within the NEA.

We’ll do on the order of 25 interviews; ask their input, especially about changes to the content of the questionnaire.

We have to proceed as if the Census is going to be able to do it, but…a plan b would be good. There’s some flexibility with the budget, so the timeline is not of huge importance.

The current chairman has been supportive—he did a great job promoting the “Reading at

Risk” report.

3.5 Summary

The discussion of the questionnaire provided numerous ideas and this summary only identifies the changes that were deemed most relevant and potentially something NEA would want to test. First, everyone agreed that making changes on most items would be risky such that we may need to consider the possibility of randomly splitting the main sample. In particular there is a danger of broadening art performance definitions in a way that could to lead to difficulty for both respondents and the interviewers in knowing what counts as art.

Asking about school performances was perhaps the easiest way to broaden the definition of arts performances. Most advisors agreed that attendance at elementary and secondary school performances was among the most common forms of arts participation, and that the SPPA’s exclusion of school events reinforces the notion of arts as “elite.” One solution would be to continue as in the past with follow-up questions about school performances (possibly limited to high school performances.)

Another area that would broaden overall participation estimates would be to create additional music categories in the music preference section. This would certainly increase participation for younger people as would expanding the media participation questions to include new ways in which people participate such as using their iPod or another digital device.

While everyone would like to get as much geographic information on respondents as possible, the Census bureau is not likely to provide much in the way of geographic identifiers. However, in lieu of detailed geographic codes, NEA should negotiate with

Census to include on the SPPA data file neighborhood measures that can be used to look at under-served areas, such as rural, high poverty, minority, immigrant, seniors etc…

The discussion of once again collecting information on barriers to participation was interesting but overall the consensus was that it would require a major redesign of the

SPPA. Other suggested changes included expanding the reading questions, possibly adding a preference list similar to the music preference list. There was lots of interest in collecting more arts education information about parents and children and/or collecting this information at the family level. There also was discussion on improving the travel/trips question to better understand the demand for cultural tourism (i.e. where the art was the primary motivation for the travel.) Finally, there was some interest in asking about a respondent’s skills, such as asking if they can play music or draw, and whether they done this activity publicly.

Appendix A: Data collection alternatives for the next SPPA

Summary:

Currently telephone surveys are suffering from decreasing frame integrity, falling response rates, and increasing costs, so the telephone random digit dial (RDD) approach used to field the 1997 SPPA is no longer a reasonable alternative. Some surveys, such as the National Election Survey, have gone away from telephone interviewing in favor of personal interviewing. The SPPA, however, requires a large sample to ensure a sufficient number of responses to examine the characteristics of participants in activities such as opera and ballet, which have relatively small participation rates. The large sample, unfortunately, makes personal interviewing too expensive. So, in effect, we have ruled out the possibility of conducting the next SPPA as a stand-alone telephone or in-person interview.

An additional problem is that the Census Bureau is busy in 2007 implementing a major software redesign of both the Current Population Survey (CPS) and the National Crime Victimization Survey (NCVS). Hence, the earliest that either of these two surveys could host the SPPA as a supplement would be May of 2008.

There are currently three potential alternatives that are being considered for fielding the next SPPA:

1) Wait until May of 2008 and conduct the SPPA as a supplement to the CPS. Given the sample size needs of the SPPA, using the NCVS would likely not be an option until late 2008 or early 2009.

2) Conduct the SPPA as a supplement to the American Time Use Survey (ATUS). The ATUS study subject matter is a more natural fit for the SPPA. The response rate for the ATUS, however, is lower than the CPS response rate and the sample consists of households who recently completed their CPS panel rotation commitment.

3) Conduct the next SPPA using an On-line Internet panel that has been recruited through acceptable probability sampling procedures. This alternative would certainly be a dramatic change for the SPPA, and this mode of data collection does have both response rate and coverage concerns. On the other hand, it is an evolving mode of data collection and substantial resources have been spent in identifying ways of improving the panel recruitment and producing estimates that benchmark well against the CPS estimates. The biggest advantage is the flexibility this mode would offer in changing or lengthening the SPPA questionnaire.

General considerations in fielding the SPPA

1) Stand-alone versus Supplement

Stand-alone survey advantages

• Increased flexibility in design and implementation

• Survey does not have to coexist with or carry sponsorship effects of another survey

• The survey could be longer, since there should be less respondent burden.

• Avoid the potential bias from having respondents being fatigued or influenced by the sponsor survey (i.e. NCVS, CPS)

• It is also possible to complete them for less money, since most of the options for being conducted as a supplement are limited and cannot be put out for bid

Stand-alone survey disadvantages

• Schedule needs to allow for more planning and development work

• Questionnaire may have to ask additional demographic questions that are currently being captured on the sponsor survey

• Would likely have to be conducted by someone other than the Census Bureau (for cost reasons) and thus the response rate will likely be lower and other issues of data quality (interviewers, trainings, scheduling) will need to be closely monitored

• Prior experience suggests that there will be a greater potential for producing estimates that differ from the prior SPPA numbers

2) Mode of Data Collection (options only for stand-alone survey)

In-person interviewing

• The SPPA started as being primarily collected in-person. In-person has the potential for achieving the highest response rates, but the downside is that is likely to be too costly. Also, clustering associated with most in-person sample designs often dramatically reduces you effective sample size.

Telephone interviewing

• The SPPA since 1985 has been primarily collected by telephone. Telephone interviewing has been popular since it is less expensive then in-person interviewing and has had low non-coverage rates, and reasonably high response rates. Unfortunately all three of these advantages have been getting worse, in particular falling response rates and coverage issues associated with increased reliance on cellular phone technology.

Traditional Mail survey

• Mail surveys are not usually chosen for household surveys because of problems with low response rates. Another problem is that there is no way to ensure that the survey is completed by a random adult or by all adults in the household. A mail survey approach though could be considered as part of a mixed mode effort.

Internet data collection

• On-line Internet panel surveys are increasing in popularity. This is in large part due to the problems of falling response rates in other modes of data collection as well as increases in Internet penetration rates. Never the less, there are still severe coverage issues that make it difficult to claim their results are representative of all adults in the U.S. Through the use of weights and improved panel recruitment they are getting better.

Mixed-mode data collection

• The SPPA has used mixed modes in the past (telephone/personal), however, the

SPPA has never been collected using a self-administered questionnaire (mail/internet). Mode effects tend to be greater when mixing self-administered with interviewer administered--though the questionnaire is not that complicated and could be self-administered. It is, however, highly recommended that any survey design that includes a self-administered component be carefully tested with the understanding that the results would be less comparable with previous SPPA results. The advantage or a mixed mode design would be the potential for increasing response rates and potentially improving the estimates.

3) Random adult respondent or interview all adults in the household

Assuming that the next SPPA is conducted as a supplement –

• Interviewing all adults in the household may require proxy interviews, unless the sponsor survey interviews all adults. On the CPS, the respondent is not a random respondent, and therefore, there needs to be an attempt made to interview other adults in the household and possibly allow some proxy interviews to be completed. Since, proxy reporting is not ideal, rules maybe needed to decide when a proxy is acceptable for all or some questions of the questions.

Advantages with random adult selection assuming a stand-alone survey –

• Less cluster effect so that the estimated survey design effect should be lower which means effective sample size will be closer to the actual sample size

• Less respondent/household burden

Advantages of interviewing all adults assuming a stand-alone survey –

• Interviewing all adults in the household lowers the cost per completed interview.

• More compatible with prior SPPA survey design (except 1997)

• Topic of interest has been shown to influence survey participation and it may also have a negative impact on efforts to select a random adult respondent. By interviewing all adult members in the household this potential bias would be eliminated. However, this would not be an advantage if the within household completion rate is low or if large of proxy interviewing is required.

4) Sample size issues

• The sample size for the SPPA has been as low as 12,349 and as high as 17,254.

The large sample size is one of the SPPA’s biggest strengths.

• A smaller sample size would put less pressure on having to reduce or limit the length of the questionnaire, but it would negatively affect the precision of some sub-population estimates. Too much reduction could raise the issue of whether any over-sampling needs to be introduced

• Likewise a larger sample size will most likely require some reduction in the length of the SPPA, but would have positive impact on the precision fro sub-population estimates.

5) Questionnaire length issues

• Longer questionnaire would most likely mean fewer completed interviews.

• Assuming the core questions remain the same, a longer questionnaire may require smaller replicates for the modules.

• If a random adult is selected, a short, core only questionnaire could be administered to other adults in the household.

• If the SPPA moves towards a self-administered format, the instrument could be longer.

• If respondent fatigue is a factor, rotating modules can help.

• Since the SPPA is asked every 5 years, the questionnaire length could be reduced by having some modules only asked every 10 years.

6) Interviewing Field Period

Assuming that the next SPPA is conducted as a supplement – Comment [T1]:

• Then the field period will likely be dictated by when the sponsor survey is able to field the supplement.

Assuming that the next SPPA is conducted as a Stand-alone survey-

• The field period would be an important consideration in the overall survey design.

• Estimates will have less potential for seasonal bias if a longer the field period is chosen and the completions are spread out across most of the year.

• A longer field period would also provide more time for extended contact efforts, which usually improve response rates.

• A longer field period will cost more and could have a negative impact on either the sample size or questionnaire length.

• A longer field period would delay availability of the data.

A longer field period has an increased potential for bias by a major media event

(ex. – 9/11, or some major artist or art form making headline news).

Appendix B: Recent Research Using SPPA Data

The bibliography below focuses on research dating from 1992 – present, and which employs or refers to SPPA data. As such, it is not a comprehensive bibliography of research on arts participation. Rather it represents different approaches to using SPPA data, and the variety of topics researchers have pursued. Many sources are drawn from RAND Corporation’s Guide to the Literature on Participation in the Arts. Additional sources come from Princeton University’s Center for Arts and Cultural Policy Studies’ Resources for Studying Public Participation in the Arts; the National Endowment for the Arts; and from searches of research databases, including Jstor, Econlit, and EBSCOHost’s Sociological Collection, and Psychology and Behavioral Sciences Collection.

The below chart corresponds to the bibliography and illustrates the variety of approaches to using SPPA data. The distinction into empirical, theoretical, and practitioner-oriented literature is borrowed from RAND, and was applied consistently. In addition, research may be concentrated on demographics, particular disciplines, or other focuses.

Orientation of research and number of studies focusing on particular topics

Orientation Demographics Discipline Other Focuses Empirical 41 Age 3 Dance 1 Communities 3 Theoretical 23 Education 3 Literature 3 High/low Culture 2 Practitioner 6 Geography 1 Museums 5 Leisure 5 Race/ethnicity 4 Music 5 Longitudinal 5 Performing Arts 3 Media 2 Theater 2 (Note: Categories are not mutually exclusive.)

Empirical research is that which describes the data—answering questions such as who, where, when, and how people participate in the arts. Theoretical research, by contrast, attempts to answer the question of why people participate. The Rand Guide also noted that many more arts participation studies are empirical than are theoretical.

AMS Planning & Research Corporation. American Participation in Theater. Santa Ana, CA: Seven Locks Press, 1996.

———. A Practical Guide to Arts Participation Research. Washington, D.C.: National

Endowment for the Arts, 1995.

———. Summary Report: 12 Local Surveys of Public Participation in the Arts.

Washington, D.C.: National Endowment for the Arts, 1993.

Aschaffenburg, Karen and Ineke Maas. "Cultural and Educational Careers: The

Dynamics of Social Reproduction." American Sociological Review 62, no. 4 (August 1997): 573-87.

Bergonzi, Louis, and Julia Smith. Effects of Arts Education on Participation in the Arts.

Santa Ana, CA: Seven Locks Press, 1996.

Borgonovi, Francesca. "Performing Arts Attendance: An Economic Approach." Applied Economics 36, no. 17 (September 20 2004).

Brudney, Jeffrey L. "Art, Revolution, and Arts Education." Society 27, no. 6 (September/October 1990): 17-20.

Cherbo, Joni Maya, and Monnie Peters. American Participation in Opera and Musical Theater 1992. Santa Ana, CA: Seven Locks Press, 1995.

Collins, Paul. "Decline and Fall and Fall and Fall. In Our Minds and in Our Cars:

Technology Killed the Literary Star--Again." The Village Voice (New York, NY), September 8 - 14 2004, News.

Deveaux, Scott. Jazz in America: Who's Listening? Santa Ana, CA: Seven Locks Press, 1994.

DiMaggio, Paul and Bruce Western. "Adjusting for the Effects of Proxy Interviews on Estimated Participation Rates in the 2002 Survey of Public Participation in the Arts." Working Draft. Princeton University, 2005.

DiMaggio, Paul and Francie Ostrower. Race, Ethnicity, and Participation in the Arts.

Santa Ana, CA: Seven Locks Press, 1992.

DiMaggio, Paul, and Francie Ostrower. "Participation in the Arts by Black and White Americans." Social Forces 68, no. 3 (March 1990): 753-79.

Falk, John. "A Framework for Diversifying Museum Audiences: Putting Heart and Head in the Right Place." Museum News September/October 1995.

Felton, Marianne V. "On the Assumed Inelasticity of Demand for the Performing Arts."

Journal of Cultural Economics 16, no. 1 (June 1992).

Gray, Charles, M. Turning On and Tuning In: Media Participation in the Arts. Santa Ana, CA: Seven Locks Press, 1995.

Heilbrun, James. "Growth, Accessibility and the Distribution of Arts Activity in the United States: 1980 to 1990." Journal of Cultural Economics 20, no. 4 (1996).

Heilbrun, James, and Charles Gray. The Economics of Art and Culture. Cambridge:

Cambridge University Press, 1993.

Hendon, William S. "The General Public's Participation in Art Museums: Visitors Differ from Non-Visitors, But Not as Markedly as Case…