PERSPECTIVE: Survey Data; Not What It Used to Be

by Michelle Riordan-Nold In public debate, I increasingly hear the phrase: data-driven decision making. At CTData part of our core mission includes advocating for the public availability of data to inform policymaking. But what does it really mean and what data are we talking about?

As people increasingly use data to inform business or policy decisions, the quality of that data becomes even more important. Using data responsibly is something creators and users need to take seriously. Part of this includes understanding the drawbacks of data collection methodologies.

I recently attended the Association of Public Data Users conference where several of the speakers discussed the increasing unreliability of survey data.

As it turns out, most of what we know in social science comes from surveys of households. For example, the unemployment rate, poverty rate, rate of inflation are all collected through household surveys. Federal household surveys are used to make macroeconomic policy, they are used in indexing government benefits, and in determining tax brackets.

Surveys tell us a host of factors about people's lives, including: what people are doing in response to government programs, their level of education and employment, and how they spend their money, just to name a few. They tell us how the economy is operating and how government programs are working or not working. However, rarely do we talk about the challenges and deficiencies of survey data, yet we rely very heavily on this data for decision making.

Surveys as an instrument for collecting reliable data are deteriorating. Over time, as Bruce Meyer, the McCormick Foundation Professor at Chicago Harris, notes, people are "less willing to participate in surveys, less willing to answer the questions, and when providing answers people are less likely to give accurate answers than they did in past. People are over surveyed." Frankly, the allure of being surveyed and giving your opinion is no longer a thrill.

As the saying goes, ‘a picture is worth a thousand words.' The chart below, taken from a recent paper by Meyer, shows that the non-response rate for five key national household surveys has been creeping up over the years. People are not responding that they receive government services even when they actually are the recipients of government programs.

But it's not just a problem of people not responding, it is also an issue that the information they provide is also quite poor. Meyer's research revealed that

"in our most used survey, that's the source of official income and poverty statistics, only about half of people receiving food stamps report it, under 40% of those receiving cash assistance report it. If you want to know who is poor you get a very bad picture from just surveys alone."

How did he figure this out? Professor Meyer linked the main household survey data to government program data (also known as administrative data). In a secure research data center, using anonymized data, he was able to look at what a recipient says in a survey to what the recipient is actually receiving.

"In surveys, the poverty rate looks much higher than what it really is; second these programs look less effective than what they are because much of the receipt is missing; in addition it looks like people who you think should be receiving these programs aren't."

What does this mean for policy?

On the one hand, it makes it look like the poor are doing much worse than they are. It also makes government programs look less effective than they are.

Meyer was involved in federal legislation that called for the establishment of a commission to look into ways that administrative data could be made more widely available to administrative agencies to determine whether programs should be expanded or contracted and to provide access to researchers both inside and outside government. The commission would be staffed by program or data experts, experts on data confidentiality and security, and with equal appointments from the House, Senate, and White House.

If these statistics were found to be so unreliable at the national level; what does it mean for Connecticut?

Relying on survey data is no longer a reliable measurement tool. It is time to look at the administrative data and examine the effectiveness of state government programs. This can be done. It should be easier with the passage last year of Public Act 15-142 that gives the Office of Policy and Management authority to link agency data and provide it to public researchers, as well as with the continued development of the P20-WIN data sharing initiative.

In order for policymakers to make 'data-driven decisions' we need to:

1. define the outcomes desired by government programs;
2. ensure the data are being collected to measure efficacy;
3. analyze the data to measure the programs; and
4. take action on the research results to ensure efficient allocation of resources.

However, this requires good administrative data that must come from those administering the programs. Let's get a commission together to ask the questions, determine the needs, and analyze the data.

With the fiscal challenges the State is facing, it's important that policymakers ensure that dollars are well spent and government programs are working. Access to better data can only lead to better government decisions.

______________________________

Michelle Riordan-Nold is Executive Director of the Connecticut Data Collaborative

PERSPECTIVE commentaries by contributing writers appear each Sunday on Connecticut by the Numbers.

LAST WEEK:  The State Budget:  What Do Demographics Have to Do With It?