Rosana Pellizzari, the medical officer of health of Peterborough, Ontario, knows a thing or two about bad data. The public health office she oversees is charged with running policy-driven health programs and services for the mid-size city and county, population 123,000, which makes it the 33rd largest metro in Canada, if that country's most recent census is to be believed.
Trouble is, she's not sure it can be. In 2010, with little fanfare or preparation, Prime Minister Stephen Harper’s conservative government decided that the next long-form census, completed in Canada every five years, would not be mandatory. As officials told the story, without citing any specific polls, the public had expressed concerns about their privacy when filling out the long-form census, as well as the threat of jail time should they decline to fill it out.
“We were all shocked,” says Marni Cappe, who in 2010 was the president of the Canadian Institute of Planners. “It sent a ripple through the community … They did it on a [June] afternoon when they thought, ‘Who would be paying attention?’”
So in 2011, Statistics Canada, the governmental body responsible for collecting and analyzing all of Canada’s statistics, sent out two versions of the census. The first, a mandatory short-form questionnaire, asked Canadians about their about age, sex, marital status, mother tongue, and the languages spoken at home. The second was the National Household Survey (NHS), a 40-page voluntary survey sent to 30 percent of Canadians. Munir Sheikh, then the head of Statistics Canada, resigned over the change in policy. “I want to take this opportunity to comment on a technical statistical issue which has become the subject of media discussion ... the question of whether a voluntary survey can become a substitute for a mandatory census," he wrote. "It can not.”
Today, four years after Canada’s first voluntary long-form census, and one year away from what looks to be its second, Rosana Pellizzari is still wondering how to deal with the dearth of data. Thirty-six percent of Peterborough residents sent the voluntary survey did not return it, which gave the metro the lowest NHS response rate of all Canadian cities.
“Essentially, it means that we’re not able to develop a clear picture of our population’s health,” says Pellizzari.
Peterborough doesn't have the money to complete its own public health surveys. So as Pellizzari and her team work on a new child health report for the region, she says Peterborough’s government is forced to use nearly 10 year-old information, from when the last mandatory census was completed. “There are indicators we are not able to derive because we’re missing the numerator and the denominator,” she says—meaning both the total number of children in Peterborough and the number of children affected by specific illnesses and diseases. “We don’t have the data.”
Though Peterborough’s situation is particularly difficult, it’s not an anomaly. In 2006, 93.5 percent of Canadians responded to the then-mandatory long-form census. In 2011, 68.6 percent returned the NHS. This is despite the fact that census officials distributed more surveys to compensate for the predicted drop in response rates—to one in three Canadians in 2011 instead of the one in five in 2006. Still, Statistics Canada withheld 2011 NHS data for 1,128 of 4,567 Canadian census subdivisions. “[A]pproximately 25 percent of geographic areas do not have reliable National Household Survey data available for their use,” Canada’s auditor general wrote in a 2014 report.
Here’s the kicker, though: This new survey, the one that provides less data than the one before it and has left academics, government officials, demographers, social planners and businesspeople wondering how to calculate exactly how many people live in Canada, and where, also cost more money. Maclean’s, the Canadian weekly, reports that Statistics Canada needed an extra $22 million to cover the “costs associated with increased questionnaire production and mail-out.”
“Voluntary surveys are simply a waste of money,” Munir Sheikh, the Chief Statistician who resigned over the switch to the voluntary census, tells CityLab in a phone interview. “[They] cannot provide you the kind of accurate information that you need to make your policy decisions. So in my view, this is the worst of both worlds.”
There seems to be little chance that the conservative government will reverse its stance on the census while it’s in power, despite the outcry from groups as varied as the Canadian Nurses Association, policy think tanks, the Canadian Chamber of Commerce and United Way Toronto. A recent attempt by Liberal Ontario MP Ted Hsu to bring back the long-form was hopeless, commentators agreed, well before it was voted down on February 4. Analysts say it will take a change in government to return to the mandatory version. Meanwhile, this has become far more than a fight over statistics. How best to distribute Canada’s census forms has become a political and ideological question, too. As one conservative commentator wrote in 2010:
"If you measure it, it matters” is the motto of those net tax-receiving organizations who only matter if they can make their case. Prime Minister Stephen Harper has tried the ideological argument against these groups for years. But ideology is by its nature debatable; removing the framework of debate is his shortcut to victory. ... I imagine that [in] Stephen Harper’s view, Canada should be a country of individual initiative, not one of collective dependence “justified” through the collection of data.
Missing the underserved
“Because of the move to the voluntary NHS, Canada is a richer, whiter, more educated country now,” says Ryan Berlin, a Vancouver-based economist and demographer with the non-profit Urban Futures Institute. Berlin is making a joke here, one that’s been making the rounds in Canadian academic conferences for the past few years. But he’s not wrong. Certain populations—low-income residents, immigrants, the disabled, aboriginal peoples, and those without a firm grasp of the English language—were far less likely to return the voluntary census. These are also often the communities most in need of social programs. The question marks are particularly disconcerting in the wake of the worldwide recession. Where are the needy? Canada isn’t entirely sure.
While statisticians with the Canadian government do have sophisticated mathematical tools to help estimate how many underserved citizens they missed, the 2011 still survey left glaring uncertainties. In one example, the NHS found that Filipinos were the most represented group among immigrants who entered Canada between 2006 and 2011. But a footnote in the Statistics Canada release notes that this result is “not in line with administrative data from the Department of Citizenship and Immigration Canada which provides the number of recent immigrants by their country of birth settling in Canada each year.” Why the gap? It could be sampling errors, it could be response patterns, or it could be an “under or over estimation of certain groups of recent immigrants in the NHS.” Officials say they just can't be certain why they don’t know what they don’t know.
And how many First Nations people live in Canada? Four percent of the reserves in the country were “incompletely enumerated” in the NHS, Statistics Canada wrote in its data release. For that reason, the agency fears that it overestimated the number of Inuit living in the country. The 2006 mandatory census found that 40 percent of all indigenous Canadian children lived in poverty, more than double the national average. Now the picture is less clear.
Smaller places are also disproportionately affected. Marc Hamel, the director general of Statistics Canada’s Census Management Office, has said that there are few alternative data sources for very small towns. “[W]e are cautioning users,” he told the Global News in 2013. “[W]e can’t say if the information is in line with reality in these locations.”
The census switch has had implications for big cities, as well. Officials from Toronto, Montreal, Vancouver and Regina brought up the NHS during a big city mayors meeting this February. Ryan Berlin, who often works on housing issues in Vancouver, says the voluntary census has already begun to affect his city’s policies. The 2011 data, for example, shows that a decreasing population of older residents of the western Canadian city are living in their own homes, and that more and more senior citizens are moving into assisted living homes. That’s a significant deviation from 2006 and 2001 numbers.
“If you take the data in that respect at face value—‘Wow, a greater proportion of seniors now need to live in collective-type dwellings’—then we need to ramp up supply of that type of housing,” Berlin says. “But that really flies in the face of a lot of things we know about seniors: that they’re living longer, living longer in couples. That is just befuddling, and it makes [crafting policy] difficult.”
Still, policymaking has rolled on. Decisions, as Sheikh, the statistician, says, “need to be made, so they are going to get made, whether you have bad data or good data. If you have bad data, you’re going to make bad policy decisions."
Are there other ways for government officials to get the specific kinds of data they need? Statistics Canada does put out other products (the Labour Force Survey and the Survey Of Employment, Payroll, and Hours, for example), that aren't dependent on census data at all.
But when it comes to community sizes, Martin Cooke, a sociologist at the University of Waterloo, writes that though the “2010 changes have been harmful,” the political reality is that Canadians need to figure out new ways to find numbers. Thanks to the Internet, in many ways marketers know more about consumers than ever before. And there’s a wealth of administrative data—hospital records, tax statements—that’s currently unavailable because of ethics and privacy concerns, but which could be anonymized and used to gain a firmer picture of what’s really happening in Canada. Why have a big census when you have Big Data?
But hospital records only capture those who go to the hospital, and information held by private companies doesn't tend to be made available to “academic troublemakers” who do not endear themselves to firms, as Frances Woolley, an economist with Carleton University, points out. Census data is available to all, and is particularly important when academics go back to review their colleagues' numbers. “When data is all in private hands, there’s no right to get ahold of it,” Woolley says.
Academics also stress that census data is often used as a benchmark, to check whether other data sets derived from alternative sources are correct. Now there appears to be no universally-acknowledged set of numbers against which to check one’s own work.
The U.S. census has its own opponents
So could this happen in the U.S.? Though the idea lives on the fringe of American politics, it has some supporters. Once again, the fight is highly political. Republican Representative Ted Poe of Texas has introduced legislation to make the American Community Survey, the U.S. equivalent of the detailed long-form census, voluntary. (Currently, those who don’t fill out the mandatory ACS technically face fines, though the bureau hasn't gone after anyone for failing to fill it out in decades.) "I think it’s an example of the government forcibly trying to find out more information from people," Poe told Politico. "It violates their privacy. It’s no business of the government to have this information.”
In March 2014*, that legislation received a hearing in the House Oversight Committee, though Chairman Darrell Issa decided not to follow through with the bill after it was savaged by his own party's called witnesses. Republican Senator Rand Paul of Kentucky—a presumptive presidential candidate—has introduced parallel legislation in the Senate.
“This is a very U.S.-centric country,” says Phil Sparks, who co-directs the non-profit Census Project, a coalition of groups that advocate for the census. “There’s some recognition of the disaster that befell StatsCanada … State and local government organizations, planning agencies, unions, civil rights groups, professional organizations, they’re acutely aware. Now I’m trying to convey that on Capitol Hill.”
The bigger problem here—both in the U.S. and Canada—may be that most citizens don’t fully understand what happens when they send that lengthy questionnaire back to the census bureau. Berlin, the Vancouver economist, had a real estate developer give him a call recently to ask for a profile of a specific community.
“I just had to explain, ‘There are no data to answer your questions.’ In his mind, the data was just there, because [I'm] a researcher,” Berlin says. “People don’t have a great understanding of how data collection and analysis works. I don’t think the average person understands that.”
Illustration by Mark Byrnes with elements from Wikimedia Commons under a Creative Commons license.
*CORRECTION: A previous version of this article misidentified the date of the House Oversight Committee hearing on the American Community Census. It was 2014, not 2013.