REUTERS/Yiorgos Karahalis

Incomplete data, inconsistent definitions, and an outsider's perspective can all trip up attempts to measure cities against one another

Besides its coverage of a wide spectrum of urban issues—from mystifying crosswalks to the false economics of sprawl—TheAtlanticCities.com aims to share ideas about best practices. And for those who think about cities, one important way to do that is to find innovative ways to rank certain urban performance indicators as a means of determining what makes some regions succeed. A pertinent recent example: the Global Creativity Index, developed by our own Richard Florida and his team at the Martin Prosperity Institute at the University of Toronto.

Academics who do this work know that reliable and current data is crucial but elusive. For example, national or state/provincial level economic data may not be sufficiently granular to reveal trends at the urban or municipal level. There are also boundary issues: for the purposes of collecting statistics to generate indicators, does “the city” constitute the political entity, the census metropolitan area, or, in the case of large urbanized regions, the agglomeration of several CMAs?

The Global City Indicators Facility, a research unit at the University of Toronto founded by director Patricia McCarney in 2009, is building a city-by-city database of up-to-date, meticulously delineated statistical information on a wide range of urban metrics—everything from crime rates to building permits—that can be used by municipal officials to benchmark their decisions. The project, overseen by a board representing member cities, traces its roots to concerns from World Bank officials about sketchy data on urban regions.

The University of Loughborough’s Globalization and World Cities Research Network adopts yet another approach, measuring a broad range of economic and private sector indicators to determine the commercial interconnectedness between large cities as a means of assessing the global hierarchy of urban regions.

Despite such efforts to inject academic rigor to the business of measuring urban performance, the rankings that tend to get the most public attention often suffer from serious methodological shortcomings, according to Zack Taylor, a Toronto-based planning consultant and Ph.D. candidate who has written a trenchant study (“Lies, Damn Lies and Statistics: A Critical Examination of City Ranking Studies”) on the shortcomings of highly publicized city rankings produced by private organizations and extensively promoted in the media.

His 78-page report, commissioned by Canada’s Intergovernmental Committee on Economic and Labour Force Development, aims to shine an analytical light on the growing number of privately-commissioned urban rankings, including those produced by Mercer Consulting, Mastercard, KPMG, UBS, PriceWaterhouse Coopers and others. These studies attract wide-spread media attention, and find their way into the promotional material produced by municipal and regional economic development agencies and local chambers of commerce.

Taylor, founder of Metapolis Consulting, points out that the ostensible purpose of such rankings is widely overlooked: the sponsors aim to sell their proprietary data to multi-national corporations as a means of helping HR teams establish cost-of-living allowances for senior managers and executives posted to off-shore offices. “They can’t tell the story of the actual experience of the people who live there because they are meant to evaluate the cities from the perspective of outsiders,” says Taylor, who describes that discovery as a “revelation.”

The means by which these studies are promoted to the media also suggest the sponsoring firms clearly recognize the brand bounce associated with the inevitable coverage.

The problems aren’t just limited to mandate. After examining several leading studies over a period of several years, Taylor discovered that annual scores can shift unpredictably due to methodological issues, such as fluctuations in global currency markets, changes in evaluation criteria, incomplete data, inconsistent definitions of what constitutes a particular urban region (i.e., the census metropolitan area, political boundaries, etc.), and the choice of cities being evaluated.

In the report, he also notes there’s an important but often unacknowledged distinction between a city’s score and its ranking: due to very small shifts in the data, a city’s overall ranking may slip relative to the others on the list, thereby magnifying the impression that something serious has gone amiss, even though a city’s overall score may have barely budged.

So when it comes to publicizing such findings, Taylor has this bit of sage advice for journalists, readers, and economic development officials alike: “Be skeptical.” 

About the Author

Most Popular

  1. Design

    These Sneakers Are Your Free Transit Pass

    A new BVG-Adidas collaboration means unlimited travel along Berlin’s public transit network for the rest of 2018. That is if you can find a pair.

  2. Environment

    Britain's Next Megaproject: A Coast-to-Coast Forest

    The plan is for 50 million new trees to repopulate one of the least wooded parts of the country—and offer a natural escape from several cities in the north.

  3. Equity

    How Today's Developers Maintain Jim Crow Housing Segregation

    A private Brooklyn developer and a Louisiana public housing authority share the belief that racial division in housing has its plusses.

  4. A small accessory dwelling unit—known as an ADU—is attached to an older single-family home in a Portland, Oregon, neighborhood.
    Design

    The Granny Flats Are Coming

    A new book argues that the U.S. is about to see more accessory dwelling units and guides homeowners on how to design and build them.

  5. Transportation

    On Paris Metro, Drug Abuse Reaches a Boiling Point

    The transit workers’ union says some stations on Line 12 are too dangerous to stop at. What will the city do?