Emily Badger is a former staff writer at CityLab. Her work has previously appeared in Pacific Standard, GOOD, The Christian Science Monitor, and The New York Times. She lives in the Washington, D.C. area.
"Sentiment analysis" of social media could change the way you report potholes forever.
You probably know about a couple of ways to log formal gripes with your city about things like downed tree limbs, crummy garbage service, or graffiti. You could call up your councilmember, or file a 3-1-1 ticket, or even turn up for a public meeting. Many cities have long been trying to modernize this process for the 21st century, by allowing residents to track responses to their complaints online, or by letting them dash off geotagged missives by smart phone app.
But what if technology could go just one step further? What if you never even had to contact City Hall about that needed road repair on Main Street, because City Hall already saw your tweets? (And we know you spend a lot of time on Twitter moaning about #potholes).
We wrote back in January about this idea, after some academics at Purdue proved that they could track live customer satisfaction on the Chicago Transit Authority’s ‘L’ trains by data-mining riders’ tweets. Now the concept – social sentiment analysis – is coming to whole cities. IBM, under its Smarter Cities program, is now marketing software to municipalities that claims it can determine aggregated public opinion – positive, neutral and negative – in the millions of data points that make up our social media ramblings. Its most impressive trick, as an algorithm, is that it can read natural language (as opposed to the Proper English you were taught in freshman composition, which none of us speak online). IBM (which, yes, is an occasional advertiser on this website) even claims that its software can reason out the difference in intent between sarcasm and sincerity.
Municipal governments are, not surprisingly, a little slower to this leading edge of social media analysis. For-profit companies have already figured out how to deploy software that trolls the Internet for consumer reaction to products and brands. Even the federal government has expressed an interest in similarly data-mining social media for clues about upcoming civil unrest or disease outbreaks (in America and abroad).
The challenge for cities is how they might parse social media sentiment about not just one dish detergent (or one frequently Googled query during flu season), but about numerous interlocking indexes of civic life. Are parents in Chicago supportive of the teachers’ strike? What are New York subway riders saying about that new trash program? Or Los Angelenos about the crackdown on pot dispensaries? Is there a flare-up of graffiti concern on the west side of the city? Or a collision on the east side about to erupt into an all-out traffic jam?
Our world now spits out by the second unconnected data points on all of these questions. This is both a boon and a problem. “Those inputs are now starting to overwhelm structures,” says Chris O’Connor, vice president for IBM Industry Solutions Software, “whether that’s a public structure like a city, or a private structure like the people who run a football stadium or an airport.”
Cities have always had old-school, plodding ways of collecting smaller quantities of this information. In addition to those 3-1-1 reports you file, you may have taken a phone survey for a government agency or responded to a mailer.
"There are a bunch of analog versions where you’re more intrusive in asking the person to take a minute to respond," O’Connor says. "We’re trying to go where the people are already publicly expressing themselves."
If the thought bothers you of a government-deployed algorithm listening in on your public expression, maybe you don’t want to complain about potholes on Twitter after all. These sentiment analysis tools work precisely because we’ve made so many of our private thoughts publicly available in cyber space. From a city’s perspective, the Internet is the new town hall.
Before IBM unveiled its Social Sentiment Index this week, it tested the system in India, scouring social media in Mumbai, Bangalore and New Delhi around the subject of traffic. People in Bangalore, it turns out, view their parking situation three times more negatively than commuters in Mumbai and New Delhi, although Delhites appear much more stressed when it comes to traffic than residents of the other two cities. Sentiment analysis is valuable when it aggregates the moods of so many people, but the big picture that emerges could also point to specific action.
never been happier to realize trash day is delayed a day for a holiday :D— Lauren Siegert (@ljs39) September 5, 2012
"Think about health care services, think about the pothole, think about the bus and transportation capability that could be rerouted and moved as a result of understanding things a few minutes sooner," O’Connor says. And, yes, you’re probably tweeting about problems faster than bureaucracies can report them. "Five minutes one way or the other makes a huge difference in a traffic jam."