Jason Sayer is a freelance journalist based in London and New York, covering subjects related to the built environment.
So far, major forays into our augmented world have been pretty harmless. But with technological advancements and unchecked intrusions by private companies, the future could be terrifying.
I’m sitting at New York University (NYU), in a Brooklyn Tandon School of Engineering building with wires sprawling across my body while I stare at a 98-inch TV screen. Glasses with inward-facing cameras are tracking my pupils; plaster wrapped around my finger is measuring my perspiration; cheek pads are monitoring whether or not I’m smiling; another device looks at my heart-rate; and an electroencephalogram (EEG) headset—a contraption with wet squidgy nodes on the end of prongs, similar to a head massager—monitors my brain activity. All while my face is being filmed.
The researchers want to quantifiably measure my response and stress levels to two slightly different virtual environments. On the screen I am looking at a basic scene created in Google SketchUp. Mock-ups of an existing NYU building are identical minus a couple of minor differences: one has wider windows and lighter wall colors. Within the two environments I have to navigate my way up a set of stairs onto a mezzanine level, open a door, switch on a thermostat, and return.
Hardly thrilling (that’s the point), but Semiha Ergan, an assistant professor at NYU, is keen to see if the architectural theory behind such rudimentary design aspects is true and can quantitatively measure how much of a difference there is. Though the researchers couldn’t disclose the architecture firms actively interested in this research, I was told that a few well-known “three-letter” firms were keeping a close eye on things.
As you might have guessed, Ergan’s hypothesis was right. In the supposedly less stressful environment that had wider windows, I perspired less, my heart-rate was lower, and my facial expressions along with the EEG monitoring also indicated lower stress levels. The ramifications of this research pertain not only to virtual reality (VR) but augmented reality (AR) services, in which computer-generated visuals are superimposed on a tech user's view of the real world
Unfortunately, Ergan’s test didn’t include Balloon Dog, Jeff Koons’s AR-only sculpture in Central Park. The virtual sculpture appeared exclusively on Snapchat screens in New York’s Central Park in late 2017 when users visited the area. Chilean artist Sebastian Errazuriz decided he wanted to vandalize the work using virtual graffiti as part of a “stance against an imminent AR corporate invasion.” After submitting his version to Snapchat without a response, he placed Balloon Dog in the same geo-tagged location as Snapchat’s using his own app, ARNYC.
The response gained traction online—Hyperallergic soon declared it, “the first act of vandalism to occur in the realm of augmented reality.” Errazuriz’s “vandalism” and the attention it received may be a preview of what to expect from augmented additions to the public realm through social media apps.
Inevitably, advertising will join the augmented public realm. It already has in less immersive ways, including televised sports broadcasts that project advertisements onto to the field of play. Likewise, visual recreations of dead artists can now appear on a concert stage.
Advertising online is hyper-targeted thanks to services like Google AdSense, which employs user-tracking cookies to show the most relevant ads for a user as determined by algorithms. This system relies on “impressions”—essentially, each time an advert is shown by Google, either on a Google search or on a advert box operated by Google—something which, as Ergan’s test has shown, could be tracked easily when monitoring eye movement if actuated in a virtual or augmented environment. But what happens when the echo chamber of social media is married with AR and VR? “The effects could be the real-world amplification of what we're seeing online,” architecture historian Owen Hopkins told CityLab. “It could drastically affect our everyday experience of reality.”
Physical ads have a propensity to gain cultural significance. The iconic yellow arrowhead signs of In-n-Out Burger across the west coast have become allied with the culture surrounding the fast food restaurant itself. In the U.K., Catford’s “Big Cat,” a locally famous piece of mimetic architecture in South East London that advertises the Catford shopping center, evokes so much civic pride that residents campaigned (successfully) to save it when developers proposed its demolition. But AR advertising lacks phenomenological qualities and is bereft of permanency, all of which can diminish a sense of place. How would both these scenarios have played out in a world with AR advertising? Would a claim for the preservation of a virtual object—an AR version of Long Island City’s 5pointz, for example—have any merit, say if the firm that implemented that AR decided to discontinue it?
Stefan Simon, director of the Institute for the Preservation of Cultural Heritage (IPCH) at Yale University, describes the advent of digital and virtual technology in architecture as the “4th industrial revolution.” Speaking to CityLab, Simon warned that without developing the proper tools to save it, much of our digital heritage will be lost. “Do people have an agency to save these bits and bites? Yes, of course they do,” he argues. “Does the conservation community have the proper tools in place? No.”
Simon references Vint Cerf, an internet pioneer who a few years ago said that future historians looking to study the 21st century will find an "information black hole." This is because, according to Simon, the programs needed to view our digital files will soon become be obsolete. “Obsolescence is a key factor in this process [and] migration of data alone won’t help to mitigate this threat,” he says.
In the more immediate future, it is perhaps the printed and sign-based advertising industry where AR could be most disruptive. Companies can’t place their ads on a wall or billboard without permission, but with AR it is easy to see how advertisers could abuse their free reign. Media theorist Franco Berardi aruges in Precarious Rhapsodies that the rising overload of information in the average person’s daily life has already “produced a saturation of human attention that has reached pathological levels.” Will people even want to engage or be able to process AR-heavy spaces?
Japanese filmmaker Keiichi Matsuda’s “Hyper-Reality” demonstrated in 2015 how dangerous a dependency on AR could be, with supermarkets devoid of any meaningful physical design, filled instead with blank spaces for AR to overlay and fill. AR is perhaps even more intrusive on the street, bamboozling the unnamed protagonist in the short film with an overload of adverts and information. The city had become subjective; shared experience in public spaces instead eschewed for personalized and augmented perceptions of the environment.
But, with Semiha Ergan’s results applied, AR could in fact lower stress. Some aspects of it in Matsuda’s “Hyper-Reality” are genuinely beneficial—even life-saving, such as the red warning arrows that indicate the path of oncoming vehicles.
Space Popular, an architecture firm in Bangkok, is also exploring the positive possibilities of AR. Run by Lara Lesmes and Fredrik Hellberg, the pair proposed what they call a “multidimensional memorial” for Cubryna Park in Warsaw, Poland. Named Kazymierzowsky Rebound, the project sees a basic steel structure act as a template for AR renditions of former versions of the Kazimierzowski Palace. The palace dates back to the the 17th century when it was built for King Władysław IV in 1641 and has been significantly transformed five times since. With the steel being allied physically with recovered stone fragments of former palaces, the project, according to Hellberg, “links the incredibly real to the wholly virtual.”
“This is good because perception of what is real and what is not facilitates a better understanding of place,” he says. Hellberg is also concerned with future architects using AR. “Interaction with architecture will soon be dynamic and architects need to be involved in the cognitive process of seeing,” he said. “We will be hiring someone to amplify experiences of spaces through augmentation and most young architects will be designing virtual/augmented enhancements.” Hopkins views this pessimistically, asking rhetorically, “Do architects just become video game designers?”
If we are to take the quantified data from NYU’s research, then AR’s use by architects in some cases merely needs to be algorithmic and—like video games—scripted by computers. “This completely breaks down the architect/client relationship, which is the bedrock of the profession since it became a profession,” Hopkins says.
Like Matsuda and Hopkins, Hellberg is wary of AR’s darker prospects. “Local governments will need regulation for [AR] in the future because it will become intrusive,” Hellberg warns. “We need to create some sort of order. That will be the last phase of this digital revolution.” Errazuriz echoed this sentiment when he told The New York Times last fall that the public should charge Snapchat for its use of “virtual public space.”
Some may argue that Snapchat having AR autonomy isn’t a bad thing. After all, you do choose to use the app. However, there is the possibility of it being bought out, as Koons’s Balloon Dog demonstrated. This doesn’t pertain just to advertising, but privacy as well. Residents in Neo Bankside, across the street from the Switch House viewing platform at the Tate Modern, complain about nosy museum-goers peering into their homes. It’s entirely possible that Snapchat could one day charge those residents to put fake blinds or curtains in their windows when Snapchat users point their lenses at them.
It is not far-fetched to envision Privately Owned Augmented Public Spaces coming to fruition. Though only hypothetical, these would be similar to Privately Owned Public Spaces (POPS). Indeed, POPS are a precedent worthwhile studying to imagine what “POAPS” could be like. Business Improvement Districts—a form of POPS —adopt a “clean and safe” mantra and can sometimes result in aggressive efforts to relocate the homeless or restrict recreational activities and demonstrations.
The debate surrounding a right to augmented space gets further off the ground when we consider moving on from app-based AR, which relies on smartphones, to technology akin to RideOn goggles, which are currently being developed for extreme sport usage. Who is to say the owners of POPS will not edit out the homeless through their augmented view of the world? On a larger and more terrifying level, governments could even alter, or rather augment, their cities to hide politically inconvenient signs of dissent or distress.
AR is not yet advanced enough for humans to not be able to distinguish what is real and what is not. It is perhaps this “faulty” aspect that makes a project like Space Popular’s Kazymierzowsky Rebound so effective. In a similar vein, Pokémon Go’s charm is in part due to its endearingly unrealistic graphics. The app, released in 2016, presents AR as a force for good. In a 2017 study of the game’s play in Australia, researchers Lisa Jacka and Lachlan Hartley Yee found that AR added depth to player’s experience of public space, heightened their awareness of it, and encouraged exploration of urban areas.
Moreover, the findings shed light on urban planning principles. Players, for example, preferred outside, well-lit, and walkable environments where they could see others around them. The researchers subsequently argued that AR games could supplement placemaking efforts, and also supply framework for planners to see how people move through space in real time. There would also be a ton of data just waiting to be monetized.
Digital artist Philipp Schaerer hints at what a world with extremely realistic AR could be like in his Bildbauten series, where he constructs fake buildings made from photographs of real buildings. The result is an array of post-truth architecture, a critique on how contemporary renderings distort the relationship between image and architecture. Belgian artist Filip Dujardin meanwhile, uses a similar technique, but creates images that are closer to the edge of plausibility. Dujardin’s surrealist explorations conjure a sense of misunderstanding. Whereas with Schaerer, whose images we mistakenly believe, Dujardin infers a heuristic approach that relies on our knowledge of the built environment. If Schaerer and Dujardin tell us what a hyper-realistic virtual environment could look like, German artist Josef Schulz perhaps best indicates what an AR environment could look like with AR “switched off,” or perhaps with a real-life Adblocker switched on. Schulz essentially removed the clutter from architecture, making “the real appear virtual.”
Virtual artwork, face filters, “holograms” of musicians, and Pokémon Go are the first major forays into our augmented world, and they appear benevolent. However, with companies like Facebook and Google collecting vast amounts of user data, twinned with the receptive technologies demonstrated by NYU’s Engineering Department, the scope and reach of AR may not be so fun for the average city dweller in the near future.