Open Road

With self-driving cars, head-up displays will become an immersive experience

It’s been nearly three decades since the first, rather rudimentary, head-up displays starting showing up in passenger cars. Since its introduction, the head-up function has been constrained to a small piece of windshield real estate displaying relatively simple information. As cars increasingly become sensor-driven automated mobility machines, however, things are set to change dramatically.

Head-up displays will evolve into immersive experiences in which broad swaths of the windshield and windows display a wide range of information. Navdy has already taken traditional head-up display technology and morphed it into, what it calls, an augmented driving device — using augmented reality technology to project turn-by-turn directions, text messages and notifications, as well as a car’s diagnostic information as a transparent image over the road ahead. But these features are doled out in small doses; the goal is to show just enough info in the driver’s line of sight to help make smart driving decisions, but not to overwhelm or distract.

Imagine if your car can already “see” its surroundings via lidar, radar, sonar and video sensors—and it can use onboard supercomputers powered by smart algorithms to know precisely where you are in the lane, what obstacles are around you, and a detailed itinerary for your trip. That’s what self-driving and semi-autonomous vehicles are built to do.

“Augmented reality in cars is coming faster than we thought,” said Jules White, assistant professor of electric engineering and computer science at Vanderbilt University. “It’s because automated driving has taken over the auto industry and we’ve seen major advances in computer vision and deep learning.”

Professor White explained that presenting a large arrow on the windshield could help prevent a driver from missing a critical turn. However, today’s designers of head-up displays would never obscure the driver’s view of a potential roadside hazard or a pedestrian crossing the street. As cars start to drive themselves, though, they will have a deep awareness of their surroundings: “When and how you show information to drivers is just as important as what you show them,” White told me. “The question of when and how is driven by the car being aware of what’s happening on the road around you.”

Simply put: In situations that are entirely safe (or when the car takes over more of the driving), information about routing options, road conditions or even roadside-dining choices could be precisely placed in the three-dimensional virtual space ahead. Unlike today, the information would be liberated from a small corner of the windshield. Color-coded pathways and highlighted objects would look as if they are actually on the road. It’s a shift from “screen-fixed” navigation to “world-fixed” augmented reality.

Jaguar augmented reality windshield
Jaguar announced it was working on augmented reality windshields back in 2014

If the car’s sensors detect a child chasing a ball across the street, for example, the screen would suppress navigation instructions and draw all the driver’s attention to the threat of an accident. Based on complex algorithms and user preferences, the car would make contextual split-second decisions about what to show the driver. That opens up a world of possibilities for richer interactions between car and driver.

In the future, drivers of fully autonomous cars will barely need to pay attention to the road. In that scenario, you could imagine the use of augmented reality to present a panoply of menu options on the windshield, from web-powered points of interest to streaming movies.

That might sound like science fiction, but there’s a long and growing list of major car companies and automotive suppliers producing advanced head-up concept cars, some using gesture controls (like the Navdy device). In 2014, researchers from Virginia Tech proclaimed: “Windshield-based optical see-through displays may enable commercially available systems within the next few years.”

That’s optimistic, according to Karlheinz Blankenbach, a German physicist, and professor of displays and electric engineering at Pforzheim University near Stuttgart. “Augmented reality requires a large field-of-vision, which is technically feasible,” he told me. “But at this stage, those are too costly and bulky for mass production.”

The reality is that the shift to maximize augmented reality’s potential in production cars will happen gradually. Navdy remains the first step on that ladder, providing the pinnacle of what’s possible today as an aftermarket product for anyone and almost any car.

As fully autonomous features in vehicles become ubiquitous, there will initially be the need for drivers to pay at least a little attention in case a human override is needed. This means we need stimulation to prevent us from simply nodding off, and advanced augmented reality technology may be one solution to achieve that. After all, experiencing a windshield that’s seemingly plucked from the pages of a William Gibson novel—who wouldn’t stay awake for that?

Navdy Staff

Look forward