GENERATIVE AI, AUGMENTED REALITY AND TELEOPERATED PARKING – THE DIGITAL EXPERIENCE IN THE BMW OF THE FUTURE AT CES 2024.
Las Vegas. The in-car digital experience of both the present day and the future is the dominant theme of the BMW Group’s presence at CES 2024. New infotainment products for the current BMW model portfolio will be shown at the event. At the same time, the BMW Group is demonstrating innovative development projects that offer a glimpse into the future. Visitors can teleoperate a BMW iX and guide it around the Silver Lot on the exhibition site, discover as a passenger the potential offered by augmented reality glasses during a journey and experience a new generative AI capability of the BMW Intelligent Personal Assistant powered by the Alexa large language model (LLM), enabling quick answers and how-to responses on vehicle features.
"BMW is synonymous with both the ultimate driving machine and the ultimate digital experience," says Frank Weber, Member of the Board of Management responsible for BMW Group Development. "At the CES we are showing more content, more customisation and more gaming. This is all underpinned by our powerful, in-house developed BMW Operating System. And we will take a look to the future, of course, with perfectly integrated augmented reality and strong, reliable artificial intelligence at the interaction between human and machine."
BMW Operating System 9 makes it possible: new gaming experiences from the ConnectedDrive Store in 2024.
The optional "BMW Digital Premium" offering allows BMW customers with BMW Operating System 9 to access an extended range of products from the BMW ConnectedDrive Store. As well as on-demand functions for the vehicle, the store also offers an ever-evolving selection of third-party apps in the categories music & audio, news & magazines and gaming for the infotainment system. More games are now being made available via this channel for in-car enjoyment. And occupants will also be able to play using proper gaming controllers in future. BMW is presenting this new form of gaming experience for the first time at CES 2024. For example, "Beach Buggy Racing 2" offers customers an action-packed racing game that evokes memories of the great console classics. Two players in the vehicle can compete against one another in split-screen mode using their gaming controllers. Any standard gaming controller with Bluetooth capability can be connected to the infotainment system in just a few steps. The controller link-up will be made available over the air for BMW Operating System 9 during 2024.
The AirConsole App already lets BMW customers play single and multiplayer games in their car. The "casual games" offered are easy to pick up for anyone; the user’s smartphone serves as a controller, allowing simple control of gameplay that is quick to learn. Additions are constantly being made to the choice of games in the in-car AirConsole App.
BMW Digital Premium is optionally available for models with BMW Operating System 9 and can be obtained from the ConnectedDrive Store either online, in the MyBMW App or directly from the car. BMW Digital Premium allows customers to enjoy in-car use of all the apps available in their country as well as covering the necessary data usage. And the package offers more besides, including enhanced navigation functions and added customisation options.
Top entertainment – with the Video App in BMW Operating Systems 8.5 and 9.
In addition to the BMW Theatre Screen with Amazon Fire TV built-in in the rear compartment of the BMW 7 Series, BMW customers are now able to watch a variety of video content on the central display too. The already very succesful TiVo operating system for smart TVs was optimised for in-car video streaming in the space of just a few months by developers from the BMW Group working together with XPERI Inc. The result is the integration of the DTS AutoStage Video Service powered by TiVo. The entertainment programme comprises both live channels and on-demand media libraries covering the categories news, sport, entertainment for kids, movies and TV series.
First launched in the new BMW 5 Series, the platform will now be rolled out to further executive and luxury models over the air. Since the end of 2023, customers driving compact models with BMW Operating System 9 have also been able to access the Video App with BMW Digital Premium. These will be followed by further models over the course of 2024, such as the BMW X2 and all models in the new MINI family.
The Video App is currently accessible in the USA, UK, South Korea, France, Italy, Germany and Spain, and will soon be coming to Japan. A number of portals are available internationally, including Pluto TV, Bloomberg, TED, Yahoo! and Chili. These are accompanied by market-specific streaming services:
> Germany: JOYN, ZDF, tagesschau, Deutsche Welle.
> Italy: RaiPlay
> France: TF1
> USA: Tubi, Gusto TV, Journy
> Japan: Hulu Japan, NHK Plus, NTV NEWS NNN, TVer, WOWOW
The portfolio of video streaming platforms will keep expanding and will be gradually rolled out to other countries.
Ready for the wearables of the future – enhanced driving experience with XREAL Air 2 augmented reality glasses.
The CES show also sees the BMW Group demonstrate for the first time how augmented reality (AR) glasses are set to enrich the driving experience in future. Visitors can test the possible uses of AR glasses for themselves on a drive through Las Vegas. Wearing the glasses, they can see how navigation instructions, hazard warnings, entertainment content, information on charging stations and supporting visualisations in parking situations are embedded perfectly into the real-world environment by the "XREAL Air 2". AR and mixed reality (MR) devices will become increasingly popular in the next few years, thanks to technological advances and entry-level models that are more affordable for customers. In future, AR and MR devices will be able to offer both drivers and passengers enhanced information and enjoyable experiences to complement the displays fitted in the vehicle.
The BMW Group has been working on the development of AR/MR technology since 2008 and investigating possible applications for head-mounted displays in research projects. The BMW engineers, together with their development partners, have now succeeded in showing stable augmented reality and mixed reality content to passengers in a dynamically moving vehicle for the first time. The displayed image remains securely embedded in the environment even in challenging driving situations, such as when turning, driving over large bumps or accelerating. To achieve this, the tracking system of the AR/MR device in question has to be linked to the vehicle’s sensor data.
The BMW Group has consistently built up its pool of experience with AR/MR devices over the years and presented the results to the public. These include the initial MINI Augmented Vision project (2015), the collaboration with Meta Reality Labs (2022 & 2023), the ConnectedRide Smartglasses from BMW Motorrad and the M Mixed Reality Driving Experience. The Augmented View function in current BMW and MINI models shows drivers a live video stream on the display that is augmented with additional information for optimum guidance when using the navigation system. In the area of highly innovative, vehicle-related XR applications, the BMW Group has filed patents for almost 70 inventions from its developers in recent years.
"Thanks to the recent research collaborations with Meta Reality Labs and XREAL, we have made considerable progress in our preparations for integrating external XR devices into our driving experience in production models. We are proud to be leading the way here and are well prepared to offer our customers breathtaking XR experiences in the future," says Peter Lehnert, Vice President Research, New Technologies and Innovations Digital Car at the BMW Group.
"We are thrilled at the prospect of being able to digitally embellish the real world by combining car tech with AR tech," comments Peng Jin, co-founder of XREAL. "And this could lead to entirely new experiences while travelling by car and open up new potential for AR tech."
The BMW Group has successfully carried out pioneering work on digital technologies in the past – such as vehicle access using the BMW Digital Key – and has been a driving force behind the industry-wide standardisation of such innovations. The company is open to collaboration with further partners in the area of AR/MR too. The goal here is to establish an industry standard for connecting the CE industry’s XR devices with vehicles.
Development demo: BMW voice assistant becomes the ultimate vehicle expert with the help of the Alexa Large Language Model (LLM).
At CES, the BMW Group and Amazon showcase voice assistant LLM capabilities based on a current development project that the companies are working on in preparation for a potential series rollout. This makes the BMW Intelligent Personal Assistant the ultimate vehicle expert thanks to generative AI that is powered by the Alexa LLM. At CES, visitors experience how the voice assistant provides quick instructions and answers about vehicle functions in a much more human, conversation-like manner, while at the same time being able to control some vehicle functions.
The BMW Intelligent Personal Assistant has been using artificial intelligence for speech processing for a long time and this demo is the latest proof point of the continued innovation BMW and its partners are bringing to customers. Complex processing capabilities, which enable human-like interactions and dialogs, have not yet been integrated into BMW series vehicles. Such capabilities are now made possible by LLMs, which are AI models that are designed to process human language at a high level and generate responses accordingly. The BMW Group is also taking careful steps to build a safe and trusted experience for customers, and continues to test and iterate these LLM capabilities over time with its partners.
Since the introduction of the first BMW Intelligent Personal Assistant in 2018, language interaction has been an essential part of BMW iDrive. Today, the voice assistant gives customers control over many vehicle functions so that they can keep their hands on the wheel and their eyes on the road. Leveraging Amazon’s Alexa Custom Assistant — a solution that is based on Alexa technology and enables companies to create their own customized voice assistant — the next-generation of the BMW Intelligent Personal Assistant will soon become more powerful over the course of the year in vehicles with BMW Operating System 9.
Joint development project with Valeo paves the way for teleoperated valet parking.
The BMW Group and Valeo have entered a strategic partnership to work on the joint development of next-generation automated Level 4 parking solutions. This includes the development of solutions for automated valet parking (AVP). Here, the customer leaves their vehicle in a designated drop-off area. The vehicle then searches for a vacant parking space itself, manoeuvres into the space and later exits it again, and can even drive back to the pick-up area for collection by the driver. Either an autonomous driving system or a teleoperated valet system can be used as the underlying technology for this.
CES 2024 visitors will have the opportunity to try out teleoperated valet parking for themselves. Teleoperation offers complete control in situations that could be very challenging for fully autonomous systems, such as in complex or unfamiliar environments.
The teleoperator uses sophisticated technology and live camera images to guide and move the vehicle so that it can be parked and retrieved again in a carefully controlled manner. Possible future applications for this technology could include parking at events, at airports or in the logistics sector.
