AUTONOMOUS DRIVING: PETABYTES CONSTANTLY UNDER CONTROL
>> Elektrobit efficiently and securely orchestrates vast amounts of data with Azure and the EB Assist Test Lab
Autonomous driving already counts as a herculean task for developers in the automotive industry: using an unruly mix of software tools—some commercially available, others of their own devising—they must process the enormous amounts of data generated by fleets of test vehicles. Elektrobit, a subsidiary of Continental AG, is bringing structure to this vital development process. An expandable, cloud-based development lab provides engineers with the freedom to concentrate on their actual jobs, making development teams fit for the even more complex tasks in the future.
One to two gigabytes. Second after second. That’s the amount of data generated by each of today’s test vehicles sent to the streets to help develop autonomous driving. This adds up to at least 10 terabytes per day. By contrast, an Ultra HD Blu-ray transmits 4K movies to TV at a rate of just 0.01 gigabytes per second. The reason the amount of data is snowballing is that the test vehicles are equipped with an ensemble of cameras, light detection and ranging (lidar) sensors, and radar sensors, in addition to the components fitted to each vehicle’s bus.
Developers at every auto maker use sensor data to fine-tune and validate the algorithms required for autonomous driving. Top-priority algorithms include those that detect static objects, such as traffic lights and pillars, as well as those for moving objects, such as pedestrians, other vehicles, and falling trees. To get the job done, engineers often have to rely on an unwieldy mix of tools developed in-house, updated by additional purchases. This makes it hard enough to handle the petabytes of datasets the company has today. However, given that each new generation of vehicles generate more and more data―requiring increasingly complex software―development teams need a scalable, user-friendly, and reliable solution for managing and retrieving data.
"Then there’s the fact that tools used in development won’t play any part in selling the finished car," says Simon Tiedemann, Product Manager at Elektrobit Automotive GmbH. Elektrobit (EB) is a subsidiary of Continental AG that specializes in developing automotive software. According to Tiedemann, a standardized solution would allow developers to efficiently validate and manage test data.
The next—and only sensible—destination: The cloud
Elektrobit has established such a solution—the EB Assist Test Lab, a fully cloud-based development environment. "Given the amount of data to be processed and the speed with which the IT infrastructure needs to be scaled, purely local data storage is out of the question," Tiedemann says, adding: "We would’ve needed hundreds of servers to pull this off. Today, it’s so important to collaborate with partners spread all over the world as well as one’s own global development teams. An on-premises solution basically makes that impossible."
Elektrobit’s management chose Microsoft Azure as the underlying cloud platform for the EB Assist Test Lab. Another argument in favor of Azure was that both Elekrobit and its customers have had positive experiences with Microsoft’s cloud offers and would automatically have faith in a solution based on them. What’s more, Tiedemann’s team liked that Microsoft enlists the help of partners if they have greater expertise in subareas. "Since Elektrobit follows the same philosophy, this approach resonated with us. Some problems can be solved only by a combined effort," Tiedemann says.
Operated either via browser or mobile app, EB Assist Test Lab delivers information such as which driving scenarios have already been sufficiently covered—and it does so very quickly. In addition, the Azure-based software features a tool for extracting all metadata contained within the data stream, including types of road, weather during the test drive, and location. Customers can easily add new data formats and analysis functions themselves via plug-ins.
Developers using the EB Assist Test Lab can also request additional data to process a given task, so the test drive department can then generate exactly what they need. All the coordination required for this happens within the solution, which avoids any media disruption caused by switching to email or instant messaging.
Accessibility is the key
To get the raw data from the cars to the cloud as quickly as possible, the EB solution offers a variety of channels: turbo upload (up to 700 megabytes per second) via ExpressRoute from a node operated by Microsoft partner Equinix; continuous upload of small, important data packets via wireless 5G; data transfers from the customer’s network; or via Microsoft Azure Data Box. One way or another, the data finds its way into the test lab entity.
Once the data has been fed in, customers can supplement algorithms to analyze that data with their own functions via plug-ins. During data processing in the cloud, the customer engages hundreds of Azure computing nodes to complete this time-sensitive task, releasing them once the job is over. "Our customers can up the tempo even more by reserving GPU-based nodes through Azure," Tiedemann observes.
In addition to the software development kits (SDKs) for plug-ins, the virtual test lab also offers a REST API as well as other well-documented interfaces. These SDKs can be used, for example, to connect available third-party software so customers can label the data or simulate test drives. "There’s tried-and-true special software for this—from Playment, Hella Aglaia, or IPG Automotive—so we’re not trying to reinvent the wheel. Instead, we prefer to let our customers connect their tools to the Assist Test Lab, without losing the benefits of central data storage, or fussing with converters and scripts like in the old days," Tiedemann says.
The next stage in expanding the lab involves an interface to the customer’s on-premises data storage. "All manufacturers and developers have already invested in their own server capacity and want to continue using the data they have stored there," Tiedemann notes.
Secure and expandable
Available since the end of 2019, the first version of the solution already offered customers a connection to Microsoft Azure Active Directory (Azure AD) to unify user account management. To guarantee the security of the sensitive test data, Elektrobit installs the entire Assist Test Lab via each customer’s Azure deployment setup. This puts the databases fully under the control of the respective user company.
"Since they no longer have to develop so many tools on their own, our customers can achieve tool-cost savings in the very high double-digit percentage range," Tiedemann explains. This is in addition to the savings on the compute and storage hardware and expensive operations. Since companies aren’t required to purchase and constantly run the compute nodes that feed in the raw data, costs can be greatly reduced. The practical benefits of the efficient software designed are tantamount. Since everything comes from a single source and all functions are controlled via a user-friendly interface, vast amounts of data can be orchestrated more efficiently.
In addition to the functions already available to enrich, label, and trace the data―for example, "show me all datasets with overtaking maneuvers that took place in the rain in Germany" ―Elektrobit wants to collaborate with customers to develop functions for managing test cases, performing the tests themselves, and post-processing. "Thanks to the flexibility of the cloud, we don’t have to develop everything in isolation. Instead, we can address test customers’ practical requirements on an ongoing basis," concludes Tiedemann.
