Denso Delivers an IoT Prototype per Week with Kubernetes
The need for a flexible platform
Founded in 1949 after spinning off Toyota, Denso has grown to become one of the world’s largest suppliers of automotive components and spare parts. With the global connected car market projected to reach $166 billion by 2025, the company branched out into software development by creating a Digital Innovation Department to work on edge and cloud products for vehicles. To achieve this, Denso needed a flexible platform that would enable the company to develop and deploy apps for vehicles in quick iterations.
Expanding services to include edge and cloud (Image credit)
Seiichi Koizumi
In addition, there were other technical issues in developing edge and cloud systems for vehicles. According to Seiichi Koizumi, R&D Product Manager at Denso, the major challenges included:
- Computing resources on electronic control units (ECU)—managing vehicle subsystems—are limited.
- There are no easy ways to update applications on multiple vehicles at once.
- Vehicles have poor Internet connections, especially when moving.
- Apps for vehicles are hard to run and test without specialized hardware.
Limited computation power on ECUs (Image credit)“There are barriers to developing applications for vehicles. It’s not the same as developing cloud applications.” —Seiichi Koizumi, Denso
To address the challenges mentioned above, Koizumi’s team utilized Kubernetes to develop Misaki, a centralized management platform for connected cars.
What is Misaki?
JunKai Yong
According to a technical paper written by Denso Software Engineers JunKai Yong and Aman Gupta along with Koizumi, Misaki is an end-to-end mobility framework for edge and cloud environments.
Misaki breaks down and deploys Kubernetes features on the edge and the cloud by resource availability. Using functionalities of service mesh and digital twin technologies, the framework integrates both the edge and the cloud.
Misaki’s architecture (Image credit)With the microservices approach of Kubernetes, Misaki enabled developers to quickly create, test, and deploy their apps.
Additionally, Kubernetes provided much needed resource flexibility as even modern vehicles with high-performance ECUs still lacked the computing power to run certain apps, noted Koizumi. With Kubernetes, vehicles serve as worker nodes, which can then migrate heavy workloads to master nodes in the cloud when necessary.
Vehicles acting as worker nodes (Image credit)“Applications will sometimes consume a lot of the resources in a vehicle. In this case, high-priority apps will keep running in the vehicle, and low-priority apps are offloaded into the cloud. This way, applications will keep running, and workloads in the vehicle remain stable.” —Seiichi Koizumi, Denso
Misaki also benefits from the Kubernetes service mesh, which reduces the reliance on stable Internet connections. With the service mesh, apps can store requests in a queue, while the network is unstable.
Service mesh storing application requests (Image credit)
How does it work?
Aman Gupta
According to Gupta, the team considers Misaki a “vehicle cluster” consisting of edge and cloud nodes connected by Kubernetes. The master nodes reside in the cloud, while worker nodes are on the edge.
“In a real-world scenario, we can have a single cloud instance for a car, or a single cloud instance can be shared among a few cars.” —Aman Gupta, Denso
Misaki is made up of two parts: Misaki Orchestrator and Misaki Service Mesh. The first part, Misaki Orchestrator, consists of four components: Misaki-UI, Misaki-api, Digital-twin, and Misaki-k8s-agent.
Misaki Orchestrator’s components (Image credit)Implemented in Nuxt.js, Misaki-UI serves as a dashboard for administrators, enabling them to see a list of vehicles, as well as a list of apps that can be deployed to the vehicles. Once an administrator chooses an application to deploy, the request is sent to Misaki-api.
“Applications here are basically Helm charts. These Helm charts reside in a chart repository, which can be hosted anywhere. We currently use ChartMuseum, which is hosted on our EKS cluster.”
—Aman Gupta, Denso
Misaki-api is a REST API implemented in Go. The primary function of the component is to render Helm charts into a manifest list for Kubernetes.
Digital-twin is a database for manifests. It enables clients to connect through WebSocket.
Misaki-k8s-agent applies manifests to vehicle clusters. In this manner, applications can be deployed to devices on the edge or cloud.
Misaki Orchestrator’s workflow (Image credit)“Misaki-api sends a rendered Kubernetes manifest and stores it in a vehicle ID in Digital-twin. Once it is changed, Misaki-k8s-agent fetches it from Digital-twin via WebSocket.” —Aman Gupta, Denso
Misaki Service Mesh consists of a control plane and Envoy sidecars. The control plane is a centralized unit that manages policies and configuration for each sidecar.
Misaki Service Mesh’s components (Image credit)Envoy sidecars are responsible for routing all incoming and outgoing requests from applications. This abstracts network operations, enabling developers to focus on building applications.
Envoy routes traffic for microservices (Image credit)In addition, Misaki Service Mesh provides a queue acting as a proxy server. The queue stores HTTP requests from apps when the network disconnects and resends the stored HTTP requests when the network reconnects.
Scaling up 130 factories
Yuta Kuroda
Besides Misaki, Denso utilized Kubernetes to build an Internet of Things (IoT) platform to connect and scale up 130 of its factories all over the world.
According to a presentation led by Denso Software Engineers Yuta Kuroda, Hiroo Sawada, and Dan Yakabe, the company needed a system that would enable scaling depending on the number of connected equipment, unstructured data in different formats from multiple sources, and loose coupling through middleware.
An architecture of the IoT platform (Image credit)The team developed a Kubernetes-based IoT platform on Google Cloud Platform (GCP). Mosquitto, Kafka, and other middleware are in pods within Google Kubernetes Engine (GKE). According to the team, the middleware is loosely coupled by having in-house applications between them, so even if changes are made to the middleware, simply swapping out in-house apps enables end-to-end delivery to BigQuery.
Hiroo Sawada
“We used Kubernetes to enable scaling out for 130 factories in a short time. We applied rules to the relational database for converting from various factory data formats, and then carry out streaming extract, transform, and load operations. We utilized loose coupling between middleware through in-house applications, and we created our own framework to achieve a system for building various apps in short times.” —Hiroo Sawada, Denso
Prior to Kubernetes, critical layer features took 2–3 years to implement. Now, Denso reduced its development cycle for noncritical software to just two months.
Dan Yakabe
Today, the company has a vehicle edge computer, a private Kubernetes cloud, and managed Kubernetes on GKE, Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). Besides Helm and Envoy, the organization has also added Prometheus, Fluentd, and Istio to its Kubernetes deployments.
As of 2021, Denso’s Digital Innovation Department, which started out with two members, has grown to over 50. The company is also able to release 10 new apps per year and introduce a new prototype on a weekly basis.
Misaki is still a prototype in active development. The Denso team is working on new challenges and tasks, such as updating Kubernetes worker nodes on the edge, deciding between multiple master nodes or a single master node for multiple worker nodes, as well as experimenting on lightweight Kubernetes to minimize resource usage on the edge. Anyone interested in tracking updates can check out the project’s GitHub repository.
Want details? Watch the videos!
In this video, Seiichi Koizumi and Aman Gupta provide an overview of Misaki.
Further reading
- Lyft Runs 300,000+ Containers in a Multicluster Kubernetes Environment
- BMW Delivers IoT Services to 1M Car Owners by Using IBM Cloud Foundry
- Porsche Invests in Blockchain Startups to Secure Car Data and Access
About the experts
Aman Gupta is Software Engineer at Denso. He is responsible for the development of the company’s Kubernetes client, as well as rolling updates in Internet of Things devices. Aman is involved in designing architecture for seamless data flows and setting up continuous integration/delivery pipelines for development and production environments. He is also highly proficient in working with UNIX environments.
Seiichi Koizumi was Product Manager at Denso, where he was responsible for the Misaki project. Koizumi has eight years of experience managing multinational engineering teams. His expertise includes connected platforms, edge computing, data engineering, and information retrieval.
JunKai Yong is Software Engineer at Denso. He is passionate about engineering and hopes to contribute to the advancement of technology and society. Yong works with the Mobility-as-a-Service Research and Development Department in Denso. Previously, he was Director of Publics at Nanyang Technology Univesity and Media Editor at WWHCurrent.
Dan Yakabe is Software Engineer at Denso. In 2002, he joined a systems integrator company where he developed smart meter systems. Yakabe also has worked in the financial industry building security systems. His comprehensive experience developing machine-to-machine technologies, security systems, and APIs have proven useful in building Denso’s IoT platform.
Yuta Kuroda is Software Engineer at Denso. He joined the organization in 2008, starting in production engineering and later moving on to core systems. Kuroda works with the Factory IoT Development Team. His interests include Kubernetes, Ansible, continuous integration/delivery, as well as other areas related to infrastructure.
Hiroo Sawada is Software Engineer at Denso. He previously worked at a major electronics components company where he gained experience in a variety of things, including quality assurance and production engineering. Sawada was eventually assigned to work on autonomous control technologies coordinated with manufacturing execution systems.
