The emergence of cloud computing has prompted the explosive growth of the Internet of Things. Under the rising trend of device scale and business complexity, edge computing has also rapidly developed because it can bring computing power closer to the network edge and devices, thus reducing the cost of cloud performance.
Admittedly, IoT edge computing is still in the early stages of development, and there are many challenges to be solved. For example, it is necessary to have the ability to process business data quickly and respond to abnormal situations quickly in a heterogeneous environment where a large number of software and communication protocols are extremely complex. In addition, in most cases, for security or other considerations, edge nodes are physically unable to directly access from cloud nodes, making deployment difficult and cloud-edge management impossible. These issues threaten business continuity, stability, and availability.
Now, enterprises and developers can find solutions to the problems above through the open-source community. Recently, OpenYurt and the open-source project eKuiper reached a formal partnership and completed integration. Starting from version v0.4.0, OpenYurt will officially support the deployment and management of eKuiper. The two sides will jointly help developers easily and efficiently solve the streaming data processing and O&M challenges in IoT edge computing scenarios.
Data stream processing capabilities are required in many IoT edge computing scenarios. The so-called streaming data refers to a set of sequential, large, fast, and continuous data sequences. In general, streaming data can be regarded as a collection of dynamic data that grows indefinitely over time, which can help users understand the status of system equipment in real-time and respond quickly to abnormal situations.
At the edge, computing resources (CPU, memory, etc.) are not as abundant as they are on the cloud. Traditional data stream processing frameworks (like Apache Spark or Apache Flink) have large installation packages, complicated deployment structures and processes, and high run-time consumption. Therefore, they are not suitable for running on these resource-constrained edge devices (industrial personal computers, gateways, or devices, such as x86 or ARM servers with low configurations). eKuiper was designed and developed to solve these problems on IoT edge devices.
The predecessor of eKuiper was the Kuiper project. It was officially open-sourced in 2019 by EMQ, an open-source IoT data infrastructure software provider. In June 2021, the Kuiper project joined the LF Edge Foundation and changed its name to eKuiper, starting to operate as an independent project. The essence of eKuiper is a lightweight IoT data analysis and stream processing software that can run on various resource-constrained edge devices. It is hoped that data stream processing at the edge can have the capabilities of projects, such as Spark and Flink.
As shown in the following figure, the overall architecture of eKuiper is roughly divided into three parts:
In eKuiper, you can manage one or more eKuiper instances using dashboards. Typically, these dashboards are deployed in cloud nodes to manage eKuiper instances across multiple edge nodes. As mentioned earlier, since edge nodes are physically inaccessible from cloud nodes in most cases, deployment becomes difficult, and efficient eKuiper cloud-edge management cannot be performed.
OpenYurt has changed this situation.
Cloud-native technology has become ubiquitous and has been applied to new application loads, new computing forms, and new physical boundaries. As the technical cornerstone of cloud-native, containers and Kubernetes are being carried by more computing forms. The rich forms are also moving from the traditional central cloud to edge computing and terminals.
Generally speaking, the computing scale is large, and the business is complex in the edge scenario. The workload management model of native Kubernetes is far from meeting the landing requirements of cloud-native edge computing in reality. Moreover, the cloud-edge network is connected through the public network, and the network connection has several uncontrollable factors, which may lead to unstable factors in the operation of edge services. Additionally, the edge nodes are generally located inside the firewall of the user network. This will result in the objective condition that the cloud-edge network can only be connected in one direction, which brings many challenges to the native Kubernetes O&M monitoring. Finally, it is inevitable to face the diverse and heterogeneous types of edge resources, which makes it difficult to support edge standardization.
OpenYurt is the first cloud-native open-source platform for edge computing built on native Kubernetes that does not intrude into Kubernetes in the industry. OpenYurt is the core framework of Alibaba Cloud ACK@Edge. In May 2020, Alibaba Cloud contributed to the industry by making it open-source and donating it to CNCF to achieve a more neutral and open community environment. It became the preferred project for edge cloud-native platforms in ecologically compatible scenarios.
As shown in the following figure, the architecture design of OpenYurt is very simple, which is a typical center-edge mode. Yurt Controller Manager, Yurt App Manager, and Tunnel Server components are added on the cloud (Kubernetes Master). The YurtHub and Tunnel Agent components are added on the edge (Kubernetes Worker).
This architecture design provides the following enhancements to native Kubernetes capabilities in edge scenarios:
OpenYurt enables users to manage applications running in edge infrastructures like in cloud infrastructures.
Starting from version v0.4.0, the OpenYurt project officially supports the deployment of eKuiper and its dashboards. Users can take advantage of the yurt-tunnel, a cloud-edge tunnel, to implement configuration management of eKuiper instances through the Web management console of eKuiper, enabling efficient and lightweight IoT edge stream processing capabilities.
The eKuiper and eKuiper Dashboard components are directly deployed using native Helm charts. The eKuiper component runs entirely on the edge, while the eKuiper Dashboard runs on the cloud (the control side). At the same time, the yurt-tunnel component of OpenYurt uses mechanisms, such as DNS and iptables DNAT, to intercept cloud-edge network communication automatically. Therefore, the eKuiper component can issue commands from the cloud to the edge without being aware of the cloud-edge tunnel.
Based on the methods above, users can directly use the dashboard on the OpenYurt platform to easily manage eKuiper in edge scenarios. They can also use the dashboard in the browser to check whether the service is healthy in time.
In the future, the two sides will continue exploring efficient cloud-native edge middleware solutions for the IoT industry and attracting more lightweight IoT-related middleware to join the OpenYurt ecosystem. As such, edge middleware can truly obtain deployment support in a cloud-native way. EMQ will also officially become a member of the cloud-native IoT SIG as a contributor to eKuiper, initiated by OpenYurt after Alibaba Cloud, VMWare, and Intel.
We hope more developers will support and join the OpenYurt and eKuiper communities to build an open-source ecosystem for cloud-native IoT.
If you want to learn more about LF Edge eKuiper and CNCF OpenYurt, please check the following references:
Alibaba Clouder - September 17, 2020
Alibaba Developer - January 11, 2021
Alibaba Developer - July 9, 2021
Alibaba Cloud Community - February 3, 2023
Alibaba Clouder - September 16, 2020
Alibaba Developer - June 19, 2020
Link IoT Edge allows for the management of millions of edge nodes by extending the capabilities of the cloud, thus providing users with services at the nearest location.Learn More
A unified, efficient, and secure platform that provides cloud-based O&M, access control, and operation audit.Learn More
Managed Service for Grafana displays a large amount of data in real time to provide an overview of business and O&M monitoring.Learn More
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.Learn More
More Posts by Alibaba Cloud Native Community