Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog OpenYurt Teamed with eKuiper to Solve the Processing Problems of Edge Streaming Data in IoT Scenarios

OpenYurt Teamed with eKuiper to Solve the Processing Problems of Edge Streaming Data in IoT Scenarios

This article discusses the new partnership between OpenYurt and eKuiper.

By OpenYurt Community

The emergence of cloud computing has prompted the explosive growth of the Internet of Things. Under the rising trend of device scale and business complexity, edge computing has also rapidly developed because it can bring computing power closer to the network edge and devices, thus reducing the cost of cloud performance.

Admittedly, IoT edge computing is still in the early stages of development, and there are many challenges to be solved. For example, it is necessary to have the ability to process business data quickly and respond to abnormal situations quickly in a heterogeneous environment where a large number of software and communication protocols are extremely complex. In addition, in most cases, for security or other considerations, edge nodes are physically unable to directly access from cloud nodes, making deployment difficult and cloud-edge management impossible. These issues threaten business continuity, stability, and availability.

Now, enterprises and developers can find solutions to the problems above through the open-source community. Recently, OpenYurt and the open-source project eKuiper reached a formal partnership and completed integration. Starting from version v0.4.0, OpenYurt will officially support the deployment and management of eKuiper. The two sides will jointly help developers easily and efficiently solve the streaming data processing and O&M challenges in IoT edge computing scenarios.

eKuiper: Lightweight, Open-Source Software for IoT Data Analysis and Stream Processing

1

Data stream processing capabilities are required in many IoT edge computing scenarios. The so-called streaming data refers to a set of sequential, large, fast, and continuous data sequences. In general, streaming data can be regarded as a collection of dynamic data that grows indefinitely over time, which can help users understand the status of system equipment in real-time and respond quickly to abnormal situations.

At the edge, computing resources (CPU, memory, etc.) are not as abundant as they are on the cloud. Traditional data stream processing frameworks (like Apache Spark or Apache Flink) have large installation packages, complicated deployment structures and processes, and high run-time consumption. Therefore, they are not suitable for running on these resource-constrained edge devices (industrial personal computers, gateways, or devices, such as x86 or ARM servers with low configurations). eKuiper was designed and developed to solve these problems on IoT edge devices.

The predecessor of eKuiper was the Kuiper project. It was officially open-sourced in 2019 by EMQ, an open-source IoT data infrastructure software provider. In June 2021, the Kuiper project joined the LF Edge Foundation and changed its name to eKuiper, starting to operate as an independent project. The essence of eKuiper is a lightweight IoT data analysis and stream processing software that can run on various resource-constrained edge devices. It is hoped that data stream processing at the edge can have the capabilities of projects, such as Spark and Flink.

As shown in the following figure, the overall architecture of eKuiper is roughly divided into three parts:

2

  • Sources is on the left, which represents the data source. The data source may be MQTT Broker at the deployment edge in OpenYurt or message queues, files, and databases.
  • Sinks is on the right, which represents the location where the data is stored after being processed. It refers to the target system. The target can be MQTT, files, and databases. It can also call HTTP services.
  • The eKuiper runtime is in the middle, and the top layer is data business logic processing. This layer provides SQL and rule parser. SQL processor processes data and generates an SQL execution plan. The following layer includes stream runtime and SQL runtime, which run the final execution plan. The bottom layer is Storage, which stores some information that needs to be persisted during operation.

In eKuiper, you can manage one or more eKuiper instances using dashboards. Typically, these dashboards are deployed in cloud nodes to manage eKuiper instances across multiple edge nodes. As mentioned earlier, since edge nodes are physically inaccessible from cloud nodes in most cases, deployment becomes difficult, and efficient eKuiper cloud-edge management cannot be performed.

OpenYurt has changed this situation.

OpenYurt: A Non-Intrusive Cloud-Native Intelligent Platform for Edge Computing

3

Cloud-native technology has become ubiquitous and has been applied to new application loads, new computing forms, and new physical boundaries. As the technical cornerstone of cloud-native, containers and Kubernetes are being carried by more computing forms. The rich forms are also moving from the traditional central cloud to edge computing and terminals.

Generally speaking, the computing scale is large, and the business is complex in the edge scenario. The workload management model of native Kubernetes is far from meeting the landing requirements of cloud-native edge computing in reality. Moreover, the cloud-edge network is connected through the public network, and the network connection has several uncontrollable factors, which may lead to unstable factors in the operation of edge services. Additionally, the edge nodes are generally located inside the firewall of the user network. This will result in the objective condition that the cloud-edge network can only be connected in one direction, which brings many challenges to the native Kubernetes O&M monitoring. Finally, it is inevitable to face the diverse and heterogeneous types of edge resources, which makes it difficult to support edge standardization.

OpenYurt is the first cloud-native open-source platform for edge computing built on native Kubernetes that does not intrude into Kubernetes in the industry. OpenYurt is the core framework of Alibaba Cloud ACK@Edge. In May 2020, Alibaba Cloud contributed to the industry by making it open-source and donating it to CNCF to achieve a more neutral and open community environment. It became the preferred project for edge cloud-native platforms in ecologically compatible scenarios.

As shown in the following figure, the architecture design of OpenYurt is very simple, which is a typical center-edge mode. Yurt Controller Manager, Yurt App Manager, and Tunnel Server components are added on the cloud (Kubernetes Master). The YurtHub and Tunnel Agent components are added on the edge (Kubernetes Worker).

4

This architecture design provides the following enhancements to native Kubernetes capabilities in edge scenarios:

  • Edge Unitization: Yurt App Manager is used to manage edge resources scattered in different regions from the perspective of unitization. It also provides independent lifecycle management, upgrades, scaling, and closed-loop traffic capabilities for businesses in each region. In addition, the business does not need any adaptation or transformation.
  • Edge Autonomy: YurtHub is a transparent proxy with caching capabilities. It is added to each edge node, so the cloud-edge network can be disconnected. If the node or service is restarted, the locally cached data can be used to restore the service.
  • Cloud-Edge Collaboration (O&M Monitoring): A secure encrypted channel for cloud-edge two-way authentication is provided for edge nodes located inside the firewall through the cooperation of Tunnel Server and Tunnel Agent. Even in edge computing scenarios with single-direction network connection from edge to cloud, users can still run native Kubernetes O&M commands (such as kubectl proxy/logs/exec/port-forward/attach). At the same time, central O&M monitoring systems (such as Prometheus and Metrics Server) can also obtain monitoring data at the edge through the cloud-edge channel.
  • Cloud-Native Ecosystem Compatibility: All functions enhance Kubernetes capabilities by Add-on or controller, thus ensuring 100% compatibility with Kubernetes and the cloud-native community ecosystem. OpenYurt also provides YurtCtl that can be used for one-click conversion between native Kubernetes and OpenYurt clusters.

OpenYurt enables users to manage applications running in edge infrastructures like in cloud infrastructures.

Use OpenYurt to Deploy and Manage eKuiper

Starting from version v0.4.0, the OpenYurt project officially supports the deployment of eKuiper and its dashboards. Users can take advantage of the yurt-tunnel, a cloud-edge tunnel, to implement configuration management of eKuiper instances through the Web management console of eKuiper, enabling efficient and lightweight IoT edge stream processing capabilities.

5

The eKuiper and eKuiper Dashboard components are directly deployed using native Helm charts. The eKuiper component runs entirely on the edge, while the eKuiper Dashboard runs on the cloud (the control side). At the same time, the yurt-tunnel component of OpenYurt uses mechanisms, such as DNS and iptables DNAT, to intercept cloud-edge network communication automatically. Therefore, the eKuiper component can issue commands from the cloud to the edge without being aware of the cloud-edge tunnel.

Based on the methods above, users can directly use the dashboard on the OpenYurt platform to easily manage eKuiper in edge scenarios. They can also use the dashboard in the browser to check whether the service is healthy in time.

6

Planning on Community Cooperation

In the future, the two sides will continue exploring efficient cloud-native edge middleware solutions for the IoT industry and attracting more lightweight IoT-related middleware to join the OpenYurt ecosystem. As such, edge middleware can truly obtain deployment support in a cloud-native way. EMQ will also officially become a member of the cloud-native IoT SIG as a contributor to eKuiper, initiated by OpenYurt after Alibaba Cloud, VMWare, and Intel.

We hope more developers will support and join the OpenYurt and eKuiper communities to build an open-source ecosystem for cloud-native IoT.

References

If you want to learn more about LF Edge eKuiper and CNCF OpenYurt, please check the following references:

IOT Cloud Computing Developers O&M Edge Computing Data Processing Stream Computing Kubernetes cloud native OpenYurt eKuiper
0 0 0
Share on

Read previous post:

Fluid Practice of HAOMO.AI: Cloud-Native AI Makes Cars "Smarter"

Read next post:

ChaosBlade: From the Chaos Engineering Experiment Tool to the Chaos Engineering Platform

Alibaba Cloud Native Community

489 posts | 48 followers

You may also like

Comments

Alibaba Cloud Native Community

489 posts | 48 followers

Related Products

  • Link IoT Edge

    Link IoT Edge allows for the management of millions of edge nodes by extending the capabilities of the cloud, thus providing users with services at the nearest location.

    Learn More

  • Bastionhost

    A unified, efficient, and secure platform that provides cloud-based O&M, access control, and operation audit.

    Learn More

  • Managed Service for Grafana

    Managed Service for Grafana displays a large amount of data in real time to provide an overview of business and O&M monitoring.

    Learn More

  • Function Compute

    Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.

    Learn More

More Posts by Alibaba Cloud Native Community

See All