Best Practices for Flink on Zeppelin Stream Computing Processing

This article is an overview of the best practices for Flink on Zeppelin stream computing processing taken from a recent lecture.

Lecturer: Jian Feng, Head of Alibaba Cloud EMR Data Development Platform

Content Framework:

Big Data Overview
Flink Learning Framework
Best Practices for Stream Computing on EMR Studio
Presentation

I. Big Data Overview

Big Data Processing ETL (Data → Data)
Big Data Analysis BI (Data → Dashboard)
Machine Learning Platform for AI (Data → Model)

II. Flink Learning Framework

Flink Essentials

Stateful
Time
Flink Architecture
Flink API
Flink Configuration
Flink Log

Stateful

1. Why

Timeliness of stream computing
Unbounded for stream computing

2. When

Window
Join
Pattern

3. How

Statebackend

Time

Event Time
Processing Time
Watermark

Flink Architecture

Flink API

Flink Configuration

Cluster Configuration
Job Configuration
Statebackend
Resource Manager
SQL/Python

Reference: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/

Flink Log

III. Best Practices for Stream Computing on EMR Studio

EMR Studio Features

Compatible with Open-Source Components

EMR Studio is optimized and enhanced based on the open-source software Apache Zeppelin, Jupyter Notebook, and Apache Airflow.

Supports connecting to multiple clusters
Adapt to multiple computing engines
Interactive development + seamless connection of job scheduling
Applicable to a variety of big data application scenarios
Separated computing and storage

Flink Clients

Flink on Zeppelin (Phase 1) – Interactive Flink Client

Flink on Zeppelin (Phase 2) – Interactive JobManager

Flink on Zeppelin Main Feature

IV. Demonstration

For specific product introduction and demonstration, you can click the following link to watch the playback: https://developer.aliyun.com/live/247106 (Video in Chinese)

V. Related Product

E-MapReduce: https://www.alibabacloud.com/product/emapreduce

EMR Studio (Beta): https://help.aliyun.com/document_detail/208107.html (Article in Chinese)

Realtime Compute for Apache Flink: https://www.alibabacloud.com/product/realtime-compute

0 0 0

Share on

Big Data Consulting for Data Technology Solution

Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.

Learn More

Big Data Consulting Services for Retail Solution

Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.

Learn More