×
Community Blog Best Practices for Flink on Zeppelin Stream Computing Processing

Best Practices for Flink on Zeppelin Stream Computing Processing

This article is an overview of the best practices for Flink on Zeppelin stream computing processing taken from a recent lecture.

Lecturer: Jian Feng, Head of Alibaba Cloud EMR Data Development Platform

Content Framework:

  • Big Data Overview
  • Flink Learning Framework
  • Best Practices for Stream Computing on EMR Studio
  • Presentation

I. Big Data Overview

  • Big Data Processing ETL (Data → Data)
  • Big Data Analysis BI (Data → Dashboard)
  • Machine Learning Platform for AI (Data → Model)

1

II. Flink Learning Framework

Flink Essentials

  • Stateful
  • Time
  • Flink Architecture
  • Flink API
  • Flink Configuration
  • Flink Log

Stateful

1.  Why

  • Timeliness of stream computing
  • Unbounded for stream computing

2.  When

  • Window
  • Join
  • Pattern

3.  How

  • Statebackend

Time

  • Event Time
  • Processing Time
  • Watermark

Flink Architecture

2

Flink API

3

Flink Configuration

  • Cluster Configuration
  • Job Configuration
  • Statebackend
  • Resource Manager
  • SQL/Python

Reference: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/

Flink Log

4

III. Best Practices for Stream Computing on EMR Studio

EMR Studio Features

  • Compatible with Open-Source Components

EMR Studio is optimized and enhanced based on the open-source software Apache Zeppelin, Jupyter Notebook, and Apache Airflow.

5

  • Supports connecting to multiple clusters
  • Adapt to multiple computing engines
  • Interactive development + seamless connection of job scheduling
  • Applicable to a variety of big data application scenarios
  • Separated computing and storage

Flink Clients

6

Flink on Zeppelin (Phase 1) – Interactive Flink Client

7

Flink on Zeppelin (Phase 2) – Interactive JobManager

8

Flink on Zeppelin Main Feature

9

IV. Demonstration

For specific product introduction and demonstration, you can click the following link to watch the playback: https://developer.aliyun.com/live/247106 (Video in Chinese)

V. Related Product

E-MapReduce: https://www.alibabacloud.com/product/emapreduce

EMR Studio (Beta): https://help.aliyun.com/document_detail/208107.html (Article in Chinese)

Realtime Compute for Apache Flink: https://www.alibabacloud.com/product/realtime-compute

0 0 0
Share on

Alibaba EMR

62 posts | 6 followers

You may also like

Comments

Alibaba EMR

62 posts | 6 followers

Related Products