The message delivery feature allows you to send Flink job startup logs, resource usage information, and job events to external message queues or storage systems in real time. This feature supports use cases like data persistence, log retention, seamless integration, and real-time analysis, enabling ad-hoc queries on historical data for troubleshooting, performance optimization, and audit analysis. This topic will guide you through configuring message delivery and viewing the delivered messages.
Background
You can deliver three types of messages to an external system in the same or a different region:
Category | Description | Delivery timing |
Job startup logs | Logs produced throughout the startup process, from environment initialization to JobManager startup and DAG graph generation. | Job startup logs are delivered once when the job starts successfully or reaches the final status (failed or finished). |
Resource usage |
Important Although resource usage messages facilitate resource tracking and management, they do not provide alerting functionality. | For an active namespace, resource usage messages are sent every 30 seconds. |
Job events | The startup status at each time point during job startup. | Job events are delivered upon their generation. |
Prerequisites
An Alibaba Cloud Simple Log Service (SLS) project and a logstore are created.
Usage notes
Supported receiver system
Currently, messages can only be delivered to SLS.
Billing
You are not billed on the message delivery feature. However, using SLS features, such as the logstore index, incurs traffic fees.
To query and analyze logs in SLS, you need to enable indexing, which generates index traffic and takes up storage space. You can decide whether to create indexes.
For more information, see Billing overview.
Security
You can set server-side encryption for the logstore, which will be inherited by the delivery of session records. For more information, see Data encryption.
Supported message types and configurations
This feature does not support job runtime logs, which can be output to OSS, SLS, or Kafka through other methods. For more information, see Configure log export.
Changes to message delivery configurations take up to 10 seconds to take effect.
Procedure
Step 1: Configure message delivery
Go to the message delivery configuration page.
Log on to the Realtime Compute for Apache Flink management console.
Click Console in the Actions column of the target workspace.
The development console appears.
In the left-side navigation pane, choose .
Configure SLS message delivery.
Click the Message Delivery tab and turn on the Deliver to SLS switch.
Configure SLS message delivery.
Parameter
Description
Authorization Mode
Valid values:
STS Token: Used for intra-region message delivery. You need to provide your SLS project and logstore.
AccessKey: Used for cross-region message delivery. You need to provide your SLS project, logstore, and AccessKey pair.
Log service project
The name of your SLS project.
Log Service Log Library
Your SLS logstore name.
Endpoint
The endpoint of the SLS project.
When the authorization mode is set to STS Token, the system automatically sets the endpoint based on the Flink workspace's region. Conversely, when the authorization mode is AccessKey, you need to manually specify the endpoint.
AccessKeyId
The AccessKey ID and secret of your Alibaba Cloud account.
ImportantTo ensure security, provide your AccessKey pair by using variables: Click the drop-down arrow to select existing variables or click
on the right side of the fields to create new variables.For more information about variable management and how to view the AccessKey ID and secret values, see Manage variables and How do I view the AccessKey pair of an account?
AccessKeySecret
Click Save.
Step 2: View delivered messages
Procedure
Click Open the SLS console on the right side of Log Service Log Library.
View the raw logs.
Field descriptions
The following tables describe the fields in the three types of messages.
Job startup logs
Field | Description |
messageType | The message type. The value is fixed to |
deploymentId | The job deployment ID. |
deploymentname | The name of the job deployment. |
jobId | The job instance ID. |
tag | The job tag. This field is empty if no tag is configured. |
length | The total length of the log. |
offset | The starting position of this log entry when logs are sharded. |
content | The details of the job startup log. |
workspace | The workspace ID. |
namespace | The namespace name. |
messageId | The message ID. |
timestamp | The timestamp. |
Resource usage
Field | Description |
messageType | The message type. The value is fixed to |
namespaceTotalCpuMemory | The total number of Compute Units (CUs) in the namespace. |
namespaceTotalCpu | The total number of CUs in the namespace. |
namespaceTotalMemory | The total amount of memory in the namespace. |
namespaceUsedCpuMemory | The number of consumed CUs in the namespace. |
namespaceUsedCpu | The number of consumed CUs in the namespace. |
namespaceUsedMemory | The amount of memory used in the namespace. |
resourceQueueName | The queue name . |
resourceQueueTotalCpuMemory | The total number of CUs in the queue. |
resourceQueueTotalCpu | The total number of CUs in the queue. |
resourceQueueTotalMemory | The total amount of memory in the queue. |
resourceQueueUsedCpuMemory | The number of consumed CUs in the queue. |
resourceQueueUsedCpu | The number of consumed CUs in the queue. |
resourceQueueUsedMemory | The amount of memory used in the queue. |
workspace | The workspace ID. |
namespace | The namespace name. |
messageId | The message ID. |
timestamp | The timestamp. |
Job events
Field | Description |
messageType | The message type. The value is fixed to |
deploymentId | The ID of the job deployment. |
deploymentname | The name of the job deployment. |
jobId | The job instance ID. |
tag | The job tag. This field is empty if no tag is configured. |
eventId | The event ID. |
eventName | The event name. |
content | Details of the job startup log. |
workspace | The workspace ID. |
namespace | The namespace name. |
messageId | The message ID. |
timestamp | The timestamp. |
References
For information about how to export logs for a single job, see Configure log export.
For information about how to view logs in the development console, see View boot logs and operational logs, View the events of a deployment, View the exception logs of a deployment, and View the logs of a historical job.