Fully managed Flink allows you to view metrics to check whether data of a deployment is normal. You can view the metrics of a deployment on the Metrics tab of the Deployments page in the console of fully managed Flink or configure parameters to report metrics to other platforms. This topic describes how to report metrics to a self-managed Prometheus service, Simple Log Service, Kafka, a self-managed platform, or multiple platforms.
Precautions
You can configure parameters to report metrics only to a self-managed Prometheus service, Simple Log Service, or Kafka. You can also configure parameters to collect and report metrics to multiple platforms.
Report metrics to multiple platforms
If you want to collect and report metrics to multiple platforms, you must modify the configurations of the metrics.reporters parameters. If metrics are reported to multiple platforms, the metric collection cost increases. For more information, see Report metrics to multiple platforms.
Report metrics to a platform other than the console of fully managed Flink
Application Real-Time Monitoring Service (ARMS) or CloudMonitor is no longer activated together with Realtime Compute for Apache Flink. In this case, metrics are not displayed in the console of fully managed Flink. You can view the metrics on the platform to which the metrics are reported.
The existing alert configurations in the console of fully managed Flink become invalid. For more information, see Configure alert rules. You can no longer configure alert rules in the console of fully managed Flink. You must configure alert rules on the platform to which the metrics are reported.
Report metrics to a self-managed Prometheus service
Make sure that a network connection is established between fully managed Flink and the self-managed Prometheus service. Then, log on to the console of fully managed Flink and perform the following operations: In the left-side navigation pane, click Deployments. On the Deployments page, click the name of the desired deployment in the Name column. On the Configuration tab, click Edit in the upper-right corner of the Parameters section, add the following code to the Other Configuration field, and then change the values of the parameters related to the self-managed Prometheus service. For more information about the configuration method, see How do I configure parameters for deployment running?
metrics.reporters: promgatewayappmgr
metrics.reporter.promgatewayappmgr.groupingKey: 'deploymentName={{deploymentName}};deploymentId={{deploymentId}};jobId={{jobId}}'
metrics.reporter.promgatewayappmgr.jobName: '{{deploymentName}}'
metrics.reporter.promgatewayappmgr.factory.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory
metrics.reporter.promgatewayappmgr.host: pushgateway host
metrics.reporter.promgatewayappmgr.port: pushgateway portIn the code, you must replace pushgateway host with the hostname of your Pushgateway and replace pushgateway port with the port number of your Pushgateway. The system automatically replaces the values of the deploymentName, deploymentId, and jobId parameters.
The network connection that is established between fully managed Flink and a self-managed Prometheus service must meet the following requirements:
If the self-managed Prometheus service resides in the same virtual private cloud (VPC) as the workspace of fully managed Flink, you must make sure that the CIDR block of fully managed Flink can access the security group of the self-managed Prometheus service.
If the self-managed Prometheus service has a public endpoint and resides in a different VPC from the workspace of fully managed Flink, you must configure Internet access for fully managed Flink. For more information, see How does a fully managed Flink service access the Internet?
If the self-managed Prometheus service has only a VPC endpoint and resides in a different VPC from the workspace of fully managed Flink, you must establish a connection between the VPCs. For more information, see How does fully managed Flink access a service across VPCs?
Report metrics to Simple Log Service
Make sure that a network connection is established between fully managed Flink and Simple Log Service. Then, log on to the console of fully managed Flink and perform the following operations: In the left-side navigation pane, click Deployments. On the Deployments page, click the name of the desired deployment in the Name column. On the Configuration tab, click Edit in the upper-right corner of the Parameters section, add the following code to the Other Configuration field, and then change the values of the parameters related to Simple Log Service. For more information about the configuration method, see How do I configure parameters for deployment running?
metrics.reporters: sls
metrics.reporter.sls.factory.class: org.apache.flink.metrics.sls.SLSReporterFactory
metrics.reporter.sls.project: the project of Simple Log Service
metrics.reporter.sls.logStore: the Logstore of Simple Log Service
metrics.reporter.sls.endPoint: the endpoint of Simple Log Service
metrics.reporter.sls.accessId: the AccessKey ID that is used to connect to Simple Log Service
metrics.reporter.sls.accessKey: the AccessKey secret that is used to connect to Simple Log Service
metrics.reporter.sls.extraTags: deploymentId={{ deploymentId }};deploymentName={{ deploymentName}};namespace={{ namespace }}In the code, you must configure the parameters related to Simple Log Service. The system automatically replaces the values of the deploymentId, deploymentName, and namespace parameters. For more information about the AccessKey pair of an account, see How do I view information about the AccessKey ID and AccessKey secret of the account?
Report metrics to Kafka
Make sure that a network connection is established between fully managed Flink and Kafka. Then, log on to the console of fully managed Flink and perform the following operations: In the left-side navigation pane, click Deployments. On the Deployments page, click the name of the desired deployment in the Name column. On the Configuration tab, click Edit in the upper-right corner of the Parameters section, add the following code to the Other Configuration field, and then change the values of the parameters related to Kafka. For more information about the configuration method, see How do I configure parameters for deployment running?
metrics.reporters: monitor
metrics.reporter.monitor.factory.class: org.apache.flink.metrics.monitor.KafkaReporterFactory
metrics.reporter.monitor.kafka.bootstrap.servers: the addresses of the Kafka brokers
metrics.reporter.monitor.topicName: the name of the Kafka topic
metrics.reporter.monitor._FLINK_CLUSTER_NAME: '{{ deploymentName }}'
metrics.reporter.monitor._JOB_NAME: '{{ deploymentName }}'
metrics.reporter.monitor._NAMESPACE_NAME: '{{ namespace }}'In the code, you must configure the parameters related to the self-managed Kafka cluster. The system automatically replaces the values of the deploymentName and namespace parameters.
Report metrics to multiple platforms
Call an ARMS API operation to report metrics to a self-managed platform
If you select Alibaba Cloud Managed Service for Prometheus when you purchase the workspace of fully managed Flink, you can call an ARMS API operation to obtain the metrics of fully managed Flink and integrate the metrics to your self-managed platform. In this case, you can view metrics on your self-managed platform. You can also view metric curves and configure alert rules in the console of fully managed Flink.
For more information about the metrics of operators, see Operator Metric.
Report metrics to the console of fully managed Flink and other platforms
If you want to view metric curves and configure alert rules in the console of fully managed Flink and also view metrics on other platforms, you can configure parameters to report metrics to multiple platforms. This increases the metric collection cost.
For example, if you want to view metrics in the console of fully managed Flink and Simple Log Service, you can log on to the console of fully managed Flink and perform the following operations: In the left-side navigation pane, click Deployments. On the Deployments page, click the name of the desired deployment in the Name column. On the Configuration tab, click Edit in the upper-right corner of the Parameters section, add the following code to the Other Configuration field, and then change the values of the parameters related to Simple Log Service.
metrics.reporters: jmx,promappmgr,sls
metrics.reporter.sls.factory.class: org.apache.flink.metrics.sls.SLSReporterFactory
metrics.reporter.sls.project: the project of Simple Log Service
metrics.reporter.sls.logStore: the Logstore of Simple Log Service
metrics.reporter.sls.endPoint: the endpoint of Simple Log Service
metrics.reporter.sls.accessId: the AccessKey ID that is used to connect to Simple Log Service
metrics.reporter.sls.accessKey: the AccessKey secret that is used to connect to Simple Log Service
metrics.reporter.sls.extraTags: deploymentId={{ deploymentId }};deploymentName={{ deploymentName}};namespace={{ namespace }}If the metrics.reporters parameters contain jmx,promappmgr, you can view metrics in the console of fully managed Flink. For more information about other parameters, see Report metrics to Simple Log Service.
Report metrics to multiple platforms other than the console of fully managed Flink
After metrics are reported to multiple platforms other than the console of fully managed Flink, metrics are not displayed in the console of fully managed Flink. You must view the metrics on the platforms to which the metrics are reported. For more information, see Precautions.
For example, if you want to report metrics to Simple Log Service and Kafka, you can add the following code to the Other Configuration field in the Parameters section of the Configuration tab in the console of fully managed Flink.
metrics.reporters: sls,monitor
metrics.reporter.sls.factory.class: org.apache.flink.metrics.sls.SLSReporterFactory
metrics.reporter.sls.project: the project of Simple Log Service
metrics.reporter.sls.logStore: the Logstore of Simple Log Service
metrics.reporter.sls.endPoint: the endpoint of Simple Log Service
metrics.reporter.sls.accessId: the AccessKey ID that is used to connect to Simple Log Service
metrics.reporter.sls.accessKey: the AccessKey secret that is used to connect to Simple Log Service
metrics.reporter.sls.extraTags: deploymentId={{ deploymentId }};deploymentName={{ deploymentName}};namespace={{ namespace }}
metrics.reporter.monitor.factory.class: org.apache.flink.metrics.monitor.KafkaReporterFactory
metrics.reporter.monitor.kafka.bootstrap.servers: the addresses of the Kafka brokers
metrics.reporter.monitor.topicName: the name of the Kafka topic
metrics.reporter.monitor._FLINK_CLUSTER_NAME: '{{ deploymentName }}'
metrics.reporter.monitor._JOB_NAME: '{{ deploymentName }}'
metrics.reporter.monitor._NAMESPACE_NAME: '{{ namespace }}'For more information, see Report metrics to Simple Log Service and Report metrics to Kafka.