Log Service:Usage notes

Features

CloudLens for SLS supports the following features:

• The application allows you to manage all projects and Logstores that meet specified conditions within your Alibaba Cloud account in a centralized manner. For more information about conditions, see Limits.

• The application allows you to enable log collection for instance logs and global logs with a few clicks and manage the log collection status in a centralized manner. Instance logs are classified into important logs, detailed logs, and job operational logs. Global logs are classified into audit logs, billing logs, error logs, and metrics.

  • Important logs record the consumption delay of consumer groups for each Logstore, and record the errors, heartbeats, and statistics of Logtail. The logs are stored in a Logstore named internal-diagnostic_log of a specified project.

  • Detailed logs record the operations that create, modify, update, and delete resources in each project. Detailed logs also record data read and write operations. The logs are stored in a Logstore named internal-operation_log of a specified project.

  • Job operational logs record the error and latency information about jobs such as data import jobs of the new version, data shipping jobs of the new version, and Scheduled SQL jobs. The logs are stored in a Logstore named internal-diagnostic_log of a specified project.

  • Audit logs record the operations that create, modify, update, and delete resources in all projects within your Alibaba Cloud account. The logs are stored in a Logstore named internal-audit_log of a specified project.

  • Billing logs record billable resource data. The logs are generated by Cost Manager. The logs are stored in Tablestore and are automatically updated on a daily basis. For more information, see Use the new Cost Manager.

    Important

    • The first time the log collection feature is enabled for billing logs, the feature takes approximately 1 to 2 hours to import logs. The duration during which logs are imported is based on the amount of billing logs.

    • Only Alibaba Cloud public cloud users can enable log collection for billing logs.

    • You cannot disable log collection for billing logs after the feature is enabled.

  • Error logs record the errors of the operations that create, modify, update, and delete resources and the errors of data read and write operations in all projects within your Alibaba Cloud account. The logs are stored in a Logstore named internal-error_log of a specified project.

  • The metrics of each project and each Logstore are recorded, including traffic, requests, and latency. The metrics are stored in a Metricstore named internal-monitor-metric of a specified project.

  For more information, see Service logs.

  

• CloudLens for SLS provides alert monitoring rules such as baseline alert monitoring rules, interval-valued comparison alert monitoring rules, and periodicity-valued comparison alert monitoring rules. You can use the alert monitoring rules to monitor the usage of resources such as projects, Logtail, and consumer groups. CloudLens for SLS supports different notification methods, such as text messages, DingTalk messages, emails, voice calls, and custom webhooks. For more information, see Configure alerts.

• The application provides the following types of reports: Overview, Access Monitoring, Collection Monitoring, Operation Monitoring, Job Monitoring, Quota Monitoring, and Billable Resource Monitoring. For more information, see View data reports.

• The application allows you to view quota information about all projects and Logstores that meet specified conditions within your Alibaba Cloud account. For more information about conditions, see Limits.

  Note

  CloudLens for SLS updates quota information at an interval of 12 hours.

  

Scenarios

• Development O&M

  Developers can use CloudLens for SLS to monitor and manage assets such as projects and Logstores in an efficient manner. In addition, the developers can query and analyze the instance logs of a project in CloudLens for SLS to efficiently troubleshoot errors. Instance logs are classified into detailed logs, important logs, and job operational logs.

• IT O&M

  IT O&M personnel can use the alerting feature of CloudLens for SLS to identify and handle the exceptions of project service status, traffic, quotas, and Logtail metrics at the earliest opportunity.

• Security audit

  The audit logs of CloudLens for SLS record the operations that create, update, and delete resources in a project. This allows security auditors to analyze and audit operations within a specific period of time.

Assets

• Projects, Logstores, and Metricstores

  • When you enable log collection for important logs or job operational logs, you can select a project based on your business requirements. After log collection is enabled for important logs or job operational logs, Simple Log Service generates a Logstore named internal-diagnostic_log in the project.

  • When you enable log collection for detailed logs, you can select a project based on your business requirements. After log collection is enabled for detailed logs, Simple Log Service generates a Logstore named internal-operation_log in the project.

  • When you enable log collection for audit logs, you can select a project based on your business requirements. After log collection is enabled for audit logs, Simple Log Service generates a Logstore named internal-audit_log in the project.

  • When you enable log collection for error logs, you can select a project based on your business requirements. After log collection is enabled for error logs, Simple Log Service generates a Logstore named internal-error_log in the project.

  • When you enable log collection for metrics, you can select a project based on your business requirements. After log collection is enabled for metrics, Simple Log Service generates a Metricstore named internal-error_log in the project.

• Dashboards

  Dashboard		Description
  Overview	Access Overview	Displays information about Simple Log Service projects in charts. The charts include Number of requests, Request Success Rate, Requests with Status Code 4xx, Requests with Status Code 5xx, Write traffic (raw size), Write traffic (after compression), Write traffic compression ratio, and Read Traffic.
  Access Monitoring	Access Traffic Monitoring	Displays information about access to Simple Log Service in charts. The charts include Total Requests, Clients, Users, Top 10 SourceIPs with Most Read Traffic, Top 10 SourceIPs with Most Write Traffic, Requests, Write Traffic, Trend of Write Traffic to Shards, Trend of Read Traffic from Shards, Read/Write Traffic/Minute, and Request IP Distribution.
  	Access Exception Monitoring	Displays information about abnormal access to Simple Log Service in charts. The charts include Total Requests, Percentage of Failed Requests, Quota Exceeded, Error Status Distribution, Logstore with Excessive Write Traffic, Error Requests, and Trend of Request Latency.
  	Consumer Group Monitoring	Displays information about consumer groups in charts. The charts include Consumer Group, Logstore, Shard, Fall Behind, Data, Consumer Group, Top 10 Latency, and Trend of Consumer Group Latency.
  Collection Monitoring	Logtail Overall Status	Displays information about Logtail in charts. The charts include Active Logtail, Data Traffic, Status, CPU, Logtail Overall Status, Trend of CPU, Trend of Memory, and Data Sending Traffic.
  	Logtail File Collection Monitoring	Displays information about the files from which you want to collect logs in charts. The charts include Logtail File Collection, Machine Collection, Collection File Distribution, Trend of Log Collection, Trend of Average Latency, Trend of Parse Failure, and Trend of Send.
  	Logtail Exception Monitoring	Displays information about Logtail exceptions in charts. The charts include Active Logtail, Restart Logtail List, Restart Clients, and All Alarm.
  Operation Monitoring	Operation Monitoring	Displays information about Simple Log Service operation records in charts. The charts include Logstore Related Operation Count, Logstore Related Fail Operation, and Clients.
  Job Monitoring	Data Processing Monitor	Displays information about data import jobs of the new version, data shipping jobs of the new version, and Scheduled SQL jobs in charts. The charts include Ingest Success, Ingest Failed, Deliver Success, Deliver Failed, Ingest PubNet, Deliver PubNet, Process Speed, Process Lags, Process Error, and Process State.
  Quota Monitoring	Quota Monitoring	Displays information about quota exceeded events in charts. The charts include Overview of Resource Quota (80%) Warning, QuotaExceed, LogStore Usage Distribution, MachineGroup Usage Distribution, Logtail Config Usage Distribution, Project Write Inflow Usage Distribution, Project Overview of Basic Quota Usage, Project Overview of Write/Read Quota Usage, and Project Quota Exceeded Statistics.
  Billable Resource Monitoring	Billable Resource Monitoring	Displays information about billable resources of Simple Log Service in charts. The charts include Storage Size(Hot Storage), Index Size(Log), Read/Write Traffic, Billing Items, Top Project, and Top Logstore.

Billing

For more information, see Billable items of pay-by-feature.

Limits

The following table describes the regions where CloudLens for SLS allows you to monitor and manage assets.

Precautions

If you enable a CloudLens application, Simple Log Service automatically checks whether a project whose name is in the aliyun-product-data-<Alibaba Cloud account ID>-cn-heyuan format exists within your Alibaba Cloud account. If the project does not exist, Simple Log Service automatically creates the project.