Alibaba Cloud CDN allows you to process log data in Data Lake Analytics (DLA) and persist the log data in Object Storage Service (OSS). You can also use DLA to analyze log data. This topic describes how to enable log storage.

Prerequisites

Before you enable log storage, make sure that the following requirements are met:

  • OSS is activated. To activate OSS, go to the product page of OSS.
  • DLA is activated. To activate DLA, go to the product page of DLA.
  • DLA and OSS are activated in the accelerated region of Alibaba Cloud CDN.

Background information

Alibaba Cloud CDN supports the log storage feature. This feature is integrated with OSS and DLA. When you enable log storage, the system automatically creates the AliyunServiceRoleForDCDNLogDelivery service-linked role and grants the role the permissions on OSS and DLA. Alibaba Cloud CDN can assume this role to automatically deliver logs to OSS and DLA. For more information about the service-linked role, see Manage the SLR for log storage.

Scenarios

By default, the log storage feature of Alibaba Cloud CDN retains log data for 30 days. If you want to retain log data for a longer period of time or analyze log data, you can enable DLA for log storage. DLA allows you to persist and analyze log data based on business requirements.

Billing rules

Log storage is free of charge. After you enable log storage, you are charged for OSS and DLA.
  • For more information about the billing rules of OSS, see OSS Pricing.
  • For more information about the billing rules of DLA, see Billing method.
    Notice
    • After you enable log storage, logs are delivered by DLA. You are charged for DLA even if you do not use DLA to analyze data.
    • If you use DLA only to deliver log data but do not analyze log data, DLA consumes compute units (CUs) only for log delivery. No CU is consumed for log analytics.
    • If you use DLA to analyze log data, CUs are consumed based on the number of log entries and the complexity of the log analytics model. CU consumption increases with the number of log entries or the complexity of the analytics model.

Enable log storage

Notice
  • Logs are collected from points of presence (POPs) and then directly delivered to OSS. Log integrity is not guaranteed. If you want to acquire complete log data, we recommend that you go to the Log Download tab to download logs. Make sure that the fields that you specify are the same as the fields in the logs.
  • By default, RAM users cannot enable or manage log storage. You must grant the RAM users the required permissions. For more information, see Grant a RAM user permissions on log storage.
  • If you want to grant a RAM user the permissions to manage log storage, make sure that the authorized scope is set to Alibaba Cloud Account. For more information, see Step 2: Grant permissions to the RAM user.
  • If you want to manage log storage as a RAM user, make sure that the authorized scope of the RAM user is set to Alibaba Cloud Account. Otherwise, log storage may be disabled for some domain names.
  1. Log on to the Alibaba Cloud CDN console.
  2. In the left-side navigation pane, choose Logs > Offline Logs.
  3. Click the DLA for Log Storage and Analytics tab.
  4. Click Activate Now.
  5. Configure the Storage Location parameter.
    1. In the Activate Log Storage dialog box, configure the Storage Location parameter.
      Note
      • An OSS bucket is created in the specified region. You cannot change the region that you specified. Logs are delivered to the OSS bucket.
      • If a region is specified when log storage is enabled for Dynamic Route for CDN (DCDN), you cannot specify a region when you enable log storage for Alibaba Cloud CDN. By default, the regions for the log storage feature in Alibaba Cloud CDN and DCDN are the same.
      Service country and region Storage country and region Description
      Chinese mainland
      • China (Shanghai)
      • China (Beijing)
      • China (Hangzhou)
      		 Three storage regions are supported: China (Shanghai), China (Beijing), and China (Hangzhou). Select a region based on your business requirements.
      Other countries and regions Singapore (Singapore) Only this region is supported.
      India India (Mumbai) Only this region is supported.
      Europe Germany (Frankfurt) Only this region is supported.
      US US (Silicon Valley) Only this region is supported.
    2. Click Activate Data Lake Analytics and Next.
  6. Configure the Delivery Rule parameter.
    1. In the Activate Log Storage dialog box, configure the Log Fields parameter, and select the accelerated domain names for which you want to enable log storage.
      Note You can view and select up to 500 accelerated domain names in the console. If you want to enable log storage for more accelerated domain names, submit a ticket.
      You can specify one or more transfer fields. You cannot modify the transfer fields that you specified. The following table describes the supported fields.
      Field Description Example
      contentType The file type. text/html
      domain The information about the accelerated domain name. example.com
      hitInfo The cache hit status. HIT
      http2 The HTTP/2 protocol. HTTP2
      httpCode The HTTP status code. 504 , 404 , 302 , 200
      method The request method. Valid values: GET and POST. GET and POST
      refer The Referer header in HTTP requests. "-"
      remoteIP The IP address of the client that initiated the request. 192.168.15.75
      reqSize The size of the request. Unit: bytes. 129
      respSize The size of the response. Unit: bytes. 129
      rt The response time. Unit: milliseconds. 1
      schema The protocol over which the request is transmitted. Valid values: HTTP and HTTPS. HTTP and HTTPS
      traceID The unique ID of the request. d35ba34115550716522547264e
      ua The information about the proxy of the client. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36
      unixtime The timestamp of the request. Unit: seconds. 160734****
      urlPath The Uniform Resource Identifier (URI) of the request. No domain name information is included. /index.html
      urlRawQuery The query string, which is the parameters that follow the question mark (?) in the URL. x=1&y=1
      userlnfo The custom log fields. N/A
    2. Click Activate.

      You can modify log storage rules or disable log storage on the Log Storage tab.

Disable log storage

  1. Log on to the Alibaba Cloud CDN console.
  2. In the left-side navigation pane, choose Logs > Offline Log.
  3. Click the DLA for Log Storage and Analytics tab.
  4. Click Close Delivery Task.
  5. Click OK.

You can delete the AliyunServiceRoleForCDNLogDelivery service-linked role. For more information, see Delete AliyunServiceRoleForCDNLogDelivery.

FAQ

What is the format of the files that are delivered to OSS after log storage is enabled?

The files that are delivered to OSS are in the Optimized Row Columnar (ORC) format. We recommend that you use DLA to analyze ORC files.