Alibaba Cloud Dynamic Route for CDN (DCDN) allows you to process log data in Data Lake Analytics (DLA) and persist the log data to Object Storage Service (OSS). You can also use DLA to analyze the log data. This topic describes how to enable log storage.

Prerequisites

Before you enable log storage, make sure that the following requirements are met:

  • OSS is activated. To activate OSS, go to the product page of OSS.
  • DLA is activated. To activate DLA, go to the product page of DLA.
  • DLA and OSS are activated in the acceleration region of DCDN.

Background information

DCDN supports the log storage feature, which is integrated with OSS and DLA. When you enable log storage, the system automatically creates the service-linked role AliyunServiceRoleForDCDNLogDelivery and grants the role permissions to access OSS and DLA for log delivery and storage. For more information about the SLR, see SLR for log delivery.

Scenarios

DCDN allows you to download logs over the previous 30 days. If you want to retain log data for a longer period of time or analyze log data, you can enable the log storage feature for data storage and analysis.

Billing

Log storage is free of charge. After log storage is enabled, you are charged for OSS and DLA.
  • For more information about the billing rules of OSS, see OSS Pricing.
  • For more information about the billing rules of DLA, see Billing overview.
    Notice
    • After log storage is enabled, logs are delivered by DLA. You are charged for DLA even if you do not use DLA to analyze data.
    • If you use DLA only to deliver log data but do not analyze log data, DLA consumes compute units (CUs) only for log delivery. No CU is consumed for log analytics.
    • If you use DLA to analyze log data, CUs are consumed based on the number of log entries and the complexity of the log analytics model. CU consumption increases if the number of log entries increases or the complexity of the analytics model increases.

Enable log storage

Notice
  • Logs are collected from DCDN nodes and then directly delivered to OSS. Log integrity is not guaranteed. If you want to acquire complete log data, we recommend that you go to the Log Download tab in the DCDN console to download logs. Make sure that the fields that you specify are the same as those in the logs.
  • By default, Resource Access Management (RAM) users are not allowed to enable or manage log storage. You must first grant the RAM users the required permissions. For more information, see Grant a RAM user permissions on log storage.
  • If you want to grant a RAM user permissions to manage log storage, make sure that the authorized scope is set to Alibaba Cloud Account. For more information, see Step 2: Grant permissions to a RAM user.
  • If you want to manage log storage as a RAM user, make sure that the authorized scope of the RAM user is set to Alibaba Cloud Account. Otherwise, log storage may be disabled for some domain names.
  1. Log on to the DCDN console.
  2. In the left-side navigation pane, choose Data Center > Logs > Offline Log.
  3. Click the Log Storage tab.
  4. Click Activate Now.
  5. Set Storage Region.
    1. In the Activate Log Storage dialog box, set Storage Region.
      Note
      • An OSS bucket is created in the specified region. The region cannot be changed after it is specified. Logs are delivered to the OSS bucket.
      • By default, the regions in which you can use the log storage feature are the same as that you specify when you enable log storage.
      Service country and region Storage country and region Description
      Chinese mainland
      • China (Shanghai)
      • China (Beijing)
      • China (Hangzhou)
      Three storage regions are supported: China (Shanghai), China (Beijing), and China (Hangzhou). Select a region based on your business requirements.
      Other countries and regions Singapore Only this region is supported.
      India India (Mumbai) Only this region is supported.
      Europe Germany (Frankfurt) Only this region is supported.
      US US (Silicon Valley) Only this region is supported.
    2. Click Activate Data Lake Analytics and go to the next step.
  6. Set Transfer Rule.
    1. In the Activate Log Storage dialog box, set Transfer Field and select the accelerated domain names for which you want to enable log storage.
      Note You can view and select up to 500 domain names in the DCDN console. If you want to enable log storage for more domain names, submit a ticket.
      You can specify one or more transfer fields. The transfer fields cannot be modified after they are specified. The following table describes the supported fields.
      Field Description Example
      contentType The file types. text/html
      domain The accelerated domain name. example.com
      hitInfo The cache hit result. HIT
      http2 The HTTP/2 protocol. HTTP2
      httpCode The HTTP status code. 504 , 404 , 302 , 200
      method The request method. Valid values: GET and POST. GET and POST
      refer The Referer header in HTTP requests. "-"
      remoteIP The IP address of the client that initiated the request. 192.168.15.75
      reqSize The size of the request. Unit: bytes. 129
      respSize The size of the response. Unit: bytes. 129
      rt The response time. Unit: milliseconds. 1
      schema The protocol over which the request was transmitted. Valid values: HTTP and HTTPS. HTTP and HTTPS
      traceID The unique ID of the request. d35ba34115550716522547264e
      ua The information about the proxy of the client. Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36
      unixtime The timestamp of the request. Unit: seconds. 160734****
      urlPath The Uniform Resource Identifier (URI) of the request. No domain name information is included. /index.html
      urlRawQuery The query string, which is the parameters that follow the question mark (?) in the URL. x=1&y=1
      userlnfo The custom log fields. None
    2. Click Activate.

      After you enable log storage, you can modify log storage rules or disable log storage on the Log Storage tab.

Disable log storage

  1. Log on to the DCDN console.
  2. In the left-side navigation pane, choose Data Center > Logs > Offline Log.
  3. Click the Log Storage tab.
  4. Click Close Delivery Task.
  5. Click OK.

You can delete the SLR AliyunServiceRoleForDCDNLogDelivery. For more information, see Delete AliyunServiceRoleForDCDNLogDelivery.

FAQ

What is the format of the files that are delivered to OSS after log storage is enabled?

Files delivered to OSS are in the Optimized Row Columnar (ORC) format. We recommend that you use DLA to analyze ORC files.