This topic describes how to import data from MaxCompute to Log Service. After you import data from MaxComputeto Log Service, you can query, analyze, and transform the data in Log Service.

Prerequisites

Create a data import configuration

  1. Log on to the Log Service console.
  2. In the Import Data section, click the Data Import tab. Then, click MaxCompute - Data Import.
  3. Select the project and Logstore. Then, click Next.
  4. In the Configure Import Settings step, create a data import configuration.
    1. On the Specify Data Source tab, configure the following parameters.
      ParameterDescription
      Config NameThe name of the data import configuration.
      MaxCompute ProjectThe name of the MaxCompute project that stores the data to import.
      TableThe name of the MaxCompute table that stores the data to import.
      Partition DescriptionThe descriptions of partitions in the MaxCompute table. For more information, see Partition.

      If the table contains multiple partitions, separate the descriptions of multiple partitions with commas (,).

      AccessKey IDThe AccessKey ID that you use to access MaxCompute. For more information, see AccessKey pair.
      Important Make sure that your AccessKey pair has the required permissions to access the MaxCompute project that you want to manage.
      AccessKey SecretThe AccessKey secret that you use to access MaxCompute.
      EndpointThe endpoint of the MaxCompute project. For more information, see Endpoints.
      Tunnel EndpointThe tunnel endpoint of the MaxCompute project. For more information, see Endpoints.
    2. Click Preview to preview the import result.
    3. Confirm the result and click Next.
    4. On the Specify Data Type tab, configure the following parameters.
      ParameterDescription
      Use System Time
      • If you turn on Use System Time, the time field of a parsed log indicates the system time at which the log is imported.
      • If you turn off Use System Time, you must manually specify the time field and time format.
      Note We recommend that you turn on Use System Time. You can configure an index for the time field and use the index to query logs. If you import historical data to a Logstore and the data is generated earlier than the current time minus the data retention period of the Logstore, you cannot query the data in the Log Service console. For example, if you import data that was generated seven days ago to a Logstore whose data retention period is seven days, no results are returned when you query the data in the Log Service console.
      TimeIf you turn off Use System Time, you must specify a field to extract the log time.
      Time FormatIf you turn off Use System Time, you must specify a time format that is supported by Java SimpleDateFormat. The time format is used to parse the time field or the strings that are extracted by using a specified regular expression. For more information about the syntax of the time format, see Class SimpleDateFormat. For more information about common time formats, see Time formats.
      Important The Java SimpleDateFormat class does not support UNIX timestamps. If you want to use UNIX timestamps, set Time Format to epoch.
      Time ZoneIf you turn off Use System Time, you must specify a time zone. The time zone is used for the parsed log time. If the specified time format includes information about the time zone, this parameter does not take effect.
    5. Optional:After you configure the parameters, click Test.
    6. If the test is successful, click Next.
    7. Optional:On the Specify Scheduling Interval tab, configure the following parameters.
      ParameterDescription
      Import IntervalThe interval at which data is imported from MaxCompute to Log Service.
      Execute NowIf you turn on Execute Now, the data import task is immediately run.
    8. After the configuration is complete, click Next.
  5. Preview data, configure indexes, and then click Next.
    By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.
    Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
  6. Click Log Query. You are redirected to the query and analysis page of your Logstore. On the page, check whether the import task of MaxCompute data is successful.
    Wait for approximately 1 minute. If you can query the MaxCompute data, the import task is successful.

View the data import configuration

After you create the data import configuration, you can view the configuration and related reports in the Log Service console.

  1. In the Projects section, click the project to which the data import configuration belongs.
  2. Find and click the Logstore to which the data import configuration belongs, choose Data Import > Data Import and then click the data import configuration.
  3. On the Import Configuration Overview page, view the basic information about the data import configuration and the related reports.

What to do next

On the Import Configuration Overview page, you can perform the following operations:

  • Modify the data import configuration

    To modify the data import configuration, click Modify Settings. For more information, see In the Configure Import Settings step, create a data import configuration.

  • Delete the data import configuration
    To delete the data import configuration, click Delete Configuration.
    Warning After a data import configuration is deleted, it cannot be restored. Proceed with caution.