This topic describes how to import data from MaxCompute to Log Service. After you store log files in MaxCompute, you can import these files as MaxCompute data to Log Service. Then you can search, analyze, and transform the data in Log Service.

Prerequisites

Import data

  1. Log on to the Log Service console.
  2. In the Import Data section, click the MaxCompute card.
  3. In the Specify Logstore step, select the destination project and Logstore, and then click Next.
    You can also click Create Now to create a project and a Logstore.
  4. Configure data import.
    1. On the Specify Data Source tab, set the parameters. The following table describe the parameters.
      Parameter Description
      Config Name The name of the Logtail configuration file.
      MaxCompute Project The name of the MaxCompute project where the data to be imported resides.
      Table The name of the table where the data to be imported resides.
      Partition Description The description of MaxCompute table partitions. You must specify this parameter if you import data from a partitioned table. For more information, see Partition.

      Separate multiple-level partitions with commas (,).
      AccessKey ID The AccessKey ID that you use to access MaxCompute.
      Note The Alibaba Cloud account or RAM user to which the AccessKey ID belongs must be granted access to the MaxCompute project.
      AccessKey Secret The AccessKey secret that you use to access MaxCompute.
      Endpoint The endpoint of the MaxCompute project. For more information, see Configure endpoints.
      Tunnel Endpoint The tunnel endpoint that you use to access the MaxCompute project. For more information, see Configure endpoints.
    2. Click Preview to preview the data.
    3. Click Next.
    4. On the Specify Data Type tab, set the required parameters. The following table describes the parameters.
      Parameter Description
      Use System Time
      • If you turn on the Use System Time switch, the time field of a parsed log entry is the system time when the log entry is imported.
      • If you turn off the Use System Time switch, you must manually configure the time field and format.
      Note We recommend that you turn on the Use System Time switch. You can configure an index for the time field and use the index for log queries. If you import data that is generated earlier than the current time minus the data retention period of the Logstore, the data cannot be queried in the Log Service console. For example, if the data retention period is seven days and you import data that was generated seven days ago, no results can be found in the Log Service console.
      Time Field If you turn off the Use System Time switch, you must specify a field to extract the log time.
      Time Format If you turn off the Use System Time switch, you must specify a time format by using the Java SimpleDateFormat class. The time format is used to parse time fields, or parse strings that are extracted by regular expressions. For more information, visit Class SimpleDateFormat.
      Note The Java SimpleDateFormat class does not support Unix timestamps. If you want to use Unix timestamps, you can set the time format to epoch.
      Time Zone If you turn off the Use System Time switch, you must specify a time zone to parse the time zone of the log time. This parameter is invalid if the time zone information already exists in the log format.
    5. Optional:After you specify the required parameters, click Test.
    6. After the test succeeds, click Next.
    7. Optional:On the Specify Scheduling Interval tab, set the required parameters. The following table describes the parameters.
      Parameter Description
      Import Interval The interval at which the MaxCompute data is imported to Log Service.
      Import Now If you turn on the Import Now switch, the MaxCompute data is immediately imported.
    8. Click Next.
  5. In the Configure Query and Analysis step, set indexes.
    Indexes are configured by default. You can reconfigure the indexes as needed. For more information, see Enable and configure the index feature for a Logstore.
    Note
    • You must set Full Text Index or Field Search. If you set both of them, the settings of Field Search prevail.
    • If the data type of an index is Long or Double, the Case Sensitive and Delimiter settings are unavailable.
  6. Click Next.

View the data import configurations

After you create data import configurations, you can view the configuration details and relevant statistical report in the Log Service console.

  1. In the Projects section, click the target project.
  2. Find the Logstore to which the data import configurations belong, choose Data Import > Data Import, and then click the name of the data import configurations.
  3. On the Import Configuration Overview page, view the configuration details and statistical report.

References

On the Import Configuration Overview page, you can perform the following operations:

  • Modify the data import configurations

    Click Modify Settings to modify the data import configurations. For more information, see Configure data import.

  • Delete the data import configurations.
    You can click Delete Configuration to delete the data import configurations.
    Note The data import configurations cannot be recovered after they are deleted.

Appendix

The following table describes the time format syntax.
Character Description Examples
G Epoch mark AD
y 4-digit year number 2001
M Month name or number July or 07
d Day of the month as a number 10
h Hour (12-hour clock, 0 to 12) 12
H Hour (24-hour clock, 0 to 23) 22
m Minute 30
s Second 55
S Millisecond 234
E Day of the week (Sunday to Saturday) Tuesday
D Day of year 360
F Day of a week in a month 2 (second Wed. in July)
w Week number of a year 40
W Week number of a month 1
a AM/PM PM
k Hour (24-hour clock, 1 to 24) 24
k Hour (12-hour clock, 0 to 11) 10
z Time Zone Eastern Standard Time
' Text delimiter Delimiter
" Single quotation mark None