This topic describes how to import data from MaxCompute to Log Service. After you
store log files in MaxCompute, you can import these files as MaxCompute data to Log
Service. Then you can search, analyze, and transform the data in Log Service.
Import data
- Log on to the Log Service console.
- In the Import Data section, click the MaxCompute - Data Import card.
- Select the newly created project and Logstore. Then, click Next.
- Configure data import.
- On the Specify Data Source tab, set the parameters. The following table describe the parameters.
Parameter |
Description |
Config Name |
The name of the Logtail configuration file. |
MaxCompute Project |
The name of the MaxCompute project where the data to be imported resides. |
Table |
The name of the table where the data to be imported resides. |
Partition Description |
The description of MaxCompute table partitions. You must specify this parameter if
you import data from a partitioned table. For more information, see Partition.
Separate multiple-level partitions with commas (,).
|
AccessKey ID |
The AccessKey ID that you use to access MaxCompute.
Note The Alibaba Cloud account or RAM user to which the AccessKey ID belongs must be granted
access to the MaxCompute project.
|
AccessKey Secret |
The AccessKey secret that you use to access MaxCompute. |
Endpoint |
The endpoint of the MaxCompute project. For more information, see Endpoints.
|
Tunnel Endpoint |
The tunnel endpoint that you use to access the MaxCompute project. For more information,
see Endpoints.
|
- Click Preview to preview the data.
- Click Next.
- On the Specify Data Type tab, set the required parameters. The following table describes the parameters.
Parameter |
Description |
Use System Time |
- If you turn on the Use System Time switch, the time field of a parsed log entry is the system time when the log entry
is imported.
- If you turn off the Use System Time switch, you must manually configure the time field and format.
Note We recommend that you turn on the Use System Time switch. You can configure an index for the time field and use the index for log queries.
If you import data that is generated earlier than the current time minus the data
retention period of the Logstore, the data cannot be queried in the Log Service console.
For example, if the data retention period is seven days and you import data that was
generated seven days ago, no results can be found in the Log Service console.
|
Time Field |
If you turn off the Use System Time switch, you must specify a field to extract the log time.
|
Time Format |
If you turn off the Use System Time switch, you must specify a time format by using the Java SimpleDateFormat class.
The time format is used to parse time fields, or parse strings that are extracted
by regular expressions. For more information, see Class SimpleDateFormat and Time formats.
Note The Java SimpleDateFormat class does not support Unix timestamps. If you want to use
Unix timestamps, you can set the time format to epoch.
|
Time Zone |
If you turn off the Use System Time switch, you must specify a time zone to parse the time zone of the log time. This
parameter is invalid if the time zone information already exists in the log format.
|
- Optional:After you specify the required parameters, click Test.
- After the test succeeds, click Next.
- Optional:On the Specify Scheduling Interval tab, set the required parameters. The following table describes the parameters.
Parameter |
Description |
Import Interval |
The interval at which the MaxCompute data is imported to Log Service. |
Import Now |
If you turn on the Import Now switch, the MaxCompute data is immediately imported.
|
- Click Next.
- Preview data, configure indexes, and then click Next.
By default, full-text indexing is enabled for Log Service. You can also configure
field indexes based on collected logs in manual or automatic mode. For more information,
see
Configure indexes.
Note If you want to query and analyze logs, you must enable full-text indexing or field
indexing. If you enable both full-text indexing and field indexing, the system uses
only field indexes.
View the data import configurations
After you create data import configurations, you can view the configuration details
and relevant statistical report in the Log Service console.
- In the Projects section, click the target project.
- Find the Logstore to which the data import configurations belong, choose , and then click the name of the data import configurations.
- On the Import Configuration Overview page, view the configuration details and statistical report.
References
On the Import Configuration Overview page, you can perform the following operations: