This topic describes how to import data from MaxCompute to Log Service. After you import data from MaxComputeto Log Service, you can query, analyze, and transform the data in Log Service.
Create a data import configuration
- Log on to the Log Service console.
- In the Import Data section, click the Data Import tab. Then, click MaxCompute - Data Import.
- Select the project and Logstore. Then, click Next.
- In the Configure Import Settings step, create a data import configuration.
- On the Specify Data Source tab, configure the following parameters.
Parameter | Description |
---|
Config Name | The name of the data import configuration. |
MaxCompute Project | The name of the MaxCompute project that stores the data to import. |
Table | The name of the MaxCompute table that stores the data to import. |
Partition Description | The descriptions of partitions in the MaxCompute table. For more information, see Partition. If the table contains multiple partitions, separate the descriptions of multiple partitions with commas (,). |
AccessKey ID | The AccessKey ID that you use to access MaxCompute. For more information, see AccessKey pair.
Important Make sure that your AccessKey pair has the required permissions to access the MaxCompute project that you want to manage. |
AccessKey Secret | The AccessKey secret that you use to access MaxCompute. |
Endpoint | The endpoint of the MaxCompute project. For more information, see Endpoints. |
Tunnel Endpoint | The tunnel endpoint of the MaxCompute project. For more information, see Endpoints. |
- Click Preview to preview the import result.
- Confirm the result and click Next.
- On the Specify Data Type tab, configure the following parameters.
Parameter | Description |
---|
Use System Time | - If you turn on Use System Time, the time field of a parsed log indicates the system time at which the log is imported.
- If you turn off Use System Time, you must manually specify the time field and time format.
Note We recommend that you turn on Use System Time. You can configure an index for the time field and use the index to query logs. If you import historical data to a Logstore and the data is generated earlier than the current time minus the data retention period of the Logstore, you cannot query the data in the Log Service console. For example, if you import data that was generated seven days ago to a Logstore whose data retention period is seven days, no results are returned when you query the data in the Log Service console. |
Time | If you turn off Use System Time, you must specify a field to extract the log time. |
Time Format | If you turn off Use System Time, you must specify a time format that is supported by Java SimpleDateFormat. The time format is used to parse the time field or the strings that are extracted by using a specified regular expression. For more information about the syntax of the time format, see Class SimpleDateFormat. For more information about common time formats, see Time formats.
Important The Java SimpleDateFormat class does not support UNIX timestamps. If you want to use UNIX timestamps, set Time Format to epoch. |
Time Zone | If you turn off Use System Time, you must specify a time zone. The time zone is used for the parsed log time. If the specified time format includes information about the time zone, this parameter does not take effect. |
- Optional:After you configure the parameters, click Test.
- If the test is successful, click Next.
- Optional:On the Specify Scheduling Interval tab, configure the following parameters.
Parameter | Description |
---|
Import Interval | The interval at which data is imported from MaxCompute to Log Service. |
Execute Now | If you turn on Execute Now, the data import task is immediately run. |
- After the configuration is complete, click Next.
- Preview data, configure indexes, and then click Next.
By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click
Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see
Create indexes.
Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
- Click Log Query. You are redirected to the query and analysis page of your Logstore. On the page, check whether the import task of MaxCompute data is successful.
Wait for approximately 1 minute. If you can query the MaxCompute data, the import task is successful.
View the data import configuration
After you create the data import configuration, you can view the configuration and related reports in the Log Service console.
- In the Projects section, click the project to which the data import configuration belongs.
- Find and click the Logstore to which the data import configuration belongs, choose and then click the data import configuration.
- On the Import Configuration Overview page, view the basic information about the data import configuration and the related reports.
What to do next
On the Import Configuration Overview page, you can perform the following operations:
- Modify the data import configuration
To modify the data import configuration, click Modify Settings. For more information, see In the Configure Import Settings step, create a data import configuration.
- Delete the data import configuration
To delete the data import configuration, click
Delete Configuration.
Warning After a data import configuration is deleted, it cannot be restored. Proceed with caution.