This topic describes how to import data from MaxCompute to Log Service. After you
store log files in MaxCompute, you can import these files as MaxCompute data to Log
Service. Then you can search, analyze, and transform the data in Log Service.
Import data
- Log on to the Log Service console.
- In the Import Data section, click the MaxCompute card.
- In the Specify Logstore step, select the destination project and Logstore, and then click Next.
You can also click Create Now to create a project and a Logstore.
- Configure data import.
- On the Specify Data Source tab, set the parameters. The following table describe the parameters.
Parameter |
Description |
Config Name |
The name of the Logtail configuration file. |
MaxCompute Project |
The name of the MaxCompute project where the data to be imported resides. |
Table |
The name of the table where the data to be imported resides. |
Partition Description |
The description of MaxCompute table partitions. You must specify this parameter if
you import data from a partitioned table. For more information, see Partition.
Separate multiple-level partitions with commas (,).
|
AccessKey ID |
The AccessKey ID that you use to access MaxCompute.
Note The Alibaba Cloud account or RAM user to which the AccessKey ID belongs must be granted
access to the MaxCompute project.
|
AccessKey Secret |
The AccessKey secret that you use to access MaxCompute. |
Endpoint |
The endpoint of the MaxCompute project. For more information, see Configure endpoints.
|
Tunnel Endpoint |
The tunnel endpoint that you use to access the MaxCompute project. For more information,
see Configure endpoints.
|
- Click Preview to preview the data.
- Click Next.
- On the Specify Data Type tab, set the required parameters. The following table describes the parameters.
Parameter |
Description |
Use System Time |
- If you turn on the Use System Time switch, the time field of a parsed log entry is the system time when the log entry
is imported.
- If you turn off the Use System Time switch, you must manually configure the time field and format.
Note We recommend that you turn on the Use System Time switch. You can configure an index for the time field and use the index for log queries.
If you import data that is generated earlier than the current time minus the data
retention period of the Logstore, the data cannot be queried in the Log Service console.
For example, if the data retention period is seven days and you import data that was
generated seven days ago, no results can be found in the Log Service console.
|
Time Field |
If you turn off the Use System Time switch, you must specify a field to extract the log time.
|
Time Format |
If you turn off the Use System Time switch, you must specify a time format by using the Java SimpleDateFormat class.
The time format is used to parse time fields, or parse strings that are extracted
by regular expressions. For more information, visit Class SimpleDateFormat.
Note The Java SimpleDateFormat class does not support Unix timestamps. If you want to use
Unix timestamps, you can set the time format to epoch.
|
Time Zone |
If you turn off the Use System Time switch, you must specify a time zone to parse the time zone of the log time. This
parameter is invalid if the time zone information already exists in the log format.
|
- Optional:After you specify the required parameters, click Test.
- After the test succeeds, click Next.
- Optional:On the Specify Scheduling Interval tab, set the required parameters. The following table describes the parameters.
Parameter |
Description |
Import Interval |
The interval at which the MaxCompute data is imported to Log Service. |
Import Now |
If you turn on the Import Now switch, the MaxCompute data is immediately imported.
|
- Click Next.
- In the Configure Query and Analysis step, set indexes.
Indexes are configured by default. You can reconfigure the indexes as needed. For
more information, see
Enable and configure the index feature for a Logstore.
Note
- You must set Full Text Index or Field Search. If you set both of them, the settings
of Field Search prevail.
- If the data type of an index is Long or Double, the Case Sensitive and Delimiter settings
are unavailable.
- Click Next.
View the data import configurations
After you create data import configurations, you can view the configuration details
and relevant statistical report in the Log Service console.
- In the Projects section, click the target project.
- Find the Logstore to which the data import configurations belong, choose , and then click the name of the data import configurations.
- On the Import Configuration Overview page, view the configuration details and statistical report.
References
On the Import Configuration Overview page, you can perform the following operations:
Appendix
The following table describes the time format syntax.
Character |
Description |
Examples |
G |
Epoch mark |
AD |
y |
4-digit year number |
2001 |
M |
Month name or number |
July or 07 |
d |
Day of the month as a number |
10 |
h |
Hour (12-hour clock, 0 to 12) |
12 |
H |
Hour (24-hour clock, 0 to 23) |
22 |
m |
Minute |
30 |
s |
Second |
55 |
S |
Millisecond |
234 |
E |
Day of the week (Sunday to Saturday) |
Tuesday |
D |
Day of year |
360 |
F |
Day of a week in a month |
2 (second Wed. in July) |
w |
Week number of a year |
40 |
W |
Week number of a month |
1 |
a |
AM/PM |
PM |
k |
Hour (24-hour clock, 1 to 24) |
24 |
k |
Hour (12-hour clock, 0 to 11) |
10 |
z |
Time Zone |
Eastern Standard Time |
' |
Text delimiter |
Delimiter |
" |
Single quotation mark |
None |