|
Parameter |
Type |
Description |
Example |
|---|---|---|---|
|
object |
S3 import configuration. |
||
| awsRegion |
string |
The S3 region. |
ap-northeast-1 |
| awsAccessKey |
string |
The AWS AccessKey ID. |
AKIxxxxxxxxxxYJE |
| awsAccessKeySecret |
string |
The AWS Secret Access Key. |
123xxxxxxxxxxfjf |
| bucket |
string |
The S3 bucket name. |
s3bucket |
| compressionCodec |
string |
The compression type. Valid values: none, snappy, and gzip. |
none |
| encoding |
string |
The encoding format. |
UTF-8 |
| format |
object |
The data format. |
|
|
any |
The data format. |
{"type": "line"} |
|
| interval |
string |
The interval to check for new files. |
30m |
| pattern |
string |
A regular expression to filter file paths. |
.* |
| prefix |
string |
A prefix to filter file paths. |
prefix |
| awsUseSQS |
boolean |
Specifies whether to use Amazon Simple Queue Service (SQS). |
false |
| awsSQSQueueUrl |
string |
The SQS queue URL. |
https://sqs.ap-northeast-1.amazonaws.com/123456788/chifan |
| useAwsSQSOnly |
boolean |
Specifies whether to use only SQS to fetch new files. |
false |
| tagPackId |
boolean |
Specifies whether to enable context-aware browsing. |
false |
| startTime |
integer |
Ingest files modified after this time. |
1714274081 |
| endTime |
integer |
Ingest files modified before this time. |
1714360481 |
| timeField |
string |
The field from which to extract the time. |
__time__ |
| timeFormat |
string |
The format of the time field. |
yyyy-MM-dd HH:mm:ss |
| timePattern |
string |
The regular expression to extract the time. |
[0-9]{0,2}\/[0-9a-zA-Z]+\/[0-9:,]+ |
| timeZone |
string |
The time zone of the time field. |
GMT+08:00 |
| processorId |
string |
The ID of the write processor. |
ingest-processor-1756802123-953901 |
| endpoint |
string |
The S3 endpoint. |
s3.us-east-1.amazonaws.com |