This topic describes common parameter tuning for Taildir Source, File Channel, and HDFS Sink configurations.
Taildir Source
| Parameter | Description |
| filegroups | Splits a folder into multiple parts to increase concurrent reads from the source. |
| batchSize | The default value is 100. The number of data rows to read at a time. Increase this value to improve throughput. |
File Channel
| Parameter | Description |
| checkpointInterval | The default value is 30 seconds (s). Decrease this value to shorten the checkpoint writing interval. |
| useDualCheckpoints | The default value is false. Set this parameter to true to prevent the system from rereading all events from the beginning if a checkpoint is lost during a restart. |
| maxFileSize | The default value is 1.6 GB. The size of a single DATA file. Decrease this value so the file channel can remove expired files sooner and reduce disk usage. |
| capacity | The default value is 1,000,000. The maximum number of events that a channel can retain. Increase this value to optimize throughput. You can also multiply this value by the size of a single event to estimate disk usage. |
| transactionCapacity | The default value is 10,000. The maximum capacity of a single transaction supported by the channel. |
HDFS Sink
| Parameter | Description |
| hdfs.batchSize | The default value is 100. The number of events in each batch operation when writing content to HDFS. Increase this value to improve throughput. Note Set this parameter to the same value as the batchSize parameter in the Source configuration. The value must not exceed the value of the transactionCapacity parameter in the Channel configuration. |
| hdfs.threadsPoolSize | The default value is 10. The number of HDFS I/O threads. Adjust this value based on the machine configuration. |
| hdfs.useLocalTimeStamp | The default value is false. Specifies whether to use the local timestamp. Set this parameter to true to add a timestamp to the event header. |
| hdfs.rollInterval | The default value is 30 seconds (s). The interval at which a temporary file is rolled over to become an object file. Set this parameter to 0 to disable time-based rollover. |
| hdfs.rollSize | The default value is 1024 bytes. The file size at which a temporary file is rolled over to become an object file. Set this parameter to 0 to disable size-based rollover. |
| hdfs.rollCount | The default value is 10 events. The number of events at which a temporary file is rolled over to become an object file. Set this parameter to 0 to disable event count-based rollover. |
| hdfs.minBlockReplicas | The default value is the number of HDFS replicas. The minimum number of replicas for an HDFS file block. This parameter is usually set to 1 to ensure that files are rolled over correctly. |