Use JindoSDK with Flume to write data to the OSS-HDFS service - Object Storage Service

Apache Flume is a distributed, reliable, and high-availability (HA) system. It efficiently collects, aggregates, and moves large amounts of log data from many different data sources to a centralized data store. Flume uses flush() calls to ensure transactional writes. When you use JindoSDK to write data to the OSS-HDFS service, the data is immediately visible after a flush, which prevents data loss.

Prerequisites

For example, if your deployment environment is an Alibaba Cloud ECS instance, you need to purchase an ECS instance.
You have created a Hadoop environment. For more information, see Create a Hadoop runtime environment.
You have deployed Apache Flume. For more information, see Apache Flume.
You have enabled and authorized the OSS-HDFS service. For more information, see Enable the OSS-HDFS service.

Procedure

Connect to the ECS instance. For more information, see Connect to an ECS instance.
Configure JindoSDK.
1. Download the latest JindoSDK JAR package. For the download link, see GitHub.
2. Decompress the downloaded installation package.
  
  The following example shows how to decompress jindosdk-x.x.x-linux.tar.gz. If you use a different version of JindoSDK, replace the package name with the actual one.
```
tar -zxvf jindosdk-x.x.x-linux.tar.gz -C /usr/lib
```
3. Configure the JINDOSDK_HOME environment variable.
```
export JINDOSDK_HOME=/usr/lib/jindosdk-x.x.x-linux
export PATH=$JINDOSDK_HOME/bin:$PATH
```
4. Configure the HADOOP_CLASSPATH environment variable.
```
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:${FLUME_HOME}/lib/*
```
  Important
  On each node, deploy the required files to the lib folder in the Flume root directory and set the environment variables.
5. Configure the FLUME_CLASSPATH environment variable.
```
cp ${FLUME_HOME}/conf/flume-env.sh.template ${FLUME_HOME}/conf/flume-env.sh
echo "FLUME_CLASSPATH=/usr/lib/jindosdk-x.x.x-linux/lib/*" >>  ${FLUME_HOME}/conf/flume-env.sh
```

Configure the sink.

The following code provides an example of the sink configuration:

# Configure the OSS sink. Set your_bucket to a bucket for which the OSS-HDFS service is enabled.
xxx.sinks.oss_sink.hdfs.path = oss://${your_bucket}/flume_dir/%Y-%m-%d/%H

# The maximum number of events in a Flume transaction. Flush at least 32 MB of data at a time to prevent small flushes from affecting overall performance and generating many staging files.
# The unit of batchSize is the number of events, which is the number of log entries. To set this parameter, first estimate the average event size. For example, if the average event size is 200 bytes and you expect to flush 32 MB of data at a time, set batchSize to approximately 160,000 (32 MB / 200 bytes).
xxx.sinks.oss_sink.hdfs.batchSize = 100000

...
# Specifies whether to partition Hadoop Distributed File System (HDFS) files by time and round down the timestamp. The default value is true.
xxx.sinks.oss_sink.hdfs.round = true
# This parameter is used with xxx.sinks.oss_sink.hdfs.roundUnit when xxx.sinks.oss_sink.hdfs.round is set to true. For example, if xxx.sinks.oss_sink.hdfs.roundUnit is set to minute and this parameter is set to 60, data within a 60-minute period is written to a single file. This means a new file is generated every 60 minutes.
xxx.sinks.oss_sink.hdfs.roundValue = 15
# The time unit for time-based partitioning. Default value: minute. Valid values: second, minute, and hour.
xxx.sinks.oss_sink.hdfs.roundUnit = minute
# The fixed prefix for new files that Flume creates in the HDFS folder.
xxx.sinks.oss_sink.hdfs.filePrefix = your_topic
# The file size in bytes that triggers a file roll. A new file is created when the current file reaches this size. A value of 0 indicates that files are not split by size.
xxx.sinks.oss_sink.rollSize = 3600
# The number of threads that each sink instance opens for HDFS I/O operations, such as open and write.
xxx.sinks.oss_sink.threadsPoolSize = 30
...

For more information about the sink configuration parameters, see Apache Flume.

FAQ

Error: "org.apache.flume.conf.ConfigurationException: Component has no type. Cannot configure.user_sink"

To resolve this issue, add the following configuration to the Hadoop core-site.xml file.

<!-- Configure the JindoOSS implementation class -->
fs.AbstractFileSystem.oss.impl com.aliyun.jindodata.oss.OSS 
fs.oss.impl com.aliyun.jindodata.oss.OssFileSystem
<!-- Configure the AccessKey and Endpoint -->
fs.oss.credentials.provider com.aliyun.jindodata.auth.SimpleAliyunCredentialsProvider 
fs.oss.accessKeyId LTAI******** 
fs.oss.accessKeySecret KZo1********
fs.oss.endpoint {regionId}.oss-dls.aliyuncs.com