After you install a third-party plug-in, you can use the plug-in to customize the features that you require without the need to modify the Flume source code. For example, you can use a third-party plug-in to add data sources and storage components. You can also use JindoFS SDK to write data from Flume to Object Storage Service (OSS). This topic describes how to install a third-party plug-in.
Prerequisites
- The JAR package of the third-party plug-in that you want to install is obtained.
- An E-MapReduce (EMR) cluster is created, and the Flume service is selected from the optional services when you create the cluster. For more information about how to create a cluster, see Create a cluster.
Procedure
You can use one of the following methods to install a third-party plug-in:
- Method 1: Modify the FLUME_CLASSPATH parameter to add a JAR package.
- On the Configure tab of the Flume service in the EMR console, enter FLUME_CLASSPATH in the search box.
- Modify the value of the FLUME_CLASSPATH parameter to add the obtained JAR package. Separate multiple classpaths with colons (:).
- Save the configuration.
- Click Save.
- In the dialog box that appears, configure the Execution Reason parameter, turn on Automatically Update Configurations, and then click Save.
Note In this example, an EMR cluster is used. For a non-EMR cluster, you can change the value of theFLUME_CLASSPATH
parameter on the flume-env.sh tab based on your business requirements. - Method 2: Upload a third-party JAR package to the $FLUME_HOME/lib/ directory. Note In this example,
$FLUME_HOME
specifies the installation directory of Flume. When you configure a third-party plug-in, replace the directory with the actual directory. In an EMR cluster, the installation directory of Flume is /opt/apps/FLUME/flume-current/lib. - Method 3: Upload a third-party JAR package to the $FLUME_HOME/plugins.d/ directory Related subdirectories:
- $FLUME_HOME/plugins.d/lib: stores the JAR package of the plug-in.
- $FLUME_HOME/plugins.d/libext: stores the JAR package on which the plug-in depends.
- $FLUME_HOME/plugins.d/native: stores the local library file on which the plug-in depends, such as the .so file.
Note In an EMR cluster, common plug-ins are directly placed in the /opt/apps/FLUME/flume-current/lib directory. If you need to upload plug-ins with complex dependencies, especially plug-ins with native dependencies, you can create a /opt/apps/FLUME/flume-current/plugins.d directory and deploy the plug-ins by using Method 3.