After you install a third-party plug-in, you can use the plug-in to customize the features that you require without the need to modify the Flume source code. For example, you can use a third-party plug-in to add data sources and storage components. You can also use JindoFS SDK to write data from Flume to Object Storage Service (OSS). This topic describes how to install a third-party plug-in.

Prerequisites

  • The JAR package of the third-party plug-in that you want to install is obtained.
  • An E-MapReduce (EMR) cluster is created, and the Flume service is selected from the optional services when you create the cluster. For more information about how to create a cluster, see Create a cluster.

Procedure

You can use one of the following methods to install a third-party plug-in:
  • Method 1: Modify the FLUME_CLASSPATH parameter to add a JAR package.
    1. On the Configure tab of the Flume service in the EMR console, enter FLUME_CLASSPATH in the search box.
    2. Modify the value of the FLUME_CLASSPATH parameter to add the obtained JAR package. Separate multiple classpaths with colons (:). flume_jar
    3. Save the configuration.
      1. Click Save.
      2. In the dialog box that appears, configure the Execution Reason parameter, turn on Automatically Update Configurations, and then click Save.
    Note In this example, an EMR cluster is used. For a non-EMR cluster, you can change the value of the FLUME_CLASSPATH parameter on the flume-env.sh tab based on your business requirements.
  • Method 2: Upload a third-party JAR package to the $FLUME_HOME/lib/ directory.
    Note In this example, $FLUME_HOME specifies the installation directory of Flume. When you configure a third-party plug-in, replace the directory with the actual directory. In an EMR cluster, the installation directory of Flume is /opt/apps/FLUME/flume-current/lib.
  • Method 3: Upload a third-party JAR package to the $FLUME_HOME/plugins.d/ directory
    Related subdirectories:
    • $FLUME_HOME/plugins.d/lib: stores the JAR package of the plug-in.
    • $FLUME_HOME/plugins.d/libext: stores the JAR package on which the plug-in depends.
    • $FLUME_HOME/plugins.d/native: stores the local library file on which the plug-in depends, such as the .so file.
Note In an EMR cluster, common plug-ins are directly placed in the /opt/apps/FLUME/flume-current/lib directory. If you need to upload plug-ins with complex dependencies, especially plug-ins with native dependencies, you can create a /opt/apps/FLUME/flume-current/plugins.d directory and deploy the plug-ins by using Method 3.