After you install a third-party plug-in, you can use the plug-in to customize the features that you require, without the need to modify Flume source code. For example, you can use a third-party plug-in to add data sources and storage components. You can also use JindoFS SDK to write data from Flume to Object Storage Service (OSS). This topic describes how to install a third-party plug-in.

Prerequisites

  • The JAR package of the third-party plug-in that you want to install is obtained.
  • An E-MapReduce (EMR) cluster is created, and Flume is selected from the optional services when you create the cluster. For more information about how to create a cluster, see Create a cluster.

Procedure

You can use one of the following methods to install a third-party plug-in:
  • Method 1: Modify the flume-env.sh configuration file and add the content of a specified JAR package to the file.
    Note In this example, an EMR cluster is used. The configuration file of Flume is stored in the /etc/ecm/flume-conf/ directory. If your cluster is not an EMR cluster, replace the directory with the actual directory.
    1. Log on to your cluster in SSH mode. For more information, see Log on to a cluster.
    2. Run the following command to go to the directory in which the configuration file of Flume is stored: 
      cd /etc/ecm/flume-conf/
    3. Edit the flume-env.sh.template file.
      1. Run the following command to open the flume-env.sh.template file: 
        vim flume-env.sh.template
      2. Press the I key to enter the edit mode.
      3. In the flume-env.sh.template file, set the FLUME_CLASSPATH parameter to the storage path of the JAR package that you want to add.
      4. Press Esc to exit the edit mode. Then, enter :wq to save and close the file.
  • Method 2: Upload a specific JAR package to the $FLUME_HOME/lib/ directory.
    Note In this example, $FLUME_HOME is the installation directory of Flume. When you configure a third-party plug-in, replace the directory with the actual directory. In an EMR cluster, the installation directory of Flume is /usr/lib/flume-current/lib.
  • Method 3: Upload a specific JAR package to a subdirectory of the $FLUME_HOME/plugins.d/ directory based on your business requirements.
    Related subdirectories:
    • $FLUME_HOME/plugins.d/lib: stores the JAR package of the plug-in.
    • $FLUME_HOME/plugins.d/libext: stores the JAR package on which the plug-in depends.
    • $FLUME_HOME/plugins.d/native: stores the local library file on which the plug-in depends, such as the .so file.
Note In an EMR cluster, the installation packages of plug-ins are stored in the /usr/lib/flume-current/lib directory. To upload a local library file on which a specific plug-in depends, create a /usr/lib/flume-current/plugins.d/native directory.