E-MapReduce:Synchronize HDFS audit logs to HDFS

Prerequisites

A data lake cluster has been created and the Flume service is selected. For more information, see Create a cluster.

Procedure

1. Navigate to the cluster services page.

   1. Log on to the E-MapReduce console.

   2. In the top navigation bar, select a region and a resource group as needed.

   3. On the Cluster Management page, find the cluster that you want to manage and click Services in the Actions column.

2. On the Services page, click Configure in the Flume service section.

3. Configure the Flume agent on the core-1-1 node and save the configuration.

   The following configuration parameters adhere to open-source Flume standards. For more information, see the Apache Flume User Guide.

   1. On the Configure page, click the flume-conf.properties tab.

   2. From the drop-down lists at the top, select Independent Node Configuration and core-1-1.

   3. Modify the parameters in flume-conf.properties based on your requirements.

      default-agent.sinks = default-sink
      default-agent.sources = default-source
      default-agent.channels = default-channel
      default-agent.sinks.default-sink.type = hdfs
      default-agent.sinks.default-sink.channel =  default-channel
      default-agent.channels.default-channel.type = file
      default-agent.sources.default-source.type = avro
      default-agent.sinks.default-sink.hdfs.path = hdfs://master-1-1:9000/path
      default-agent.sinks.default-sink.hdfs.fileType = DataStream
      default-agent.sinks.default-sink.hdfs.rollSize = 0
      default-agent.sinks.default-sink.hdfs.rollCount = 0
      default-agent.sinks.default-sink.hdfs.rollInterval = 86400
      default-agent.sinks.default-sink.hdfs.batchSize = 51200
      default-agent.sources.default-source.bind = 0.0.0.0
      default-agent.sources.default-source.port = ****
      default-agent.sources.default-source.channels =  default-channel
      default-agent.channels.default-channel.transactionCapacity = 10000
      default-agent.channels.default-channel.dataDirs = ****
      default-agent.channels.default-channel.checkpointDir = ****
      default-agent.channels.default-channel.capacity = 1000000

      Parameter	Description
      default-agent.sinks	The names of all sinks. Example: default-sink.
      default-agent.sources	The names of all sources. Example: default-source.
      default-agent.channels	The names of all channels. Example: default-channel.
      default-agent.sinks.default-sink.hdfs.path	The HDFS path. For a high-availability (HA) cluster: hdfs://emr-cluster/path For a non-HA cluster: hdfs://master-1-1:9000/path
      default-agent.sinks.default-sink.hdfs.fileType	This parameter must be set to DataStream.
      default-agent.sinks.default-sink.hdfs.rollSize	The file size in bytes to trigger a roll. When a temporary file reaches this size, the sink rolls it into a destination file. If you set this parameter to 0, files are not rolled based on size.
      default-agent.sinks.default-sink.hdfs.rollCount	The number of events to trigger a roll. When the event count reaches this value, the sink rolls the temporary file into a destination file. If you set this parameter to 0, files are not rolled based on the number of events.
      default-agent.sinks.default-sink.hdfs.rollInterval	The time interval in seconds at which to roll the file. Example: 86400.
      default-agent.sinks.default-sink.hdfs.batchSize	The number of events to batch before flushing to HDFS. Example: 51200.
      default-agent.sinks.default-sink.channel	The channel name for the default-sink.
      default-agent.sources.default-source.bind	The IP address to bind to. Set this to 0.0.0.0 to bind to all interfaces on the machine.
      default-agent.sources.default-source.port	The port to listen on. Configure this parameter as needed.
      default-agent.sources.default-source.channels	The channel name for the default-source.
      default-agent.channels.default-channel.transactionCapacity	The maximum number of events that the channel can process in a single transaction. The default value is 10000.
      default-agent.channels.default-channel.dataDirs	The path where the channel stores event data. This parameter is optional. The default path is ~/.flume/file-channel/data.
      default-agent.channels.default-channel.checkpointDir	The path where the channel stores checkpoints. This parameter is optional. The default path is ~/.flume/file-channel/checkpoint.
      default-agent.channels.default-channel.capacity	The capacity of the channel. Configure this parameter based on your HDFS roll settings. This parameter is optional. The default value is 1000000.

   4. Save the configuration.
      1. At the bottom of the page, click Save.
      2. In the dialog box, enter a reason for the change and click OK.

4. Follow the instructions in Step 3 to configure the Flume agent on the core-1-2 node and save the configuration.

5. Configure the Flume agent on the master node group and save the configuration.

   The following configuration parameters adhere to open-source Flume standards. For more information, see the Apache Flume User Guide.

   1. From the drop-down lists at the top, select Independent Node Configuration and master-1-1.

   2. Modify the parameters in flume-conf.properties based on your requirements.

      default-agent.sinks = default-sink k1
      default-agent.sources = default-source
      default-agent.channels = default-channel
      default-agent.sources.default-source.type = taildir
      default-agent.sinks.default-sink.type = avro
      default-agent.sinks.default-sink.channel =  default-channel
      default-agent.channels.default-channel.type = file
      default-agent.sources.default-source.filegroups = f1
      default-agent.sources.default-source.filegroups.f1 = /mnt/disk1/log/hadoop-hdfs/hdfs-audit.log.*
      default-agent.sources.default-source.positionFile = ~/.flume/taildir_position.json
      default-agent.sources.default-source.channels =  default-channel
      default-agent.sources.default-source.batchSize = 2000
      default-agent.sources.default-source.ignoreRenameWhenMultiMatching = true
      default-agent.channels.default-channel.checkpointDir = ****
      default-agent.channels.default-channel.dataDirs = ****
      default-agent.channels.default-channel.capacity = ****
      default-agent.channels.default-channel.transactionCapacity = 2000
      default-agent.sinkgroups = g1
      default-agent.sinkgroups.g1.sinks = default-sink k1
      default-agent.sinkgroups.g1.processor.type = failover
      default-agent.sinkgroups.g1.processor.priority.default-sink = 10
      default-agent.sinkgroups.g1.processor.priority.k1 = 5
      default-agent.sinks.default-sink.hostname = ****
      default-agent.sinks.default-sink.port = ****
      default-agent.sinks.k1.hostname = ****
      default-agent.sinks.k1.port = ****
      default-agent.sinks.default-sink.batch-size = 2000
      default-agent.sinks.k1.batch-size = 2000
      default-agent.sinks.k1.type = avro
      default-agent.sinks.k1.channel = default-channel

      Parameter	Description
      default-agent.sinks	The names of all sinks. Example: default-sink k1.
      default-agent.sources	The names of all sources. Example: default-source.
      default-agent.channels	The names of all channels. Example: default-channel.
      default-agent.sources.default-source.filegroups.f1	The path of the log files. The default path is /mnt/disk1/log/hadoop-hdfs/hdfs-audit.log.*.
      default-agent.sources.default-source.positionFile	The path where Flume stores the position file. This parameter is optional. The default path is ~/.flume/taildir_position.json.
      default-agent.channels.default-channel.checkpointDir	The path where the channel stores checkpoints.
      default-agent.channels.default-channel.dataDirs	The path where the channel stores event data.
      default-agent.channels.default-channel.capacity	Set this parameter based on your HDFS roll settings.
      default-agent.sources.default-source.batchSize	The maximum number of messages the source writes to the channel in a single batch. Example: 2000.
      default-agent.channels.default-channel.transactionCapacity	The maximum number of events that a channel can take from a source or push to a sink per transaction. Example: 2000.
      default-agent.sources.default-source.ignoreRenameWhenMultiMatching	When a Flume taildir source uses a wildcard in a file group to match rotated log4j files, data duplication can occur. Set this parameter to true to prevent this issue.
      default-agent.sinkgroups	The names of all sink groups. Example: g1.
      default-agent.sinkgroups.g1.sinks	The names of all sinks in the g1 sink group. Example: default-sink k1.
      default-agent.sinkgroups.g1.processor.type	The processor type for the g1 sink group. Valid values: default: The default processor. failover: The failover processor. load_balance: The load balancing processor.
      default-agent.sinkgroups.g1.processor.priority.default-sink	The priority of the default-sink within the g1 sink group. A higher value indicates a higher priority. Example: 10.
      default-agent.sinkgroups.g1.processor.priority.k1	The priority of the k1 sink within the g1 sink group. A higher value indicates a higher priority. Example: 5.
      default-agent.sinks.default-sink.hostname	The IP address of the core-1-1 node.
      default-agent.sinks.default-sink.port	The port of the Flume agent on the core-1-1 node.
      default-agent.sinks.k1.hostname	The IP address of the core-1-2 node.
      default-agent.sinks.k1.port	The port of the Flume agent on the core-1-2 node.
      default-agent.sinks.default-sink.batch-size	The number of events that default-sink sends in each batch. Example: 2000.
      default-agent.sinks.k1.batch-size	The number of events that k1 sends in each batch. Example: 2000.
      default-agent.sinks.k1.type	The sink type. Example: avro.
      default-agent.sinks.k1.channel	The channel for the sink. Example: default-channel.

   3. Save the configuration.
      1. At the bottom of the page, click Save.
      2. In the dialog box, enter a reason for the change and click OK.

6. Start the Flume agent.

   1. In the upper-right corner, choose More > Restart.

   2. In the dialog box that appears, enter a reason in the Execution Reason field and click OK.

   3. In the Confirm dialog box, click OK.

      After the service restarts, the Flume agent synchronizes the HDFS audit logs to HDFS.

      The Flume agent logs are stored in /var/log/emr/flume/default-agent/flume.log.