Configure an HDFS client to access LindormDFS - Lindorm

LindormDFS is compatible with the Hadoop Distributed File System (HDFS) API. This topic shows you how to configure an open-source Hadoop HDFS client to connect to LindormDFS and run common file operations.

Prerequisites

Before you begin, ensure that you have:

Java installed. JDK 1.7 or later is required.
The client IP address added to the Lindorm whitelist. For details, see Set whitelists.

Network requirements

If your application runs on an Elastic Compute Service (ECS) instance, the ECS instance and the Lindorm instance must meet both of the following conditions:

They are in the same region. Use the same availability zone to reduce network latency.
They use the same virtual private cloud (VPC).

Download the client

Download the Hadoop 2.7.3 software development kit (SDK), hadoop-2.7.3.tar.gz, from the Apache official website.

Set up Hadoop

You can configure the HDFS client in two ways:

Console (recommended): Generate the configuration file directly from the Lindorm console. See Generate the configuration file from the console.
Manual: Edit the XML configuration files yourself. See Configure manually below.

Configure manually

Decompress the SDK package.
```
tar -zxvf hadoop-2.7.3.tar.gz
```

Set the HADOOP_HOME environment variable.

export HADOOP_HOME=/${Hadoop_installation_folder}/hadoop-2.7.3

Change to the Hadoop directory.
```
cd $HADOOP_HOME
```
Add the JAVA_HOME environment variable to etc/hadoop/hadoop-env.sh. This example assumes Java is installed in /opt/install/java.
```
export JAVA_HOME=/opt/install/java
```

Edit etc/hadoop/hdfs-site.xml and add the following properties. Replace ${Instance ID} with your actual instance ID. These properties configure the Hadoop client for high availability (HA) mode, so that it automatically fails over between the two LindormDFS NameNodes (nn1 and nn2).

Note

To configure multiple instances, copy the entire <property> block for each additional instance, replace ${Instance ID} with the ID of that instance, and paste all <property> blocks inside the <configuration> element.

<configuration>
  <property>
    <!-- Logical name for the LindormDFS nameservice -->
    <name>dfs.nameservices</name>
    <value>${Instance ID}</value>
  </property>
  <property>
    <!-- Failover proxy provider for the nameservice -->
    <name>dfs.client.failover.proxy.provider.${Instance ID}</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
  <property>
    <!-- Enable automatic HA failover between NameNodes -->
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>
  <property>
    <!-- Identifiers for the two NameNodes in this nameservice -->
    <name>dfs.ha.namenodes.${Instance ID}</name>
    <value>nn1,nn2</value>
  </property>
  <property>
    <!-- RPC address of the first NameNode -->
    <name>dfs.namenode.rpc-address.${Instance ID}.nn1</name>
    <value>${Instance ID}-master1-001.lindorm.rds.aliyuncs.com:8020</value>
  </property>
  <property>
    <!-- RPC address of the second NameNode -->
    <name>dfs.namenode.rpc-address.${Instance ID}.nn2</name>
    <value>${Instance ID}-master2-001.lindorm.rds.aliyuncs.com:8020</value>
  </property>
</configuration>

Common operations

Replace ${Instance ID} with your instance ID before running these commands.

Upload a file

# Create a directory in LindormDFS
$HADOOP_HOME/bin/hadoop fs -mkdir hdfs://${instanceID}/test

# Create a local file and upload it
echo "test" > test.log
$HADOOP_HOME/bin/hadoop fs -put test.log hdfs://${Instance ID}/test

List files

$HADOOP_HOME/bin/hadoop fs -ls hdfs://${instance_id}/test

Download a file

$HADOOP_HOME/bin/hadoop fs -get hdfs://${Instance ID}/test/test.log

Generate the configuration file from the console

Use the Lindorm console to generate the hdfs-site.xml and core-site.xml configuration content automatically, without manual XML editing.

Log on to the Lindorm console.
In the upper-left corner, select the region where the instance is deployed.
On the Instances page, click the instance ID, or click View Instance Details in the Actions column.
In the left navigation pane, click Database Connections.
On the Database Connections page, click the LindormDFS tab.
Click Activate Now.
After Underlying File Access is activated, click Generate Configuration Items to generate the hdfs-site and core-site configurations.