How do I mount an HDFS file system to a Data Transport device? -

If your data is stored in Hadoop Distributed File System (HDFS), you must make sure that the HDFS can be accessed, test the network connectivity between a Data Transport device and the HDFS, and then migrate data by using the Data Transport device. This topic describes how to configure a service IP address, mount an HDFS file system to a Data Transport device, and test the connectivity between the HDFS and the Data Transport device.

Prerequisites

The device where the HDFS file system is deployed is connected to the Ethernet port or optical port of a Data Transport device directly over an Ethernet cable or over a network switch.
The Ethernet cable, optical fiber cable, and optical modules are connected properly. The port connection indicator functions as expected.
The HDFS file system from which the data is migrated supports no authentication or Kerberos authentication.

Step 1: Configure a service IP address for the Data Transport device

Log on to the Data Transport device. For more information, see Install a Data Transport device.

Step 2: Perform operations on the HDFS cluster

The HDFS file system supports no authentication or Kerberos authentication. In the case of no authentication, you must use a username and password to log on to the HDFS cluster. Perform operations based on your business requirements.

No authentication

Check whether the HDFS cluster can be accessed.
1. Log on to the HDFS cluster and confirm the IP address of the primary node for the HDFS cluster. Example: 192.168.24.247.
2. Confirm the HDFS service port. Example: 8020.
  In the HDFS configuration file core-site.xml, the value of fs.defaultFS is the port number of the HDFS service.
3. Run the following command to test whether a device in the cluster can access the HDFS: hadoop fs -ls <IP address of the primary node for the HDFS cluster>:<HDFS service port>.
  Example: hadoop fs -ls hdfs://192.168.24.247:8020.
  Note
  - If the HDFS directory information is returned, the HDFS cluster can be accessed.
  - If an access denied message is returned, the IP address or the port number of the HDFS cluster may be invalid. Check the IP address and the port number and try again.
Test the connectivity between the Data Transport device and the HDFS cluster.
1. Log on to the Data Transport device and run the ping <IP address of the primary node for the HDFS cluster> command to test the connectivity between the Data Transport device and the HDFS cluster. Example: ping 192.168.24.247.
2. Run the telnet <IP address of the primary node for the HDFS cluster> <HDFS service port> command to test the connectivity between the Data Transport device and the HDFS cluster. Example: telnet 192.168.24.247 8020.
3. If the expected output information is returned, the Data Transport device can access the HDFS cluster.

Kerberos authentication

Check whether the Kerberos authentication succeeds.
1. Run the kinit PrincipcalName -k -t <keytab path> command.
2. Run the hdfs dfs -ls command.
  If the directories can be listed correctly, the Kerberos authentication succeeds.
Check whether the HDFS cluster can be accessed.
1. Log on to the HDFS cluster and confirm the IP address of the primary node for the HDFS cluster. Example: 192.168.24.247.
2. Confirm the HDFS service port. Example: 8020.
  In the HDFS configuration file core-site.xml, the value of fs.defaultFS is the port number of the HDFS service.
3. Select a device in the cluster to test whether the HDFS cluster can be accessed. Run the following command: hadoop fs -ls <IP address of the primary node for the HDFS cluster>:<HDFS service port>.
  Example: hadoop fs -ls hdfs://192.168.24.247:8020.
  Note
  - If the HDFS directory information is returned, the HDFS cluster can be accessed.
  - If an access denied message is returned, the IP address or the port number of the HDFS cluster may be invalid. Check the IP address and the port number and try again.
Test the connectivity between the Data Transport device and the HDFS cluster.
1. Log on to the Data Transport device and run the ping <IP address of the primary node for the HDFS cluster> command to test the connectivity between the Data Transport device and the HDFS cluster. Example: ping 192.168.24.247.
2. Run the telnet <IP address of the primary node for the HDFS cluster> <HDFS service port> command to test the connectivity between the Data Transport device and the HDFS cluster. Example: telnet 192.168.24.247 8020.
3. If the expected output information is returned, the Data Transport device can access the HDFS cluster.
Copy the specified files to the Data Transport device.
1. Find the core-site.xml, hdfs-site.xml, krb5.conf, and keytab files in the HDFS cluster and download these files. You can find the files in the following ways:
  - Run the find / -name core-site.xml command to find the location of the core-site.xml file.
  - Run the find / -name hdfs-site.xml command to find the location of the hdfs-site.xml file.
  - Run the find / -name krb5.conf command to find the location of the krb5.conf file.
  - Run the find / -name keytab command to find the location of the keytab file.
2. Download and copy the files to the /mnt/cube/software/hdfs/ directory of the Data Transport device.
3. In the krb5.conf file, check whether the domain name or the IP address of the Kerberos server is specified by the kdc parameter. If the kdc parameter specifies the domain name of the Kerberos server, you must configure the hosts file on the Data Transport device. You can configure the hosts file in the following way:
  Run the vim /etc/hosts command to add the IP address and the domain name of the Kerberos server in the <IP address of the Kerberos server> <Domain name of the Kerberos server> format.