Use the HDFS shell to manage files in LindormDFS. LindormDFS is fully compatible with HDFS, so you can use standard Hadoop FileSystem shell commands to list, upload, download, delete, and manage permissions on files.
Prerequisites
Before you begin, ensure that you have:
Downloaded the client. See Download the client
Configured Apache Hadoop. See Configure Apache Hadoop
Usage notes
LindormDFS uses a high availability (HA) architecture with multiple NameNodes. Configure automatic failover to prevent access failures if the primary NameNode goes down. See Configure Hadoop.
Common commands
All commands follow this pattern:
${HADOOP_HOME}/bin/hadoop fs -<command> [options] <args>Replace ${Instance ID} with your Lindorm instance ID in all examples below.
Upload data
put
Copies one or more files from the local file system to LindormDFS. When - is specified as the source, reads from standard input (stdin) and writes to LindormDFS.
${HADOOP_HOME}/bin/hadoop fs -put test.txt hdfs://${Instance ID}/copyFromLocal
Identical to put, but the source path must be a local path.
moveFromLocal
Moves files from the local file system to LindormDFS. Unlike put, the source files are deleted after the copy completes.
appendToFile
Appends one or more source files from the local file system to a destination file in LindormDFS.
${HADOOP_HOME}/bin/hadoop fs -appendToFile localfile hdfs://${Instance ID}/remotefileProcess data
cp
Copies files from the source path to the destination path. When copying from multiple source paths, the destination must be a directory.
${HADOOP_HOME}/bin/hadoop fs -cp hdfs://${Instance ID}/src hdfs://${Instance ID}/dstmv
Moves files from the source path to the destination path. When moving from multiple source paths, the destination must be a directory. Files cannot be moved between different file systems.
mkdir
Creates the specified directories, including any missing parent directories—similar to mkdir -p in Unix.
${HADOOP_HOME}/bin/hadoop fs -mkdir -p hdfs://${Instance ID}/path/to/newdirls
Lists files and directories.
For a file: returns
File name <Number of replicas> File size Modification date Modification time Permissions User ID Group IDFor a directory: returns each direct child with
Folder name <dir> Modification date Modification time Permissions User ID Group ID
${HADOOP_HOME}/bin/hadoop fs -ls hdfs://${Instance ID}/du
Displays the size of each file in the directory or the size of a specified file.
df
Views file usage.
cat
Copies the content of a file to standard output (stdout).
${HADOOP_HOME}/bin/hadoop fs -cat hdfs://${Instance ID}/myfile.txtchown
Changes the owner of a file. Requires superuser permissions.
| Option | Description |
|---|---|
-R | Recursively changes ownership through the directory structure |
chgrp
Changes the group of a file. Requires superuser permissions or ownership of the file.
| Option | Description |
|---|---|
-R | Recursively changes the group through the directory structure |
chmod
Changes the permissions on a file. Requires superuser permissions or ownership of the file.
| Option | Description |
|---|---|
-R | Recursively changes permissions through the directory structure |
rm
Deletes a specified file. You can delete only non-empty directories and files.
rm moves the file to the Trash folder rather than permanently deleting it. To permanently delete files, run expunge to empty the Trash.
${HADOOP_HOME}/bin/hadoop fs -rm hdfs://${Instance ID}/${filename}expunge
Permanently deletes all files in the Trash folder.
${HADOOP_HOME}/bin/hadoop fs -expungeDownload data
get
Copies files from LindormDFS to the local file system.
| Option | Description |
|---|---|
-ignorecrc | Skips cyclic redundancy check (CRC) validation on downloaded files |
-crc | Saves CRC checksums alongside the downloaded files |
${HADOOP_HOME}/bin/hadoop fs -get hdfs://${Instance ID}/${filename} .getmerge
Concatenates all files in a source directory into a single local destination file.
${HADOOP_HOME}/bin/hadoop fs -getmerge hdfs://${Instance ID}/srcdir/ localfile.txtcopyToLocal
Identical to get, but the destination path must be a local path.
To see all commands supported by the HDFS shell, run:
${HADOOP_HOME}/bin/hadoop fs -helpFor the full command reference, see Apache HDFS FileSystem shell.