All Products
Search
Document Center

Use the HDFS shell to connect to and use LindormDFS

Last Updated: Jul 09, 2021

Use the HDFS shell

You can use the HDFS shell to manage the files stored in LindormDFS. For example, you can query files, delete files, manage file permissions, and modify file names. This topic describes some of the basic HDFS shell commands. You can run the help command to view all the commands. For more information, see Introduction to the Apache HDFS shell. Before you use the HDFS shell, configure the environment. For more information, see Use open source HDFS clients to access LindormDFS.

Frequently used commands

  1. Frequently used commands to upload data from your local file system to LindormDFS

    1. put: copies files from one or more source paths on the local file system to LindormDFS. You can also run this command to read the input from a standard input (stdin) and write the input to LindormDFS. 
    2. copyFromLocal: This command is similar to the put command except that the source path must be a local path. 
    3. moveFromLocal: moves files from the local file system to LindormDFS. 
    4. appendToFile: appends one or more source files from the local file system to LindormDFS.
  2. Common commands to perform operations on data stored in LindormDFS

    1. cp: copies a file from the source path to the destination path. You can run this command to copy files from multiple source paths to the destination path. The destination path must point to a directory. 
    2. mv: moves files from the source path to the destination path. You can run this command to copy files from multiple source paths to the destination path. The destination path must point to a directory. Files cannot be moved between different file systems. 
    3. chown: changes the owner of one or more files. The -R option recursively makes the change through the directory structure. The user must have superuser permissions to run this command. 
    4. chgrp: changes the group to which a file belongs. The -R option recursively makes the change through the directory structure. The user who runs the command must be the owner of the file or a superuser. 
    5. chmod: changes the permissions on a file. The -R option recursively makes the change through the directory structure. The user who runs the command must be the owner of the file or a superuser. 
    6. mkdir: accepts a path URI as an argument and creates a directory. The command is similar to the mkdir -p command in UNIX. mkdir -p creates parent directories along the path. 
    7. du: displays the size of each file in the directory or displays the size of a specified file. 
    8. df: displays the free storage. 
    9. cat: copies the content of a specified file in a path to a standard output (stdout). 
    10. rm: deletes a specified file. You can delete only non-empty directories and files 
    11. ls: returns details of the file in the following format:
              File name <Number of replicas> file size modification date modification time permission user ID group ID
              This command returns a list of the direct children for a directory. This operation is the same as that in UNIX. The returned list is in the following format:
              Directory name <dir>modification date modification time permission user ID group ID
  3. Frequently used commands to download data from LindormDFS to your local file system

    1. get: copies files to the local file system from LindormDFS. You can use the -ignorecrc option to copy files that failed the cyclic redundancy check (CRC). You can use the-crc option to copy files and CRC information. 
    2. getmerge: accepts a source directory and a destination file as inputs and concatenates files from the source destination file into the destination local file. 
    3. copyToLocal: This command is similar to the put command except that the destination path must be a local path.

Examples

  1. Query the files in a directory. ${Instance ID} specifies the instance ID of Lindorm. Replace the Instance ID value in ${Instance ID} with the actual instance ID.

    ${HADOOP_HOME}/bin/hadoop fs -ls hdfs://${Instance ID}/
    Command output
  2. Upload a file. ${Instance ID} specifies the instance ID of Lindorm. Replace the Instance ID value in ${Instance ID} with the actual instance ID.

    ${HADOOP_HOME}/bin/hadoop fs -put test.txt hdfs://${Instance ID}/
    Screenshot of a file upload
  3. Download a file to a local directory. ${Instance ID} specifies the instance ID of Lindorm. Replace the Instance ID value in ${Instance ID} with the actual instance ID.

    ${HADOOP_HOME}/bin/hadoop fs -get hdfs://${Instance ID}/xxxxx

    xxxxx specifies the name of the file that you want to download.

    Screenshot of a file download
  4. Delete a file. ${Instance ID} specifies the instance ID of Lindorm. Replace the Instance ID value in ${Instance ID} with the actual instance ID.

    ${HADOOP_HOME}/bin/hadoop fs -rm hdfs://${Instance ID}/xxxxx

    Screenshot of a file deletionxxxxx specifies the name of the file that you want to delete. If you delete the file, the file is moved to the Trash directory. If you need to permanently remove the file, run the hadoop fs -expunge command to empty the Trash directory.