Hadoop Distributed File System (HDFS) allows the administrator to configure quotas for the number of file and directory names and the amount of space used for individual directories. Name quotas and space quotas operate independently, but the management and implementation methods of the two types of quotas are similar. This topic describes the two types of quotas. This topic also describes how to configure the quotas.

Name quota

The name quota is a hard limit on the number of file and directory names in the tree rooted at a directory. If the name quota is exceeded when you create files or directories, the creation fails. The name quota is an attribute of a directory. HDFS automatically checks the name quota of a directory when you create or rename files or directories in the directory. If you have configured a name quota for a directory, the name quota remains unchanged after you rename the directory.

A new name quota can be configured for a directory even if the number of names in the directory has exceeded the new name quota. A newly created directory has no space quota. Long.Max_Value specifies the name quota. For example, a name quota of 1 causes a directory to remain empty. This is because the name quota counts towards the directory.

Space quota

The space quota is a hard limit on the number of bytes used by files in the tree rooted at a directory. If the space quota does not allow a full block to be written, the block allocation fails. Each replica of a block counts against the space quota. For example, if a file is N bytes in size and has three replicas, the space quota that is used by the file is 3N. The name quota is an attribute of a directory. HDFS automatically checks the name quota of a directory when you create or rename files or directories in the directory. If you have configured a space quota for a directory, the space quota remains unchanged after you rename the directory. If a rename operation conflicts with the specified configuration, the rename operation fails.

A new name quota can be configured for a directory even if the number of names in the directory has exceeded the new name quota. A newly created directory has no space quota. Long.Max_Value specifies the name quota. If the space quota for a directory is 0, you can still create files in the directory, but you cannot add blocks to the files. A directory does not occupy disk space. Therefore, directories do not count against the space quota.

Sample commands

  • Administrative commands
    Quotas are managed by a set of commands available only to the administrator.
    • Set the name quota to N for some directories. 
      hdfs dfsadmin -setQuota <N> <directory>...<directory>

      The configuration fails if N is a non-positive number, one of the specified directories does not exist, or one of the specified directories is about to exceed the new quota.

    • Remove the name quota for some directories. 
      hdfs dfsadmin -clrQuota <directory>...<directory>
    • Set the space quota to N for some directories. 
      hdfs dfsadmin -setSpaceQuota <N> <directory>...<directory>
    • Remove the space quota for some directories. 
      hdfs dfsadmin -clrSpaceQuota <directory>...<directory>
  • Reporting command
    Report the quota value, the total number of names that are used, and the total number of bytes that are used. 
    hadoop fs -count -q [-h] [-v] [-t [comma-separated list of storagetypes]] <directory>...<directory>