Hadoop Distributed File System (HDFS) allows the administrator to configure quotas for the number of file and directory names and the amount of space used for individual directories. Name quotas and space quotas operate independently, but the management and implementation methods of the two types of quotas are similar. This topic describes the two types of quotas and describes how to configure the quotas.

Name quota

The name quota is a hard limit on the number of file and directory names in the tree rooted at a directory. If the quota is exceeded when you create files or directories, the creation fails. Name quota is an attribute of a directory. HDFS automatically checks the quota of a directory when you create or rename files or directories in the current directory. If you have configured a quota for a directory, the quota is retained after you rename the directory.

A new quota can be configured for a directory even if the number of names in the directory has exceeded the new quota. A newly created directory has no name quota. Long.Max_Value specifies the quota. For example, a quota of 1 causes a directory to remain empty. This is because the quota counts towards the directory.

Space quota

The space quota is a hard limit on the number of bytes used by files in the tree rooted at a directory. If the quota does not allow a full block to be written, the block allocation fails. Each replica of a block counts against the quota. For example, if a file is N in size and has three replicas, the quota that is used by the file is 3N. Space quota is an attribute of a directory. HDFS automatically checks the quota of a directory when you create or rename files or directories in the current directory. The quota remains unchanged for renamed directories. If a rename operation conflicts with the specified configuration, the rename operation fails.

A new quota can be configured for a directory even if the number of bytes used in the directory has exceeded the new quota. A newly created directory has no space quota. Long.Max_Value specifies the quota. If the quota for a directory is 0, you can still create files in the directory, but you cannot add blocks to the files. A directory does not occupy disk space. Therefore, directories do not count against the space quota.

Sample commands

  • Administrative commands
    Quotas are managed by a set of commands available only to the administrator.
    • Set the name quota to N for some directories.
      hdfs dfsadmin -setQuota <N> <directory>...<directory>

      The configuration fails if N is a non-positive number, one of the specified directories does not exist, or one of the specified directories is about to exceed the new quota.

    • Remove the name quota for some directories.
      hdfs dfsadmin -clrQuota <directory>...<directory>
    • Set the space quota to N bytes for some directories.
      hdfs dfsadmin -setSpaceQuota <N> <directory>...<directory>
    • Remove the space quota for some directories.
      hdfs dfsadmin -clrSpaceQuota <directory>...<directory>
  • Reporting command
    Report the quota value, the total number of names that are used, and the total number of bytes that are used.
    hadoop fs -count -q [-h] [-v] [-t [comma-separated list of storagetypes]] <directory>...<directory>