Alluxio Cache Policies and Data Lifecycle Management - E-MapReduce - Alibaba Cloud - E-MapReduce

Alluxio provides distributed data caching by using the memory and disks of local nodes in an E-MapReduce (EMR) cluster. You can configure cache tiers, set eviction policies, and manage data lifecycle.

Prerequisites

An E-MapReduce cluster has been created with the Alluxio service enabled. For more information, see Create a cluster.
You are logged on to the cluster. For more information, see Log on to a cluster.

Background

By default, E-MapReduce uses a two-tier cache with both memory and disk. The default allocation is 10% of a node's memory and 30% of its disk space. To modify this allocation, on the Alluxio service configuration page, search for configuration items that start with alluxio.worker.tieredstore. Key configuration items include: level0.alias (default: MEM), level0.dirs.quota (default: ${alluxio.worker.ramdisk.size}), level1.alias (default: SSD), level1.dirs.quota (default: 10GB), level0.watermark.high.ratio (default: 0.95), and level1.watermark.low.ratio (default: 0.7).

For more information about caching, see Caching.

Cache policies

When a client writes a new block, Alluxio writes it to the level 0 storage tier by default. If level 0 does not have enough space, Alluxio tries the next storage tier. If no space is found on any tier, Alluxio evicts existing data to store the new block. The default eviction policy is LRUAnnotator, which evicts blocks in least-recently-used order.

When a client reads a block, if the data is already in Alluxio, the client reads it directly from the corresponding worker node. If the data is not in Alluxio, the system caches it first and then serves it from the worker node for subsequent reads.

Manage data lifecycle

The following table lists the common commands for data lifecycle management.

Command	Description
free	Frees data from the cache.
load	Loads data into the Alluxio cache.
persist	Persists a file or directory in Alluxio to the Under File Storage (UFS).
setTtl	Sets a time-to-live (TTL) for a file or directory in milliseconds.

free

Frees data from the cache.

Freeing data removes it from the Alluxio cache but not from the Under File Storage (UFS). The data remains available, but subsequent access may be slower because clients must read it from the UFS again.

Syntax
```
alluxio fs free <path>
```
Example: Free all data in the tmp directory from the cache.
```
alluxio fs free /tmp
```
Expected output:
```
/tmp was successfully freed from Alluxio space.
```

load

Loads data into the Alluxio cache.

Syntax
```
alluxio fs load <path>
```
Example: Load all data in the /tmp3/logs directory into the Alluxio cache.
```
alluxio fs load /tmp3/logs
```
Expected output:
```
/tmp3/logs loaded
```

persist

Persists a file or directory in Alluxio to the Under File Storage (UFS).

Persistence writes data from Alluxio storage back to the UFS, regardless of whether it has been modified. This ensures data is recoverable if an Alluxio failure occurs.

Syntax
```
alluxio fs persist <path>
```
Example: Persist the tmp directory in Alluxio to the UFS.
```
alluxio fs persist /tmp
```
Expected output:
```
persisted file /tmp with size 46
```

setTtl

Sets a time-to-live (TTL) for a file or directory in milliseconds.

When the current time exceeds the file's creation time plus the TTL duration, the action parameter determines the operation to perform. The delete action (default) removes the file from both Alluxio and the UFS. The free action removes the file only from Alluxio.

Syntax

alluxio fs setTtl [--action delete|free] <path> <time to live>

Examples:

Deletes the tmp directory after one minute.

alluxio fs setTtl /tmp 60000

Expected output:

TTL of path '/tmp' was successfully set to 60000 milliseconds, with expiry action set to DELETE

Frees the dir directory from the cache after one day.

alluxio fs setTtl --action free /dir 86400000

Expected output:

TTL of path '/dir' was successfully set to 86400000 milliseconds, with expiry action set to FREE