Fix for SmartData data read errors

Impact

• Affected components: All components with the SmartData cache feature enabled.
  Important If SmartData is deployed on your cluster but you do not use the cache, you can ignore this notice.

  SmartData supports two cache modes: JindoFS Cache mode and JindoFS Block mode.

• Affected versions:
  • E-MapReduce versions: 3.30.x, 4.5.x, 3.32.x, 4.6.x, 3.33.x, 4.7.x, 3.34.x, 4.8.x, 3.35.x, and 4.9.x.
  • SmartData versions: 3.0.x, 3.1.x, 3.2.x, 3.3.x, 3.4.x, and 3.5.x.
• Severity: Critical. A fix is strongly recommended. This issue occurs intermittently and can cause data integrity problems.
• jfs.cache.data-cache.enabletrueSymptoms: If your cluster uses JindoFS Cache mode (with the parameter set to ) or JindoFS Block mode (which enables cache by default), there is a small probability that cached data can become corrupted. This can cause jobs to fail when reading data. For example, jobs might report errors such as a failure to parse ORC or Parquet files, or HBase might report HFile format errors.

Solution

The cache corruption issue is caused by a defect in the small file merge (compaction) process of the Storage Service. You can prevent this issue by disabling this optimization in the configuration and restarting the SmartData service. If the issue has already occurred, immediately disable the cache to mitigate the impact and restore your online services. If you use only JindoFS Cache mode and not JindoFS Block mode, you can clear all cache data from the cluster using a tool. This action formats the cache system and removes all potentially corrupted cache blocks. After the cleanup is complete, you can re-enable the cache.

Procedure

Proactive fix

If your cluster is not yet affected by this issue, you can prevent it by disabling the small file merge optimization.

1. In the EMR console, go to the SmartData service page and add a custom configuration.
   1. On the storage configuration page for the SmartData service, click Custom Configuration.
   2. In the Add Configuration Itemstorage.compaction.enablefalse dialog box, add a configuration item with Key set to and Value set to .
   3. Click OK.
2. Restart Jindo Storage Service.
   1. On the SmartData service page, choose Actions > Restart Jindo Storage Service.
   2. In the Cluster Activities dialog box, enter a reason for the action and click OK.
   3. In the Confirm dialog box, click OK.

Emergency fix

If this issue has already occurred, follow these steps to restore your services and repair the cache:

1. This defect is caused by corrupted cache data. You can eliminate its impact by disabling the cache. On the client configuration page of the SmartData service, use the following settings to disable the cache.
   • jfs.data-cache.enablefalseFor JindoFS Block mode: Add a custom configuration item with Key set to and Value set to .
   • jfs.cache.data-cache.enablefalseFor JindoFS Cache mode: Change the value of the parameter to .
2. Rerun the affected jobs.

   After you rerun the jobs, they should operate normally. If the problem persists, the issue may have a different root cause that requires separate investigation.

   For long-running services such as Presto, Impala, and HBase, you must restart their respective service components for the configuration to take effect. For On-YARN compute components such as Hive and Spark, you only need to rerun the jobs for the configuration to take effect.

3. Repair the cache.
   • For JindoFS Block mode: submit a ticket to request a component upgrade.
   • For JindoFS Cache mode:
     Note Because the cache is already disabled, the following operations will not affect your services.
     1. In the EMR console, stop the SmartData service.
     2. Upload the format_cache.sh script to the master node of the cluster and run the following command as the hadoop user.

        sh format_cache.sh

     3. On the storagestorage.compaction.enablefalse configuration page of the SmartData service in the EMR console, add a configuration item with Key set to and Value set to .
     4. In the EMR console, start the SmartData service.
     5. jfs.cache.data-cache.enabletrueRe-enable the cache by changing the value of the parameter to .