You can use the JindoTable access frequency collection feature to identify hot and cold data. This helps reduce overall storage costs and improve the cache hit rate.
Data collection
JindoTable collects access records for Hive tables. The supported engines are Spark and Hive. The collected data is stored in the namespace of the SmartData service on the cluster.
Access record collection is enabled by default. To disable it, see Disabling access record collection.
Query access data
- Syntax
jindo table -accessStat <-d [days]> <-n [topNums]>daysandtopNumsare positive integers. Ifdaysis set to 1, the command queries all access records from 00:00 on the current day (local time) to the current time. - Function
Returns the top N most frequently accessed tables or partitions within a specified time range.
- Example: Query for the 20 most frequently accessed tables or partitions over the last seven days.
jindo table -accessStat -d 7 -n 20
For more information about JindoTable, see Use JindoTable.
Disabling access record collection
- In the top navigation bar, select the region and resource group.
- Click the Clusters tab.
- On the Clusters page, click Details in the Actions column for your cluster.
- Modify the parameter values.
Remove specific content from the following parameter values:
- For the Hive service:
- In the left-side navigation pane, click .
- Click the Configure tab.
- Click the hive-site tab.
- Search for the hive.exec.post.hooks parameter and delete com.aliyun.emr.table.hive.HivePostHook from its value.
- For the Spark service:
- In the left-side navigation pane, click .
- Click the Configure tab.
- Click the spark-defaults tab.
- Search for the spark.sql.queryExecutionListeners parameter and delete com.aliyun.emr.table.spark.SparkSQLQueryListener from its value.
- For the Hive service:
- Save the configuration.
- In the upper-right corner, click Save.
- In the Confirm dialog box, enter a reason for the change and select Auto-update Configuration.
- Click OK.
- Restart the services.
- For the Hive service:
- In the upper-right corner, choose .
- In the Cluster Activities dialog box, configure the parameters.
- Click OK.
- ConfirmIn the confirmation dialog box, click OK.
- For the Spark service:
- In the upper-right corner, choose .
- In the Cluster Activities dialog box, configure the parameters.
- Click OK.
- ConfirmIn the confirmation dialog box, click OK.
- For the Hive service: