If you want to purchase an E-MapReduce (EMR) Presto cluster or an EMR Trino cluster and you do not know how to select specifications or configure the memory for the cluster, you can refer to the recommendations in this topic. This topic describes how to select appropriate specifications and memory configurations for an EMR Trino cluster based on your business requirements.
Recommendations on specifications selection of EMR Trino clusters
A Trino cluster consists of one coordinator and multiple workers. For more information, see Trino overview. In an EMR Trino cluster, the coordinator is deployed on the master node and the workers are deployed on core nodes or task nodes by default. Trino does not natively support high availability. The Trino coordinator is deployed on one master node regardless of whether the purchased Trino cluster is a high-availability cluster. A single coordinator node can schedule up to hundreds of worker nodes.
To create an EMR Trino cluster, you need to select only the Trino engine and the required data lake components. The following content provides recommendations on specifications selection when you create an EMR Trino cluster.
If you select the same specifications for the master node and core nodes, we recommend that you purchase 5 to 20 worker nodes. You can also configure more than 20 worker nodes.
If a small number of worker nodes are required, the specifications of the master node can be half of the specifications of core nodes. We recommend that the specifications of the master node, especially the number of vCPUs, be not much lower than the specifications of core nodes. Otherwise, service performance and stability are affected.
If a large-scale cluster is required, we recommend that you increase the number of worker nodes rather than increasing the specifications of a single node.
To prevent issues such as insufficient memory in an EMR Trino cluster, we recommend that you enable auto scaling for the cluster to perform scale-out when you create the cluster. If the load of the cluster decreases, auto scaling also allows you to perform scale-in. For more information, see Overview.
If you want to create an EMR cluster in which only the Trino component is installed, we recommend that you do not select unnecessary components such as HDFS and Hive when you create the cluster. You need to select only the Trino component and the data lake components that you want to use. This helps improve scalability and reduce resource usage.
When you use Trino to query data, both vCPU and memory resources are consumed. vCPUs directly determine the query speed, and the memory capacity determines whether a query task can be successfully executed. The memory capacity also has a small impact on the query speed. You can purchase EMR Trino clusters based on your SQL requirements and business requirements. If you do not have special requirements, general-purpose instances can meet your daily needs. The following table describes the typical specifications of an EMR Trino cluster.
Node type and quantity | Number of vCPUs for a cluster | Memory size for a cluster |
1 Master | 16 vCPUs | 64 GB |
5 Core | 16 vCPUs | 64 GB |
Recommendations on memory configurations of EMR Trino clusters
Insufficient memory is the main cause of query task failures in EMR Trino clusters. The following parameters are related to the memory of EMR Trino clusters. The units of the parameters are GB and MB.
query.max-memory-per-node
query.max-memory
query.max-total-memory-per-node (This parameter has been removed in Trino Release 369. The Trino component deployed in DataLake clusters does not contain this parameter.)
query.max-total-memory
memory.heap-headroom-per-node
To modify the preceding parameters, perform the following steps:
Modify Java virtual machine (JVM) parameters.
When you configure memory parameters, you must modify JVM parameters first.
On the Configure tab of the Trino service page in the EMR console, click the jvm.config subtab and change the value that follows -Xmx.
For a Trino cluster in standalone mode or a cluster in which only the Trino service is deployed, you can change the value to about 70% of the physical memory of the cluster. This way, both efficiency and stability can be ensured.
If the cluster has large configurations, such as cluster memory of more than 128 GB, you can increase the value. However, we recommend that you do not set this parameter to an excessively large value, especially when a large number of Native method requests are received. In this case, you must decrease the value based on the actual situation. If you do not decrease the value, the process may be stopped by the operating system due to insufficient memory. As a result, worker nodes are restarted and tasks fail.
If the configurations of the master node and core nodes are different, you can change the configurations of each node group or node.
For the clusters with the specifications listed in Recommendations on specifications selection of EMR Trino clusters, you can set the value that follows -Xmx to a value from 45G to 50G.
Configure memory parameters.
On the Configure tab of the Trino service page in the EMR console, click the config.properties subtab and change the related parameter values.
Parameter
Description
Default value
Recommended value
query.max-memory-per-node
The size of memory that is required to perform a query on a single worker node.
2 GB
Set the parameter to a value that is less than or equal to the difference between the JVM memory and the value of memory.heap-headroom-per-node. This ensures that the memory of a single worker node does not exceed 70% of the JVM memory. If the concurrency is high, we recommend that you properly decrease the value.
query.max-memory
The total size of memory that is required to perform a query on all nodes in a cluster.
4 GB
Set this parameter to a value that is calculated based on the following formula: Value of query.max-memory-per-node × Number of worker nodes.
query.max-total-memory
The total size of memory that is consumed to perform a query on a cluster. The value of this parameter must be greater than or equal to the value of the
query.max-memoryparameter.6 GB
If the concurrency is not high, set this parameter to the same value as the query.max-memory parameter. If the concurrency is high, the value of the query.max-total-memory parameter cannot exceed the maximum memory capacity of the cluster, which is equal to 70% of JVM memory multiplied by the number of worker nodes. Proportionally decrease the values of query.max-memory and query.max-memory-per-node based on your business requirements. You can decrease the value of query.max-memory to the half of query.max-total-memory. In this case, you must set the query.max-memory-per-node parameter to the value of query.max-memory divided by the number of worker nodes.
memory.heap-headroom-per-node
The reserved JVM heap memory.
30% of the JVM memory
Retain the default value if you have no special requirements.
You can configure query.max-memory-per-node and memory.heap-headroom-per-node for each node. We recommend that you globally configure query.max-memory and query.max-total-memory for all nodes in a cluster.
For clusters with the specifications listed in Recommendations on specifications selection of EMR Trino clusters, you can set query.max-memory-per-node to a value that ranges from 30 GB to 35 GB, and set query.max-memory and query.max-total-memory to values that range from 150 GB to 165 GB. If services on worker nodes are frequently stopped during queries, you can try to gradually decrease the value of query.max-memory-per-node.
After the configurations are saved and deployed, restart all nodes to make the memory configurations take effect.
Other related configurations
If you want to limit the resources that are used by a single query, you can use the resource group feature. For more information, see Resource groups.
If the query speed is slow, you can change the value of the task.concurrency parameter to adjust the concurrency. For more parameters, see Task properties.