E-MapReduce:Resource isolation

Limits

The resource isolation feature is available for E-MapReduce (EMR) clusters of V5.7.0 or a later minor version.

Feature overview

The resource isolation feature isolates resources for queries performed in a cluster by different tenants. This reduces resource consumption in data queries and improves resource utilization in the cluster.

When you start a data query job, StarRocks selects a classifier for the job based on the job information. The classifier that is most suitable for the job is selected and the resource group to which the classifier belongs provides resources for the job.

Terms

resource group

classifier

• A classifier is used to match a data query job. The classifier that is most suitable for the job is selected and the resource group to which the classifier belongs provides resources for the job. A resource group can be associated with one or more classifiers.

• The following parameters must be configured when you select a classifier:

  • user: the username.

  • role: the role of the user.

  • query_type: the query type. Only SELECT queries are supported.

  • source_ip: the IP address from which the query is initiated. The value is a CIDR block.

• Classifiers and data query jobs are matched based on the following rules:

  • The job information meets the conditions of a classifier.

  • If the job information meets the conditions of multiple classifiers, StarRocks calculates and compares the matching scores. Only the classifier that has the highest matching score is selected. The matching score is calculated based on the following rules:

    • If the username of the job meets the condition, one point is added to the score.

    • If the role of the user meets the condition, one point is added to the score.

    • If the query type meets the condition, the following points are added to the score: 1 + 1/Value of the query_type parameter of the classifier.

    • If the IP address from which the query is initiated meets the condition, the following points are added to the score: 1 + (32 - cidr_prefix)/64.

  Examples:

  • If a classifier requires more conditions, the classifier has a higher score.

    -- In this example, Classifier B that requires more conditions has a higher score. 
    classifier A (user='Alice') classifier B (user='Alice', source_ip = '192.168.**.**/24')

  • If the numbers of conditions are the same, the classifier that has the most precise descriptions of conditions has the highest score.

    -- In this example, Classifier B requires a smaller range of IP addresses, and therefore has a higher score. 
    classifier A (user='Alice', source_ip = '192.168.**.**/16')
    classifier B (user='Alice', source_ip = '192.168.**.**/24')
    
    -- In this example, Classifier C supports fewer query types, and therefore has a higher score. 
    classifier C (user='Alice', query_type in ('select'))
    classifier D (user='Alice', query_type in ('insert','select', 'ctas'))

Usage

Enable the resource group feature

-- Enable a pipeline engine and the resource group feature for the current session: 
SET enable_pipeline_engine = true;
-- Globally enable a pipeline engine and the resource group feature: 
SET GLOBAL enable_pipeline_engine = true;

Create resource groups and classifiers

• Syntax

  CREATE RESOURCE GROUP group_name
  TO (
      user='string',
      role='string',
      query_type in ('select'),
      source_ip='cidr'
  ) -- Create classifiers. Separate multiple classifiers with commas (,). 
  WITH (
      "cpu_core_limit" = "INT",
      "mem_limit" = "m%",
      "concurrency_limit" = "INT",
      "type" = "normal" -- The type of the resource group. The value is fixed to normal. 
  );

• Example

  CREATE RESOURCE GROUP rg1
  TO
      (user='rg1_user1', role='rg1_role1', query_type in ('select'), source_ip='192.168.**.**/24'),
      (user='rg1_user2', query_type in ('select'), source_ip='192.168.**.**/24'),
      (user='rg1_user3', source_ip='192.168.**.**/24'),
      (user='rg1_user4'),
      (db='db1')
  WITH (
      'cpu_core_limit' = '10',
      'mem_limit' = '20%',
      'type' = 'normal',
      'big_query_cpu_second_limit' = '100',
      'big_query_scan_rows_limit' = '100000',
      'big_query_mem_limit' = '1073741824'
  );

(Optional) Specify a resource group

SET resource_group = 'group_name';

Query resource groups and classifiers

Manage quotas for resource groups and classifiers

• Modify the quota for a resource group.

  Syntax

  ALTER RESOURCE GROUP <yourResourceName> WITH (
      'cpu_core_limit' = '10',
      'mem_limit' = '20%'
  );

• Create or delete classifiers.

  Syntax

  • Create a classifier.

    ALTER RESOURCE GROUP <yourResourceName> ADD CLASSIFIER[,...];

  • Delete specific classifiers.

    ALTER RESOURCE GROUP <yourResourceName> DROP (CLASSIFER_ID_1, CLASSIFIER_ID_2, ...);

  • Delete all classifiers.

    ALTER RESOURCE GROUP <yourResourceName> DROP ALL;

Delete a resource group

DROP RESOURCE GROUP <yourResourceName>;