Automatic Peak-Based Scaling for Compute Groups - Hologres - Alibaba Cloud - Hologres

Use cases

Consider a scenario where a data platform team uses the init_warehouse for large ETL jobs, high-volume bulk imports, and near real-time data ingestion, while Business Team A and Business Team B use warehouse 1 and warehouse 2 for queries. The following are typical use cases for scheduled scaling:

For near real-time and real-time data ingestion, continue to use the init_warehouse virtual warehouse. If the data volume is stable throughout the day, no elasticity plan is needed. If there is a predictable, recurring write peak within a specific time window (typically less than 16 hours), create an elasticity plan to provision additional elastic resources during that period.
For business team queries, continue to use separate virtual warehouses to ensure workload isolation. If the query volume is stable, no elasticity plan is necessary. If query requests follow a predictable daily cycle, create an elasticity plan to add extra elastic resources during peak query hours.
For large ETL jobs and high-volume bulk imports, we recommend using Serverless Computing resources. Serverless Computing provides additional on-demand compute resources without requiring advance provisioning. This improves instance stability, reduces out-of-memory (OOM) errors, and you pay only for what you use. For more information, see Serverless Computing. To learn how to use it, see Use Serverless Computing.

Key concepts

For definitions of instance-level and virtual warehouse-level compute resources, see Terminology.

The following is an example of an instance's resource configuration:

Category	Resource details
Instance	96 reserved CUs, with 64 CUs allocated and 32 CUs unallocated.
	32 elastic CUs.
	Total compute resources: 96 + 32 = 128 CUs.
Virtual warehouse init_warehouse	32 reserved CUs, 16 elastic CUs, for a total of 48 CUs.

Billing

Instance reserved resources: Compute resources dedicated to your virtual warehouse instance and are billed according to your instance's billing method (subscription or pay-as-you-go).
Instance elastic resources: Additional compute resources provisioned by the scheduled scaling feature. The cost is calculated as follows: Cost = Actual elastic resource usage (CU-hours) × Unit price. For specific unit prices, see Billing overview. Bills are settled hourly, and the fees are automatically deducted from your account.
Note
- The system records elastic resource usage every minute. At the end of each hour, it calculates the total usage for that hour, converts the units, generates a bill, and automatically deducts the fee from your account.
- Instance elastic resources are provisioned independently of unallocated instance resources. Even if your instance has unallocated reserved resources, scheduled scaling will still provision new elastic resources instead of using the unallocated ones.

Limitations

Scheduled scaling is available only for virtual warehouse instances. It is not supported for general-purpose or read-only replica instances.
Scheduled scaling is supported in Hologres v2.2.21 and later.

Regional availability:

The scheduled scaling feature is currently in public preview. To request access, use your Alibaba Cloud main account to fill out the Hologres Scheduled Scaling Public Preview Application.

Region	Availability	Description
China (Hangzhou), China (Shanghai), China (Beijing), and China (Shenzhen)	Supported	You can use this feature directly after your application is approved.
China (Shanghai) Finance Cloud, China (Beijing) Gov Cloud, China (Shenzhen) Finance Cloud, Japan (Tokyo), Malaysia (Kuala Lumpur), and Indonesia (Jakarta)	Not supported	Trial applications are not available.
China (Chengdu), China (Hong Kong), Singapore, Germany (Frankfurt), US (Silicon Valley), US (Virginia), and UAE (Dubai)	Available by ticket	To request access, follow these steps: Fill out the application form: Hologres Scheduled Scaling Public Preview Application. Submit a ticket for further processing.

Usage notes

Permissions required for scheduled scaling:
- To create an elasticity plan, you must use an Alibaba Cloud main account or a RAM user with the AliyunHologresWarehouseFullAccess permission. This permission grants read-only access to the Hologres console and configuration rights for the scheduled scaling feature. For authorization instructions, see Grant permissions to a RAM user.
- The account must have Superuser permissions in the instance to create an elasticity plan and to ensure that each scheduled scaling operation triggers successfully. For authorization instructions, see Grant development permissions to a RAM user for an instance.
Warning
If the Superuser permission is revoked from the account that created an elasticity plan, all subsequent scheduled scaling operations will fail. To prevent failures from permission changes, we recommend using a dedicated operations account to manage elasticity plans.
Scaling operations can affect running queries and writes:
- In Hologres v2.2, scheduled scale-out and scale-in operations cause a brief interruption of about 15 seconds for queries and writes on the virtual warehouse.
- Starting with Hologres v3.0, scale-out operations do not interrupt queries or writes, although the virtual warehouse status shows as "Processing". Scale-in operations still cause an interruption of about 15 seconds.
- Starting with Hologres v3.1, scale-in operations do not affect most queries and writes. However, if a long-running query or write that needs more than 60 minutes to complete is in progress when a scale-in is triggered, the task will fail 60 minutes after the scaling operation begins.
Once an elasticity plan is configured for a virtual warehouse:
- You cannot stop or delete the virtual warehouse, nor can you manually scale in its resources.
- You can only scale out resources for the virtual warehouse through the Hologres console. Manually executing the hg_alter_warehouse command to scale out is not supported.
- You can only create new virtual warehouses through the Hologres console. Manually executing the hg_create_warehouse command is not supported.
Elastic resources for scheduled scaling are pay-as-you-go resources and their availability is not guaranteed. We recommend that you configure Cloud Monitor alerts for scaling failure events, as described in Monitoring and alerts for scheduled scaling.

Procedure

Virtual warehouse resource management

Navigate to the Virtual Warehouse Management page.
1. Log on to the Hologres console and select a region in the top navigation bar.
2. In the left-side navigation pane, click Instances, then click the target Instance ID.
3. In the left-side navigation pane of the instance details page, click Virtual Warehouse Management.
View the resource usage of the virtual warehouse instance.

On the Virtual Warehouse Resource Management tab, you can view the resource usage of your virtual warehouse instance, including instance reserved resources (both allocated and unallocated) and instance elastic resources.
Manage virtual warehouse resources.
- Click Create Virtual Warehouse. In the Create Virtual Warehouse dialog box, enter a Virtual Warehouse Name and specify the Virtual Warehouse Resource to create a new virtual warehouse.
- For a virtual warehouse with a Status of Running, you can perform Modify Configuration, Restart, Stop, and Rebalance operations. For more information about rebalancing, see Shard rebalancing (Rebalance).
  
  Note
  You cannot Stop the default init_warehouse virtual warehouse of an instance.

Elasticity plans

Timeline

Navigate to the Virtual Warehouse Management page and click the Virtual Warehouse Scaling Plan tab.
From the Instance/Virtual Warehouse drop-down list, select an instance or a specific virtual warehouse to view its elasticity plan timeline. The timeline shows future scheduled scaling operations, not historical resource metrics.

Configuration

On the Virtual Warehouse Scaling Plan tab, click the icon to the left of the target virtual warehouse, then click + Add Time Period. Configure the start and end times for the daily active period, set the amount of elastic compute resources, and click Save in the Actions column to apply the changes.
You can Edit or Delete an existing elasticity plan.

Important

The amount of elastic compute resources cannot exceed the amount of reserved compute resources.
Each virtual warehouse supports a maximum of five elastic time periods.
When you delete an elasticity plan, the virtual warehouse status temporarily changes to "Processing", even if the resource count does not change. Read and write operations are not affected during this time.

When you save an elasticity plan, if the change affects the current elastic resource amount, Hologres immediately performs the corresponding scaling operation. See the following examples:

Example 1

Assume the current time is 16:00. You add a new elasticity plan to virtual warehouse A, which has 64 reserved CUs:

If the active time period is 15:00–18:00 with 32 elastic CUs, the total compute resources of virtual warehouse A will immediately scale out to 96 CUs upon saving.
Example 2
Assume the current time is 16:00. You modify an existing elasticity plan for virtual warehouse A, which has 64 reserved CUs:
- Before modification: The active period is 15:00–18:00 with 32 elastic CUs.
- After modification: The active period is 17:00–18:00 with 32 elastic CUs.
Upon saving, the total compute resources of virtual warehouse A will immediately scale in from 96 CUs to 64 CUs.

Monitoring and alerts for scheduled scaling

You can monitor elasticity plan execution in the following ways.

Elasticity plan execution logs

Navigate to the Virtual Warehouse Management page and click the Elastic Event Execution Logs tab.
Select a time range to view the execution history of past elasticity plans. This tab contains the Scheduled Scaling and Auto Scaling subtabs. You can filter by date range, and the table displays information such as Execution Time, Virtual Warehouse, Execution Status, Event Type, Reserved Compute Resources, and Target Elastic Compute Resources.

Monitoring metrics

In the monitoring metrics section of the Hologres console, you can view the Warehouse_timed_elastic_cores(Count) metric, which represents the number of cores provisioned by scheduled scaling. You can configure alert rules for this metric as needed. For more information, see Monitoring metrics.

Cloud Monitor events

All scaling operations performed by Hologres elasticity plans are recorded as events in Cloud Monitor.

Navigate to the Event Center of Cloud Monitor. On the System Event tab, select Hologres from the Event Monitoring section to monitor scheduled scaling events. The event names are as follows:
- Instance:Warehouse:TimedElastic:Start: A scheduled scaling operation for a virtual warehouse has started.
- Instance:Warehouse:TimedElastic:Finish: A scheduled scaling operation for a virtual warehouse has completed.
- Instance:Warehouse:TimedElastic:Failed: A scheduled scaling operation for a virtual warehouse has failed.

You can configure notifications and alerts based on these Cloud Monitor events. For more information, see Use system events for alerting.

The following is an example of a scaling failure event:

{
    "Status": "Failed",
    "InstanceName": "<instance_id>",
    "ResourceId": "<instance_resource_id>",
    "Content": {
        "ScaleType": "ScaleDown",
        "ScheduleId": "xxxxxx",
        "TimedElasticCPU": 0,
        "WarehouseId": "2",
        "WarehouseName": "<warehouse_name>"
    },
    "Product": "hologres",
    "Time": 1722852008000,
    "Level": "WARN",
    "RegionId": "<region>",
    "Id": "<event_id>",
    "GroupId": "0",
    "Name": "Instance:Warehouse:TimedElastic:Failed"
}

ActionTrail

Operations in the Hologres console, such as editing an elasticity plan, and the resulting scaling operations are recorded in ActionTrail. For more information, see Audit event logs.