Realtime Compute V1.0 and V2.0 provide AutoConf for you to improve the job performance. However, AutoConf requires you to frequently restart the job. To resolve this issue, the AutoScale feature is offered in Realtime Compute V3.0 and later. After you start a job, Realtime Compute adjusts the job configuration to reach the preset performance goal based on resource configuration rules. This process does not require any manual operations.

Note
  • AutoScale is only applicable to Realtime Compute V3.0 and later.
  • Before you upgrade Realtime Compute to V3.0, delete all PlanJSON files generated in Realtime Compute earlier than V3.0, and reacquire configuration files.

Enable AutoScale

You can enable AutoScale when you publish a job.

  1. Go to the job editing page on the Realtime Compute development platform.
    1. Log on to the Realtime Compute Console.
    2. In the top navigation bar, click Development.
    3. In the Jobs section, double click the target job name or folder in the Development field.
  2. Click Publish in the top navigation bar to go to the Publish New Version page.
  3. In the Initial Resources step, select a resource type and click Next.
    • Use last time Auto Scaling: uses the PlanJSON file for the latest auto scaling to start the job. You can select Use last time Auto Scaling when the following conditions are met:
      • The job is published with AutoScale enabled and adopts the latest configuration change.
      • The job is in the suspended state.
      • Obtain the AutoScale configuration. To obtain the AutoScale configuration, click Resource Configuration in the right-side navigation pane and choose Actions > Acquire Auto Scaling Configuration.
    • Default: uses the resource configuration generated for the latest auto scaling to start the job. You can select this option to publish a new job or an existing job of which the job logic is not modified.
    • Manually Configured Resources: uses manually configured resources to start the job. Select this option to manually configure resources or modify the AutoScale configuration.
  4. In the Check step, check the job and click Next.
  5. In the Resource Configuration step, set AutoScale parameters and click Next.
    Parameter Description
    AutoScale Specifies whether to enable AutoScale. To enable the feature, select Enable.
    Maximum CUs for PlanJSON The maximum available CUs for the job. One CU consists one core and 4 GB random access memory (RAM). The value of this parameter must be smaller than the number of available CUs for the project.
    Optimization Policy The policy for optimizing the job configuration. Valid value: Data Pending Time. Realtime Compute optimizes the job configuration based on the optimization policy and expected value.
    Expected Value The threshold for data pending time. When data from the upstream data storage pends for a time period exceeding the threshold, Realtime Compute triggers AutoScale to adjust the number of concurrent tasks for the job.
    Note For example, set the expected data pending time to 5 seconds. When the data from the upstream data storage pends for more than 5 seconds before entering the Realtime Compute job, Realtime Compute keeps reducing concurrent tasks for the job until the pending time is shortened to less than 5 seconds.
  6. In the Publish File step, click Publish.
  7. Start the job. For more information, see Start a job.

Disable AutoScale

Note You can disable AutoScale for a job only after the job is published with AutoScale enabled.
You can disable AutoScale for a running job.
  1. Go to the job editing page on the Realtime Compute development platform.
    1. Log on to the Realtime Compute Console.
    2. In the top navigation bar, click Development.
    3. In the Jobs section, double click the target job name or folder in the Development field.
  2. Click Stop Auto Configuration in the upper-right corner.
  3. Click OK.
Note The actions in the Auto Configuration column on the job administration page are available only when AutoScale is enabled during the job publishing.

View information about AutoScale

Note You can view the information about AutoScale on the Administration page.
  • AutoScale metrics
    You can choose Curve Charts > Overview to view information about AutoScale.
    Metric Description
    Successes and Failures The number of AutoScale successes and failures.
    CPU Usage The CPU resources used for auto scaling.
    Memory Usage The memory resources used for auto scaling.
  • Information about PlanJSON files generated for auto scaling
    On the Realtime Compute development platform, click the Properties and Parameters tab, choose Resource Configuration > Plans, and then select the target version to view information about the corresponding PlanJSON file generated for auto scaling.

Pay attention to potential issues of AutoScale

  • The job may be automatically restarted.

    When you enable AutoScale, Realtime Compute adjusts the parallelism and resources based on the actual volume of data streams. As a result, Realtime Compute may automatically restart the job to adjust resources when the volume of data streams increases or decreases.

  • Data transmission may be delayed for a short time.

    When the volume of data streams reaches the bottom, Realtime Compute triggers AutoScale to reduce the parallelism and resources. As a result, the resources may become insufficient for throughput and data transmission when the volume of data streams increases.

  • The job may fail to be resumed.

    When AutoScale does not work well for a job, the job may be delayed, and Realtime Compute may keep adjusting the job configuration. In this case, the job cannot be resumed.

  • The parallelism may be reduced and then increased.

    For a job that uses window functions or aggregate functions, when the state data increases, the performance for accessing the state data degrades, and the parallelism is reduced when the job is started. When the job is running, the parallelism increases with the accumulation of the state data until the volume of the state data becomes steady.

FAQ

  • Q: What can I do if AutoScale cannot be triggered?

    A: Follow these steps to troubleshoot the issue:
    • Check whether the job fails frequently. To enable AutoScale, you must make sure that the job logic is correct and the job can run stably.
    • View JobManager logs to check whether any system exception exists.
  • Q: What can I do if AutoScale does not take effect?

    A: Follow these steps to troubleshoot the issue:
    1. Check whether the amount of resources consumed by the job has reached the specified upper limit.
    2. Check the source node logic to find whether too many operators are connected to the source node. If too many operators are connected to the source node, edit the PlanJSON file to remove some operators. For more information, see Optimize performance by manual configuration.
    3. View JobManager logs to check whether any system exception exists.