Best practices for online service high availability

Updated at:
Copy as MD

Best practices to minimize service interruptions and improve system stability and reliability for online services.

Quick start guide

Get started with ApsaraDB for SelectDB by reading the following documents.

  • (Required) Quick Start

    Covers ApsaraDB for SelectDB basic concepts, instance purchase and usage, and table design to help you get started with ApsaraDB for SelectDB.

  • (Optional) Data migration

    Migrate data from various data sources, such as MySQL, PostgreSQL, and Doris, to ApsaraDB for SelectDB.

  • (Optional) Performance Testing

    Run performance tests using Star Schema Benchmark, TPC-H Benchmark, and TPC-DS Benchmark.

  • (Optional) Solutions

    Observability and data lakehouse.

Release and change specifications

  • (Required) Test in advance

    Test new features on a test instance before release. For high-load services, run performance stress tests to evaluate production-level performance.

  • (Required) Grayscale release

    Release changes during off-peak hours using a phased rollout (10%→50%→100%). Observe the service 10–30 minutes between phases. Monitor closely during the first business peak—off-peak testing may not reveal all issues.

O&M recommendations

  • (Required) Business monitoring

    ApsaraDB for SelectDB provides monitoring and alerting. Key metrics to track: average and P99 query response time, query success rate, data import speed, CPU utilization, and memory usage. Configure alerts in Set alert rules.

  • Capacity management

    For high-load services, run optimization and stress tests to determine maximum QPS. Scale out resources based on business growth or before promotional events.

  • Version updates

    ApsaraDB for SelectDB fixes bugs through minor version updates (three or four-part version numbers). Upgrade to the latest patch version promptly. Test in a non-production environment first; contact technical support for emergency rollbacks. Example: version 4.0.4.2 (released February 6, 2025) received 12 updates over six months, reaching 4.0.6.1.

  • Service isolation

    Use separate instances for independent business scenarios. When different services share data, use multi-compute group architecture—multiple physical compute queues within one instance sharing data from the read-write instance.

Operational drills

  • Instance changes

    Common operations such as upgrades and scale-outs can cause transient connection interruptions. Rehearse these in advance and ensure your application retries failed connections during ApsaraDB for SelectDB instance changes.

  • Fault recovery

    ApsaraDB for SelectDB supports fault recovery options: switching to a new compute group, restarting a failed instance, or restarting a stalled compute group. Rehearse these procedures in advance.

High availability architecture

  • (Optional) Service throttling

    ApsaraDB for SelectDB supports Workload Groups—logical task queues that control resource usage by request type or service. Use Workload Groups to set resource limits and throttle traffic during bursts.

  • (Optional) Multi-zone disaster recovery

    ApsaraDB for SelectDB supports multi-zone disaster recovery. If a zone fails, the system automatically switches over with an RTO of approximately 10 seconds.

  • (Optional) Data backup and recovery

    Enable daily backups for sensitive online services. Perform manual backups before important changes. Use backup data to recover quickly from critical failures.