PolarDB-DDB: A High-Performance, Cost-Effective Cloud-Native Alternative to DynamoDB

This article introduces PolarDB-DDB, a new feature for PolarDB for PostgreSQL that makes it a high-performance, cost-effective, and API-compatible alternative to Amazon DynamoDB.

PolarDB-DDB: A High-Performance, Cost-Effective Cloud-Native Alternative Compatible with DynamoDB

We are excited to announce the release of PolarDB's DynamoDB Compatibility feature (PolarDB-DDB) as part of the latest version of PolarDB for PostgreSQL. This new capability is highly compatible with the Amazon DynamoDB API and includes a comprehensive migration solution. With PolarDB-DDB, you can transition your existing business systems without modifying your code by connecting through native DynamoDB drivers (such as the Python or Go SDK).

PolarDB-DDB boasts enterprise-level capabilities, including instant elastic scaling, massive storage, and high availability, fully meeting the demands for high performance and reliability of modern applications. Furthermore, PolarDB-DDB provides a cost-effective solution that significantly outperforms DynamoDB, thereby empowering businesses to build efficient, stable, and economical data infrastructures perfect for large-scale NoSQL data processing and real-time access applications.

Why Choose PolarDB-DDB?

While developers love DynamoDB for its flexible schemaless data model, its costs can escalate quickly with growing data volume and workloads.

PolarDB-DDB is our answer: a solution designed specifically to be compatible with the DynamoDB API. It retains the flexibility of NoSQL data models while offering enhanced data processing capabilities, more comprehensive cloud-native features, and superior cost-effectiveness.

At its core, PolarDB-DDB is powered by a mature PostgreSQL engine, leveraging efficient JSONB storage, flexible indexing, complete ACID transaction capability, and robust SQL analysis. In addition, backed by a vibrant open-source community, PolarDB-DDB is able to continuously provide ongoing feature enhancements, plugin extensions, and security updates, ensuring long-term stable evolution.

Key Advantages of PolarDB-DDB

1. Convenience and Scalability:

Highly compatible with the DynamoDB API, requiring minimal code changes while offering a schemaless, elastic, and easily scalable experience.
Supports complex queries, batch processing, and ETL using SQL, allowing seamless integration with BI tools and enterprise systems, while maintaining API simplicity.

2. Elasticity and Automated Maintenance:

Provides instant elastic scaling (serverless mode); computing resources can dynamically adjust to business needs.
Uses a multi-reader, single-writer architecture with built-in automatic load balancing to support high-concurrency scenarios.
Separates compute and storage, automatically expanding storage based on demand, accommodating data growth from terabytes to petabytes.

3. Fully Managed Service:

There’s no need for hardware management, master-slave replication, or backup recovery, significantly reducing operational burdens.

4. Ecosystem Integration and Innovation:

Deeply integrates into the PostgreSQL ecosystem, compatible with mainstream BI tools, data pipelines, programming languages, and heterogeneous database connections, while allowing SQL and NoSQL usage together, aiding efficient data governance, business expansion, and multi-cloud deployment.

5. High Cost-Effectiveness:

Under equivalent specifications or performance conditions, PolarDB-DDB can achieve over 30% cost savings compared to DynamoDB through improved performance and resource utilization. This cost advantage is especially pronounced in scenarios involving heavy writes and high concurrent queries.

In summary, PolarDB-DDB combines the flexibility of NoSQL and the powerful analytical capabilities of SQL, greatly simplifying data architecture, enabling businesses to benefit from high cost-Effectiveness, ease of maintenance, and sustainable volutive without compromise.

Architecture Overview: How to Achieve Compatibility and High Performance?

Database Proxy Layer

PolarDB-DDB uses a dedicated proxy layer to efficiently convert DynamoDB API requests and translate them to the PostgreSQL protocol. Logic for protocol parsing, authentication, and conversion is centralized at the proxy layer. Communication between the proxy and clients uses the HTTP protocol of DynamoDB, while standard PostgreSQL protocol is used between the proxy and database compute nodes. This bridging and decoupled architecture effectively avoids embedding DynamoDB's specific HTTP interfaces and authentication mechanisms into the PostgreSQL kernel, thus preserving the simplicity and stability of the PostgreSQL architecture. Additionally, this design reuses PolarDB’s connection pool, load balancing, and read-write separation capabilities, mitigating maintenance burdens arising from protocol coupling.

Here's the workflow for a complete DynamoDB request:

The client initiates an HTTP request via the DynamoDB endpoint.
The database proxy authenticates the request using the key in the request header.
Once authenticated, the protocol converter interprets the requests and translates them into JSON-targeted PostgreSQL extension protocol messages.
Then, the router intelligently sends requests to optimal compute nodes based on request type (read/write) and current node load conditions.
When responding, the database proxy converts PostgreSQL's response data back to a DynamoDB-compatible HTTP format, ensuring a consistent API experience for the client.

This decoupled architecture preserves the stability of the PostgreSQL core while reusing PolarDB’s mature connection pool, load balancing, and read-write separation capabilities, supporting hybrid queries (SQL & NoSQL) that enhance system flexibility.

Database Compute Nodes

It’s worth noting that enabling PolarDB-DDB doesn’t introduce extra compute node. PolarDB clusters with PolarDB-DDB feature use a standard primary-replica architecture, scaling from a basic two-node cluster to a cluster comprising one primary node and up to fifteen read-only nodes. The primary node handles all writes and transaction processing, while read-only nodes distribute read loads, significantly boosting concurrency capabilities. In the event of a primary node failure, automatic failover ensures high availability. Through a clear separation of storage and compute, the computing nodes leverage a standard PostgreSQL core, reusing its mature JSONB storage engine and query optimization. The system minimally reconfigures only to adapt string sorting rules, document operators, built-in functions, and datatype semantics to accurately reflect DynamoDB API behavior.

To improve user experience, the system internally manages DynamoDB metadata, concealing the traditional relational database DB/Schema concepts, presenting users with a view consistent with DynamoDB's user-region dimensional table management.

Quick Start Guide: 5 Steps to Get Started

Step 1: Create a Cluster

You can use PolarDB-DDB in either of the following ways:

Option 1: Purchase New PolarDB Cluster and Enable PolarDB-DDB Feature

Log onto the Alibaba Cloud console and navigate to the PolarDB console.
Click on "Create Cluster".
In the cluster purchasing interface, select "Create Primary Cluster," choose PostgreSQL-Compatible as the database engine, and enable DynamoDB Compatibility to enable PolarDB-DDB feature. Other configurations can be set as needed.

Option 2: Enable PolarDB-DDB Feature for Existing Clusters

In addition to purchasing new clusters, PolarDB-DDB can also be activated for existing PolarDB for PostgreSQL clusters. Ensure the following database and proxy version requirements are met:

Database proxy version >= 2.3.59
Database kernel version >= 2.0.14.17.35.0.

Step 2: Create PolarDB-DDB Access Address

Once DynamoDB Compatibility is enabled, the system will automatically create a DynamoDB endpoint for the cluster. The DynamoDB endpoint is the key to unlocking the full power of PolarDB-DDB.

You may also create a customized DynamoDB endpoint pointed to specified node with many other configurations. To do so, click “Create Custom Cluster Endpoint,” and select Endpoint Type as DynamoDB. Here, you have two options on the data consistency settings (defaulting to eventual consistency). You can choose global consistency if stronger data consistency is required. Once global consistency is selected, all read requests under this address will uniformly adopt the global consistency strategy. Please note that global consistency cannot be set in requests via ConsistentRead=True; the consistency type must be configured in advance for the endpoint.

Step 3: Create PolarDB-DDB Account

After generating a DynamoDB endpoint, create a dedicated account to access PolarDB-DDB.

Select “Accounts” and click “Create Account.” Choose the Account Type as DynamoDB Account, then customize your own account name and password.

After the account is created, the system will generate your access credentials.

Access Key ID: This is your Account Name.
Secret Access Key: This is a system-generated key (an MD5 hash of your password). For security, this is not shown in plaintext.

Step 4: Configure Your Network

For clients using Alibaba Cloud ECS or other cloud resources, it is recommended to connect to PolarDB-DDB via private endpoint. Ensure your applications are in the same VPC, or that the ECS IP is added to the database whitelist.

For local testing, you can apply for a public endpoint. However, this is recommended only for temporary testing. Be sure to deactivate public access promptly after testing, to reduce security risks.

Step 5: Validation

Here’s an example of Python code using the boto3 SDK to connect to the PolarDB-DDB:

import boto3
from botocore.exceptions import ClientError
import time

# Configure your PolarDB-DDB endpoint (DynamoDB endpoint in step 2), access key (Account Name in step 3), secret access key (Key in step 3), and region
endpoint_url = "http://<your-polardb-ddb-endpoint>"
aws_access_key_id = "<your-access-key-id>"
aws_secret_access_key = "<your-secret-access-key>"
region_name = ""

# Create connection
dynamodb = boto3.resource(
    'dynamodb',
    aws_access_key_id=aws_access_key_id,
    aws_secret_access_key=aws_secret_access_key,
    region_name=region_name,
    endpoint_url=endpoint_url
)

table_name = 'usertable'

# 1. Create table
def create_table():
    try:
        table = dynamodb.create_table(
            TableName=table_name,
            KeySchema=[
                {'AttributeName': 'userid', 'KeyType': 'HASH'}  # Primary Key
            ],
            AttributeDefinitions=[
                {'AttributeName': 'userid', 'AttributeType': 'S'}
            ],
        )
        print("Creating table. Waiting for it to be active...")
        table.meta.client.get_waiter('table_exists').wait(TableName=table_name)
        print("Table created and active!")
        return table
    except ClientError as e:
        if e.response['Error']['Code'] == 'ResourceInUseException':
            print("Table already exists.")
            return dynamodb.Table(table_name)
        else:
            print("Error:", e)
            raise

# 2. Insert some data
def put_items():
    table = dynamodb.Table(table_name)
    users = [
        {'userid': 'user01', 'name': 'Alice', 'age': 24},
        {'userid': 'user02', 'name': 'Bob', 'age': 30},
        {'userid': 'user03', 'name': 'Charlie', 'age': 28}
    ]
    for user in users:
        table.put_item(Item=user)
    print("Inserted items.")

# 3. Query (scan) data
def scan_table():
    table = dynamodb.Table(table_name)
    response = table.scan()
    items = response.get('Items', [])
    print(f"Scanned {len(items)} items:")
    for item in items:
        print(item)

# 4. Delete table
def delete_table():
    table = dynamodb.Table(table_name)
    table.delete()
    print("Deleting table. Waiting for deletion...")
    table.meta.client.get_waiter('table_not_exists').wait(TableName=table_name)
    print("Table deleted.")

if __name__ == '__main__':
    create_table()
    put_items()
    scan_table()
    delete_table()

Expected output:

Creating table. Waiting for it to be active...
Table created and active!
Inserted items.
Scanned 3 items:
{'age': Decimal('24'), 'name': 'Alice', 'userid': 'user01'}
{'age': Decimal('30'), 'name': 'Bob', 'userid': 'user02'}
{'age': Decimal('28'), 'name': 'Charlie', 'userid': 'user03'}
Deleting table. Waiting for deletion...
Table deleted.

Compatibility Assessment and Data Migration

Compatibility Assessment

To efficiently evaluate the feasibility of migrating applications built on DynamoDB to PolarDB-DDB, we provide a lightweight and secure scanning tool called polar_ddb_code_digest. This tool analyzes local project code, identifies DynamoDB API calls and parameter usage, and outputs a desensitized interface mapping report to help you evaluate compatibility.

Key features of polar_ddb_code_digest include:

Recursive Project Code Scanning: The tool recursively scans project code to identify supported source files (e.g., .py files).
DynamoDB SDK API Call Identification: Based on the AWS official interface specifications, the tool uses syntactic analysis to recognize DynamoDB SDK API calls and extract:
- Type of API calls (e.g., DynamoDB’s CRUD - PutItem, GetItem, UpdateItem, Query, etc.).
- Parameter names and structures used by each API call.
- Built-in constants and type definitions of SDK.
- Dictionary structures containing DynamoDB parameter keywords (e.g., KeyConditionExpression, FilterExpression).
Parameter Classification and Desensitization: The tool categorizes and sensitizes parameters following predefined rules, generating a complete API usage profile. All output data is desensitized, retaining only interface names, parameter names, and types; original values (like primary keys and conditional values) are not recorded, ensuring data security and no sensitive data is exposed.
Structured Report Generation: Outputs a structured report that includes the scanned directory path, a list of used DynamoDB APIs with call frequencies, and parameter types and usage.

After generating the sanitized report, users can evaluate compatibility against the "PolarDB-DDB Command Support List" or submit the sanitized report to the Alibaba Cloud technical support team for professional migration advice once internal compliance approval is complete.

Currently, the tool supports the Go SDK v2 and the Python Boto3 driver. For detailed command usage or inquiries regarding other SDKs (such as Java, Node.js, etc.), contact us for free technical support.

Authentication Code Modification

Upon confirming command compatibility, you only need to adjust the connection code. Below, we detail how to adapt based on two common user habits, with key changes only in AK/SK and region settings.

Scenario 1: Credentials specified directly in code

If the original code is as follows:

aws_access_key_id = "<your-access-key-id>"
aws_secret_access_key = "<your-secret-access-key>"
region_name = "cn-north-1"

# Create connection
dynamodb = boto3.resource(
    'dynamodb',
    aws_access_key_id=aws_access_key_id,
    aws_secret_access_key=aws_secret_access_key,
    region_name=region_name
)

Modify it to:

# Add the PolarDB-DDB Connection String (the DynamoDB endpoint in step 2)
endpoint_url = "http://<your-polardb-ddb-endpoint>"
aws_access_key_id = "<user-name>"
aws_secret_access_key = "<md5-secret-key>"
# Set region to ""
region_name = ""

# Create connection
dynamodb = boto3.resource(
    'dynamodb',
    aws_access_key_id=aws_access_key_id,
    aws_secret_access_key=aws_secret_access_key,
    region_name=region_name,
    endpoint_url=endpoint_url
)

Scenario 2: Using local configuration files (.aws/credentials and .aws/config)

Modify the access key and secret access key in the ~/.aws/credentials file:

[default]
aws_access_key_id = <user_name>
aws_secret_access_key = <md5-secret-key>

Change the region to empty in the ~/.aws/config file:

[default]
region =

Mapping DynamoDB Concepts to PolarDB-DDB

This section helps you gain a deeper understanding of the adaptation logic and principles.

To connect your existing application, you only need to update a few key parameters. This table shows the direct mapping between DynamoDB's credentials and PolarDB-DDB's connection parameters.

DynamoDB	PolarDB-DDB	How to configure
Access Key ID	PolarDB-DDB AccountName	Use the account name you created in the PolarDB console.
Secret Access Key	System-Generated Key	Use the key provided in the console (an MD5 hash of your password).
Region	Endpoint URL	Set the region_name parameter to an empty string ("") and add the endpoint_url parameter with your PolarDB-DDB connection string.

Access Key and Secret Access Key

In AWS, the Access Key and Secret Access Key are identity credentials used for programmatic access via SDKs, scripts, or tools:

Access Key ID: Functions as a username to identify your account.
Secret Access Key: Functions as a password, visible only upon initial creation.

To ensure compatibility with DynamoDB client drivers, PolarDB-DDB uses the following mapping:

The Account Name serves as the Access Key ID.
The Key (MD5 hash of the account password) serves as the Secret Access Key. (The Key automatically generated and displayed not in plaintext on the console.)

Region and Endpoint

When creating tables in the DynamoDB console, you must specify the region where the data resides, and connections also require the corresponding region configuration. On Alibaba Cloud, this concept corresponds to the region selected when purchasing an cluster. For example, to migrate a DynamoDB data located in the cn-north-1 region, you can create a PolarDB-DDB cluster in Alibaba Cloud’s China (Beijing) region.

Since the region and endpoint for a PolarDB-DDB cluster are determined during cluster creation, clients do not need to locate services by region. Instead, they connect directly by explicitly configuring the endpoint.

The Migration Process: A Step-by-Step Guide

PolarDB-DDB features a database migration tool built on nimo-shake, designed to facilitate seamless migration from AWS DynamoDB. The data migration tool leverages certain DynamoDB commands and streaming capabilities, such as Scan (for full table traversal) and Streams (for change data capture).

The tool primarily uses query and insert commands for full synchronization, capturing data changes for incremental synchronization. Below is a preliminary overview of the migration process:

The process is broken down into four key phases.

Step 1: Full Synchronization (nimo-shake Migration Tool)

The Migration tool, nimo-shake, handles the initial full data copy.
- Schema Migration: The tool first uses ListTables to fetch all table names from DynamoDB, filter the tables that need to be migrated (configurable), and then create tables in PolarDB-DDB using CreateTable.
- Data Migration: The Fetcher concurrently pulls all existing data from DynamoDB via Scan, placing it into a queue, from which the Executor retrieves the data and uses BatchWriteItem to batch write to PolarDB-DDB.

Step 2: Incremental Synchronization (nimo-shake Migration Tool)

Once full synchronization completes, incremental synchronization will start automatically. The StreamListener monitors for changes occurring after the migration point via DynamoDB Streams, notifying the StreamFetcher to fetch incremental data and synchronizing it to PolarDB-DDB through the Executor. Incremental synchronization will record checkpoint information to support resuming from pauses.

Step 3: Consistency Verification (nimo-full-check Verification Tool)

Validate consistency via bulk retrieval of DynamoDB data through Scan, matching the partition key and sort key to query corresponding records in PolarDB-DDB with GetItem, thus verifying consistency and generating a difference report.

(Optional) Reverse Synchronization

Prior to fully switching over to the PolarDB-DDB cluster, you can capture incremental data changes from PolarDB-DDB through logical replication and convert these increments into JSON for writing back to DynamoDB using a Python script.

Step 4: Business Cutover

During the cutover to the PolarDB-DDB cluster, halt writes to all accounts except for on-going data synchronization.
Once incremental data is caught up, suspend writes to source DynamoDB.
Update the database connection string in application to the PolarDB-DDB endpoint.
Pause the forward synchronization link (DynamoDB->PolarDB-DDB).
(if applicable)Initiate the reverse synchronization link.
Resume write permissions for both PolarDB-DDB and DynamoDB.
Go live after cutover.

Important Considerations

Performing a full data migration will consume resources on both the source and target databases and may increase server load. If your database workload is high or your server specifications are low, this could add significant pressure. We advise thorough evaluations before executing data migration, ideally performing it during low-traffic periods or after appropriate resource scaling.

For detailed configuration options and usage instructions, contact us at https://www.alibabacloud.com/contact-sales

Community

PolarDB-DDB: A High-Performance, Cost-Effective Cloud-Native Alternative to DynamoDB

PolarDB-DDB: A High-Performance, Cost-Effective Cloud-Native Alternative Compatible with DynamoDB

Why Choose PolarDB-DDB?

Key Advantages of PolarDB-DDB

Architecture Overview: How to Achieve Compatibility and High Performance?

Database Proxy Layer

Database Compute Nodes

Quick Start Guide: 5 Steps to Get Started

Step 1: Create a Cluster

Step 2: Create PolarDB-DDB Access Address

Step 3: Create PolarDB-DDB Account

Step 4: Configure Your Network

Step 5: Validation

Compatibility Assessment and Data Migration

Compatibility Assessment

Authentication Code Modification

Mapping DynamoDB Concepts to PolarDB-DDB

The Migration Process: A Step-by-Step Guide

Step 1: Full Synchronization (nimo-shake Migration Tool)

Step 2: Incremental Synchronization (nimo-shake Migration Tool)

Step 3: Consistency Verification (nimo-full-check Verification Tool)

Step 4: Business Cutover

Important Considerations

PolarDB Always Free Program

Read previous post:

Read next post:

ApsaraDB

You may also like

Comments

ApsaraDB

Related Products

PolarDB for PostgreSQL

PolarDB for Xscale

PolarDB for MySQL

AnalyticDB for PostgreSQL