Pony.ai's IaC practice with Terraform
The Pony.ai DevOps team shares how they adopted Terraform to fully automate their cloud infrastructure—covering their technology selection process, architectural decisions, and the solutions they built to handle code reuse and multi-environment deployments.
Background
Founded in 2016, Pony.ai is a global autonomous vehicle technology company with R&D centers in Silicon Valley, Guangzhou, Beijing, Shanghai, and Shenzhen. The company holds autonomous driving test and operation permits across multiple locations in the U.S. and China, and works with automakers including Toyota, Hyundai, FAW Group, and GAC Group.
Pony.ai's China operations run on Alibaba Cloud, hosting critical services such as data labeling, Robotaxi, and Robotruck platforms. These services depend on Elastic Compute Service (ECS), ApsaraDB RDS (RDS), Server Load Balancer (SLB), Bastionhost, and Security Center—making infrastructure management increasingly complex for the DevOps team.
To address this, the team defined three core objectives:
Reviewable deployments: Every stage of an operational activity—from requirement gathering and architectural design to coding and deployment—must be precise and traceable.
Version-controlled deployments: A complete, auditable history of all infrastructure changes, with the ability to roll back to a specific version.
Consistent multi-environment deployments: No discrepancies between development, staging, and production environments that could cause environment-specific failures.
Technology selection
The team evaluated three mainstream approaches for managing public cloud resources:
Cloud provider console: Direct management through the GUI.
Custom management system: Building or purchasing a system that calls cloud provider APIs.
Infrastructure as Code (IaC) frameworks: Using code to define and manage infrastructure.
IaC is the established standard for infrastructure automation and the most widely used framework for multi-cloud management. Paired with Git, IaC directly addresses traceability and versioning goals—every deployment and change is managed as code, and rollbacks are as simple as reverting to a previous Git branch.
Within the IaC ecosystem, Terraform is a leading open-source tool with a strong track record in enterprise production environments. By adopting Terraform, the DevOps team could focus on writing business logic for their infrastructure rather than building a custom orchestration system from scratch.
Given Pony.ai's multi-cloud strategy and hybrid cloud architecture, Terraform's standardization, ease of use, and active community made it the clear choice.
Architecture and implementation
The team implemented a Terraform-based IaC solution with a Git-centric workflow, as shown below:
For configuration files, the team chose JSON over HashiCorp Configuration Language (HCL) to stay consistent with their existing JSON-based applications and simplify code review.
Code is organized by business service. If a service needs an SLB, certificates, and ECS instances, all those resources are defined in a single Terraform file for that service. The following is a simplified example:
{
"output": {
"ecs_instance_1-private-ip": {
"value": "${alicloud_instance.ecs_instance_1.private_ip}"
},
"ecs_instance_2-private-ip": {
"value": "${alicloud_instance.ecs_instance_2.private_ip}"
},
"ponyai_business_1-slb-address": {
"value": "${alicloud_slb.ponyai_business_1-slb.address}"
}
},
"provider": {
"alicloud": {
"region": "alicloud_region"
}
},
"resource": {
"alicloud_instance": {
"ecs_instance_1": {
"availability_zone": "availability_zone_1",
"data_disks": [
{
"category": "cloud_essd",
"name": "data_volume",
"size": "xx"
}
],
"host_name": "ecs_instance_1",
"image_id": "image_id_1",
"instance_name": "ecs_instance_1",
"instance_type": "ecs_instance_type",
"internet_charge_type": "PayByTraffic",
"internet_max_bandwidth_out": 10,
"key_name": "key_name_1",
"security_groups": [
"security_groups_1"
],
"system_disk_category": "cloud_essd",
"system_disk_size": "xx",
"tags": {
"host_name": "ecs_instance_1"
},
"vswitch_id": "vswitch_id_1"
},
"ecs_instance_2": {
"availability_zone": "availability_zone_2",
"data_disks": [
{
"category": "cloud_essd",
"name": "data_volume",
"size": "xx"
}
],
"host_name": "availability_zone_2",
"image_id": "image_id_1",
"instance_name": "availability_zone_2",
"instance_type": "ecs_instance_type",
"internet_charge_type": "PayByTraffic",
"internet_max_bandwidth_out": 10,
"key_name": "key_name_1",
"security_groups": [
"security_groups_1"
],
"system_disk_category": "cloud_essd",
"system_disk_size": "xx",
"tags": {
"host_name": "availability_zone_2"
},
"vswitch_id": "vswitch_id_2"
}
},
"alicloud_slb": {
"slb-1": {
"address_type": "internet",
"internet_charge_type": "PayByTraffic",
"name": "slb_name",
"specification": "slb_specification"
}
},
"alicloud_slb_listener": {
"slb-listener-1": {
"backend_port": "xx",
"bandwidth": -1,
"frontend_port": "xx",
"health_check": "on",
"health_check_connect_port": "xx",
"health_check_domain": "domain_name",
"health_check_type": "check_type",
"health_check_uri": "uri_1",
"load_balancer_id": "${alicloud_slb.slb-1.id}",
"protocol": "protocol_1",
"scheduler": "scheduler_1",
"server_certificate_id": "${alicloud_slb_server_certificate.slb-certificate-1.id}",
"server_group_id": "${alicloud_slb_server_group.slb-server-group-1.id}"
}
},
"alicloud_slb_server_certificate": {
"slb-certificate-1": {
"alicloud_certificate_id": "xx",
"alicloud_certificate_name": "xx",
"name": "certificate_1"
}
},
"alicloud_slb_server_group": {
"slb-server-group-1": {
"load_balancer_id": "${alicloud_slb.slb-1.id}",
"name": "slb-server-group",
"servers": {
"port": "xx",
"server_ids": [
"${alicloud_instance.ecs_instance_1.id}",
"${alicloud_instance.ecs_instance_2.id}"
]
}
}
}
},
"terraform": {
"backend": {
"s3": {
"bucket": "bucket_name",
"dynamodb_table": "table",
"key": "key_1",
"profile": "profile_1",
"region": "region_1"
}
},
"required_providers": {
"alicloud": {
"source": "aliyun/alicloud",
"version": "xx"
}
}
}
}Business challenges
As the Terraform codebase grew, two problems emerged:
For resources like ECS instances, engineers only cared about a small set of parameters—
instance_type,instance_name,availability_zone—but had to define the full configuration every time.Deploying the same service across environments (development, staging, production) meant near-identical configurations with only minor differences, such as using an
slb.s2.mediumSLB in production versusslb.s1.smallin test. Copying and rewriting the code for each environment hurt readability and maintainability.
Solutions
To solve these problems, the team introduced Jsonnet—an open-source data templating language—to generate the Terraform JSON files. Jsonnet lets them abstract repetitive boilerplate into a reusable library of utility functions. For example, they built a generateEcs function that encapsulates all the default ECS parameters:
generateEcs(instance_name,
availability_zone,
vswitch_id,
security_groups,
instance_type,
host_name,
data_volume_size=null,
system_disk_size=null,
internet_charge_type="PayByTraffic",
image_id="ubuntu_18_04_x64_20G_alibase_20200914.vhd",
key_name="bootstrap-bot",
system_disk_category="cloud_essd",
internet_max_bandwidth_out=10,
data_disk_category="cloud_essd"): {
instance_name: instance_name,
availability_zone: availability_zone,
vswitch_id: vswitch_id,
security_groups: security_groups,
instance_type: instance_type,
internet_charge_type: internet_charge_type,
image_id: image_id,
system_disk_category: system_disk_category,
[if system_disk_size != null then "system_disk_size"]:
system_disk_size,
key_name: key_name,
internet_max_bandwidth_out: internet_max_bandwidth_out,
host_name: host_name,
data_disks: if data_volume_size != null then [
{
name: "data_volume",
size: data_volume_size,
category: data_disk_category,
},
] else [],
tags: {
host_name: host_name,
},
}Engineers call this function to generate the full configuration, reducing the code needed to provision multiple instances to a concise loop:
alicloud_instance: {
[host_config.host_name]:
ecsUtils.generateEcs(
instance_name=host_config.host_name,
availability_zone=host_config.az,
security_groups=$.ecs_security_groups,
host_name=host_config.host_name,
instance_type=$.ecs_instance_type,
vswitch_id=vpc_output["vswitch-public-" + host_config.az].value,
data_volume_size=$.ecs_data_volume_size,
system_disk_size=$.ecs_system_disk_size
)
for host_config in host_configs
},Any parameter changes go in the shared utility function rather than in each individual resource definition.
Multi-environment configuration
For services that run across environments where only a few parameters differ, the team defines a base template. Each environment's configuration imports that base template and overrides only the fields that change.
This results in the following directory structure per service:
generated/main.tf.json is the JSON file Terraform executes. It is generated by Jsonnet from main.tf.json.jsonnet. The file main.tf.json.jsonnet.output holds the outputs produced after Terraform applies the configuration.
├── alicloud-region
│ ├── dev
│ │ ├── generated
│ │ │ └── main.tf.json
│ │ ├── main.tf.json.jsonnet
│ │ └── main.tf.json.jsonnet.output
│ ├── prod
│ │ ├── generated
│ │ │ └── main.tf.json
│ │ ├── main.tf.json.jsonnet
│ │ └── main.tf.json.jsonnet.output
│ └── staging
│ ├── generated
│ │ └── main.tf.json
│ ├── main.tf.json.jsonnet
│ └── main.tf.json.jsonnet.output
└── ponyai_business_1_base.libsonnetThe production environment imports the base template and sets its own SLB specification:
local base = import "../../ponyai_business_1_base.libsonnet";
base {
name: "ponyai_business_1_prod",
environment: "prod",
region: "alicloud_region",
slb_specification: "slb.s2.medium"
}The development environment imports the same base template and overrides only the name and specification:
local base = import "../../ponyai_business_1_base.libsonnet";
base {
name: "ponyai_business_1_dev",
environment: "dev",
region: "alicloud_region",
slb_specification: "slb.s1.small"
}Cross-stack references
Jsonnet also solves the dependency problem between infrastructure components. Creating an ECS instance requires a Virtual Private Cloud (VPC) ID, but the VPC is typically managed in a separate Terraform file. Rather than hardcoding the generated VPC ID, Pony.ai's workflow reads it from the output file of the VPC stack.
A main.tf.json file creates the VPC and writes its ID to main.tf.json.jsonnet.output in its own directory:
├── ali-cloud-region
│ ├── dev
│ │ ├── generated
│ │ │ └── main.tf.json
│ │ ├── main.tf.json.jsonnet
│ │ └── main.tf.json.jsonnet.output
│ └── prod
│ ├── generated
│ │ └── main.tf.json
│ ├── main.tf.json.jsonnet
│ └── main.tf.json.jsonnet.outputThe resulting main.tf.json.jsonnet.output file looks like this:
{
"vpc_id": {
"sensitive": false,
"type": "string",
"value": "vpc_id_for_ponyai"
},
"vswitch-id": {
"sensitive": false,
"type": "string",
"value": "vswitch_public_id_for_ponyai"
}
}Any service that needs these values imports the output file directly:
{
"ali-cloud-region": {
prod: import "./ali-cloud-region/prod/main.tf.json.jsonnet.output",
}
}This layered abstraction lets the Pony.ai DevOps team take full advantage of the Terraform ecosystem while keeping inter-service dependencies clean and explicit.
Business outcomes
Adopting IaC with Terraform and Git has delivered concrete improvements across the entire infrastructure lifecycle. All infrastructure parameters are explicitly defined in code—when a service needs two ECS instances with specific sizing and disk configurations, every detail lives in a single file.
Before each deployment, the team reviews the full pull request (PR) in Git and discusses the changes. This review step also creates a historical record of every infrastructure change, providing complete traceability.
Because changes are reviewable at the parameter level, the team can confirm that the final deployment matches the original design. Discrepancies caught during the Terraform coding phase can be corrected before they reach production. Developers are also required to run self-tests before opening a PR, avoiding multiple review cycles on broken code.
This transformation produced four key outcomes:
Faster: Infrastructure provisioning is no longer a sequence of repetitive, manual console operations. The shorter production cycle lets the company respond more quickly to business decisions and market opportunities.
More controllable: Every change is versioned and auditable, raising the organization from manual console operations to a traceable, trustworthy management system.
More efficient: Collaboration across teams—including internationally distributed ones—has improved significantly, reducing delays from time zone differences and varied working styles.
More secure: Human error in production is much less likely. Combining automated, code-driven processes with environment-specific approval workflows gives Pony.ai a robust safeguard for business operations.
Conclusion
Management model upgrade
The IaC philosophy now underpins all infrastructure development at Pony.ai. The team has abstracted a broad set of Alibaba Cloud components—including ECS, VPC, Object Storage Service (OSS), Public DNS, Resource Access Management (RAM), Simple Log Service, and user management—into internal, reusable functions.
Business model upgrade
When a team needs a cloud resource, they call one of these pre-built, vetted functions. The DevOps team focuses on maintaining and improving the internal library, which drives a sustained increase in operational efficiency across the organization.
Operations model upgrade
Today, more than 20 business units at Pony.ai are deployed and managed 100% through this IaC workflow. Every component has a clear, versioned history; changes go through rigorous review; and the team can reject non-compliant deployments before they reach production. The result is an online environment that is clean, reliable, and scalable.
About the author
The Pony.ai DevOps team
This article was contributed by an external author, and all copyrights belong to them. Alibaba Cloud assumes no responsibility for the content.