Resource Orchestration Service:Deploy a Spark cluster

What is an Apache Spark cluster

Apache Spark is a general-purpose computing engine designed for large-scale data processing. A Spark cluster consists of one Master node and multiple Worker nodes:

The sample template installs the following software versions:

Cluster edition vs. standalone edition

Feature: deployment on an existing VPC

Unlike other ROS deployment solutions, this template requires using your existing VPC, vSwitches, and security groups without creating new network resources. This design is suitable for:

• Standardized network environments where you need to add Spark clusters to existing VPCs

• Multiple projects sharing one VPC to avoid duplicate network resource creation

Resources created

Results after deployment

After successful template execution, you will obtain:

• One Spark Master ECS instance (bound with elastic public IP, providing cluster management Web UI)

• Multiple Spark Worker ECS instances (managed by Auto Scaling Group, default 2 instances)

• All nodes installed with JDK 1.8.0, Hadoop 2.7.7, Scala 2.12.1, Spark 2.1.0

• Master and Worker nodes automatically configured for cluster networking

• SparkWebSiteURL provided in stack Outputs for accessing Spark management console

Architecture diagram

                          User access
                            ↓
                     EIP → Master ECS (port 8080)
                        ↙      ↓      ↘
                Worker-1   Worker-2   Worker-N
                   ↑          ↑          ↑
              Auto Scaling Group management (ESS ScalingGroup)
                             ↑
                    OOS lifecycle hooks
              (automatically install Spark and join cluster during creation)

Deployment parameters

This template requires using your existing VPC network resources. Configure the following parameters during deployment:

Deployment methods

# Prompt
Deploy a Spark distributed cluster environment in an existing VPC.
Requirements:
1. Use existing VPC, vSwitches, and security groups (do not create new network resources).
2. Create 1 Master ECS node bound with elastic public IP.
3. Create Auto Scaling Group to manage Worker nodes, starting with 2 Workers.
4. Use CentOS 7 image for all nodes.
5. Automatically install JDK 1.8, Hadoop 2.7.7, Scala 2.12.1, Spark 2.1.0 via UserData scripts.
6. Automatically configure cluster networking between Master and Worker nodes.
7. Use OOS templates to create lifecycle hooks for automatic Spark installation and joining cluster when Workers scale out.
8. Open inbound port 8080 in security group.
9. Output `SparkWebSiteURL` in the stack Outputs.

Post-deployment operations

Q1: Deployment fails with error "The specified InstanceType is not available"

Cause: The selected ECS instance type is unavailable in the specified zone.

Solution:

• Change the zone where your vSwitch resides and redeploy

• Check which instance types are available in the target zone on the ECS pricing page.

• Spark clusters have high memory requirements. It is recommended to choose at least 4 vCPUs and 8 GB RAM for the Master node, and at least 2 vCPUs and 4 GB RAM for each Worker node.

Q2: Deployment succeeds but cannot access SparkWebSiteURL

Troubleshooting steps:

1. Check security group: Ensure inbound rules allow TCP port 8080

2. Check EIP association: Verify elastic IP is associated to Master ECS instance

3. Connect to the Master and check service status:

   # Check Spark Master process
   ps -ef | grep spark
   # Check port 8080 listening
   netstat -tlnp | grep 8080
   # Manually start Spark cluster if not running
   $SPARK_HOME/sbin/start-all.sh

4. View UserData execution logs: cat /var/log/messages | grep cloud-init

Q3: Worker nodes not registered to Master

Troubleshooting steps:

1. View registered Worker list in Spark Web UI (port 8080)

2. SSH login to Worker nodes and check if Spark processes are running

3. Verify Worker nodes are in same VPC/vSwitch as Master node

4. View Worker logs: cat $SPARK_HOME/logs/spark-*-worker-*.out