All Products
Search

This Product

    Resource Orchestration Service:Install Spark in standalone mode, 安装Spark单机版

    Document Center

    Resource Orchestration Service:Install Spark in standalone mode, 安装Spark单机版

    Last Updated:Sep 11, 2026

    In this tutorial, you will have a standalone Spark environment for data processing, interactive queries, or machine learning using a public template of ROS.

    Overview

    What is Apache Spark

    Apache Spark is a general-purpose computing engine designed for large-scale data processing. Spark uses Scala as its application framework and enables in-memory distributed datasets, which support interactive queries and iterative workloads.

    The template installs the following software versions:

    Component

    Description

    Version

    JDK

    Java Development Kit, the Java runtime environment

    1.8.0

    Hadoop

    Distributed system infrastructure that provides HDFS distributed storage

    2.7.7

    Scala

    Programming language on which the Spark core framework is built

    2.12.1

    Spark

    General-purpose big data computing engine that supports SQL queries, stream processing, and machine learning

    2.1.0

    Deployment on an existing VPC

    This template uses an existing VPC, vSwitch, and security group, and does not create network resources. This design applies to the following scenarios:

    • You already run a standardized network environment and want to add a Spark service to the current VPC.

    • Multiple projects share one VPC, and duplicate network resources must be avoided.

    Resources created

    The template creates the following resources:

    No.

    Resource

    ROS resource type

    Description

    1

    ECS instance

    ALIYUN::ECS::Instance

    Compute node that runs Spark in standalone mode

    2

    Elastic IP address (EIP)

    ALIYUN::VPC::EIP

    Provides internet access to the ECS instance

    3

    EIP association

    ALIYUN::VPC::EIPAssociation

    Associates the EIP with the ECS instance

    Deployment results

    After the template runs, you have the following results:

    • One ECS instance that runs CentOS 7 and is associated with an EIP.

    • JDK 1.8.0, Hadoop 2.7.7, Scala 2.12.1, and Spark 2.1.0 installed on the ECS instance.

    • The SparkWebSiteURL output of the stack, which opens the Spark web UI.

    Note

    To access the SparkWebSiteURL over the public network, add inbound rules for ports 8088 and 8080 in the security group. For more information, see Security groups.

    Architecture diagram

    安装 Spark 单机版

    Prerequisites

    Before you use this template, make sure that the following conditions are met:

    1. Account permissions: Your Alibaba Cloud account must have the permissions to create ECS, VPC, and EIP resources.

    2. Existing network resources: A VPC, a vSwitch, and a security group must already be created. For instructions, see VPCs and vSwitches and Create an instance.

    Deployment

    Deployment parameters

    This template uses existing VPC network resources. Specify all of the following parameters when you create the stack. The region that you select must be the same as the region where the existing VPC, vSwitch, and security group reside.

    Parameter

    Type

    Description

    Example

    Existing VPC ID

    String

    The ID of the VPC. You can obtain the ID as described in VPCs and vSwitches.

    vpc-bp1m6fww66xbntjyc****

    VSwitch Zone ID

    String

    The zone ID of the vSwitch in the VPC.

    China (Hangzhou) Zone K

    VSwitch ID

    String

    The ID of the vSwitch in the VPC. You can obtain the ID as described in VPCs and vSwitches.

    vsw-bp183p93qs667muql****

    Business security group ID

    String

    The ID of the security group for the ECS instance. You can obtain the ID as described in Security groups.

    sg-bp15ed6xe1yxeycg7o****

    Instance specification

    String

    Spark is memory-intensive, so select an instance type with at least 2 vCPUs and 4 GiB of memory.

    ecs.c5.large

    Image ID

    String

    The image of the ECS operating system. Default value: centos_7.

    centos_7

    Instance password

    String

    The logon password of the ECS instance. The password must be 8 to 30 characters in length and must contain uppercase letters, lowercase letters, digits, and special characters.

    -

    Public IP bandwidth

    Number

    The bandwidth of the public IP address. Valid values: 1 to 100. Unit: Mbit/s.

    5

    Disk type

    String

    The category of the system disk. Valid values:

    • cloud_efficiency (ultra disk),

    • cloud_ssd (standard SSD),

    • cloud_essd (ESSD)

    For more information, see Disks.

    cloud_efficiency

    System disk space

    Number

    The size of the system disk. Valid values: 40 to 500. Unit: GB.

    40

    Deployment methods

    You can deploy the template by using one of the following methods. Method 1 deploys the public template directly and is recommended. Method 2 deploys the same environment through ROS IaC Code.

    Method 1: Use the ROS public template (recommended)

    Create a stack from the public template in the ROS console:

    1. Log on to the ROS console.

    2. In the left-side navigation pane, choose Templates > Public Templates > Sample Templates.

    3. Enter Spark in the search box, find the Installs Spark on an ECS instance (existing VPC) template, and click Create Stack on the template card.

    4. On the Configure Parameters page, enter a Stack name and configure the parameters described.

    5. Click Next: Check and Confirm, and then click Create.

    6. In the left-side navigation pane, choose Deployment > Resource stack. On the Stacks page, click the ID of the stack that you created, and then check the stack status on the Stack Information tab. Wait until the status changes to CREATE_COMPLETE.

    7. Click the Output tab of the stack and obtain the value of SparkWebSiteURL.

    8. Open the URL in a browser. If you can log on to the Spark web UI, the deployment succeeds.

    Method 2: Use ROS IaC Code

    IaC Code is an AI infrastructure-as-code assistant for cloud infrastructure. It generates ROS templates from natural language descriptions and deploys them. For more information, see IaC Code quick start.

    # Prompt
Create an ECS instance in an existing VPC and deploy a standalone Spark development and test environment.
Requirements:
1. Use an existing VPC, vSwitch, and security group. Do not create network resources.
2. Create an EIP and associate it with the ECS instance.
3. Use a CentOS 7 image for the ECS instance.
4. Use an instance user data script to install JDK 1.8, Hadoop 2.7.7, Scala 2.12.1, and Spark 2.1.0.
5. Configure the JAVA_HOME, HADOOP_HOME, SCALA_HOME, and SPARK_HOME environment variables.
6. Start the Spark service and return SparkWebSiteURL as an output.
7. Allow inbound traffic on ports 8088 and 8080 in the security group.

    Post-deployment operations

    Verify the Spark environment

    1. On the Output tab of the stack, obtain the value of SparkWebSiteURL.

    2. Open the URL in a browser. If the Spark web UI appears, the deployment succeeded.

    3. Connect to the ECS instance and run the following commands to verify the environment:

    java -version          # Check the JDK version
hadoop version         # Check the Hadoop version
scala -version         # Check the Scala version
spark-shell --version  # Check the Spark version

    FAQ

    The deployment fails with the error "The specified InstanceType is not available"

    Cause: the selected ECS instance type is out of stock in the specified zone.

    Solutions:

    • Switch the vSwitch to another zone and deploy the stack again.

    • Check which instance types are available in the target zone on the ECS pricing page.

    • Spark is memory-intensive. Select an instance type with at least 2 vCPUs and 4 GiB of memory.

    The deployment succeeds, but SparkWebSiteURL is not accessible

    Troubleshoot the issue in the following order:

    1. Make sure that the inbound rules of the security group allow traffic on TCP ports 8088 and 8080. For instructions, see Security groups.

    2. Visit the ECS console - instances, Go to the instance details page and confirm that the Elastic IP address has been successfully bound to the instance..

    3. Connect to the ECS instance and check the Spark process and the port listening status.

      # Check the Spark process
ps -ef | grep spark
# Check whether port 8080 is listening
netstat -tlnp | grep 8080
# Start Spark manually if it is not running
$SPARK_HOME/sbin/start-all.sh

    4. If the Spark installation is not complete, run cat /var/log/messages | grep cloud-init to check the execution log of the instance user data (UserData) script and locate the failure cause.

    How do I deploy Spark in cluster mode?

    Resolution: ROS provides a public template for Spark in cluster mode, which creates a multi-node Spark cluster.

    1. Log on to the ROS console.In the left-side navigation pane, choose Templates > Public Templates > Sample Templates. Find Spark Cluster Edition (Existing VPC).

    2. This template will create multiple ECS instances (1 Master + N Workers) and automatically configure cluster communication.

    3. Before deployment, it is recommended to delete the standalone edition resource stack first to avoid resource conflicts.

    To learn more about resource deployment issues, see the FAQ.

    References