ALIYUN::EMR::Cluster is used to create an E-MapReduce (EMR) cluster.

Syntax

{
  "Type": "ALIYUN::EMR::Cluster",
  "Properties": {
    "SshEnable": Boolean,
    "EasEnable": Boolean,
    "WhiteListType": String,
    "InitCustomHiveMetaDB": Boolean,
    "IoOptimized": Boolean,
    "HostGroup": List,
    "Config": List,
    "KeyPairName": String,
    "VpcId": String,
    "AutoRenew": Boolean,
    "RelatedClusterId": String,
    "BootstrapAction": List,
    "InstanceGeneration": String,
    "DepositType": String,
    "VSwitchId": String,
    "NetType": String,
    "UserDefinedEmrEcsRole": String,
    "Name": String,
    "ClusterType": String,
    "ZoneId": String,
    "IsOpenPublicIp": Boolean,
    "OptionSoftWareList": List,
    "Configurations": String,
    "MasterPwd": String,
    "MachineType": String,
    "EmrVer": String,
    "SecurityGroupName": String,
    "MetaStoreConf": String,
    "SecurityGroupId": String,
    "LogPath": String,
    "Period": Integer,
    "HighAvailabilityEnable": Boolean,
    "UseCustomHiveMetaDB": Boolean,
    "UserInfo": List,
    "ChargeType": String,
    "MetaStoreType": String,
    "AuthorizeContent": String,
    "UseLocalMetaDb": Boolean,
    "ClickHouseConf": Map,
    "ResourceGroupId": String,
    "Tags": List
  }
}

Properties

Property Type Required Editable Description Constraint
SshEnable Boolean No No Specifies whether to enable SSH. Valid values:
  • true
  • false
EasEnable Boolean No No Specifies whether to create a high-security cluster. Valid values:
  • true
  • false
WhiteListType String No No The type of the whitelist. Valid values:
  • IP: IP address
  • SecurityGroup: security group
InitCustomHiveMetaDB Boolean No No A reserved property. You do not need to specify this property. None
IoOptimized Boolean No No Specifies whether to enable I/O optimization. Default value: true. Valid values:
  • true
  • false
HostGroup List Yes No Details of the node group. For more information, see HostGroup properties.
Config List No No Details of the custom configuration items. For more information, see Config properties.
KeyPairName String No No The name of the key pair. None
VpcId String No No The ID of the virtual private cloud (VPC). None
AutoRenew Boolean No No Specifies whether to enable auto-renewal for the subscription cluster. Valid values:
  • true
  • false
RelatedClusterId String No No The ID of the EMR cluster that is associated with the gateway cluster. This property takes effect when you set the ClusterType property to GATEWAY.
BootstrapAction List No No Details of the bootstrap action that you want to configure for the cluster. For more information, see BootstrapAction properties.
InstanceGeneration String No No The family of the Elastic Compute Service (ECS) instance. None
VSwitchId String No No The ID of the vSwitch. None
NetType String Yes No The type of the network. Set the value to VPC.
UserDefinedEmrEcsRole String No No The ECS application role that allows internal access from ECS to other Alibaba Cloud services, such as Object Storage Service (OSS). None
Name String Yes Yes The name of the cluster. The name must be 1 to 64 characters in length, and can contain letters, digits, hyphens (-), and underscores (_).
ClusterType String Yes No The type of the cluster. Valid values:
  • HADOOP
  • KAFKA
  • DRUID
  • ZOOKEEPER
  • DATA_SCIENCE
  • GATEWAY
ZoneId String Yes No The ID of the zone. None
IsOpenPublicIp Boolean No No Specifies whether to use public IP addresses. Valid values:
  • true: uses public IP addresses. If you use public IP addresses, the default bandwidth is 8 Mbit/s.
  • false: does not use public IP addresses.
OptionSoftWareList List No No The list of available software. None
Configurations String No No A reserved property. You do not need to specify this property. None
MasterPwd String No No The SSH password that is used to access the master node. The password must be 8 to 30 characters in length. The password must contain at least three of the following character types: uppercase letters, lowercase letters, digits, and special characters.
MachineType String No No The type of the node. None
EmrVer String Yes No The version of EMR. None
SecurityGroupName String No No The name of the security group. If you do not specify the SecurityGroupId property, the value of the SecurityGroupName property is used as the name of the security group that you create.

After the cluster is created, you can query the ID of the security group on the cluster details page. The default security group policy is applied to the security group. The default policy allows inbound traffic only on port 22 and outbound traffic on all ports.

DepositType String No No The hosting type of the cluster. None
SecurityGroupId String No No The ID of the security group. If you want to use an existing security group, the default security group policy is applied to the security group. The default policy allows inbound traffic only on port 22 and outbound traffic on all ports.
LogPath String No No The OSS path in which you want to store EMR logs. None
Period Integer No No The subscription duration of the cluster. This property is required when you set the ChargeType property to PrePaid.

Valid values: 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 24, and 36

Unit: months.

HighAvailabilityEnable Boolean No No Specifies whether to enable high availability for the cluster. Valid values:
  • true: enables high availability. If you want to enable high availability for a cluster, make sure that the cluster contains at least two master nodes.
  • false: disables high availability.
UseCustomHiveMetaDB Boolean No No A reserved property. You do not need to specify this property. None
UserInfo List No No The information about the Knox account. For more information, see UserInfo properties.
ChargeType String Yes No The billing method of the cluster. Valid values:
  • PostPaid
  • PrePaid
AuthorizeContent String No No A reserved property. You do not need to specify this property. None
UseLocalMetaDb Boolean Yes No Specifies whether to use the built-in MySQL database of the cluster as the Hive metadatabase. Valid values:
  • true
  • false
MetaStoreConf String No No The configurations of the metadata. This property is required when you set the MetaStoreType property to user_rds.

Specify the MetaStoreConf property in the following format: {"dbUrl":"jdbc:mysql://xxxxxx", "dbUserName":"username", "dbPassword":"password"}.

MetaStoreType String No No The type of the metadata. Valid values:
  • local: the built-in MySQL database of the cluster
  • dlf: the Data Lake Formation (DLF) metadata service
  • user_rds: the self-managed ApsaraDB RDS service
ClickHouseConf Map No No The configurations of the ApsaraDB for ClickHouse cluster. None
ResourceGroupId String No Yes The ID of the resource group. None
Tags List No Yes The tags that you want to add to the cluster. For more information, see Tags properties.

Tags syntax

"Tags": [
  {
    "Value": String,
    "Key": String
  }
]

Tags properties

Property Type Required Editable Description Constraint
Key String Yes No The key of the tag. The tag key must be 1 to 128 characters in length, and cannot contain http:// or https://. The tag key cannot start with aliyun or acs:.
Value String No No The value of the tag. The tag value can be up to 128 characters in length, and cannot contain http:// or https://. The tag value cannot start with aliyun or acs:.

HostGroup syntax

"HostGroup": [
  {
    "Comment": String,
    "SysDiskType": String,
    "DiskCapacity": Integer,
    "NodeCount": Integer,
    "ClusterId": String,
    "DiskCount": Integer,
    "CreateType": String,
    "DiskType": String,
    "AutoRenew": Boolean,
    "HostGroupType": String,
    "SysDiskCapacity": Integer,
    "VSwitchId": String,
    "ChargeType": String,
    "Period": Integer,
    "HostKeyPairName": String,
    "HostPassword": String,
    "HostGroupId": String,
    "InstanceType": String,
    "GpuDriver": String,
    "HostGroupName": String
  }
]

HostGroup properties

Property Type Required Editable Description Constraint
Comment String No No A reserved property. You do not need to specify this property. None
SysDiskType String Yes No The category of the system disk that you want to attach to the node group. Valid values:
  • CLOUD_EFFICIENCY: ultra disk
  • CLOUD_SSD: standard SSD
  • CLOUD: basic disk
DiskCapacity Integer Yes No The capacity of the data disk that you want to attach to the node group. Unit: GB.
NodeCount Integer Yes No The number of nodes in the node group. None
ClusterId String No No A reserved property. You do not need to specify this property. None
DiskCount Integer Yes No The number of data disks that you want to attach to the node group. None
CreateType String No No The creation type of the node group. Valid values:
  • ON-DEMAND: on-demand creation
  • MANUAL: manual creation
DiskType String Yes No The category of the data disk that you want to attach to the node group. Valid values:
  • CLOUD_EFFICIENCY: ultra disk.
  • CLOUD_SSD: standard SSD.
  • LOCAL_DISK: local disk. If you use an ECS instance, you must set the DiskType property to LOCAL_DISK.
  • CLOUD: basic disk.
AutoRenew Boolean No No Specifies whether to enable auto-renewal for the subscription cluster. Valid values:
  • true
  • false
HostGroupType String Yes No The type of the node group. Valid values:
  • MASTER: master node group
  • CORE: core node group
  • TASK: task node group
SysDiskCapacity Integer Yes No The capacity of the system disk that you want to attach to the node group. Unit: GB.
VSwitchId String No No The vSwitch ID of the node group. None
ChargeType String Yes No The billing method of the node group. Valid values:
  • PostPaid
  • PrePaid
Period Integer No No The subscription duration of the node group. This property is required when you set the ChargeType property to PrePaid.

Valid values: 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 24, and 36

Unit: months.

HostKeyPairName String No No The name of the key pair that is used to access the node group. This property takes effect when you set the ClusterType property to GATEWAY.
HostPassword String No No The password that is used to access the node. This property takes effect when you set the ClusterType property to GATEWAY.
HostGroupId String No No A reserved property. You do not need to specify this property. None
InstanceType String Yes No The type of the instance. None
GpuDriver String No No The GPU driver. None
HostGroupName String No No The name of the node group. None

Config syntax

"Config": [
  {
    "Encrypt": String,
    "ConfigKey": String,
    "FileName": String,
    "ServiceName": String,
    "Replace": String,
    "ConfigValue": String
  }
]

Config properties

Property Type Required Editable Description Constraint
Encrypt String No No A reserved property. You do not need to specify this property. None
ConfigKey String No No The name of the custom configuration item. None
FileName String No No The name of the file that contains the custom configuration item. None
ServiceName String No No The name of the service to which the custom configuration item belongs. None
Replace String No No A reserved property. You do not need to specify this property. None
ConfigValue String No No The value of the custom configuration item. None

BootstrapAction syntax

"BootstrapAction": [
  {
    "Path": String,
    "Name": String,
    "Arg": String
  }
]

BootstrapAction properties

Property Type Required Editable Description Constraint
Path String No No The OSS path in which you want to store the bootstrap action script. None
Name String No No The name of the bootstrap action. None
Arg String No No The parameter of the bootstrap action. None

UserInfo syntax

"UserInfo": [
  {
    "UserName": String,
    "Password": String,
    "UserId": String
  }
]

UserInfo properties

Property Type Required Editable Description Constraint
UserName String No No The username of the Knox account. None
Password String No No The password of the Knox account. None
UserId String No No The RAM user ID of the Knox account. None

Return values

Fn::GetAtt

  • ClusterId: the ID of the cluster.
  • HostGroups: the list of node groups in the cluster.
  • MasterNodePubIps: the list of public IP addresses of master nodes in the cluster.
  • MasterNodeInnerIps: the list of private IP addresses of master nodes in the cluster.

Examples

{
  "ROSTemplateFormatVersion": "2015-09-01",
  "Resources": {
    "Cluster": {
      "Type": "ALIYUN::EMR::Cluster",
      "Properties": {
        "SshEnable": {
          "Ref": "SshEnable"
        },
        "EasEnable": {
          "Ref": "EasEnable"
        },
        "WhiteListType": {
          "Ref": "WhiteListType"
        },
        "InitCustomHiveMetaDB": {
          "Ref": "InitCustomHiveMetaDB"
        },
        "IoOptimized": {
          "Ref": "IoOptimized"
        },
        "HighAvailabilityEnable": {
          "Ref": "HighAvailabilityEnable"
        },
        "Config": {
          "Ref": "Config"
        },
        "KeyPairName": {
          "Ref": "KeyPairName"
        },
        "VpcId": {
          "Ref": "VpcId"
        },
        "AutoRenew": {
          "Ref": "AutoRenew"
        },
        "RelatedClusterId": {
          "Ref": "RelatedClusterId"
        },
        "BootstrapAction": {
          "Ref": "BootstrapAction"
        },
        "InstanceGeneration": {
          "Ref": "InstanceGeneration"
        },
        "VSwitchId": {
          "Ref": "VSwitchId"
        },
        "NetType": {
          "Ref": "NetType"
        },
        "Name": {
          "Ref": "Name"
        },
        "ClusterType": {
          "Ref": "ClusterType"
        },
        "ZoneId": {
          "Ref": "ZoneId"
        },
        "IsOpenPublicIp": {
          "Ref": "IsOpenPublicIp"
        },
        "OptionSoftWareList": {
          "Ref": "OptionSoftWareList"
        },
        "Configurations": {
          "Ref": "Configurations"
        },
        "MasterPwd": {
          "Ref": "MasterPwd"
        },
        "MachineType": {
          "Ref": "MachineType"
        },
        "EmrVer": {
          "Ref": "EmrVer"
        },
        "SecurityGroupName": {
          "Ref": "SecurityGroupName"
        },
        "DepositType": {
          "Ref": "DepositType"
        },
        "HostGroup": {
          "Ref": "HostGroup"
        },
        "SecurityGroupId": {
          "Ref": "SecurityGroupId"
        },
        "LogPath": {
          "Ref": "LogPath"
        },
        "Period": {
          "Ref": "Period"
        },
        "UserDefinedEmrEcsRole": {
          "Ref": "UserDefinedEmrEcsRole"
        },
        "UseCustomHiveMetaDB": {
          "Ref": "UseCustomHiveMetaDB"
        },
        "UserInfo": {
          "Ref": "UserInfo"
        },
        "ChargeType": {
          "Ref": "ChargeType"
        },
        "AuthorizeContent": {
          "Ref": "AuthorizeContent"
        },
        "UseLocalMetaDb": {
          "Ref": "UseLocalMetaDb"
        }
      }
    }
  },
  "Parameters": {
    "SshEnable": {
      "Type": "Boolean",
      "Description": "Indicates whether SSH is enabled.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    },
    "EasEnable": {
      "Type": "Boolean",
      "Description": "Indicates whether the cluster is a high-security cluster.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    },
    "WhiteListType": {
      "Type": "String",
      "Description": "Not required."
    },
    "InitCustomHiveMetaDB": {
      "Type": "Boolean",
      "Description": "A reserved parameter. Not required.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    },
    "IoOptimized": {
      "Default": true,
      "Type": "Boolean",
      "Description": "Indicates wether I/O optimization is enabled. Default value: true.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    },
    "HighAvailabilityEnable": {
      "Type": "Boolean",
      "Description": "Indicates whether the cluster is a high-availability cluster. A value of true indicates\nthat two master nodes are required.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    },
    "Config": {
      "Type": "Json",
      "Description": ""
    },
    "KeyPairName": {
      "Type": "String",
      "Description": "The name of the key pair."
    },
    "VpcId": {
      "Type": "String",
      "Description": "The ID of the VPC. A value is required when NetType=vpc."
    },
    "AutoRenew": {
      "Type": "Boolean",
      "Description": "Indicates whether the subscription cluster is auto-renewed.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    },
    "RelatedClusterId": {
      "Type": "String",
      "Description": "The ID of the primary cluster (when the cluster that you create is a Gateway cluster)."
    },
    "BootstrapAction": {
      "Type": "Json",
      "Description": ""
    },
    "InstanceGeneration": {
      "Type": "String",
      "Description": "The generation of the ECS instances."
    },
    "VSwitchId": {
      "Type": "String",
      "Description": "The ID of the Vswitch. A value is required when NetType=vpc."
    },
    "NetType": {
      "Type": "String",
      "Description": "The type of the network."
    },
    "Name": {
      "Type": "String",
      "Description": "The name of the cluster. The name can be 1 to 64 characters in length and only contain\nChinese characters, letters, numbers, hyphens (-), and underscores (_)."
    },
    "ClusterType": {
      "Type": "String",
      "Description": "The type of the cluster. Allowd values: HADOOP, KAFKA, DRUID, ZOOKEEPER, DATA_SCIENCE, GATEWAY.",
      "AllowedValues": [
        "HADOOP",
        "KAFKA",
        "DRUID",
        "ZOOKEEPER",
        "DATA_SCIENCE",
        "GATEWAY"
      ]
    },
    "ZoneId": {
      "Type": "String",
      "Description": "The zone ID."
    },
    "IsOpenPublicIp": {
      "Type": "Boolean",
      "Description": "Indicates whether a public IP address is assigned. A value of true indicates that\na bandwidth of 8 MB is set by default.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    },
    "OptionSoftWareList": {
      "Type": "Json",
      "Description": "The list of optional services."
    },
    "Configurations": {
      "Type": "String",
      "Description": "Not required."
    },
    "MasterPwd": {
      "MinLength": 8,
      "Type": "String",
      "Description": "The SSH password for the master node. The password must meet the following requirements.\nLength constraints: Minimum length of 8 characters. Maximum length of 30 characters.\nIt must contain three types of characters (uppercase letters, lowercase letters, numbers,\nand special symbols).",
      "MaxLength": 30
    },
    "MachineType": {
      "Type": "String",
      "Description": "The type of the machine."
    },
    "EmrVer": {
      "Type": "String",
      "Description": "The version of EMR."
    },
    "SecurityGroupName": {
      "Type": "String",
      "Description": "The name of the security group to create. If the ID of the security group is not specified,\nthis name is used to create a new security group. After the cluster is created, you\ncan view the ID of the security group on the Cluster Management page. The default\nsecurity group policy is applied to this security group: Only port 22 is open at the\ninbound and all ports are open at the outbound. You need to specify either SecurityGroupId\nor SecurityGroupName."
    },
    "DepositType": {
      "Type": "String",
      "Description": "The hosting type."
    },
    "HostGroup": {
      "Type": "Json",
      "Description": ""
    },
    "SecurityGroupId": {
      "Type": "String",
      "Description": "The ID of the security group. You can create a security group in the ECS console and\nuse it. Note: If you use an existing security group, the default security group policy\nis applied to this security group: Only port 22 is open at the inbound and all ports\nare open at the outbound. You need to specify either SecurityGroupId or SecurityGroupName."
    },
    "LogPath": {
      "Type": "String",
      "Description": "The log path in OSS."
    },
    "Period": {
      "Type": "Number",
      "Description": "The length of the subscription. Unit: months. Valid values: 1, 2, 3, 4, 5, 6, 7, 8,\n9, 12, 24, and 36. A value is required when ChargeType=PrePaid.",
      "AllowedValues": [
        1,
        2,
        3,
        4,
        5,
        6,
        7,
        8,
        9,
        12,
        24,
        36
      ]
    },
    "UserDefinedEmrEcsRole": {
      "Type": "String",
      "Description": "The role that is assigned to EMR for calling ECS resources."
    },
    "UseCustomHiveMetaDB": {
      "Type": "Boolean",
      "Description": "A reserved parameter. Not required.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    },
    "UserInfo": {
      "Type": "Json",
      "Description": ""
    },
    "ChargeType": {
      "Type": "String",
      "Description": "The billing method. Valid values: PostPaid and PrePaid. PostPaid: pay-as-you-go. PrePaid:\nsubscription.",
      "AllowedValues": [
        "PostPaid",
        "PrePaid"
      ]
    },
    "AuthorizeContent": {
      "Type": "String",
      "Description": "Not required."
    },
    "UseLocalMetaDb": {
      "Type": "Boolean",
      "Description": "Indicates whether the local Hive metadatabase is used.",
      "AllowedValues": [
        "True",
        "true",
        "False",
        "false"
      ]
    }
  },
  "Outputs": {
    "ClusterId": {
      "Description": "The ID of the cluster.",
      "Value": {
        "Fn::GetAtt": [
          "Cluster",
          "ClusterId"
        ]
      }
    }
  }
}

For more examples, visit Cluster.json and Cluster.yml.