edit-icon download-icon

Add scheduling resources

Last Updated: Apr 24, 2018

Project administrator can create and modify scheduling resources on the Resource page of the DataWorks console. When the default scheduling resource cannot connect to your complex network environment, the data integration provides a powerful capability to connect networks in complex network environment. The data integration uses Agents to open data transmission synchronization between any network environments. For more information, see Database data sync (without public network IP) and Data sync when the network of data source (both sides) is disconnected.

Agent deployment of data integration

Purchase ECS instances

For more information, see Purchase ECS instances.

Note:

  • We recommend using CentOS6, CentOS7, and AliyunOS.

  • If the added ECS needs to run MaxCompute or synchronization tasks, verify whether the current Python version of ECS is later than 2.6.5. (The Python version of CentOS5 is 2.4 while that of other operating systems is later than 2.6.)

  • Make sure that a public IP is available to ECS.

  • 8 CPU cores and 16 GB memory are recommended for ECS.

View the hostname and intranet IP address of ECS

  1. Log on to Alibaba Cloud account.
  2. Go to ECS console, select Instances to view the hostname and IP address.

The instance ID is the hostname.

Activate port 8000 to read logs

Note:
You can skip the configuration for a VPC network.

Follow these steps to add security group rules:

  1. Log on to Alibaba Cloud account.
  2. Select ECS console > Instances > Manage > Security Groups and click Configure Rules to enter the rule configuration page.

  3. Click Add Security Group Rule to add intranet inbound rules.

    AddSecurityGroupRule

  4. Set the Port Range as 8000, and set the Authorization Object as 10.116.134.123.

Add the scheduling resource

  1. Log on to Alibaba Cloud account.
  2. Go to the DataWorks console > Enter Project > Resources > Resource page as the project administrator.
  3. Click additional resources group and enter a name for the added resource group.

  4. Click Next to add the purchased ECS to a resource group.

    • Network type

      • Classic network: The IP address is allocated uniformly by Alibaba Cloud. It is easy to configure and convenient to use. It is suitable for ECS users with a high operational usability.

      • VPC network: A proprietary network by using logical isolation. You can customize the network topology and IP address, and support the connection through the private line. It is suitable for users who are familiar with the network management.

    • Servers

      • Classic network: Log on to ECS, and run the hostname command to take the return value.

      • VPC network: Log on to ECS, and run the dmidecode | grep UUID command to take the return value.

    • Maximum concurrency

      • Concurrency calculation: The number of concurrency is calculated based on the number of CPU cores and the size of memory.

      • Add server: The content is related to the network type in preceding selection. If you choose the classic network, you can only add a classic network; if you choose the VPC network, you can view the network information about the VPC environment.

    Note:

    • If you enter an ECS on the VPC as the server, enter the UUID of ECS as the server name. To retrieve the UUID, log on to ECS and run the following command: dmidecode | grep UUID.

    • For example, when you run dmidecode | grep UUID, the following result is returned: UUID: 713F4718-8446-4433-A8EC-6B5B62D75A24. It means the UUID is 713F4718-8446-4433-A8EC-6B5B62D75A24.

  5. Click Next to install the agent.

    When you add a new machine, enter the following commands.

    1. Use SSH to log on to ECS as a root user.

    2. Run the following command to install the agent: wget https://alisaproxy.shuju.aliyun.com/install.sh --no-check-certificate.

    3. Run the following command to enter the username and password: sh install.sh --user_name=xxxxxxxxxx19d --password=yyyyyygh1bm --enable_uuid=false.

    4. Refresh the page to check whether the agent is installed successfully.

    5. Open the server port (Port: 8000).

Note:
If you encounter an error in running install.sh, run rm –rf install.sh in the directory where install.sh resides to delete generated files, and run install.sh again.

If status of the agent is still unavailable, you may not bind the host. In this case, follow these steps:

  1. Switch to the admin user.

  2. Run the following command to check the host binding status: hostname -i.

  3. Run the following command to add IP address and hostname: vim/etc/hosts.

  4. Refresh the page. If the information is displayed normally, then the registration of ECS machine is successful.

Note:

  • If the machine is still in unavailable state, you can restart the alisa by using the following commands. Switch to admin user, and run /home/admin/alisatasknode/target/alisatasknode/bin/serverct1 restart.
  • The command contains your AccessKey information. Therefore, you must not share or reveal this information to others.
Thank you! We've received your feedback.