Resource scheduling is used to execute/distribute the jobs dispatched by the scheduling system. The resource scheduling in Data IDE Kit can be divided into default resource scheduing and user-defined resource scheduling. The default resource scheduling cannot be modified, and user-defined resource scheduling means that the user has purchased ECS on his/her own and has configured the ECS into the scheduling server for executing/distributing jobs.The organization administrator can create new user-defined resource scheduling for executing data synchronization jobs and shell jobs, or distributing ODPS SQL jobs, OPEN MR jobs, and machine learning jobs.
Jobs are by default all executed/distributed in the default resource scheduling group. You can modify the job execution/distribution resource scheduling group in the “O&M Center - Manage Jobs (List) - Modify Resource Groups“.
When can I add resource scheduling?
- When a large number of jobs are in the “Waiting for resources” status, and the number of jobs of the current project has reached a certain threshold, and existing resource scheduling cannot meet the operational requirements any more and scheduling resource expansion is required.
- Some special jobs (for example, some shell scripts) of the project need to be executed on a specific machine. At this time, you must apply for your own scheduling resources to execute the job.
- [Note] One data synchronization job will occupy ten slots (that is, ten concurrency units), but the default resource group has limited resources. Therefore, during data production, data synchronization jobs are usually executed by self-built resource groups to ensure that the job won’t always wait for resources.
Resource schedules may contain several physical machines or ECS virtual machines for executing job schedules. The project administrator can create and modify resource schedules in “Project Configuration - Manage Resource Scheduling”. When the response performance of the default resource scheduling cannot meet your requirements, you can purchase ECS and configure it into your resource schedule to improve the execution efficiency of your scheduled jobs.
Concepts of listed items:
Resource name: The name of the scheduled resource group. It may consist of English letters, underscores, and numbers, and should not exceed 60 characters in length. Once created, the name cannot be modified.
Network type: The network type used by the ECS server added for resource scheduling. The type may involve VPC and classic networks;
Server: The name of the server contained in the currently scheduled resources;
Default resource scheduling: The default resource scheduling is a tag to mark whether the current resource scheduling is the default resource scheduling. By default, scheduled jobs are submitted to this resource group. Only one default resource scheduling is allowed in a project.
Initialize server: After an ECS server is configured, you should modify the configuration as prompted, in the interface, as the admin of the corresponding ECS. For detailed operations, see "New Scheduling Resource" in the next section;
Modify server: You can modify the server configuration in the current resource schedule, including adding/deleting a server and modifying the maximum number of concurrent jobs of the server;
Modify owner project: You can allocate the currently scheduled resources to a specified project so that only the specified project can use the scheduled resources. This action can only be performed by the primary account that activated the service.
Modify default resources: You can change the default resource schedule and specify any resource schedules under the current project, as the default resource schedule. The change in the default resource scheduling will influence all jobs. Please perform this action with caution.
New resource scheduling: You can add a new resource scheduling group. For detailed operations, see "New Resource Scheduling" in the next section.
- Classic network: IP addresses are allocated in a unified manner by Alibaba Cloud, featuring easy configuration and convenient use. This network type is suitable for users who require high ease-of-use of operations and need to use ECS quickly ;
- VPC: A logically isolated private network. You can customize network topologies and IP addresses. VPC supports private lines. VPC is suitable for users familiar with network management.
- We recommend CentOS6, CentOS7, AliyunOS ;
- If the ECS you added needs to execute ODPS jobs or synchronization jobs, you should check whether the Current ECS Python is Python 2.6.5 or newer (CentOS5 has Python 2.4, and other OS’s are nested with Python 2.6 or newer) ;
- Please ensure that the ECS has a public IP address ;
- We recommend the ECS memory to be exceeds 4GB ;
- ECS added by the user can only execute ODPS_SQL, MR, shell, synchronization jobs and other job types are not supported .
View the ECS host name and IP address ;
If you didn’t change the machine name when you purchased the ECS, the host name and IP address are shown in the screenshot below; if you have changed the machine name, please run the hostname and hostname -i commands on the ECS to get the host name and IP address .
The project administrator should enter “Project Configuration - Manage Scheduling Resources” page, click “New Resource Schedule” (or select a resource scheduling and click “Manage Servers - Modify”) to make the purchased ECS as the server and add it to the resource group ;
- When you want to make an ECS in a VPC as the server, you should fill in the ECS UUID as the server name. How to get the UUID: Log in to the ECS and run the command: dmidecode | grep UUID ;
- For example, if you run dmidecode | grep UUID, the result returned will be the UUID: 713F4718-8446-4433-A8EC-6B5B62D75A24. In this case, the UUID is: 713F4718-8446-4433-A8EC-6B5B62D75A24 .
The project administrator should enter “Project Configuration - Manage Resource Scheduling” and select a machine, then click “Initialize Server” and perform operations as prompted:
Log in to the ECS [server IP address] with the root permission (this server is the ECS you want to make operations on, and run ssh root@[service IP address], such as: ssh firstname.lastname@example.org;
Run the command (this command line is prompted on the copy interface and executed): ‘wget https://alisaproxy.shuju.aliyun.com/install.sh’;
Run the command (this command line is prompted on the copy interface and executed): ‘sh install.sh --user_name=zz_[the unique identifier of the resource schedule] --password=[AK password] --enable_uuid=false’;
Click the Fresh button on the New Server page after a while (around 15 seconds), and check whether the service status changes to “Normal”. If yes, it indicates that the newly created ECS has been successfully registered.