All Products
Search
Document Center

create_job

Last Updated: May 11, 2018

Parameter description

All classes of parameters are converted to dictionary objects containing the attribute information.

Parameter Type Description
job_desc JobDescription object, str, dict Simple description of the job, description of each task of the job object, and DAG dependencies between tasks

Response description

The create_job method returns a CreateResponse object. The following lists the attributes of the CreateResponse object. You can use response.Id to obtain the ID of the new job.

Attribute Type Description
Id str ID of the new job

For example:

  1. try:
  2. # Init a Client object.
  3. client = ...
  4. # Refer to JobDescription.
  5. job_desc = JobDescription(...)
  6. job = client.create_job(job_desc)
  7. # Print out the job id.
  8. print (job.Id)
  9. except ClientError, e:
  10. print (e)

JobDescription class

Parameter description

Parameter Type Description
properties dict, str, JobDescription object Object that contains the job description

Attribute description

Attribute Type Description
Name str Job name
Description str Brief description of a job
Priority int The value is an integer ranging from 0 to 1000. Priority increases with the value
Notification dict Notification configuration. You can configure the topic or job related events of the MNS
JobFailOnInstanceFail bool Specifies whether the failure of an instance will cause the failure of the related job
AutoRelease boolean Specifies whether the system automatically releases (deletes) a job after the job runs successfully. Default value: False
Type str Currently, a task can be described only in the form of a directed acyclic graph (DAG)
DAG dict, DAG object DAG description

2. DAG class

Parameter description

Parameter Type Description
properties dict, str, DAG object Descriptions of task mapping and dependencies between all tasks

Attribute description

Attribute Type Description
Tasks dict Mapping between all tasks names and task descriptions
Dependencies dict Dependencies between all tasks

Method description

Method Description
add_task(task_name, Adds a task
get_task(task_name) Obtains information about a task, based on the task name
delete_task(task_name) Deletes a task

For example:

  1. # Refer to the TaskDescription class.
  2. find_task = TaskDescription(...)
  3. task_dag = DAG()
  4. task_dag.add_task(task_name='Find', task=find_task)
  5. task_dag.Dependencies = {}

3. TaskDescription class

Parameter description

Parameter Type Description
properties dict, str, TaskDescription object Description of a single task

Attribute description

Attribute Type Description
Parameters dict, Parameters object Task parameter details
InputMapping dict Mapping from the OSS directory to the local directory
OutputMapping dict Mapping from the local directory to the OSS directory
LogMapping dict Mapping from the local log directory to the OSS directory
Timeout int Task timeout
InstanceCount int Number of instances in a task. It is a positive integer
MaxRetryCount int Maximum number of retries. Default value: 0
ClusterId str Cluster ID
Mounts dict, Mounts object Network mounting configurations of an instance, which is described by Mounts. Currently, NAS mounting and OSS mounting are supported
AutoCluster dict, AutoCluster object You can select only one of autoCluster and clusterId

For example:

  1. echo_task = TaskDescription()
  2. # Create map task.
  3. # Refer to Parameters documentation.
  4. echo_task.Parameters.Command.CommandLine = "echo Batchcompute Service"
  5. echo_task.Parameters.Command.PackagePath = ""
  6. echo_task.Parameters.StdoutRedirectPath = "oss://xxx/xxx/"
  7. echo_task.Parameters.StderrRedirectPath = "oss://xxx/xxx/"
  8. echo_task.Timeout = 86400 # 1-day
  9. echo_task.InstanceCount = 3
  10. echo_task.MaxRetryCount = 1
  11. # cluster_id is a cluster created in batchcompute service before.
  12. echo_task.ClusterId = cluster_id
  13. # You can now add this task description to a DAG object through its add_task method.

4. Parameters class

Parameter description

Parameter Type Description
properties dict, str, Parameters object Description of task parameters

Attribute description

Attribute Type Description
Command dict, Command object Command line parameters related to a user program
InputMappingConfig dict, InputMappingConfig object Configuration of the NFS mounting service
StdoutRedirectPath str OSS directory to which standard output is redirected
StderrRedirectPath str OSS directory to which a standard error is redirected

5. AutoCluster class

Parameter description

Parameter Type Description
properties dict, str, AutoCluster object AutoCluster information

Attribute description

Attribute Type Description
ECSImageId str ECS image ID. Images provided by the system can be used.
InstanceType str Instance type
ResourceType str Resource type. Currently, the OnDemand and Spot types are supported. Default value: OnDemand
UserData dict Key-Value mapping, which is defined by the user and obtained by using metaserver of the ECS instance
Configs Configs object Cluster configurations. For more information, see the introduction to ClusterDescription in Section 4.13.
SpotStrategy str Spot strategy for an instance. It takes effect only when ResourceType is Spot.Values: SpotWithPriceLimit (Set the maximum price for the spot instance) and SpotAsPriceGo (The system provides a price automatically, and the maximum value is the Pay-As-You-Go price).
SpotPriceLimit float Maximum price of an instance per hour (the price of each instance type, not the price per core*hour). The value can contain at most three decimals. It takes effect when SpotStrategy is SpotWithPriceLimit.
ReserveOnFail bool When a task fails, related VMs are not released, and the fees for these resources are still charged until the user deletes the job. The default value is False. It is used only in a survey.

6. Command class

Parameter description

Parameter Type Description
properties dict, str, InputMappingConfig object Configuration of the NFS mounting service

Attribute description

Attribute Type Description
CommandLine str Specifies a command for executing a user program
PackagePath str OSS directory in which a user program resides
EnvVars dict Environment variable for execution of a user program

7. InputMappingConfig class

Parameter description

Parameter Type Description
properties dict, str, InputMappingConfig object Configuration of the NFS mounting service

Attribute description

Attribute Type Description
Locale str Character set used when an OSS object is mounted to the local server. Optional values include GBK, GB2312-80, BIG5, ANSI, EUC-JP, EUC-TW, EUC-KR, SHIFT-JIS, and KSC5601.
Lock bool Specifies whether the NFS mounting service supports the network file lock.

8. Notification class

Parameter description

Parameter Type Description
properties dict, str, Command object Command line parameters related to a user program

Attribute description

Attribute Type Description
Topic Topic Object Message topic

9. Topic class

Parameter description

Parameter Type Description
properties dict, str, Command object Command line parameters related to a user program

Attribute description

Attribute Type Description
Endpoint str Endpoint of the MNS region. The format is http://${your_user_id}.mns.${region}-internal.aliyuncs.com/. Use the intranet endpoint whenever possible.
Name str Topic name
Events list List of events. Make sure that you enter the cluster related events.

10. Mounts class

Parameter description

Parameter Type Description
properties dict, str, Command object Network disk mounting configurations when a cluster is created

Attribute description

Attribute Type Description
Entries array List of network disk mount points, which is described by MountPoint.
Locale str Language used when OSS or NAS storage is mounted
Lock bool Specifies whether the file lock is supported when OSS or NAS storage is mounted
NAS dict NAS configuration
OSS dict OSS configuration

11. MountPoint class

Parameter description

Parameter Type Description
properties dict, str, Command object Network mount point

Attribute description

Attribute Type Description
Source str Source directory for network disk mounting. It can be a string starting with “nas://“ or “oss://“.
Destination str Local mount point directory of the network disk
WriteSupport bool Specifies whether the mount point is writable.

12. NAS class

Parameter description

Parameter Type Description
properties dict, str, Command object NAS configuration

Attribute description

Attribute Type Description
AccessGroup list NAS access group to which the cluster instance must be added
FileSystem list File system to be accessed

13. OSS class

Parameter description

Parameter Type Description
properties dict, str, Command object OSS configuration

Attribute description

Attribute Type Description
AccessKeyId str AccessKey ID used for OSS mounting
AccessKeySecret str AccessKeySecret used for OSS mounting
SecurityToken str Security token used for OSS mounting

Note: Take the following precautions for Mounts:

  1. The Job-level Mounts configuration overwrites the cluster-level Mounts configuration.
  2. After a cluster is modified, the RecreateInstance interface must be called to make the new Mounts configuration take effect.
  3. During NAS mounting, the source directory must start with “nas:”. Otherwise, an error occurs.