Quick start for a standard job

Prepare the job

Objective: Count the occurrences of "INFO", "WARN", "ERROR", and "DEBUG" in a log file.

This job consists of three tasks: split, count, and merge.

• The split task divides the log file into three parts.

• The count task tallies the occurrences of "INFO", "WARN", "ERROR", and "DEBUG" in each part. With InstanceCount set to 3, three count tasks run concurrently.

• The merge task combines the results of the count tasks.

Upload the data file to OSS

Download the data file for this example: log-count-data.txt

Upload log-count-data.txt to oss://your-bucket/log-count/log-count-data.txt.

bcs oss upload ./log-count-data.txt oss://your-bucket/log-count/log-count-data.txt

bcs oss cat oss://your-bucket/log-count/log-count-data.txt  # Check if the upload was successful.

The bcs o command provides several common functions for OSS. Run bcs o -h to view help information. This command is convenient for testing with small amounts of data. However, we do not recommend using it to upload or download large datasets because it does not support multithreading, which results in slow transfers. For more information about how to upload data to OSS, see Common OSS Tools.

Prepare the task programs

The job programs in this example are written in Python. Download the program package for this example: log-count.tar.gz

To extract the package, run the following command:

mkdir log-count && tar -xvf log-count.tar.gz -C log-count

After extraction, the log-count/ directory has the following structure:

log-count
  |-- conf.py    # Configuration
  |-- split.py       # split task program
  |-- count.py      # count task program
  |-- merge.py    # merge task program

Note: You do not need to modify the programs.

Submit the job

Write the job configuration

In the parent directory of log-count, create a file named job.cfg with the following content.

[DEFAULT]
job_name=log-count
description=demo
pack=./log-count/
deps=split->count;count->merge

[split]
cmd=python split.py

[count]
cmd=python count.py
nodes=3

[merge]
cmd=python merge.py

This configuration describes a multi-task job. The task order is split -> count -> merge.

For details about the cfg format, see Multi-task Support.

Submission command

bcs sub --file job.cfg -r oss://your-bucket/log-count/:/home/input/ -w oss://your-bucket/log-count/:/home/output/

• The -r and -w flags specify a read-only mount and a writable mount, respectively. For details, see Mount OSS.

• You can mount the same OSS path to different local directories. However, you cannot mount different OSS paths to the same local directory.

• Note that if you mount a directory, its path must end with a forward slash (/).

Check the job status

bcs j   # Get the job list. This command caches the list on each run. The most recently submitted job is usually listed first.
bcs ch 1  # Check the status of a cached job. Replace '1' with the index of your job from the 'bcs j' output.
bcs log 1 # View the logs for a cached job. Replace '1' with the job's index.

View the results

When the job completes, run the following command to view the results in OSS.

bcs oss cat oss://your-bucket/log-count/merge_result.json

The output should be as follows:

{"INFO": 2460, "WARN": 2448, "DEBUG": 2509, "ERROR": 2583}