E-MapReduce:Configure job time and date

How it works

When a job runs, EMR replaces each variable wildcard in the Configuration Parameters section with the computed time value. The wildcard expression defines the format and the arithmetic offset from the current time.

Syntax:

${dateexpr}
${dateexpr+Nd}
${dateexpr-Nd}
${dateexpr+Nh}
${dateexpr-Nh}

Where dateexpr is a time format string built from the specifiers in the table below.

Limitations

Before writing expressions, note these constraints:

• Expressions are case-sensitive. MM (month) and mm (minute) produce different results.

• Time variables must start with yyyy. For example, ${yyyy-MM} is valid; ${MM-dd} is not.

• Only days (d) and hours (h) are supported for arithmetic offsets. N must be an integer. To work with other time units, use parseDate() and formatDate() instead.

Format specifiers

Build a dateexpr by combining yyyy with one or more of the following specifiers:

Arithmetic expressions

The following examples use a reference time of 20160427 12:08:01.

Use parseDate and formatDate for other time units

Because arithmetic offsets are limited to days and hours, use parseDate() and formatDate() when you need to extract specific components or work with other time units.

parseDate(<parameter name>, <time format>) — converts the string value of a Configuration Parameters key into a Date object.

• <parameter name>: the key name defined in Configuration Parameters (for example, current_time)

• <time format>: the format string that matches the key's value (for example, yyyyMMddHHmmss)

formatDate(<Date object>, <time format>) — converts a Date object into a formatted string.

Combine the two functions to extract specific time components:

For example, if current_time is defined as ${yyyyMMddHHmmss-1d}, pass yyyyMMddHHmmss as the format to parseDate().

Configure a variable wildcard

1. Log on to the EMR console.

2. In the top navigation bar, select the region where your cluster resides and select a resource group.

3. Click the Data Platform tab.

4. In the Projects section, find your project and click Edit Job in the Actions column.

5. In the Edit Job pane on the left, click the job name, then click Job Settings in the upper-right corner.

6. On the Basic Settings tab, go to the Configuration Parameters section and click to add a parameter.

7. Enter a key name and set the value to a variable wildcard expression, such as ${yyyyMMdd-1d}.

   

8. Reference the key in your job script using the same key name you defined in step 7.