This topic describes the CREATE STREAM statement of Spark SQL. This statement is supported in EMR V3.23.0 and later.

Background information

You can use the SET or CREATE STREAM statement to configure WriteStream parameters. We recommend that you use CREATE STREAM to configure the required WriteStream parameters, which are checkpointLocation, outputMode, triggerType, and triggerIntervalMs.

Syntax

CREATE STREAM queryName
OPTIONS (propertyName=propertyValue[,propertyName=propertyValue]*)
INSERT INTO tbName
queryStatement;
The following table describes these parameters.
Parameter Description Default value
checkpointLocation The directory where the checkpoint file for the streaming query job is stored. No default value
outputMode The output mode of the query result. Append
triggerType The execution mode of the streaming query. ProcessingTime
triggerIntervalMs The interval between streaming queries. Unit: milliseconds. 0

Example

CREATE STREAM job1
OPTIONS(
checkpointLocation='/tmp/spark',
outputMode='Append',
triggerType='ProcessingTime'
triggerIntervalMs='3000')
INSERT INTO LargeOrders
SELECT * FROM Orders WHERE units > 1000;