This topic introduces the common keywords in Spark Streaming SQL and describes how to use these keywords.
Background information
Spark Streaming SQL is developed based on Spark Structured Streaming. All syntax and limits are compliant with Spark Structured Streaming. Therefore, you can use SQL statements to develop a streaming query job.
Usage notes
Whether an E-MapReduce (EMR) cluster supports the Spark Streaming SQL feature depends on the version of Spark.
The version of Spark that is used by EMR V3.X series is Spark 2.x. Only EMR V3.21.0 and later support Spark Streaming SQL.
The version of Spark that is used by EMR V4.X series and EMR V5.X series is Spark 3.x. Only EMR V5.8.0 and later support Spark Streaming SQL.
Common keywords
Type | Keyword |
DDL | CREATE TABLE, CREATE TABLE AS SELECT, CREATE SCAN, and CREATE STREAM |
DML | INSERT INTO and MERGE INTO |
SELECT clause | SELECT FROM, WHERE, GROUP BY, JOIN, and UNION ALL |
Use keywords as field names
If you want to use a keyword as a field name, enclose the keyword in backticks (`). Example: `value`
.