This topic describes how to develop a demo project on Spark on MaxCompute by using Java or Scala.
Download a demo project
Spark on MaxCompute provides a demo project template. We recommend that you download and copy the template to develop your application.
Run the following commands to download the demo project template:
Download and compile the Spark-1.x template
git clone https://github.com/aliyun/MaxCompute-Spark.git
cd spark-1.x
mvn clean package
Download and compile the Spark-2.x template
git clone https://github.com/aliyun/MaxCompute-Spark.git
cd spark-2.x
mvn clean packageNotice In the demo project, the scope parameter for the Spark dependency is set to provided. Do not modify this parameter. Otherwise, the submitted job will not run normally.
Spark-1.x demo project
Examples of a Spark-1.x demo project:
Spark-2.x demo project
Examples of a Spark-2.x demo project:
- WordCount example
- GraphX PageRank example
- Mllib Kmeans-ON-OSS example
- OSS UnstructuredData example
- Example of reading data from or writing data to a MaxCompute table
- Example of using PySpark to read data from or write data to a MaxCompute table
- Example of using PySpark to write data to OSS
- Spark Streaming LogHub example
- Spark Streaming DataHub example
- Spark Streaming Kafka example