MaxCompute Spark is an open source-compatible Spark computing service provided by MaxCompute. Built on a unified computing resource and dataset permission system, it provides the Spark computing framework and allows users to submit and run Spark jobs in a familiar development approach to support a wider range of data processing and analytics scenarios.
Key features
-
Support for native multi-version Spark jobs
Native community Spark runs on MaxCompute, fully compatible with Spark APIs, and supports multiple Spark versions.
-
Unified computing resources
MaxCompute Spark runs on the unified computing resources enabled for MaxCompute projects, same as other task types such as MaxCompute SQL and MapReduce.
-
Unified data and permission management
Follows the permission system of MaxCompute projects to securely query data within the user's permission scope.
-
Same user experience as open source systems
Provides native open source real-time Spark UI and query history log features.
Supported features
MaxCompute Spark currently supports the following features:
-
Offline computing: GraphX, MLlib, RDD, Spark-SQL, PySpark, and more.
-
Read and write MaxCompute table data.
-
Reference file resources in MaxCompute.
-
Access services in Alibaba Cloud VPC environments.
-
Access Alibaba Cloud OSS unstructured storage.
-
Read MaxCompute OSS external tables.
-
DataWorks Notebook
Limits
MaxCompute Spark currently does not support the following scenarios:
-
Interactive scenarios such as Spark-Shell, Spark-SQL-Shell, and PySpark-Shell are not supported.
-
Access to MaxCompute built-in functions and user-defined functions (MaxCompute UDFs) is not supported.
-
Access to MaxCompute external tables other than OSS external tables is not supported.