MaxCompute:What is MaxCompute?

Feature category

Description

Basic data warehouse capabilities

• On-demand elasticity

  Alibaba Cloud MaxCompute provides an out-of-the-box, fully managed data warehouse service. It uses a serverless architecture that decouples storage and compute. Both storage and computing resources can be independently and dynamically scaled. You do not need to plan capacity or reserve resources in advance, which lets you handle sudden business growth.

• SQL development

  MaxCompute is pre-integrated with multiple services. You can use standard SQL for direct development, which is simple and easy to learn.

Multi-scenario computing capabilities

• AI computing framework

  • The MaxFrame distributed computing framework supports Python programming interfaces, is compatible with Pandas interfaces, and performs automatic distributed computing. It is suitable for scenarios such as large-scale data processing, scientific computing, machine learning, and AI development.

  • MaxCompute seamlessly integrates with PAI. You can use the algorithm components of the machine learning platform to perform operations such as model training on MaxCompute data. It provides Data+AI integration for deep learning scenarios like large models through its cloud-native big data pre-processing capabilities.

• Incremental and full data processing

  • Alibaba Cloud MaxCompute has upgraded its architecture based on the original offline batch processing engine to launch a near real-time data warehouse solution. It uses Delta tables to implement integrated storage and management for both incremental and full data. It also introduces rich incremental computing capabilities and has upgraded the MaxCompute short query acceleration feature MaxQA (MCQA 2.0) to support returning query results in seconds.

  • Based on the Delta Table format, MaxCompute adds a series of capabilities such as incremental materialized views, Time Travel, and Stream Tables.

• Integration of offline and real-time capabilities

  Deeply integrates with the real-time data warehouse Hologres. Hologres supports batch import of MaxCompute metadata, which removes the need to manually create external tables. It also supports direct reads from the storage layer. Use Hologres to accelerate queries on MaxCompute data for a performance improvement of more than 10 times.

Open architecture

• Data lakehouse

  MaxCompute provides a Data Lakehouse 2.0 solution. This solution lets you create management objects that define the metadata of external data sources and data access methods. It also uses an external schema mapping mechanism to directly access all tables within the scope of an external data source's database or schema.

  This solution breaks down the silos between data lakes and data warehouses. It combines the flexibility and rich multi-engine ecosystem of a data lake with the enterprise-grade capabilities of a data warehouse to help you build an integrated data management platform.

• OpenLake

  MaxCompute fully supports the OpenLake solution. OpenLake is an integrated solution for big data, search, and AI built on an open and controllable data lakehouse. It uses the metadata management platform DLF to manage structured, semi-structured, and unstructured data. It provides secure access and I/O acceleration for data lakehouse tables and files. It supports multi-engine integration and peer-to-peer collaborative computing, unified development through DataWorks, and large-scale task scheduling.

• Multi-engine access

  You can directly run MaxCompute SQL tasks or tasks from third-party engines such as Spark, MapReduce, and Graph.

• Open storage

  To better integrate with the big data ecosystem and support external engine access to data in MaxCompute, MaxCompute provides an open storage API (Storage API). Mainstream third-party compute engines can call the Storage API to directly access the underlying storage of MaxCompute. This significantly improves data access and interaction efficiency.

Enterprise-grade capabilities

• Enterprise-grade operations and management

  Provides resource observation and job O&M features. You can view the usage of various resources and job details. This lets you promptly detect abnormal job conditions and issues, handle problematic jobs through the console, and optimize job execution plans and resource configurations to improve job execution efficiency and performance.

• Intelligent data warehouse

  • Provides a cost optimization feature. Based on actual job request volumes and expected resource configurations, it can generate better resource configuration plans and simulate their effects. This helps you optimize costs and improve resource utilization.

  • Provides an intelligent materialized view recommendation feature. Based on data table relationships, it suggests materialized views with a high impact index. This helps you intelligently optimize queries, improve computing efficiency, and reduce redundant calculations.

• Fine-grained access control

  MaxCompute provides fine-grained control over operations on objects such as projects, resource quotas, and Networklink objects, along with objects within a project like tables, functions, resources, and instances. It also supports control over behaviors such as Tunnel downloads, sensitive data access, and cross-project access. During project operation, you can grant fine-grained permissions based on user roles to ensure the security of all objects.

Security, disaster recovery, and stability

• Disaster recovery and fault tolerance

  • Zone-disaster recovery: If a data center becomes unavailable due to a catastrophic event such as a carrier network outage, power system failure, or infrastructure issue, the storage disaster recovery mechanism ensures that data read and write services are not interrupted and no data is lost. This meets the requirement of a recovery point objective (RPO) of 0.

• Dynamic data masking

  MaxCompute data masking is performed dynamically the moment data is read from the storage layer. This ensures security without compromising performance. It guarantees that data is masked before it enters subsequent stages such as queries, downloads, joins, and UDF calculations, thereby preventing sensitive data leakage.

  Data masking policies include masking, hashing, character replacement, value rounding, and date rounding. It supports integration with the data classification and categorization features of Data Security Guard to meet data masking needs for information such as identities, bank card numbers, addresses, and phone numbers.

• Data storage encryption

  MaxCompute supports storage encryption of data at the project level through Key Management Service (KMS). This provides data-at-rest protection to meet enterprise regulatory and security compliance requirements. Supported encryption algorithms include AES256, AESCTR, and RC4.

• IP whitelist

  In addition to access control, MaxCompute provides control based on an IP whitelist. When the whitelist feature is enabled for a MaxCompute project, only devices on the whitelist can access the project. If a device not on the whitelist attempts to access the project, it will fail authentication even if it has the correct AccessKey ID and AccessKey secret.

• SLA

  Alibaba Cloud MaxCompute is widely used across various industries and offers customers a service availability commitment of up to 99.9%. For more information, see the MaxCompute Service-Level Agreement (SLA).