Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Still Running Your Own Hive Metastore? Point Spark Straight at OSS Tables and Iceberg Just Works [OSS Tables Deep Dive]

Still Running Your Own Hive Metastore? Point Spark Straight at OSS Tables and Iceberg Just Works [OSS Tables Deep Dive]

This article explains how to connect Apache Spark directly to Alibaba Cloud OSS Tables via the Apache Iceberg REST Catalog, eliminating the need for a traditional Hive Metastore.

An old question: how many components does Spark really need to read a data lake?

Spark is the workhorse of the OSS Tables ecosystem. It creates tables, runs batch ETL, and handles complex analytics. But once you start building, you hit the gap between storing the data and getting Spark to query it — and closing that gap takes engineering work.

Metadata management is the classic example. Before Spark can read or write tables in the lake, something has to track which tables exist, what their schemas are, and where the data files live. That something is a catalog service. The traditional answer is to stand up a Hive Metastore backed by a relational database. It works, but it adds operational overhead, and every extra component widens the failure surface.

OSS Tables takes a direct approach. It is natively compatible with the Apache Iceberg REST Catalog protocol, so Spark connects through a standard Iceberg client with no catalog service to deploy. Table creation, queries, writes, and time travel all use standard SQL, and in your code you change a single line of catalog configuration.

In other words: your data already sits on OSS, so you only need to point Spark at OSS Tables to get a complete data warehouse solution.

The following end-to-end example shows how to connect Spark to OSS Tables, create a table, write data, and run a time travel query.

Prerequisites

• Spark 3.5 or later is installed, running on Java 11 or later.

• An OSS Tables Table Bucket has been created. If you have not created one, see OSS Tables.

Step 1: Prepare the environment

Download the dependency JAR files

Put the following JAR files in the $SPARK_HOME/jars directory, or pass them with the --jars parameter at startup.

JAR file Description
iceberg-spark-runtime-3.5_2.12-1.10.1.jar Iceberg Spark Runtime package, which provides Iceberg's Spark integration. Choose the package that matches your Spark version (for example, Spark 3.5 pairs with iceberg-spark-runtime-3.5_2.12).
iceberg-aws-bundle-1.10.1.jar Iceberg AWS bundle, which provides the S3FileIO implementation and the AWS SDK required for REST Catalog sigv4 signature authentication. Its version must match the Runtime package.

Configure environment variables

The Iceberg REST Catalog uses sigv4 signature authentication, and S3FileIO also needs credentials to reach the data plane. We recommend supplying both sets of credentials through environment variables. Set the following variables before you start Spark:

Note

The variable names carry the AWS_ prefix because Iceberg's sigv4 signer and S3FileIO reuse the standard AWS SDK credentials provider chain. The values you supply are your Alibaba Cloud AccessKey ID and AccessKey Secret.

export AWS_ACCESS_KEY_ID=<Alibaba Cloud AccessKey ID>
export AWS_SECRET_ACCESS_KEY=<Alibaba Cloud AccessKey Secret>
export AWS_REGION=<region, for example cn-hangzhou>
# Optional. Set this when you use STS temporary credentials
export AWS_SESSION_TOKEN=<Alibaba Cloud STS TOKEN>

Important: When you use environment variables, both the driver and the executors must be able to read them. In YARN mode, inject the variables on every cluster node; in Kubernetes mode, inject them into the pod. If you cannot do that, specify the credentials explicitly through the Spark configuration items described below.

Set Spark configuration items

Instead of environment variables, you can pass credentials explicitly in PySpark or Spark SQL through the Spark catalog configuration items. This suits multi-catalog setups and cases where setting environment variables is impractical.

# S3FileIO data plane credentials
spark.sql.catalog.oss_tables.s3.access-key-id=<Alibaba Cloud AccessKey ID>
spark.sql.catalog.oss_tables.s3.secret-access-key=<Alibaba Cloud AccessKey Secret>
spark.sql.catalog.oss_tables.client.region=<region, for example cn-hangzhou>
# Optional. Set this when you use STS temporary credentials
spark.sql.catalog.oss_tables.s3.session-token=<Alibaba Cloud STS TOKEN>
# REST Catalog signing credentials

spark.sql.catalog.oss_tables.rest.access-key-id=<Alibaba Cloud AccessKey ID>
spark.sql.catalog.oss_tables.rest.secret-access-key=<Alibaba Cloud AccessKey Secret>
spark.sql.catalog.oss_tables.rest.signing-region=<region, for example cn-hangzhou>
# Optional. Set this when you use STS temporary credentials
spark.sql.catalog.oss_tables.rest.session-token=<Alibaba Cloud STS TOKEN>

Step 2: Configure the Spark connection

OSS Tables exposes an Iceberg REST Catalog endpoint, and Spark manages table metadata through it. The endpoint format is as follows:

  • Internal endpoint: https://{region}-internal.oss-tables.aliyuncs.com/iceberg
  • Public endpoint: https://{region}.oss-tables.aliyuncs.com/iceberg

OSS Tables also exposes the access endpoint that S3FileIO uses to reach the OSS data plane, and Spark reads table data through it. The endpoint format is as follows:

  • Internal endpoint: https://oss-{region}-internal.aliyuncs.com
  • Public endpoint: https://oss-{region}.aliyuncs.com

Note: Do not set io-impl to org.apache.iceberg.hadoop.HadoopFileIO.

Iceberg is designed to avoid LIST operations, and an OSS Table Bucket — a storage type tuned specifically for Iceberg — blocks them outright. HadoopFileIO builds on file semantics over object storage and issues LIST operations to stay compatible with general-purpose scenarios, so you cannot use it to access data in an OSS Table Bucket.

Start with PySpark

The following example uses the Table Bucket name my-data-lake in region cn-hangzhou, which gives the Table Bucket ARN acs:osstables:cn-hangzhou:{accountId}:bucket/my-data-lake.

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .appName("OSS Tables Demo") \
    .config("spark.jars", "/path/to/iceberg-spark-runtime-3.5_2.12-1.10.1.jar,"
            "/path/to/iceberg-aws-bundle-1.10.1.jar") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.oss_tables", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.oss_tables.catalog-impl", "org.apache.iceberg.rest.RESTCatalog") \
    .config("spark.sql.catalog.oss_tables.uri", "https://cn-hangzhou-internal.oss-tables.aliyuncs.com/iceberg") \
    .config("spark.sql.catalog.oss_tables.warehouse", "acs:osstables:cn-hangzhou:{accountId}:bucket/my-data-lake") \
    .config("spark.sql.catalog.oss_tables.rest.sigv4-enabled", "true") \
    .config("spark.sql.catalog.oss_tables.rest.signing-region", "cn-hangzhou") \
    .config("spark.sql.catalog.oss_tables.rest.signing-name", "osstables") \
    .config("spark.sql.catalog.oss_tables.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
    .config("spark.sql.catalog.oss_tables.s3.endpoint", "https://oss-cn-hangzhou-internal.aliyuncs.com") \
    .getOrCreate()

Start with spark-sql

spark-sql \
  --jars /path/to/iceberg-spark-runtime-3.5_2.12-1.10.1.jar,/path/to/iceberg-aws-bundle-1.10.1.jar \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.oss_tables=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.oss_tables.catalog-impl=org.apache.iceberg.rest.RESTCatalog \
  --conf spark.sql.catalog.oss_tables.uri=https://cn-hangzhou-internal.oss-tables.aliyuncs.com/iceberg \
  --conf spark.sql.catalog.oss_tables.warehouse=acs:osstables:cn-hangzhou:{accountId}:bucket/my-data-lake \
  --conf spark.sql.catalog.oss_tables.rest.sigv4-enabled=true \
  --conf spark.sql.catalog.oss_tables.rest.signing-region=cn-hangzhou \
  --conf spark.sql.catalog.oss_tables.rest.signing-name=osstables \
  --conf spark.sql.catalog.oss_tables.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
  --conf spark.sql.catalog.oss_tables.s3.endpoint=https://oss-cn-hangzhou-internal.aliyuncs.com

Configuration parameters

Parameter Required Description
catalog-impl Yes Always org.apache.iceberg.rest.RESTCatalog, which selects the REST Catalog.
uri Yes The REST Catalog endpoint URL. Format:
• Internal endpoint: https://{region}-internal.oss-tables.aliyuncs.com/iceberg
• Public endpoint: https://{region}.oss-tables.aliyuncs.com/iceberg
warehouse Yes The Table Bucket ARN. Format: acs:osstables:<Region>:<Alibaba Cloud account ID>:bucket/<Table Bucket name>.
rest.sigv4-enabled Yes Always true, which enables sigv4 signature authentication.
rest.signing-region No (when credentials are supplied through environment variables) The REST Catalog signing region, which must match the region of the Table Bucket, for example cn-hangzhou. You can omit it if you set the AWS_REGION environment variable; it is required when you pass credentials through Spark configuration items.
rest.signing-name Yes The signing service name, always osstables.
io-impl Yes The FileIO implementation Iceberg uses to read and write the underlying data files, always org.apache.iceberg.aws.s3.S3FileIO.
s3.endpoint Yes The endpoint S3FileIO uses to reach the OSS data plane. It must include the https:// prefix. Format:
• Internal endpoint: https://oss-{region}-internal.aliyuncs.com
• Public endpoint: https://oss-{region}.aliyuncs.com

Step 3: Work with data using SQL

Once connected, use standard SQL statements to work with your data.

Manage namespaces

A namespace groups tables logically and plays the same role as a database.

-- List existing namespaces
SHOW NAMESPACES IN oss_tables;
-- Create a namespace
CREATE NAMESPACE oss_tables.my_namespace;
-- Drop a namespace (drop all tables in it first)
DROP NAMESPACE oss_tables.my_namespace;

Create and manage tables

-- Create a non-partitioned table
CREATE TABLE oss_tables.my_namespace.users (
id BIGINT NOT NULL COMMENT 'User ID',
name STRING COMMENT 'User name',
email STRING COMMENT 'Email',
created_at TIMESTAMP COMMENT 'Creation time'
) USING iceberg;
-- Create a partitioned table (partitioned by day)
CREATE TABLE oss_tables.my_namespace.events (
    id BIGINT NOT NULL,
    event_type STRING,
    data STRING,
    ts TIMESTAMP
) USING iceberg
PARTITIONED BY (days(ts));
-- List all tables in the namespace
SHOW TABLES IN oss_tables.my_namespace;
-- View the table schema
DESCRIBE TABLE oss_tables.my_namespace.users;
-- Drop a table (OSS Tables requires the PURGE keyword; otherwise you get the error: OSS Tables only supports dropping tables with purge enabled)
DROP TABLE oss_tables.my_namespace.users PURGE;

Write and query data

-- Insert data
INSERT INTO oss_tables.my_namespace.users VALUES
(1, 'Zhangsan', 'zhangsan@example.com', TIMESTAMP '2024-01-15 10:30:00'),
(2, 'Lisi', 'lisi@example.com', TIMESTAMP '2024-01-16 14:20:00'),
(3, 'Wangwu', 'wangwu@example.com', TIMESTAMP '2024-01-17 09:15:00');
-- Query the whole table
SELECT * FROM oss_tables.my_namespace.users;
-- Filtered query
SELECT * FROM oss_tables.my_namespace.users WHERE id = 2;
-- Aggregate query
SELECT COUNT(*) AS total FROM oss_tables.my_namespace.users;
-- Grouped aggregation
SELECT name, COUNT(*) AS cnt FROM oss_tables.my_namespace.users GROUP BY name;
-- Update data
UPDATE oss_tables.my_namespace.users SET name = 'Zhaoliu' WHERE id = 3;
-- Delete data
DELETE FROM oss_tables.my_namespace.users WHERE id = 1;
-- Verify with a query
SELECT * FROM oss_tables.my_namespace.users ORDER BY id;

Work with partitioned tables

-- Insert partitioned data
INSERT INTO oss_tables.my_namespace.events VALUES
    (1, 'click', '{"page": "home"}', TIMESTAMP '2024-01-15 10:30:00'),
    (2, 'view', '{"page": "product"}', TIMESTAMP '2024-01-15 11:00:00'),
    (3, 'click', '{"page": "detail"}', TIMESTAMP '2024-01-16 09:00:00');
-- Query with partition pruning (scans matching partitions only)
SELECT * FROM oss_tables.my_namespace.events
WHERE ts >= TIMESTAMP '2024-01-15 00:00:00'
  AND ts < TIMESTAMP '2024-01-16 00:00:00';
-- Aggregate statistics
SELECT event_type, COUNT(*) AS cnt
FROM oss_tables.my_namespace.events
GROUP BY event_type;

Time travel queries

Iceberg supports time travel queries, so you can read a data snapshot as of a point in the past.

-- View snapshot history
SELECT snapshot_id, committed_at, operation
FROM oss_tables.my_namespace.users.snapshots;
-- Query historical data by snapshot ID
SELECT * FROM oss_tables.my_namespace.users
VERSION AS OF <snapshot_id>;
-- Query data as of a specific point in time
SELECT * FROM oss_tables.my_namespace.users
TIMESTAMP AS OF TIMESTAMP '2024-01-16 00:00:00';
-- View the data file layout
SELECT * FROM oss_tables.my_namespace.users.files;

Considerations

• Version requirements: We recommend Spark 3.5 or later with Iceberg 1.10.1. The Spark version must match the version of the iceberg-spark-runtime JAR file (for example, Spark 3.5 pairs with iceberg-spark-runtime-3.5_2.12).

• Credential configuration: S3FileIO and REST Catalog sigv4 signing use the same AccessKey ID and AccessKey Secret pair. We recommend setting them through the following environment variables, which cover both REST Catalog signing and S3FileIO data access:

  • AWS_ACCESS_KEY_ID: your Alibaba Cloud AccessKey ID
  • AWS_SECRET_ACCESS_KEY: your Alibaba Cloud AccessKey Secret
  • AWS_REGION: the region of the Table Bucket, used for REST Catalog signing. Once you set this variable, you can omit rest.signing-region from the catalog configuration.

• Table format: OSS Tables currently supports only the Iceberg format. You must specify USING iceberg when you create a table.

• Data maintenance: OSS Tables has built-in file compaction, snapshot expiration, and orphan file cleanup, so you do not need to run Iceberg maintenance procedures manually in Spark. For details, see Data maintenance.

• OSS Tables endpoint support:

  • Internal endpoint: https://{region}-internal.oss-tables.aliyuncs.com/iceberg
  • Public endpoint: https://{region}.oss-tables.aliyuncs.com/iceberg

Getting started:

• OSS Tables

https://www.alibabacloud.com/help/oss/user-guide/quick-start

• JAR files

https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20251231/bhwllw/iceberg-spark-runtime-3.5_2.12-1.10.1.jar

https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20251231/bxghdg/iceberg-aws-bundle-1.10.1.jar

• Data maintenance

https://www.alibabacloud.com/help/oss/user-guide/data-maintenance

Storage Big Data Object Storage Service Apache Spark Data Lake Apache Iceberg big data analytics OSS Tables REST Catalog
0 0 0
Share on

Read previous post:

Still Batching Streaming Data into Files? Send Kafka Straight to the Lake with Kafka Connect [OSS Tables Deep Dive]

Alibaba Cloud Community

1,534 posts | 515 followers

You may also like

Comments

Alibaba Cloud Community

1,534 posts | 515 followers

Related Products

More Posts by Alibaba Cloud Community

See All