Still Batching Streaming Data into Files? Send Kafka Straight to the Lake with Kafka Connect [OSS Tables Deep Dive]

This article explains how to use Kafka Connect and the Iceberg Sink Connector to stream data directly from Kafka into Alibaba Cloud OSS Tables in real time.

By Alibaba Cloud Storage

Where Streaming Data Ends Up

In the OSS Tables ecosystem, Kafka Connect is the ingestion pipeline into the lake: it writes Kafka messages into OSS Tables as-is, adding nothing and changing nothing. Kafka carries the traffic, but it is not the destination — event streams that flow through it still have to land in a data lake. Kafka Connect is the shortest route there.

The usual approach is to write to a downstream store. There are good reasons to write to a database, to a search index, or to files in object storage. But once the data volume grows large enough, or once you need to combine several streams for offline analysis, a data lake is almost always the destination.

The hard part is the write itself. With traditional approaches, you either write your own consumer that pulls messages, converts them to Parquet files, and writes those files into OSS, or you build a dedicated ingestion pipeline. Either way, you still have to handle schema management, file rolling, exactly-once semantics, and partitioning yourself.

The Iceberg Sink Connector for Kafka Connect handles all of this for you. Because OSS Tables is compatible with the Iceberg REST Catalog protocol, you can use the connector to write Kafka messages into an OSS Tables table in real time, so your streaming data lands directly in the lake.

Step 1: Prepare the Environment

Download the Required JAR Files

Place the following JAR files in the Kafka Connect plugin directory (the directory specified by plugin.path).

JAR File	Version Requirement	Description
iceberg-aws-bundle-1.10.1.jar	Match the Iceberg version	Provides the S3FileIO implementation and the AWS SDK required for SigV4 signature authentication with the REST Catalog.
iceberg-aws-1.10.1.jar	Match the Iceberg version	Provides SigV4 signing and the S3FileIO implementation. The version must match iceberg-aws-bundle.
iceberg-parquet-1.10.1.jar	Match the Iceberg version	Adds support for writing Parquet files.
hadoop-client-runtime-3.3.6.jar	3.3.6	Hadoop runtime dependency (Iceberg loads it internally). Adjust the version as needed.
hadoop-client-api-3.3.6.jar	3.3.6	Hadoop API dependency (Iceberg loads it internally). Adjust the version as needed.
failsafe-3.3.2.jar	3.3.2	Required at runtime by the Iceberg SnapshotProducer. Without it, a ClassNotFoundException occurs.

Step 2: Create a Table Bucket

Before you write any data, create a Table Bucket and a Namespace. You can use either ossutil or the AWS CLI.

Method 1: Using ossutil

1. Install or upgrade ossutil

Install ossutil 2.3.0 or later. If ossutil is already installed, run the following command to upgrade to the latest version:

ossutil update -f

2. Configure credentials

Run the ossutil config command and enter your AccessKey ID, AccessKey Secret, and Region as prompted.

3. Create a Table Bucket

ossutil tables-api create-table-bucket --name {table-bucket-name} --endpoint http://{endpoint} --region {region}

When the command succeeds, the output includes the Table Bucket ARN. Make a note of this value.

4. Create a Namespace

ossutil tables-api create-namespace --table-bucket-arn {Table Bucket ARN} --namespace {namespace-name} --endpoint http://{endpoint}

Important: Namespace and Table names cannot contain hyphens (-); use underscores (_) instead, because these names become identifiers in SQL statements.

5. Create a Table

You can create an Iceberg table in either of the following ways:

• Create it with another compute engine, such as Spark.
• Create it with ossutil: save the table schema as a JSON file, then call create-table.

In the following example, the schema file schema.json defines three fields:

{
  "iceberg": {
    "schema": {
      "fields": [
        {"name": "event_id", "type": "string", "required": true},
        {"name": "event_time", "type": "string"},
        {"name": "event_type", "type": "string"}
      ]
    }
  }
}

Create the Table from the schema file:

ossutil tables-api create-table --table-bucket-arn <Table Bucket ARN> --namespace <namespace-name> --name <table-name> --format ICEBERG --metadata file://schema.json --endpoint http://{endpoint}

Method 2: Using the AWS CLI

OSS Tables is compatible with the S3 Tables API, so you can also manage Table Buckets with the AWS CLI.

1. Install the AWS CLI

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

2. Configure credentials

Run the aws configure command and enter your AccessKey ID, AccessKey Secret, and Region as prompted.

3. Create a Table Bucket

aws s3tables --endpoint http://{endpoint} create-table-bucket --region {region} --name {table-bucket-name}

When the command succeeds, the output includes the Table Bucket ARN.

4. Create a Namespace

aws s3tables --endpoint http://{endpoint} create-namespace --table-bucket-arn {Table Bucket ARN} --namespace {namespace-name}

5. Create a Table

• Create the table with another compute engine, such as Spark.

• Create it with the AWS CLI: first save the full request parameters to a JSON file named create-table.json, then call create-table.

{
  "tableBucketARN": "<Table Bucket ARN>",
"namespace": "<namespace-name>",
"name": "<table-name>",
  "format": "ICEBERG",
  "metadata": {
    "iceberg": {
      "schema": {
        "fields": [
          {"name": "event_id", "type": "string", "required": true},
          {"name": "event_time", "type": "string"},
          {"name": "event_type", "type": "string"}
        ]
      }
    }
  }
}

aws s3tables --endpoint http://{endpoint} create-table --cli-input-json file://{file-path}

6. Manage background maintenance tasks

OSS Tables can automatically perform background maintenance on Iceberg tables, such as file cleanup and compaction. Use the AWS CLI to query and configure these maintenance tasks.

Query the maintenance task status of a Table:

aws s3tables get-table-maintenance-job-status \
   --table-bucket-arn="{bucketArn}" \
--namespace="{namespace-name}" \
--name="{table-name}"

Configure a bucket-level maintenance policy (file cleanup):

aws s3tables put-table-bucket-maintenance-configuration \
   --table-bucket-arn {tableArn} \
   --type icebergUnreferencedFileRemoval \
   --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":4,"nonCurrentDays":10}}}'

Configure a table-level maintenance policy (small file compaction):

aws s3tables put-table-maintenance-configuration \
   --table-bucket-arn {bucketArn} \
   --type icebergCompaction \
--namespace {namespace-name} \
--name {table-name} \
   --value='{"status":"enabled","settings":{"icebergCompaction":{"targetFileSizeMB":256}}}'

Step 3: Configure Kafka Connect

OSS Tables provides an Iceberg REST Catalog endpoint. Kafka Connect connects to this endpoint through the Iceberg Sink Connector to write data. The endpoint format is:

• Internal: https://{region}-internal.oss-tables.aliyuncs.com/iceberg
• Public: https://{region}.oss-tables.aliyuncs.com/iceberg

OSS Tables also provides the endpoint that S3FileIO uses to reach the OSS data plane. Spark reads table data through this endpoint. The endpoint format is:

• Internal: https://oss-{region}-internal.aliyuncs.com
• Public: https://oss-{region}.aliyuncs.com

Connector Configuration

When you create the Iceberg Sink Connector, set the connector class to org.apache.iceberg.connect.IcebergSinkConnector and configure the following properties:

# --- Iceberg Catalog (REST) ---
iceberg.catalog.type: rest
iceberg.catalog.uri: https://{region}-internal.oss-tables.aliyuncs.com/iceberg
iceberg.catalog.rest.sigv4-enabled: true
iceberg.catalog.rest.signing-region: <Region>
iceberg.catalog.warehouse: <Table Bucket ARN>
iceberg.catalog.rest.signing-name: osstables
iceberg.catalog.rest.access-key-id: <AccessKey ID>
iceberg.catalog.rest.secret-access-key: <AccessKey Secret>
# --- Force S3FileIO (catalog returns oss:// but storage is S3-compatible) ---
iceberg.catalog.io-impl: org.apache.iceberg.aws.s3.S3FileIO
# --- S3FileIO storage configuration ---
iceberg.catalog.s3.endpoint: https://oss-{region}-internal.aliyuncs.com
iceberg.catalog.s3.access-key-id: <AccessKey ID>
iceberg.catalog.s3.secret-access-key: <AccessKey Secret>
iceberg.catalog.s3.path-style-access: true
iceberg.catalog.client.region: <Region>
# --- Data format conversion ---
key.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: false
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: false

Important: If you use a newer AWS SDK (2.20+), you may hit the signature error aws-chunked encoding is not supported with the specified x-amz-content-sha256 value. In that case, add the following JVM option to the Java startup parameters of Kafka Connect:

-Daws.requestChecksumCalculation=when_required
-Daws.responseChecksumValidation=when_required

Parameter Reference

Parameter	Required	Description
`iceberg.catalog.type`	Yes	Always rest, which selects the REST Catalog.
`iceberg.catalog.uri`	Yes	The REST Catalog endpoint URL. Format: • Internal: `https://{region}-internal.oss-tables.aliyuncs.com/iceberg` • Public: `https://{region}.oss-tables.aliyuncs.com/iceberg`
`iceberg.catalog.warehouse`	Yes	The Table Bucket ARN. Format: `acs:osstables:<Region>:<Alibaba-Cloud-Account-ID>:bucket/<Table Bucket name>`.
`iceberg.catalog.rest.sigv4-enabled`	Yes	Always `true`, which enables SigV4 signature authentication.
`iceberg.catalog.rest.signing-name`	Yes	Always `osstables`, the SigV4 signing service name for the OSS Tables endpoint.
`iceberg.catalog.io-impl`	Yes	Always `org.apache.iceberg.aws.s3.S3FileIO`, which accesses the OSS data plane over the S3 protocol.
`iceberg.catalog.s3.endpoint`	Yes	The OSS data plane endpoint. Format: • Internal: `https://oss-{region}-internal.aliyuncs.com` • Public: `https://oss-{region}.aliyuncs.com`
`iceberg.catalog.s3.path-style-access`	Yes	Always `true`, which enables path-style access.

Permission Configuration

When you access OSS Tables with a RAM user or STS temporary credentials, make sure that the identity has the permissions required for the operations it performs.

Resource Definitions

• Table Bucket ARN: acs:osstables:<Region>:<Alibaba-Cloud-Account-ID>:bucket/<bucket_name>

• Table ARN: acs:osstables:<Region>:<Alibaba-Cloud-Account-ID>:bucket/<bucket_name>/table/<table_id>

Action Definitions

The following table lists the Actions that OSS Tables supports and whether each one can be granted across accounts:

Category	Action	Cross-Account Access
Table Bucket level	`oss:CreateTableBucket`	Not allowed
	`oss:GetTableBucket`	Allowed
	`oss:ListTableBuckets`	Not allowed
	`oss:CreateNamespace`	Allowed
	`oss:GetNamespace`	Allowed
	`oss:ListNamespaces`	Allowed
	`oss:DeleteNamespace`	Allowed
	`oss:DeleteTableBucket`	Allowed
	`oss:PutTableBucketPolicy`	Not allowed
	`oss:GetTableBucketPolicy`	Not allowed
	`oss:DeleteTableBucketPolicy`	Not allowed
	`oss:GetTableBucketMaintenanceConfiguration`	Allowed
	`oss:PutTableBucketMaintenanceConfiguration`	Allowed
	`oss:PutTableBucketEncryption`	Not allowed
	`oss:GetTableBucketEncryption`	Not allowed
	`oss:DeleteTableBucketEncryption`	Not allowed
Table level	`oss:GetTableMaintenanceConfiguration`	Allowed
	`oss:PutTableMaintenanceConfiguration`	Allowed
	`oss:PutTablePolicy`	Not allowed
	`oss:GetTablePolicy`	Not allowed
	`oss:DeleteTablePolicy`	Not allowed
	`oss:CreateTable`	Allowed
	`oss:GetTable`	Allowed
	`oss:GetTableMetadataLocation`	Allowed
	`oss:ListTables`	Allowed
	`oss:RenameTable`	Allowed
	`oss:UpdateTableMetadataLocation`	Allowed
	`oss:GetTableData`	Allowed
	`oss:PutTableData`	Allowed
	`oss:GetTableEncryption`	Not allowed
	`oss:PutTableEncryption`	Not allowed
	`oss:DeleteTable`	Allowed

Mapping Between Iceberg REST Operations and Permissions

The following table lists the OSS Action required for each Iceberg REST Catalog operation:

Iceberg REST Operation	Required OSS Action
`getConfig`	`oss:GetTableBucket`
`listNamespaces`	`oss:ListNamespaces`
`createNamespace`	`oss:CreateNamespace`
`loadNamespaceMetadata`	`oss:GetNamespace`
`dropNamespace`	`oss:DeleteNamespace`
`listTables`	`oss:ListTables`
`createTable`	`oss:CreateTable`、`oss:PutTableData`
`loadTable`	`oss:GetTableMetadataLocation`、`oss:GetTableData`
`updateTable`	`oss:UpdateTableMetadataLocation`、`oss:PutTableData`、`oss:GetTableData`
`dropTable`	`oss:DeleteTable`
`renameTable`	`oss:RenameTable`
`tableExists`	`oss:GetTable`
`namespaceExists`	`oss:GetNamespace`

Community

Still Batching Streaming Data into Files? Send Kafka Straight to the Lake with Kafka Connect [OSS Tables Deep Dive]

Where Streaming Data Ends Up

Step 1: Prepare the Environment

Download the Required JAR Files

Step 2: Create a Table Bucket

Method 1: Using ossutil

1. Install or upgrade ossutil

2. Configure credentials

3. Create a Table Bucket

4. Create a Namespace

5. Create a Table

Method 2: Using the AWS CLI

1. Install the AWS CLI

2. Configure credentials

3. Create a Table Bucket

4. Create a Namespace

5. Create a Table

6. Manage background maintenance tasks

Step 3: Configure Kafka Connect

Connector Configuration

Parameter Reference

Permission Configuration

Resource Definitions

Action Definitions

Mapping Between Iceberg REST Operations and Permissions

Read previous post:

Read next post:

Alibaba Cloud Community

You may also like

Comments

Alibaba Cloud Community

Related Products

Data Lake Storage Solution

Data Lake Formation

Hybrid Cloud Distributed Storage

Big Data Consulting for Data Technology Solution