Configure automatic data maintenance for OSS Tables - Object Storage Service

OSS Tables provides data maintenance capabilities for Iceberg tables and Lance tables. For Iceberg tables, compaction, snapshot management, and unreferenced file cleanup are automatically performed. For Lance tables, you submit maintenance jobs such as compaction and historical version cleanup through APIs and query job statuses. These features continuously optimize query performance and reduce storage costs without external compute resources.

OSS Tables supports two table formats, each with a different maintenance method:

Iceberg tables: Use automatic maintenance. A backend service schedules and runs maintenance according to the configuration. You only need to enable the feature and configure its parameters; no manual triggering is required.
Lance tables: Use manual job maintenance. Submit asynchronous maintenance jobs through APIs, query their status with the returned Job ID, and delete completed or failed job records to release quota. Lance tables do not currently support unreferenced file cleanup.

Service-linked role

A backend service runs OSS Tables data maintenance jobs, including automatic maintenance for Iceberg tables and maintenance jobs for Lance tables. The service requires the AliyunServiceRoleForOssTableMaintenance service-linked role to obtain access permissions for Table Buckets. For more information about service-linked roles, see service-linked roles.

Authorization

When you first open the Table Buckets page in the OSS console, a message appears at the top of the page if the current account has not created this service-linked role: Insufficient permissions: AliyunServiceRoleForOssTableMaintenance. Click Create Now in the message to authorize the role. After authorization, the backend service can run automatic maintenance for Iceberg tables and maintenance jobs for Lance tables.

Note

If you use a Resource Access Management (RAM) user, make sure that the user has permission to create service-linked roles. An Alibaba Cloud account can attach the following permission policy to the RAM user:

{
  "Version": "1",
  "Statement": [
    {
      "Action": "ram:CreateServiceLinkedRole",
      "Resource": "*",
      "Effect": "Allow",
      "Condition": {
        "StringEquals": {
          "ram:ServiceName": "tablemaintenance.oss.aliyuncs.com"
        }
      }
    }
  ]
}

Permissions

The permission policy of the service-linked role, AliyunServiceRolePolicyForOssTableMaintenance, includes the following permissions to read table configurations, run data maintenance, and update metadata:

Permission (Action)	Purpose
oss:GetTableBucketMaintenanceConfiguration	Read bucket-level maintenance configurations
oss:GetTableMaintenanceConfiguration	Read table-level maintenance configurations
oss:ListTableBuckets	List Table Buckets
oss:GetTableBucket	Get Table Bucket information
oss:ListTables	List tables
oss:GetTable	Get table information
oss:GetTableData	Read table data files (for compaction and cleanup)
oss:PutTableData	Write table data files (to generate new files after compaction)
oss:GetTableMetadataLocation	Get the metadata file location
oss:UpdateTableMetadataLocation	Update the metadata file location (to commit changes after maintenance)

Iceberg table maintenance

Iceberg tables use automatic maintenance: a backend service schedules and runs the maintenance jobs according to the configuration, and you do not need to trigger them manually. The following sections describe each job, its runtime mechanism, and status queries. Before you enable automatic maintenance, complete the authorization described in Service-linked role.

Iceberg table automatic maintenance includes three optimization jobs that apply at different resource levels:

Unreferenced file cleanup: A bucket-level configuration that automatically identifies and deletes objects no longer referenced by any table snapshot.
Compaction: A table-level configuration that merges small files into larger files to improve query performance.
Snapshot management: A table-level configuration that automatically cleans up expired snapshots to reduce the number of metadata files and storage overhead.

After you create a Table Bucket and table, the system automatically applies the following default maintenance configurations.

Maintenance feature	Configuration level	Parameter	Default value
Unreferenced file cleanup	Bucket level	status	enabled
Unreferenced file cleanup	Bucket level	unreferencedDays	3
Unreferenced file cleanup	Bucket level	nonCurrentDays	10
Compaction	Table level	status	enabled
Compaction	Table level	targetFileSizeMB	512
Compaction	Table level	strategy	auto
Snapshot management	Table level	status	enabled
Snapshot management	Table level	maxSnapshotAgeHours	120
Snapshot management	Table level	minSnapshotsToKeep	1

Unreferenced file cleanup

Unreferenced file cleanup is a bucket-level optimization that applies to all tables in a Table Bucket. It identifies and deletes all objects that are not referenced by any table snapshot. During data writes, updates, or deletions, data files such as Parquet, Avro, and ORC files can become unreferenced by metadata because of failed write jobs or expired snapshots. These orphan files continue to consume storage space.

Important

After you enable unreferenced file cleanup, any Parquet, Avro, or ORC file that you PUT to a Table Bucket but that does not belong to a table is also treated as an unreferenced file and automatically cleaned up.

Unreferenced file cleanup uses a two-stage process. It first marks unreferenced files as non-current, controlled by unreferencedDays, and then deletes them after they have been non-current for the number of days specified by nonCurrentDays.

The following table describes the configuration parameters for unreferenced file cleanup.

Parameter	Description	Value range	Default value
status	Specifies whether to enable unreferenced file cleanup.	enabled or disabled.	enabled
unreferencedDays	The number of days to wait before marking a file as non-current after no metadata references it.	1 to 2147483647.	3
nonCurrentDays	The number of days to wait before deleting a file after it is marked as non-current.	1 to 2147483647.	10

Console

Use the OSS console to view and modify the automatic maintenance configuration of a Table Bucket.

Follow these steps:

Log on to the OSS console. In the left-side navigation pane, choose Table Buckets.
Click the name of the Table Bucket that you want to manage to open the Bucket details page.
Click the Data Maintenance tab.
In the Unreferenced file cleanup section, view the current configuration:
- Status: Indicates whether the feature is enabled.
- Unreferenced Days: The number of days to wait before marking a file as non-current after no metadata references it. The default value is 3 days.
- Non-current Days: The number of days to wait before deleting a file after it is marked as non-current. The default value is 10 days.
Click Edit to modify the configuration parameters for unreferenced file cleanup, and then click OK.

ossutil

Use ossutil to modify the unreferenced file cleanup configuration. Replace {ARN} with the ARN of your Table Bucket:

ossutil tables-api put-table-bucket-maintenance-configuration \
  --table-bucket-arn {ARN} \
  --type icebergUnreferencedFileRemoval \
  --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":5,"nonCurrentDays":15}}}'

SDK

The following code shows how to query and modify the unreferenced file cleanup configuration of a Table Bucket through an SDK.

Python

Query the unreferenced file cleanup configuration:

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.tables as oss_tables

parser = argparse.ArgumentParser(description="get table bucket maintenance configuration sample")
parser.add_argument('--region', help='The region in which the table bucket is located.', required=True)
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS Tables.')
parser.add_argument('--table-bucket-arn', help='The ARN of the table bucket.', required=True)

def main():
    args = parser.parse_args()

    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    client = oss_tables.Client(cfg)

    result = client.get_table_bucket_maintenance_configuration(
        oss_tables.models.GetTableBucketMaintenanceConfigurationRequest(
            table_bucket_arn=args.table_bucket_arn,
        )
    )

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' table bucket arn: {result.table_bucket_arn},'
          f' configuration: {result.configuration}')

if __name__ == "__main__":
    main()

Modify the unreferenced file cleanup configuration:

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.tables as oss_tables

parser = argparse.ArgumentParser(description="put table bucket maintenance configuration sample")
parser.add_argument('--region', help='The region in which the table bucket is located.', required=True)
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS Tables.')
parser.add_argument('--table-bucket-arn', help='The ARN of the table bucket.', required=True)
parser.add_argument('--type', help='The maintenance type, e.g., icebergUnreferencedFileRemoval.', required=True)

def main():
    args = parser.parse_args()

    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    client = oss_tables.Client(cfg)

    value = {
        'status': 'enabled',
        'settings': {
            'icebergUnreferencedFileRemoval': {
                'unreferencedDays': 7,
                'nonCurrentDays': 30
            }
        }
    }

    result = client.put_table_bucket_maintenance_configuration(
        oss_tables.models.PutTableBucketMaintenanceConfigurationRequest(
            table_bucket_arn=args.table_bucket_arn,
            type=args.type,
            value=value,
        )
    )

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id}')
    print(f'successfully updated maintenance configuration for: {args.table_bucket_arn}')

if __name__ == "__main__":
    main()

Go

Query the unreferenced file cleanup configuration:

package main

import (
    "context"
    "flag"
    "log"

    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/tables"
)

var (
    region         string
    tableBucketArn string
)

func init() {
    flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
    flag.StringVar(&tableBucketArn, "table-bucket-arn", "", "The arn of the table bucket.")
}

func main() {
    flag.Parse()
    if len(tableBucketArn) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table bucket arn required")
    }

    if len(region) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, region required")
    }

    cfg := oss.LoadDefaultConfig().
        WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
        WithRegion(region)

    client := tables.NewTablesClient(cfg)

    result, err := client.GetTableBucketMaintenanceConfiguration(context.TODO(), &tables.GetTableBucketMaintenanceConfigurationRequest{
        TableBucketARN: oss.Ptr(tableBucketArn),
    })

    if err != nil {
        log.Fatalf("failed to get table bucket maintenance configuration %v", err)
    }

    log.Printf("get table bucket maintenance configuration result:%#v\n", result)
}

Modify the unreferenced file cleanup configuration:

package main

import (
    "context"
    "flag"
    "log"

    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/tables"
)

var (
    region         string
    tableBucketArn string
)

func init() {
    flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
    flag.StringVar(&tableBucketArn, "table-bucket-arn", "", "The arn of the table bucket.")
}

func main() {
    flag.Parse()
    if len(tableBucketArn) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table bucket arn required")
    }

    if len(region) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, region required")
    }

    cfg := oss.LoadDefaultConfig().
        WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
        WithRegion(region)

    client := tables.NewTablesClient(cfg)

    result, err := client.PutTableBucketMaintenanceConfiguration(context.TODO(), &tables.PutTableBucketMaintenanceConfigurationRequest{
        TableBucketARN: oss.Ptr(tableBucketArn),
        Type:           oss.Ptr("icebergUnreferencedFileRemoval"),
        Value: &tables.MaintenanceValue{
            Settings: &tables.MaintenanceSettings{
                IcebergUnreferencedFileRemoval: &tables.SettingsDetail{
                    UnreferencedDays: oss.Ptr(4),
                    NonCurrentDays:   oss.Ptr(10),
                },
            },
            Status: oss.Ptr("enabled"),
        },
    })

    if err != nil {
        log.Fatalf("failed to put table bucket maintenance configuration %v", err)
    }

    log.Printf("put table bucket maintenance configuration result:%#v\n", result)
}

Java

Query the unreferenced file cleanup configuration:

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.tables.OSSTablesClient;
import com.aliyun.sdk.service.oss2.tables.models.*;

public class GetTableBucketMaintenanceConfigurationSample {

    public static void main(String[] args) throws Exception {
        String region = "cn-hangzhou";
        String tableBucketARN = "acs:osstables:cn-hangzhou:1234567890:bucket/mytable-bucket";

        try (OSSTablesClient client = OSSTablesClient.newBuilder()
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .region(region)
                .build()) {
            GetTableBucketMaintenanceConfigurationRequest request = GetTableBucketMaintenanceConfigurationRequest.newBuilder()
                    .tableBucketARN(tableBucketARN)
                    .build();

            GetTableBucketMaintenanceConfigurationResult result = client.getTableBucketMaintenanceConfiguration(request);

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());
            System.out.printf("Successfully got maintenance configuration for table bucket: %s%n", tableBucketARN);
            System.out.printf("Table Bucket ARN: %s%n", result.tableBucketARN());

            result.configuration().forEach((key, value) -> {
                System.out.printf("Configuration Type: %s, Status: %s%n", key, value.status());
            });
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}

Modify the unreferenced file cleanup configuration:

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.tables.OSSTablesClient;
import com.aliyun.sdk.service.oss2.tables.models.*;

public class PutTableBucketMaintenanceConfigurationSample {

    public static void main(String[] args) throws Exception {
        String region = "cn-hangzhou";
        String tableBucketARN = "acs:osstables:cn-hangzhou:1234567890:bucket/mytable-bucket";
        String type = "icebergUnreferencedFileRemoval";
        String status = "enabled";
        int unreferencedDays = 3;
        int nonCurrentDays = 3;

        IcebergUnreferencedFileRemovalSettings removalSettings = IcebergUnreferencedFileRemovalSettings.newBuilder()
                .unreferencedDays(unreferencedDays)
                .nonCurrentDays(nonCurrentDays)
                .build();

        TableBucketMaintenanceSettings settings = TableBucketMaintenanceSettings.newBuilder()
                .icebergUnreferencedFileRemoval(removalSettings)
                .build();

        TableBucketMaintenanceConfigurationValue value = TableBucketMaintenanceConfigurationValue.newBuilder()
                .status(status)
                .settings(settings)
                .build();

        try (OSSTablesClient client = OSSTablesClient.newBuilder()
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .region(region)
                .build()) {
            PutTableBucketMaintenanceConfigurationRequest request = PutTableBucketMaintenanceConfigurationRequest.newBuilder()
                    .tableBucketARN(tableBucketARN)
                    .type(type)
                    .value(value)
                    .build();

            PutTableBucketMaintenanceConfigurationResult result = client.putTableBucketMaintenanceConfiguration(request);

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());
            System.out.printf("Successfully configured maintenance for table bucket: %s%n", tableBucketARN);
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}

API

Use the following API operations:

Query configuration: GetTableBucketMaintenanceConfiguration
Modify configuration: PutTableBucketMaintenanceConfiguration

Compaction

Compaction is a table-level optimization. In Apache Iceberg tables, frequent streaming writes or small-batch writes can produce many small files. Queries must then open and scan too many files, which reduces query efficiency. Compaction merges multiple small files into files of the target size, reducing the number of files, improving query performance, and lowering metadata overhead.

The following table describes the configuration parameters for compaction.

Parameter	Description	Value range	Default value
status	Specifies whether to enable automatic compaction.	enabled or disabled.	enabled
targetFileSizeMB	The target size, in MB, of files after merging. The merge job merges files smaller than this size into files close to this size.	1 to 2147483647.	512
strategy	The compaction strategy. Different strategies determine how files are merged and sorted.	auto, binpack, sort, or z-order. When you set this parameter to sort or z-order, the table metadata must define a sort order.	auto

The meanings of the compaction strategies:

auto: The system automatically selects the optimal compaction strategy. It applies the sort compaction strategy to tables with a sort order and the binpack compaction strategy to tables without a sort order.
binpack: Merges files only by file size and does not change the data order. This is the fastest merge method and is suitable when ordering is not required.
sort: Sorts data based on the sort order defined in the table and then merges it. This strategy is suitable for range query scenarios. A sort order must be defined in the table metadata.
z-order: Uses a Z-Order curve to sort data across multiple columns and then merges it. This strategy is suitable for multidimensional query scenarios. A sort order must be defined in the table metadata.

View and configure compaction.

Console

View and modify compaction configuration on the Data Maintenance page of a table.

Log on to the OSS console. In the left-side navigation pane, choose Table Buckets.
Click the name of the Table Bucket that you want to manage. In Table list, click the name of the table that you want to manage to open the Table details page.
Click the Data Maintenance tab.
In the Compaction section, view the current configuration:
- Status: Indicates whether the feature is enabled.
- Target file size: The target file size after merging.
- Compaction strategy: The compaction strategy currently in use.
- Job status: Displays the execution status and time of the most recent compaction job.
Click Edit to modify the compaction configuration parameters, and then click OK.

ossutil

Use ossutil to modify the compaction configuration. Replace {ARN} with the ARN of your Table Bucket:

ossutil tables-api put-table-maintenance-configuration \
  --table-bucket-arn {ARN} \
  --namespace mynamespace \
  --name mytable \
  --type icebergCompaction \
  --value '{"status":"enabled","settings":{"icebergCompaction":{"targetFileSizeMB":256,"strategy":"binpack"}}}'

SDK

The following code shows how to query and modify the compaction configuration of a table through an SDK.

Python

Query the compaction configuration:

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.tables as oss_tables

parser = argparse.ArgumentParser(description="get table maintenance configuration sample")
parser.add_argument('--region', help='The region in which the table bucket is located.', required=True)
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS Tables.')
parser.add_argument('--table-bucket-arn', help='The ARN of the table bucket.', required=True)
parser.add_argument('--namespace', help='The namespace of the table.', required=True)
parser.add_argument('--name', help='The name of the table.', required=True)

def main():
    args = parser.parse_args()

    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    client = oss_tables.Client(cfg)

    result = client.get_table_maintenance_configuration(
        oss_tables.models.GetTableMaintenanceConfigurationRequest(
            table_bucket_arn=args.table_bucket_arn,
            namespace=args.namespace,
            name=args.name,
        )
    )

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' table arn: {result.table_arn},'
          f' configuration: {result.configuration}')

if __name__ == "__main__":
    main()

Modify the compaction configuration:

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.tables as oss_tables

parser = argparse.ArgumentParser(description="put table maintenance configuration sample")
parser.add_argument('--region', help='The region in which the table bucket is located.', required=True)
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS Tables.')
parser.add_argument('--table-bucket-arn', help='The ARN of the table bucket.', required=True)
parser.add_argument('--namespace', help='The namespace of the table.', required=True)
parser.add_argument('--name', help='The name of the table.', required=True)
parser.add_argument('--type', help='The maintenance type, e.g., icebergCompaction.', required=True)

def main():
    args = parser.parse_args()

    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    client = oss_tables.Client(cfg)

    value = {
        'status': 'enabled',
        'settings': {
            'icebergCompaction': {
                'targetFileSizeMB': 512,
                'strategy': 'auto'
            }
        }
    }

    result = client.put_table_maintenance_configuration(
        oss_tables.models.PutTableMaintenanceConfigurationRequest(
            table_bucket_arn=args.table_bucket_arn,
            namespace=args.namespace,
            name=args.name,
            type=args.type,
            value=value,
        )
    )

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id}')
    print(f'successfully updated maintenance configuration for: {args.namespace}/{args.name}')

if __name__ == "__main__":
    main()

Go

Query the compaction configuration:

package main

import (
    "context"
    "flag"
    "log"

    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/tables"
)

var (
    region         string
    tableBucketArn string
    namespace      string
    name           string
)

func init() {
    flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
    flag.StringVar(&tableBucketArn, "table-bucket-arn", "", "The arn of the table bucket.")
    flag.StringVar(&namespace, "namespace", "", "The name of the namespace.")
    flag.StringVar(&name, "name", "", "The name of the table.")
}

func main() {
    flag.Parse()

    if len(region) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, region required")
    }

    if len(tableBucketArn) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table bucket arn required")
    }

    if len(namespace) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, namespace name required")
    }

    if len(name) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table name required")
    }

    cfg := oss.LoadDefaultConfig().
        WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
        WithRegion(region)

    client := tables.NewTablesClient(cfg)

    result, err := client.GetTableMaintenanceConfiguration(context.TODO(), &tables.GetTableMaintenanceConfigurationRequest{
        TableBucketARN: oss.Ptr(tableBucketArn),
        Namespace:      oss.Ptr(namespace),
        Name:           oss.Ptr(name),
    })

    if err != nil {
        log.Fatalf("failed to get table maintenance configuration %v", err)
    }

    log.Printf("get table maintenance configuration result:%#v\n", result)
}

Modify the compaction configuration:

package main

import (
    "context"
    "flag"
    "log"

    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/tables"
)

var (
    region         string
    tableBucketArn string
    namespace      string
    name           string
)

func init() {
    flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
    flag.StringVar(&tableBucketArn, "table-bucket-arn", "", "The arn of the table bucket.")
    flag.StringVar(&namespace, "namespace", "", "The name of the namespace.")
    flag.StringVar(&name, "name", "", "The name of the table.")
}

func main() {
    flag.Parse()

    if len(region) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, region required")
    }

    if len(tableBucketArn) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table bucket arn required")
    }

    if len(namespace) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, namespace name required")
    }

    if len(name) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table name required")
    }

    cfg := oss.LoadDefaultConfig().
        WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
        WithRegion(region)

    client := tables.NewTablesClient(cfg)

    // icebergCompaction type
    result, err := client.PutTableMaintenanceConfiguration(context.TODO(), &tables.PutTableMaintenanceConfigurationRequest{
        TableBucketARN: oss.Ptr(tableBucketArn),
        Namespace:      oss.Ptr(namespace),
        Name:           oss.Ptr(name),
        Type:           oss.Ptr("icebergCompaction"),
        Value: &tables.TableMaintenanceValue{
            Status: oss.Ptr("enabled"),
            Settings: &tables.TableMaintenanceSettings{
                IcebergCompaction: &tables.IcebergCompactionSettingsDetail{
                    TargetFileSizeMB: oss.Ptr(400),
                    Strategy:         oss.Ptr("auto"),
                },
            },
        },
    })

    // icebergSnapshotManagement type
    //result, err := client.PutTableMaintenanceConfiguration(context.TODO(), &tables.PutTableMaintenanceConfigurationRequest{
    //    TableBucketARN: oss.Ptr(tableBucketArn),
    //    Namespace:      oss.Ptr(namespace),
    //    Name:           oss.Ptr(name),
    //    Type:           oss.Ptr("icebergSnapshotManagement"),
    //    Value: &tables.TableMaintenanceValue{
    //        Status: oss.Ptr("enabled"),
    //        Settings: &tables.TableMaintenanceSettings{
    //            IcebergSnapshotManagement: &tables.IcebergSnapshotManagementSettingsDetail{
    //                MaxSnapshotAgeHours: oss.Ptr(350),
    //                MinSnapshotsToKeep:  oss.Ptr(1),
    //            },
    //        },
    //    },
    //})

    if err != nil {
        log.Fatalf("failed to put table maintenance configuration %v", err)
    }

    log.Printf("put table maintenance configuration result:%#v\n", result)
}

Java

Query the compaction configuration:

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.tables.OSSTablesClient;
import com.aliyun.sdk.service.oss2.tables.models.*;

public class GetTableMaintenanceConfigurationSample {

    public static void main(String[] args) throws Exception {
        String region = "cn-hangzhou";
        String tableBucketARN = "acs:osstables:cn-hangzhou:1234567890:bucket/mytable-bucket";
        String namespace = "mynamespace";
        String name = "mytable";

        try (OSSTablesClient client = OSSTablesClient.newBuilder()
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .region(region)
                .build()) {
            GetTableMaintenanceConfigurationRequest request = GetTableMaintenanceConfigurationRequest.newBuilder()
                    .tableBucketARN(tableBucketARN)
                    .namespace(namespace)
                    .name(name)
                    .build();

            GetTableMaintenanceConfigurationResult result = client.getTableMaintenanceConfiguration(request);

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());
            System.out.printf("Table ARN: %s%n", result.tableARN());

            if (result.configuration() != null && !result.configuration().isEmpty()) {
                System.out.println("Maintenance configurations:");
                result.configuration().forEach((type, config) -> {
                    System.out.printf("  Type: %s, Status: %s%n", type, config.status());
                    if (config.settings() != null) {
                        if (config.settings().icebergCompaction() != null) {
                            System.out.printf("    Compaction - TargetFileSizeMB: %d, Strategy: %s%n",
                                    config.settings().icebergCompaction().targetFileSizeMB(),
                                    config.settings().icebergCompaction().strategy());
                        }
                        if (config.settings().icebergSnapshotManagement() != null) {
                            System.out.printf("    SnapshotManagement - MinSnapshotsToKeep: %d, MaxSnapshotAgeHours: %d%n",
                                    config.settings().icebergSnapshotManagement().minSnapshotsToKeep(),
                                    config.settings().icebergSnapshotManagement().maxSnapshotAgeHours());
                        }
                    }
                });
            } else {
                System.out.println("No maintenance configuration found.");
            }
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}

Modify the compaction configuration:

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.tables.OSSTablesClient;
import com.aliyun.sdk.service.oss2.tables.models.*;

public class PutTableMaintenanceConfigurationSample {

    public static void main(String[] args) throws Exception {
        String region = "cn-hangzhou";
        String tableBucketARN = "acs:osstables:cn-hangzhou:1234567890:bucket/mytable-bucket";
        String namespace = "mynamespace";
        String name = "mytable";
        String type = "icebergCompaction";
        String status = "enabled";
        int targetFileSizeMB = 256;
        String strategy = "auto";

        try (OSSTablesClient client = OSSTablesClient.newBuilder()
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .region(region)
                .build()) {
            IcebergCompactionSettings compactionSettings = IcebergCompactionSettings.newBuilder()
                    .targetFileSizeMB(targetFileSizeMB)
                    .strategy(strategy)
                    .build();
            TableMaintenanceSettings settings = TableMaintenanceSettings.newBuilder()
                    .icebergCompaction(compactionSettings)
                    .build();

            TableMaintenanceConfigurationValue value = TableMaintenanceConfigurationValue.newBuilder()
                    .status(status)
                    .settings(settings)
                    .build();

            PutTableMaintenanceConfigurationRequest request = PutTableMaintenanceConfigurationRequest.newBuilder()
                    .tableBucketARN(tableBucketARN)
                    .namespace(namespace)
                    .name(name)
                    .type(type)
                    .value(value)
                    .build();

            PutTableMaintenanceConfigurationResult result = client.putTableMaintenanceConfiguration(request);

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());
            System.out.printf("Successfully put table maintenance configuration for table: %s/%s, type: %s%n", namespace, name, type);
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}

API

Use the following API operations to configure compaction:

Query configuration: GetTableMaintenanceConfiguration
Modify configuration: PutTableMaintenanceConfiguration

Snapshot management

Snapshot management is a table-level optimization. Each write operation on an Apache Iceberg table creates a snapshot that records the state of the data at a point in time. As the number of writes increases, too many snapshots consume storage space and increase metadata operation overhead. Snapshot management automatically cleans up expired snapshots to reduce storage costs while ensuring data security.

Snapshot management prioritizes data security. When minSnapshotsToKeep conflicts with maxSnapshotAgeHours, the system first retains the minimum number of snapshots so that cleanup does not remove the data recovery capability that you need.

The following table describes the configuration parameters for snapshot management.

Parameter	Description	Value range	Default value
status	Specifies whether to enable snapshot management.	enabled or disabled.	enabled
maxSnapshotAgeHours	The maximum retention period for a snapshot, in hours. Snapshots older than this period are cleaned up.	1 to 2147483647.	120
minSnapshotsToKeep	The minimum number of snapshots to retain. The most recent snapshots in this quantity are retained even when they exceed the maximum retention period.	1 to 2147483647.	1

View and configure snapshot management.

Console

View and modify snapshot management configuration on the Data Maintenance page of a table.

Log on to the OSS console. In the left-side navigation pane, choose Table Buckets.
Click the name of the Table Bucket that you want to manage. In Table list, click the name of the table that you want to manage to open the Table details page.
Click the Data Maintenance tab.
In the Snapshot management section, view the current configuration:
- Status: Indicates whether the feature is enabled.
- Maximum snapshot retention period: The maximum retention period for a snapshot.
- Minimum snapshots to retain: The minimum number of snapshots to retain.
- Job status: Displays the execution status and time of the most recent snapshot cleanup job.
Click Edit to modify the snapshot management configuration parameters, and then click OK.

ossutil

Use ossutil to modify the snapshot management configuration. Replace {ARN} with the ARN of your Table Bucket:

ossutil tables-api put-table-maintenance-configuration \
  --table-bucket-arn {ARN} \
  --namespace mynamespace \
  --name mytable \
  --type icebergSnapshotManagement \
  --value '{"status":"enabled","settings":{"icebergSnapshotManagement":{"maxSnapshotAgeHours":72,"minSnapshotsToKeep":3}}}'

SDK

The following code shows how to query and modify the snapshot management configuration of a table through an SDK. Snapshot management and compaction use the same table-level maintenance configuration APIs: GetTableMaintenanceConfiguration and PutTableMaintenanceConfiguration.

Python

Query the snapshot management configuration:

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.tables as oss_tables

parser = argparse.ArgumentParser(description="get table maintenance configuration sample")
parser.add_argument('--region', help='The region in which the table bucket is located.', required=True)
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS Tables.')
parser.add_argument('--table-bucket-arn', help='The ARN of the table bucket.', required=True)
parser.add_argument('--namespace', help='The namespace of the table.', required=True)
parser.add_argument('--name', help='The name of the table.', required=True)

def main():
    args = parser.parse_args()

    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    client = oss_tables.Client(cfg)

    result = client.get_table_maintenance_configuration(
        oss_tables.models.GetTableMaintenanceConfigurationRequest(
            table_bucket_arn=args.table_bucket_arn,
            namespace=args.namespace,
            name=args.name,
        )
    )

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' table arn: {result.table_arn},'
          f' configuration: {result.configuration}')

if __name__ == "__main__":
    main()

Modify the snapshot management configuration:

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.tables as oss_tables

parser = argparse.ArgumentParser(description="put table maintenance configuration sample")
parser.add_argument('--region', help='The region in which the table bucket is located.', required=True)
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS Tables.')
parser.add_argument('--table-bucket-arn', help='The ARN of the table bucket.', required=True)
parser.add_argument('--namespace', help='The namespace of the table.', required=True)
parser.add_argument('--name', help='The name of the table.', required=True)
parser.add_argument('--type', help='The maintenance type, e.g., icebergSnapshotManagement.', required=True)

def main():
    args = parser.parse_args()

    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    client = oss_tables.Client(cfg)

    value = {
        'status': 'enabled',
        'settings': {
            'icebergSnapshotManagement': {
                'maxSnapshotAgeHours': 72,
                'minSnapshotsToKeep': 3
            }
        }
    }

    result = client.put_table_maintenance_configuration(
        oss_tables.models.PutTableMaintenanceConfigurationRequest(
            table_bucket_arn=args.table_bucket_arn,
            namespace=args.namespace,
            name=args.name,
            type=args.type,
            value=value,
        )
    )

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id}')
    print(f'successfully updated maintenance configuration for: {args.namespace}/{args.name}')

if __name__ == "__main__":
    main()

Go

Query the snapshot management configuration:

package main

import (
    "context"
    "flag"
    "log"

    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/tables"
)

var (
    region         string
    tableBucketArn string
    namespace      string
    name           string
)

func init() {
    flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
    flag.StringVar(&tableBucketArn, "table-bucket-arn", "", "The arn of the table bucket.")
    flag.StringVar(&namespace, "namespace", "", "The name of the namespace.")
    flag.StringVar(&name, "name", "", "The name of the table.")
}

func main() {
    flag.Parse()

    if len(region) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, region required")
    }

    if len(tableBucketArn) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table bucket arn required")
    }

    if len(namespace) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, namespace name required")
    }

    if len(name) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table name required")
    }

    cfg := oss.LoadDefaultConfig().
        WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
        WithRegion(region)

    client := tables.NewTablesClient(cfg)

    result, err := client.GetTableMaintenanceConfiguration(context.TODO(), &tables.GetTableMaintenanceConfigurationRequest{
        TableBucketARN: oss.Ptr(tableBucketArn),
        Namespace:      oss.Ptr(namespace),
        Name:           oss.Ptr(name),
    })

    if err != nil {
        log.Fatalf("failed to get table maintenance configuration %v", err)
    }

    log.Printf("get table maintenance configuration result:%#v\n", result)
}

Modify the snapshot management configuration:

package main

import (
    "context"
    "flag"
    "log"

    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/tables"
)

var (
    region         string
    tableBucketArn string
    namespace      string
    name           string
)

func init() {
    flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
    flag.StringVar(&tableBucketArn, "table-bucket-arn", "", "The arn of the table bucket.")
    flag.StringVar(&namespace, "namespace", "", "The name of the namespace.")
    flag.StringVar(&name, "name", "", "The name of the table.")
}

func main() {
    flag.Parse()

    if len(region) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, region required")
    }

    if len(tableBucketArn) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table bucket arn required")
    }

    if len(namespace) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, namespace name required")
    }

    if len(name) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table name required")
    }

    cfg := oss.LoadDefaultConfig().
        WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
        WithRegion(region)

    client := tables.NewTablesClient(cfg)

    // icebergSnapshotManagement type
    result, err := client.PutTableMaintenanceConfiguration(context.TODO(), &tables.PutTableMaintenanceConfigurationRequest{
        TableBucketARN: oss.Ptr(tableBucketArn),
        Namespace:      oss.Ptr(namespace),
        Name:           oss.Ptr(name),
        Type:           oss.Ptr("icebergSnapshotManagement"),
        Value: &tables.TableMaintenanceValue{
            Status: oss.Ptr("enabled"),
            Settings: &tables.TableMaintenanceSettings{
                IcebergSnapshotManagement: &tables.IcebergSnapshotManagementSettingsDetail{
                    MaxSnapshotAgeHours: oss.Ptr(350),
                    MinSnapshotsToKeep:  oss.Ptr(1),
                },
            },
        },
    })

    if err != nil {
        log.Fatalf("failed to put table maintenance configuration %v", err)
    }

    log.Printf("put table maintenance configuration result:%#v\n", result)
}

Java

Query the snapshot management configuration:

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.tables.OSSTablesClient;
import com.aliyun.sdk.service.oss2.tables.models.*;

public class GetTableMaintenanceConfigurationSample {

    public static void main(String[] args) throws Exception {
        String region = "cn-hangzhou";
        String tableBucketARN = "acs:osstables:cn-hangzhou:1234567890:bucket/mytable-bucket";
        String namespace = "mynamespace";
        String name = "mytable";

        try (OSSTablesClient client = OSSTablesClient.newBuilder()
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .region(region)
                .build()) {
            GetTableMaintenanceConfigurationRequest request = GetTableMaintenanceConfigurationRequest.newBuilder()
                    .tableBucketARN(tableBucketARN)
                    .namespace(namespace)
                    .name(name)
                    .build();

            GetTableMaintenanceConfigurationResult result = client.getTableMaintenanceConfiguration(request);

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());
            System.out.printf("Table ARN: %s%n", result.tableARN());

            if (result.configuration() != null && !result.configuration().isEmpty()) {
                System.out.println("Maintenance configurations:");
                result.configuration().forEach((type, config) -> {
                    System.out.printf("  Type: %s, Status: %s%n", type, config.status());
                    if (config.settings() != null) {
                        if (config.settings().icebergCompaction() != null) {
                            System.out.printf("    Compaction - TargetFileSizeMB: %d, Strategy: %s%n",
                                    config.settings().icebergCompaction().targetFileSizeMB(),
                                    config.settings().icebergCompaction().strategy());
                        }
                        if (config.settings().icebergSnapshotManagement() != null) {
                            System.out.printf("    SnapshotManagement - MinSnapshotsToKeep: %d, MaxSnapshotAgeHours: %d%n",
                                    config.settings().icebergSnapshotManagement().minSnapshotsToKeep(),
                                    config.settings().icebergSnapshotManagement().maxSnapshotAgeHours());
                        }
                    }
                });
            } else {
                System.out.println("No maintenance configuration found.");
            }
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}

Modify the snapshot management configuration:

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.tables.OSSTablesClient;
import com.aliyun.sdk.service.oss2.tables.models.*;

public class PutTableMaintenanceConfigurationSample {

    public static void main(String[] args) throws Exception {
        String region = "cn-hangzhou";
        String tableBucketARN = "acs:osstables:cn-hangzhou:1234567890:bucket/mytable-bucket";
        String namespace = "mynamespace";
        String name = "mytable";
        String type = "icebergSnapshotManagement";
        String status = "enabled";
        int maxSnapshotAgeHours = 72;
        int minSnapshotsToKeep = 3;

        try (OSSTablesClient client = OSSTablesClient.newBuilder()
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .region(region)
                .build()) {
            IcebergSnapshotManagementSettings snapshotManagementSettings = IcebergSnapshotManagementSettings.newBuilder()
                    .maxSnapshotAgeHours(maxSnapshotAgeHours)
                    .minSnapshotsToKeep(minSnapshotsToKeep)
                    .build();
            TableMaintenanceSettings settings = TableMaintenanceSettings.newBuilder()
                    .icebergSnapshotManagement(snapshotManagementSettings)
                    .build();

            TableMaintenanceConfigurationValue value = TableMaintenanceConfigurationValue.newBuilder()
                    .status(status)
                    .settings(settings)
                    .build();

            PutTableMaintenanceConfigurationRequest request = PutTableMaintenanceConfigurationRequest.newBuilder()
                    .tableBucketARN(tableBucketARN)
                    .namespace(namespace)
                    .name(name)
                    .type(type)
                    .value(value)
                    .build();

            PutTableMaintenanceConfigurationResult result = client.putTableMaintenanceConfiguration(request);

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());
            System.out.printf("Successfully put table maintenance configuration for table: %s/%s, type: %s%n", namespace, name, type);
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}

API

Use the following API operations to configure snapshot management:

Query configuration: GetTableMaintenanceConfiguration
Modify configuration: PutTableMaintenanceConfiguration

Runtime mechanism

Scheduling: The system backend automatically schedules maintenance jobs rather than running them at fixed times. Under normal circumstances, each type of maintenance job for the same table runs about once every 24 hours. Jobs can start later when the system is under heavy load.
Resource usage: Automatic maintenance jobs do not consume your throughput or QPS quota and do not affect normal production jobs.
Transaction isolation: Maintenance jobs run on Iceberg snapshot isolation. Production jobs can continue to write new data while files are merged. Read jobs are also unaffected because the query engine automatically selects the latest valid snapshot.

Query maintenance job status

Query the execution status of the three maintenance jobs of a table: compaction, snapshot management, and unreferenced file cleanup. Each maintenance job independently returns its own status information.

Console

View the job status of each maintenance job on the Data Maintenance page of a table.

Log on to the OSS console. In the left-side navigation pane, choose Table Buckets.
Click the name of the Table Bucket that you want to manage. In Table list, click the name of the table that you want to manage to open the Table details page.
Click the Data Maintenance tab.
View Job status in each maintenance section:
- Compaction: Displays the execution status, such as "Failed" or "Succeeded", and the last run time of the compaction job.
- Snapshot management: Displays the execution status and last run time of the snapshot cleanup job.
- Unreferenced file cleanup: A bucket-level configuration. Click Edit configuration in Table Bucket to go to the Data Maintenance page of the Table Bucket to view the configuration.

ossutil

ossutil tables-api get-table-maintenance-job-status \
  --table-bucket-arn acs:osstables:cn-hangzhou:1234567890:bucket/mytable-bucket \
  --namespace mynamespace \
  --name mytable

SDK

Python

import argparse
import alibabacloud_oss_v2 as oss
import alibabacloud_oss_v2.tables as oss_tables

parser = argparse.ArgumentParser(description="get table maintenance job status sample")
parser.add_argument('--region', help='The region in which the table bucket is located.', required=True)
parser.add_argument('--endpoint', help='The domain names that other services can use to access OSS Tables.')
parser.add_argument('--table-bucket-arn', help='The ARN of the table bucket.', required=True)
parser.add_argument('--namespace', help='The namespace of the table.', required=True)
parser.add_argument('--name', help='The name of the table.', required=True)

def main():
    args = parser.parse_args()

    credentials_provider = oss.credentials.EnvironmentVariableCredentialsProvider()

    cfg = oss.config.load_default()
    cfg.credentials_provider = credentials_provider
    cfg.region = args.region
    if args.endpoint is not None:
        cfg.endpoint = args.endpoint

    client = oss_tables.Client(cfg)

    result = client.get_table_maintenance_job_status(
        oss_tables.models.GetTableMaintenanceJobStatusRequest(
            table_bucket_arn=args.table_bucket_arn,
            namespace=args.namespace,
            name=args.name,
        )
    )

    print(f'status code: {result.status_code},'
          f' request id: {result.request_id},'
          f' table arn: {result.table_arn},'
          f' status: {result.status}')

if __name__ == "__main__":
    main()

Go

package main

import (
    "context"
    "flag"
    "log"

    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
    "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/tables"
)

var (
    region         string
    tableBucketArn string
    namespace      string
    name           string
)

func init() {
    flag.StringVar(&region, "region", "", "The region in which the bucket is located.")
    flag.StringVar(&tableBucketArn, "table-bucket-arn", "", "The arn of the table bucket.")
    flag.StringVar(&namespace, "namespace", "", "The name of the namespace.")
    flag.StringVar(&name, "name", "", "The name of the table.")
}

func main() {
    flag.Parse()

    if len(region) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, region required")
    }

    if len(tableBucketArn) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table bucket arn required")
    }

    if len(namespace) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, namespace name required")
    }

    if len(name) == 0 {
        flag.PrintDefaults()
        log.Fatalf("invalid parameters, table name required")
    }

    cfg := oss.LoadDefaultConfig().
        WithCredentialsProvider(credentials.NewEnvironmentVariableCredentialsProvider()).
        WithRegion(region)

    client := tables.NewTablesClient(cfg)

    result, err := client.GetTableMaintenanceJobStatus(context.TODO(), &tables.GetTableMaintenanceJobStatusRequest{
        TableBucketARN: oss.Ptr(tableBucketArn),
        Namespace:      oss.Ptr(namespace),
        Name:           oss.Ptr(name),
    })

    if err != nil {
        log.Fatalf("failed to get table maintenance job status %v", err)
    }

    log.Printf("get table maintenance job status result:%#v\n", result)
}

Java

import com.aliyun.sdk.service.oss2.credentials.EnvironmentVariableCredentialsProvider;
import com.aliyun.sdk.service.oss2.tables.OSSTablesClient;
import com.aliyun.sdk.service.oss2.tables.models.*;

public class GetTableMaintenanceJobStatusSample {

    public static void main(String[] args) throws Exception {
        String region = "cn-hangzhou";
        String tableBucketARN = "acs:osstables:cn-hangzhou:1234567890:bucket/mytable-bucket";
        String namespace = "mynamespace";
        String name = "mytable";

        try (OSSTablesClient client = OSSTablesClient.newBuilder()
                .credentialsProvider(new EnvironmentVariableCredentialsProvider())
                .region(region)
                .build()) {
            GetTableMaintenanceJobStatusRequest request = GetTableMaintenanceJobStatusRequest.newBuilder()
                    .tableBucketARN(tableBucketARN)
                    .namespace(namespace)
                    .name(name)
                    .build();

            GetTableMaintenanceJobStatusResult result = client.getTableMaintenanceJobStatus(request);

            System.out.printf("Status code:%d, request id:%s%n",
                    result.statusCode(), result.requestId());
            System.out.printf("Table ARN: %s%n", result.tableARN());

            if (result.jobStatus() != null && !result.jobStatus().isEmpty()) {
                System.out.println("Maintenance job status:");
                result.jobStatus().forEach((type, status) -> {
                    System.out.printf("  Type: %s, Status: %s%n", type, status.status());
                    if (status.lastRunTimestamp() != null) {
                        System.out.printf("    LastRunTimestamp: %s%n", status.lastRunTimestamp());
                    }
                    if (status.failureMessage() != null && !status.failureMessage().isEmpty()) {
                        System.out.printf("    FailureMessage: %s%n", status.failureMessage());
                    }
                });
            } else {
                System.out.println("No maintenance job status found.");
            }
        } catch (Exception e) {
            System.out.println("Error: " + e.getMessage());
        }
    }
}

API

Use GetTableMaintenanceJobStatus to query maintenance job status.

Lance table maintenance

Lance table data maintenance jobs all apply at the table level and run asynchronously: call the Submit API to submit a job, use the returned job ID to query the execution status, and delete job records after the jobs finish to release quota. Lance tables provide the following maintenance jobs:

Compaction: Merges data files (fragments) to reduce the number of small files and improve query performance.
Historical version cleanup: Cleans up expired historical versions of Lance tables to release storage space.

Complete the following authorization before use:

Create the service-linked role AliyunServiceRoleForOssTableMaintenance. For more information, see Service-linked role.
When a RAM user calls the API, grant the user the oss:PutTableMaintenanceConfiguration (submit and delete jobs) and oss:GetTableMaintenanceJobStatus (query and list jobs) permissions.

Compaction

Compaction is a table-level optimization. Frequent writes to a Lance table generate multiple small fragments. Queries must open and scan too many fragments, which reduces query efficiency. Compaction merges multiple small fragments into larger fragments, reducing the number of fragments and improving query performance.

Console

Submit a compaction job on the Data Maintenance page of a table.

Log on to the OSS console. In the left-side navigation pane, choose Table Buckets.
Click the name of the Table Bucket that you want to manage. In Table list, click the name of the table that you want to manage to open the Table details page.
Click the Data Maintenance tab.
In the Storage maintenance section, click File compaction and configure the following parameters:
- Target fragment rows: The target number of rows in each fragment after merging. The default value is 1,048,576.
- Materialize soft deletes: Specifies whether to materialize delete markers in data files.
- Soft delete ratio threshold: When the proportion of deleted rows in a fragment exceeds this threshold, compaction rewrites the fragment and materializes delete markers. This parameter takes effect only when materializing soft deletes is enabled. The default value is 10%, and the valid range is 1% to 100%.
- Maximum single file size: The maximum size of a single file after merging. If you leave this field blank, the system uses the default value of 90 GiB.
Click OK to submit the file compaction job.

API

Use the SubmitTableMaintenanceJob API to submit a compaction job. Use task_type=lance_compact when you query the job.

Historical version cleanup

Historical version cleanup is a table-level operation. Lance tables retain historical versions during writes and deletions. Historical version cleanup deletes expired versions according to the number of versions to retain or a time condition, releasing storage space.

Important

Deleted versions cannot be restored. Protected target versions are not deleted.

Console

Submit a historical version cleanup job on the Data Maintenance page of a table.

Log on to the OSS console. In the left-side navigation pane, choose Table Buckets.
Click the name of the Table Bucket that you want to manage. In Table list, click the name of the table that you want to manage to open the Table details page.
Click the Data Maintenance tab.
In the Storage maintenance section, click Version cleanup and configure the following parameters:
- Cleanup time: The time limit for historical versions to clean up. Historical versions older than this duration are cleaned up. If you leave this field blank, the system cleans up historical versions older than 14 days by default.
- Recent versions to retain: The number of most recent versions to retain. The current version is always retained.
Click OK to submit the version cleanup job.

API

Use the SubmitTableMaintenanceJob API to submit a historical version cleanup job. Use task_type=lance_cleanup when you query the job.

Query job status

Lance maintenance jobs run asynchronously. You can query a job after submission and delete its job record when its status is completed or failed. A job status can transition from submitted (submitted and waiting to run) to running (running) to completed (succeeded) or failed (failed). The submitted and running states can also transition directly to failed. Each table and job type can retain at most 1,000 job records. Completed or failed jobs do not automatically release quota. You must manually delete job records to release the quota.

Console

View the job status of each maintenance job on the Data Maintenance page of a table.

Log on to the OSS console. In the left-side navigation pane, choose Table Buckets.
Click the name of the Table Bucket that you want to manage. In Table list, click the name of the table that you want to manage to open the Table details page.
Click the Data Maintenance tab.
View Job status in each maintenance section:
- Job type: File compaction or version cleanup.
- Job ID: A unique identifier generated by the service for each submission.
- Status: The current lifecycle stage of the job.
- Created at: The time when the job was created.
- Completed at/Duration: The time when the job finished and its total duration.
- Result summary: Summary information about the job result. The content differs by job type.
- Actions: You can view the details of or delete this record.

API

Use the GetTableMaintenanceJob API to query the execution status and result of a single job.

Use the ListTableMaintenanceJobs API to list job records of the specified job type for the current table.

Use the DeleteTableMaintenanceJob API to delete job records whose status is completed or failed to release quota. Jobs in other states cannot be deleted.