Hadoop OSS Connector V2 Data Accelerator best practices - Object Storage Service

Use cases

Large-scale TPC-DS / TPC-H benchmarks
Interactive BI queries with Tableau and Superset via the Spark Thrift Server
Unified storage layer for a Lakehouse architecture
Multi-tenant data analytics platform

1. Hadoop OSS Connector V2

Hadoop OSS Connector V2 is a high-performance object storage connector that integrates Alibaba Cloud OSS as a native file system into Spark, Hive, and Presto.

V2 significantly improves performance and stability, with deep optimizations for large-scale data lake scenarios and high-concurrency analytics workloads.

Key capabilities

Supports the standard oss://bucket/path and the S3-compatible s3a:// access schemes.
Multithreaded multipart uploads and downloads for large files.
Automatic retry and intelligent connection pool management for fault tolerance.
Prefetch mechanism for faster sequential reads.
Supports the dual-domain acceleration architecture (OSS + Data Accelerator).
Compatibility has been verified for Hadoop 3.3.5 only.

Important

Current limitations:

You cannot configure a unique endpoint for each bucket (for example, fs.oss.<bucket>.endpoint). All buckets share the global fs.oss.endpoint setting.
Only v4 authentication is supported.

2. Architecture and solution

This solution uses a storage-compute separation architecture, decoupling storage from compute for elastic scaling and cost optimization:

Component	Role
OSS	Unified data lake storage for raw data and intermediate results.
Hive Metastore	Manages metadata: table structures, partitions, and schemas.
Spark on YARN	Distributed compute engine for SQL queries and data processing.

2.1 Data storage: OSS

Stores data in open formats (Parquet, ORC, CSV, JSON) for cross-engine compatibility.
A unified path format enables data sharing between components.
Leverages OSS high availability and scalability for petabyte-scale elastic storage.

2.2 Metadata management: Hive Metastore

Hive Metastore Service (HMS) manages metadata: table structures, partitions, and column statistics.
HMS uses MySQL for metadata storage. Derby is not supported in production.
Create external tables pointing to an oss:// path for data access without migration.

Example: Create an external table compatible with the OSS protocol
```
CREATE EXTERNAL TABLE ods_user (
    id BIGINT,
    name STRING
)
PARTITIONED BY (dt STRING)
STORED AS PARQUET
LOCATION 'oss://my-data-lake-bucket/dw/ods/user_info/';
```
The s3a:// protocol is also supported:
```
LOCATION 's3a://my-data-lake-bucket/dw/ods/user_info/';
```
The unified metadata layer supports both protocols, enabling smooth migration from the AWS S3 ecosystem.

2.3 Compute engine: Spark

The compute cluster uses Spark on YARN for dynamic resource scheduling.
Spark reads and writes directly to OSS via Hadoop OSS Connector V2, reducing intermediate overhead.
Write query results back to OSS for downstream analysis.

3. Installation and configuration

3.1 Prerequisites

Meet the following prerequisites before deployment:

Item	Requirement
Service activation	You have activated OSS and created a target bucket.
Region consistency	Your ECS instance and OSS bucket are in the same region, such as China (Hangzhou), to ensure internal access and minimal latency.
Internal access	The ECS instance can access OSS through an internal endpoint, such as `oss-cn-hangzhou-internal.aliyuncs.com`, to avoid public traffic costs.
Accelerator activation	You have activated Data Accelerator for OSS and obtained an acceleration endpoint, such as `cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com`.
Environment preparation	A runtime environment has been set up. The procedures in this document were validated using Hadoop 3.3.5, Spark 3.5.3, and Hive Metastore 3.1.3.
Permission configuration	You have created an AccessKey for a RAM user and granted it read and write permissions on the target bucket.
Data preparation	You have generated 5 TB of TPC-DS test data and uploaded it to OSS for performance benchmarking.

3.2 Download the JAR file

Hadoop OSS Connector V2 is available on Maven Central. Add it as a dependency or download the JAR manually.

Maven dependency (for application packaging)

<dependency>
    <groupId>com.aliyun</groupId>
    <artifactId>hadoop-oss</artifactId>
    <version>3.3.5-2.0.25-alpha-shade</version>
</dependency>

Manually download the JAR (for cluster deployment)

Download URL: https://central.sonatype.com/artifact/com.aliyun/hadoop-oss
Place the following JAR file in the $HADOOP_HOME/share/hadoop/common/lib/ and $SPARK_HOME/jars/ directories:
- hadoop-oss-3.3.5-2.0.25-alpha-shade.jar (The shaded version is recommended because it includes core dependencies.)
Note
Ensure this JAR is in the Spark classpath. Pre-install it on each node.

3.3 Core configuration

Add the following properties to $HADOOP_HOME/etc/hadoop/core-site.xml:

<configuration>
    <!-- OSS access credentials -->
    <property>
        <name>fs.oss.accessKeyId</name>
        <value>YourAccessKeyId</value>
    </property>
    <property>
        <name>fs.oss.accessKeySecret</name>
        <value>YourAccessKeySecret</value>
    </property>
    <!-- OSS internal endpoint (strongly recommended to reduce latency and traffic costs) -->
    <property>
        <name>fs.oss.endpoint</name>
        <value>oss-cn-hangzhou-internal.aliyuncs.com</value>
    </property>
    <!-- Connection tuning -->
    <property>
        <name>fs.oss.connection.maximum</name>
        <value>64</value>
    </property>
    <property>
        <name>fs.oss.connection.timeout</name>
        <value>200000</value>
    </property>
    <property>
        <name>fs.oss.attempts.maximum</name>
        <value>10</value>
    </property>
    <!-- multipart upload settings (enabled for objects larger than 20 MB) -->
    <property>
        <name>fs.oss.multipart.upload.threshold</name>
        <value>20971520</value> <!-- 20MB -->
    </property>
    <property>
        <name>fs.oss.multipart.upload.size</name>
        <value>104857600</value> <!-- 100MB per part -->
    </property>
    <!-- Use prefetch to accelerate sequential reads of large objects -->
    <property>
        <name>fs.oss.prefetch.version</name>
        <value>v2</value>
    </property>
    <property>
        <name>fs.oss.prefetch.block.size</name>
        <value>131072</value> <!-- 128KB -->
    </property>
    <property>
        <name>fs.oss.prefetch.block.count</name>
        <value>8</value>
    </property>
    <property>
        <name>fs.oss.prefetch.io.threshold</name>
        <value>2097152</value> <!-- 2MB -->
    </property>
    <!-- Buffer directories (SSD recommended) -->
    <property>
        <name>fs.oss.buffer.dir</name>
        <value>/data/oss-buffer,/mnt/disk2/oss-buffer</value>
    </property>
</configuration>

Important configuration

The Hadoop configuration must be consistent across all cluster nodes.
The directories specified by fs.oss.buffer.dir must exist and have read and write permissions.
If Data Accelerator for OSS is enabled, also configure fs.oss.acc.endpoint and fs.oss.acc.rules (Section 5.1).
To adjust the local download cache, configure fs.oss.prefetch.block.count (Section 5.2).
S3 protocol compatibility requires additional configuration (Section 5.3).

4. Verify installation

4.1 Test basic file operations

Run the following Hadoop commands to verify read and write operations:

# List the contents of an OSS directory
hadoop fs -ls oss://my-data-lake-bucket/test/
# Upload a small file
hadoop fs -copyFromLocal /tmp/test.txt oss://my-data-lake-bucket/test/
# Delete a file
hadoop fs -rm oss://my-data-lake-bucket/test/test.txt

Successful output confirms basic file operations work correctly.

4.2 Test large file upload and download

Test I/O performance using a 20 GB large file:

# Generate a test file
dd if=/dev/zero of=/tmp/large-file-20G bs=1M count=20480
# Upload to OSS (supports oss:// or s3a://)
hadoop fs -D io.file.buffer.size=4194304 -copyFromLocal -f /tmp/large-file-20G oss://your-bucket/path/20GB-file.img
# Test the download
hadoop fs -D io.file.buffer.size=4194304 -copyToLocal -f oss://your-bucket/path/20GB-file.img /dev/null

Sample output:

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2025-09-12 23:04:02,363 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2025-09-12 23:05:30,070 INFO prefetchstream.CachingInputStream: closed: oss://xxx/20GB-file.img
real    1m28.635s
user    0m41.589s
sys     0m10.346s

4.3 TPC-DS benchmark test (end-to-end validation)

Step 1: Load 5 TB TPC-DS data

Use the dsdgen tool to generate and upload data to OSS.
Create tables in Hive Metastore pointing to s3a:// or oss:// paths.

Step 2: Run a standard query

$SPARK_HOME/bin/beeline -u jdbc:hive2://localhost:10001/tpcds_bin_partitioned_orc_5000 -f q9.sql

5. Advanced configuration

5.1 Accelerate query performance

5.1.1 Fine-grained I/O acceleration

Data Accelerator is an SSD-based caching service from Alibaba Cloud that accelerates specific access patterns. Hadoop OSS Connector V2 integrates with Data Accelerator for on-demand acceleration.

Key benefits

Reduces access latency for small files, metadata, and random I/O.
Edge node caching reduces origin fetches and improves throughput.
Ideal for interactive queries, BI reports, and metadata scans.

I/O type	Description	Scenarios	Example
Prefix match acceleration	Enables caching for files with a specified path prefix, such as `/tpcds/q9/`.	Intermediate results of hot queries	`<keyPrefixes> <keyPrefix>tpcds/q9/</keyPrefix> <keyPrefix>tmp/report_</keyPrefix> </keyPrefixes>`
Suffix match acceleration	Matches specific file extensions, such as `.parquet` and `.orc`.	Table files with a uniform format	`<keySuffixes> <keySuffix>.parquet</keySuffix> <keySuffix>.orc</keySuffix> <keySuffix>.csv</keySuffix> </keySuffixes>`
File size range acceleration	Accelerates small files (0-50 MB).	Metadata and small partition files	`<IOSizeRanges> <!-- Cache small I/O requests (0-50 MB) to cover common scan operations --> <ioSizeRange> <ioType>SIZE</ioType> <minIOSize>0</minIOSize> <maxIOSize>52428800</maxIOSize> </ioSizeRange> </IOSizeRanges>`
Operation-based acceleration	Currently, only `getObject` operations are accelerated.	Read-intensive tasks	`<operations> <operation>getObject</operation> </operations>`
Head/tail access acceleration	Caches access to the file header (metadata) and tail (footers).	Structured file parsing	`<IOSizeRanges> <!-- Cache the first 2 MB, suitable for file header metadata --> <ioSizeRange> <ioType>HEAD</ioType> <ioSize>2097152</ioSize> </ioSizeRange> <!-- Cache the last 2 MB, suitable for Parquet/ORC footers --> <ioSizeRange> <ioType>TAIL</ioType> <ioSize>2097152</ioSize> </ioSizeRange> </IOSizeRanges>`

Note that only getObject operations can be accelerated.

Configure accelerator parameters

Add the following properties to core-site.xml:

<!-- Accelerator endpoint (internal network only) -->
<property>
    <name>fs.oss.acc.endpoint</name>
    <value>https://cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com</value>
</property>
<!-- Acceleration rules: Enable acceleration for small files and hot data -->
<property>
    <name>fs.oss.acc.rules</name>
    <value><![CDATA[
        <rules>
            <rule>
                <keyPrefixes>
                    <keyPrefix>tpcds/q9/</keyPrefix>
                </keyPrefixes>
                 <IOSizeRanges>
                      <!-- Cache the first 2 MB, suitable for file header metadata -->
                      <ioSizeRange>
                          <ioType>HEAD</ioType>
                          <ioSize>2097152</ioSize>
                      </ioSizeRange>
                      <!-- Cache the last 2 MB, suitable for Parquet/ORC footers -->
                      <ioSizeRange>
                          <ioType>TAIL</ioType>
                          <ioSize>2097152</ioSize>
                      </ioSizeRange>
                      <!-- Cache small I/O requests (0-50 MB) -->
                      <ioSizeRange>
                          <ioType>SIZE</ioType>
                          <minIOSize>0</minIOSize>
                          <maxIOSize>52428800</maxIOSize>
                      </ioSizeRange>
                  </IOSizeRanges>
                <operations><operation>getObject</operation></operations>
            </rule>
        </rules>
    ]]></value>
</property>

Tag case sensitivity: The tags <IOSizeRanges> and <ioSizeRange> are case-sensitive. Use them exactly as shown in this document. Sub-tags must be nested within <IOSizeRanges> and cannot be placed directly under <rule>.

ioType values

`ioType`	Description	Associated Fields
`HEAD`	Reads data from the beginning of a file.	`ioSize`
`TAIL`	Reads data backward from the end of a file.	`ioSize`
`SIZE`	Reads from any offset, as long as the request length falls within the `[minIOSize, maxIOSize]` range.	`minIOSize`, `maxIOSize`

Note

SIZE does not refer to the total file size, but to the size of a single I/O request.

5.2 Local download cache

Mechanism

Hadoop OSS Connector V2 caches prefetched data blocks to local disk for faster repeated reads.

How it works

With prefetch enabled (fs.oss.prefetch.version=v2), the system downloads subsequent data blocks from OSS in advance.
Blocks are cached in memory or written to local disk at fs.oss.buffer.prefetch.dir.
On seek or re-read, the connector loads from local cache, avoiding repeated network requests.

Key parameters

Parameter	Description
`fs.oss.prefetch.block.size`	The size of each prefetch block. Default: 128 KB.
fs.oss.prefetch.block.count	The number of prefetch blocks for a single stream. Default: 8 blocks (approximately 1 MB).
`fs.oss.prefetch.max.disk.blocks.count`	Maximum number of blocks per stream that can be cached to disk. The default is 16. Set to 0 to disable this feature. Set this to twice the value of `fs.oss.prefetch.block.count`.
`fs.oss.buffer.prefetch.dir`	The local cache directory. Default: `${env.LOCAL_DIRS:-${hadoop.tmp.dir}}/oss_prefetch`. We recommend using an SSD.

Example scenario: When running a TPC-DS query, if a Parquet file is scanned multiple times, the second read can be fully served from the local disk cache, reducing I/O latency by more than 60%.

Usage notes:

Enable this feature only with local SSD storage.
Ensure the disk specified by fs.oss.buffer.prefetch.dir has sufficient space (≥ 128 GB is recommended).

5.3 S3 protocol compatibility

Hadoop OSS Connector V2 provides full S3 protocol compatibility for smooth migration from AWS S3 or other S3-compatible platforms. Access OSS with the s3a:// protocol — no changes to existing metadata, scripts, or job code.

For data lakes with Hive tables whose LOCATION points to s3a://bucket/path, data reads go directly to Alibaba Cloud OSS without metadata updates.

Key benefits

Access OSS using the s3a:// URI.
No Hive Metastore metadata changes needed — the storage switch is transparent to applications.
Unified access model reduces operational complexity and migration costs.

Configuration

Add the following properties to $HADOOP_HOME/etc/hadoop/core-site.xml to map S3A requests to OSS:

<configuration>
    <!-- Enable the OSS implementation in S3A compatibility mode -->
    <property>
        <name>fs.AbstractFileSystem.s3a.impl</name>
        <value>org.apache.hadoop.fs.aliyun.oss.v2.OSSWithS3A</value>
    </property>
    <!-- Specify the s3a file system implementation class -->
    <property>
        <name>fs.s3a.impl</name>
        <value>org.apache.hadoop.fs.aliyun.oss.v2.AliyunOSSPerformanceFileSystem</value>
    </property>
    <!-- S3A access key (use your Alibaba Cloud access key) -->
    <property>
        <name>fs.s3a.access.key</name>
        <value>YourAccessKeyId</value>
    </property>
    <property>
        <name>fs.s3a.secret.key</name>
        <value>YourAccessKeySecret</value>
    </property>
    <!-- OSS endpoint (we recommend using an internal endpoint to improve performance) -->
    <property>
        <name>fs.s3a.endpoint</name>
        <value>oss-cn-hangzhou-internal.aliyuncs.com</value>
    </property>
    <!-- Optional: Disable SSL to avoid certificate issues -->
    <property>
        <name>fs.s3a.connection.ssl.enabled</name>
        <value>false</value>
    </property>
</configuration>

Description of the fs.AbstractFileSystem.s3a.impl configuration option:

OSSWithS3A: Supports accessing data on OSS using the s3a:// URI.

OSS: Supports accessing data on OSS using the oss:// URI.

Example

Assume you have created the following external table in Hive Metastore, which is a typical S3 scenario:

CREATE EXTERNAL TABLE ods_user (
    id BIGINT,
    name STRING,
    dt STRING
)
PARTITIONED BY (dt STRING)
STORED AS PARQUET
LOCATION 's3a://my-data-lake-bucket/dw/ods/user_info/';

With this configuration, Spark and Hive queries automatically read data from OSS through Hadoop OSS Connector V2 — no changes to table definitions or SQL scripts.

Run a query:

SELECT count(*) FROM ods_user WHERE dt = '20250405';

→ Automatically loads data from oss://my-data-lake-bucket/dw/ods/user_info/.

Compatibility limitations

Limitation	Description
Protocol limitation	Only `s3a://` is currently supported. `s3://` and `s3n://` are not supported.
Behavioral consistency	Most S3A parameters, such as those for connection pools and timeouts, are respected and behave consistently with the native S3A client.

Summary: s3a:// protocol compatibility simplifies cross-cloud migration — whether building a new data lake or migrating an existing system.

Note

In production, combine this with accelerator and prefetch configurations to improve s3a:// query performance.

6. Benchmarks and recommendations

6.1 20 GB file download throughput comparison

Version	Average throughput
Hadoop OSS Connector V1 (Apache Hadoop community version)	40 MB/s
Hadoop OSS Connector V2	232 MB/s
Hadoop OSS Connector V2 + OSS Accelerator	476 MB/s

Conclusion: Prefetching and connection optimization significantly improve throughput. Enabling the OSS Accelerator delivers even better results, especially for large sequential reads.

6.2 TPC-DS query performance

Version	Total runtime (seconds)
Hadoop OSS Connector V1 (Apache Hadoop community version)	37,816
Hadoop OSS Connector V2	27,057
Hadoop OSS Connector V2 + OSS Accelerator	23,598

Conclusion: OSS Connector V2 + Accelerator improves performance by 37.6% over V1 and 12.8% over V2 alone, reducing query response times for high-frequency OLAP scenarios.

6.3 Optimization recommendations

Production-tested recommendations to maximize Hadoop OSS Connector V2 performance in cloud data lake environments.

Core configuration optimization

Category	Best practice	Description
Storage protocol compatibility	Use the `s3a://` protocol for cross-cloud compatibility.	Enables seamless integration with the AWS S3 ecosystem without modifying metadata.
Prefetch optimization	Enable `fs.oss.prefetch.version=v2`.	Improves sequential read performance for large files. Requires a buffer size greater than 2 MB.
Connection and buffer tuning	Set the number of connections and the buffer directory appropriately.	Set `connection.maximum` to 64–128 and point `buffer.dir` to an SSD.

Acceleration and caching strategies

Category

Best practice

Description

Enable OSS Accelerator

Cache frequently accessed hot data to reduce bandwidth usage.

Significantly reduces access latency for small files and metadata.

Improve prefetch efficiency

Use an SSD for the prefetch cache disk (optional).

Use a high-speed medium, such as an Elastic Ephemeral Disk, for fs.oss.buffer.prefetch.dir as a cost-effective local cache.

If a high-speed medium is unavailable, disable this feature.

Scenario-based I/O acceleration rules

Use case	Recommended rules	Description
Reading Parquet/ORC tables	`TAIL + SIZE`	Cache file footers and small I/O requests.
Log analysis (tail -f)	`TAIL`	Cache the last 1–2 MB.
Frequent reads of metadata files	`HEAD + small file size range`	Accelerate access to small files like `_SUCCESS`.
High-frequency access to dimension tables	`keyPrefixes + keySuffixes + getObject`	Use fine-grained path matching for acceleration.

Optimization checklist

Item	Recommendation
Storage protocol	Prioritize using `s3a://` for cross-cloud compatibility.
Local cache	Point `fs.oss.buffer.dir` and `fs.oss.buffer.prefetch.dir` to high-performance disks such as an Elastic Ephemeral Disk.
Prefetch configuration	Enable `v2` mode, with `block.count=8` and `io.threshold=2MB`.
Hot data shared cache	Place the OSS Accelerator in the same availability zone as your Spark ECS instances. Start with a large cache for initial testing to find the performance ceiling, then adjust based on cache hit rate. In production, configure `IOSizeRanges` and path rules to optimize capacity and hot data access.
Local Cache Enablement	Enable this feature if you have an SSD; otherwise, disable it to avoid performance degradation.

Stress-test with TPC-DS or your actual workload. Iterate on configuration for optimal results.