TunnelBufferedWriter is an enhanced RecordWriter in the MaxCompute Tunnel SDK that simplifies large-scale data uploads. Instead of managing data blocks and connection timeouts manually, write records to a local buffer — the SDK handles batching, submission, and automatic retry.
How it works
Standard uploads with openRecordWriter() require you to:
Split data into blocks and assign each block an ID.
Upload blocks using one or more threads.
Re-upload all blocks if any block fails.
Call
session.commit([1, 2, 3, ...])with all block IDs for server verification.
TunnelBufferedWriter encapsulates this process. Call write() to append records to the local buffer. When the buffer fills, the SDK submits a batch to the server automatically. If submission fails, the SDK retries without requiring intervention from your code.
Class definition
public class TunnelBufferedWriter implements RecordWriter {
public TunnelBufferedWriter(TableTunnel.UploadSession session, CompressOption option) throws IOException;
public long getTotalBytes();
public void setBufferSize(long bufferSize);
public void setRetryStrategy(RetryStrategy strategy);
public void write(Record r) throws IOException;
public void close() throws IOException;
}Lifecycle
| Phase | Action |
|---|---|
| Create | Call uploadSession.openBufferedWriter() to get a TunnelBufferedWriter instance |
| Write | Call write(record) repeatedly to append records to the local buffer |
| Flush | The SDK automatically flushes the buffer to the server when it is full |
| Close | Call close() to flush any remaining buffered data and finalize the upload |
| Commit | Call uploadSession.commit() to notify the server that the upload is complete |
Always call close() before commit() to terminate the upload process.
Configure the buffer size
setBufferSize(long bufferSize) controls how much memory the buffer occupies, in bytes.
| Parameter | Default | Valid range |
|---|---|---|
bufferSize | 64 MB | 1 MB – 1,000 MB |
Choose a buffer size based on your throughput:
High throughput (many records per second): Keep the default 64 MB or increase it. A larger buffer reduces the number of server submissions and prevents excessive small files, which can degrade query performance.
Low throughput (few records per second): A smaller buffer causes the SDK to flush frequently, producing many small files. Keep the buffer at 64 MB or above to avoid this.
In most cases, 64 MB is the right choice. Increase it only if profiling shows that small-file overhead is affecting query performance.
Configure the retry strategy
setRetryStrategy(RetryStrategy strategy) sets the backoff policy used when a server submission fails.
Three policies are available:
| Policy | Behavior |
|---|---|
EXPONENTIAL_BACKOFF | Wait time doubles after each retry |
LINEAR_BACKOFF | Wait time increases by a fixed amount after each retry |
CONSTANT_BACKOFF | Wait time is the same between every retry |
The default EXPONENTIAL_BACKOFF configuration uses 6 retries with an initial wait of 4 seconds, increasing to 8 s, 16 s, 32 s, 64 s, and 128 s. Do not change this unless you have a specific reason — the defaults are calibrated to handle transient network issues without overwhelming the server.
To override the retry strategy:
RetryStrategy retry
= new RetryStrategy(6, 4, RetryStrategy.BackoffStrategy.EXPONENTIAL_BACKOFF);
writer = (TunnelBufferedWriter) uploadSession.openBufferedWriter();
writer.setRetryStrategy(retry);