MaxCompute:Open Storage SDK examples - Java SDK

SplitOptions

• Parameter definition

  The SplitOptions object parameters are defined as follows:

  public class SplitOptions {
  
      public static SplitOptions.Builder newBuilder() {
          return new Builder();
      }
  
      public static class Builder {
  
        public SplitOptions.Builder SplitByByteSize(long splitByteSize);
    
        public SplitOptions.Builder SplitByRowOffset();
    
        public SplitOptions.Builder withCrossPartition(boolean crossPartition);
    
        public SplitOptions.Builder withMaxFileNum(int splitMaxFileNum);
    
        public SplitOptions build();
      }
  }

• Parameter descriptions

  • SplitByByteSize(long splitByteSize)

    Splits data based on the specified splitByteSize parameter. The size of each data partition returned by the server does not exceed splitByteSize (in bytes).

    • The custom split size must be at least 10 × 1024 × 1024 (10 MB).

    • If SplitByByteSize(long splitByteSize) is not used to customize the split size, the system uses the default value of 256 × 1024 × 1024 (256 MB).

  • SplitByRowOffset()

    Splits data by row, allowing the client to read data from a specified index.

  • withCrossPartition(boolean crossPartition)

    Specifies whether to allow a single data shard to include multiple data partitions. The crossPartition parameter accepts the following values:

    • true (default): allows a single data shard to contain multiple data partitions.

    • false: Does not allow it.

  • withMaxFileNum(int splitMaxFileNum)

    When a table has many files, specify the maximum number of physical files in a single data partition to generate more data partitions.

    By default, there is no limit on the number of physical files in a single data partition.

  • build(): Creates a SplitOptions object.

• Examples

  // 1. Split data by size, set SplitSize to 256 MB
  
  SplitOptions splitOptionsByteSize = 
        SplitOptions.newBuilder().SplitByByteSize(256 * 1024L * 1024L).build()
  
  // 2. Split data by RowOffset
  
  SplitOptions splitOptionsCount = 
        SplitOptions.newBuilder().SplitByRowOffset().build()
  
  // 3. Set the maximum number of files in a single split to 1
  
  SplitOptions splitOptionsCount = 
        SplitOptions.newBuilder().SplitByRowOffset().withMaxFileNum(1).build()

ArrowOptions

• Parameter definition

  The ArrowOptions object parameters are defined as follows:

  public class ArrowOptions {
      
      public static Builder newBuilder() {
          return new Builder();
      }
  
      public static class Builder {
  
          public Builder withTimestampUnit(TimestampUnit unit);
  
          public Builder withDatetimeUnit(TimestampUnit unit);
  
          public ArrowOptions build();
      }
  
      public enum TimestampUnit {
          SECOND,
          MILLI,
          MICRO,
          NANO;
      }
  }

• Parameter descriptions

  • TimestampUnit

    Specifies the unit for Timestamp and Datetime data types. Valid values:

    • SECOND: seconds (s)

    • MILLI: milliseconds (ms)

    • MICRO: microseconds (μs)

    • NANO: nanoseconds (ns)

  • withTimestampUnit(TimestampUnit unit)

    Specifies the unit for the Timestamp data type. Default: NANO.

  • withDatetimeUnit(TimestampUnit unit)

    Specifies the unit for the Datetime data type. Default: MILLI.

• Examples

  ArrowOptions options = ArrowOptions.newBuilder()
            .withDatetimeUnit(ArrowOptions.TimestampUnit.MILLI)
            .withTimestampUnit(ArrowOptions.TimestampUnit.NANO)
            .build()

Predicate

• Parameter definition

  The SplitOptions object parameters are defined as follows:

  // 1. Binary operations
  
  public class BinaryPredicate extends Predicate {
  
    public enum Operator {
      /**
       * Binary operation operators
       */
      EQUALS("="),
      NOT_EQUALS("!="),
      GREATER_THAN(">"),
      LESS_THAN("<"),
      GREATER_THAN_OR_EQUAL(">="),
      LESS_THAN_OR_EQUAL("<=");
     
    }
  
    public BinaryPredicate(Operator operator, Serializable leftOperand, Serializable rightOperand);
  
    public static BinaryPredicate equals(Serializable leftOperand, Serializable rightOperand);
  
    public static BinaryPredicate notEquals(Serializable leftOperand, Serializable rightOperand);
  
    public static BinaryPredicate greaterThan(Serializable leftOperand, Serializable rightOperand);
  
    public static BinaryPredicate lessThan(Serializable leftOperand, Serializable rightOperand);
  
    public static BinaryPredicate greaterThanOrEqual(Serializable leftOperand,
                                                     Serializable rightOperand);
  
    public static BinaryPredicate lessThanOrEqual(Serializable leftOperand,
                                                  Serializable rightOperand);
  }
  
  // 2. Unary operations
  public class UnaryPredicate extends Predicate {
  
    public enum Operator {
      /**
       * Unary operation operators
       */
      IS_NULL("is null"),
      NOT_NULL("is not null");
    }
  
    public static UnaryPredicate isNull(Serializable operand);
    public static UnaryPredicate notNull(Serializable operand);
  }
  
  ### 3. IN and NOT IN
  public class InPredicate extends Predicate {
  
    public enum Operator {
      /**
       * IN and NOT IN operators for set membership check
       */
      IN("in"),
      NOT_IN("not in");
    }
  
    public InPredicate(Operator operator, Serializable operand, List<Serializable> set);
  
    public static InPredicate in(Serializable operand, List<Serializable> set);
  
    public static InPredicate notIn(Serializable operand, List<Serializable> set);
  }
  
  // 4. Column names
  public class Attribute extends Predicate {
  
    public Attribute(Object value);
      
    public static Attribute of(Object value);
  }
  
  // 5. Constants
  public class Constant extends Predicate {
  
    public Constant(Object value);
  
    public static Constant of(Object value);
  }
  
  // 6. Compound operations
  public class CompoundPredicate extends Predicate {
  
    public enum Operator {
      /**
       * Compound predicate operators
       */
      AND("and"),
      OR("or"),
      NOT("not");
    }
  
    public CompoundPredicate(Operator logicalOperator, List<Predicate> predicates);
  
    public static CompoundPredicate and(Predicate... predicates);
    
    public static CompoundPredicate or(Predicate... predicates);
  
    public static CompoundPredicate not(Predicate predicates);
  
    public void addPredicate(Predicate predicate);
  }
  
  // 7. Raw predicates (RawPredicate)
  // If existing methods do not meet requirements, assemble predicates based on SQL syntax
    public class RawPredicate extends Predicate {
        public RawPredicate(String rawExpr);
        public static RawPredicate of(String rawExpr);
    }

• Examples

  // 1. c1 > 20000 and c2 < 100000
  BinaryPredicate c1 = new BinaryPredicate(BinaryPredicate.Operator.GREATER_THAN, Attribute.of("c1"), Constant.of(20000));
  BinaryPredicate c2 = new BinaryPredicate(BinaryPredicate.Operator.LESS_THAN, Attribute.of("c2"), Constant.of(100000));
  CompoundPredicate predicate =
          new CompoundPredicate(CompoundPredicate.Operator.AND, ImmutableList.of(c1, c2));
  
  // 2. c1 is not null
  Predicate predicate = new UnaryPredicate(UnaryPredicate.Operator.NOT_NULL,  Attribute.of("c1"));
  
    
  // 3. c1 in (1, 10001)
  Predicate predicate =
          new InPredicate(InPredicate.Operator.IN,  Attribute.of("c1"), ImmutableList.of(Constant.of(1), Constant.of(10001)));
  
  // 4. Use RawPredicate to assemble predicates (supports all types)
    Predicate predicate = RawPredicate.of("c1 > 20000 and c2 < 100000");

EnvironmentSettings

• Parameter definition

  The EnvironmentSettings interface is defined as follows:

  public class EnvironmentSettings {
  
      public static Builder newBuilder() {
          return new Builder();
      }
  
      public static class Builder {
  
          public Builder withDefaultProject(String projectName);
  
          public Builder withDefaultSchema(String schema);
  
          public Builder withServiceEndpoint(String endPoint);
  
          public Builder withTunnelEndpoint(String tunnelEndPoint);
  
          public Builder withQuotaName(String quotaName);
  
          public Builder withCredentials(Credentials credentials);
  
          public Builder withRestOptions(RestOptions restOptions);
  
          public EnvironmentSettings build();
      }
  }

• Parameter descriptions

  • withDefaultProject(String projectName)

    Sets the project name. projectNameThe projectName parameter is the MaxCompute project name.

    • Log on to the MaxCompute console, switch the region in the upper-left corner.

    • Choose Manage Configurations > Projects to view the MaxCompute project name.

  • withDefaultSchema(String schema)

    Sets the default schema. The schema parameter is the MaxCompute schema name. For more information about schemas, see Schema operations.

  • withServiceEndpoint(String endPoint)

    Sets the service endpoint.Endpoint.

  • withTunnelEndpoint(String tunnelEndPoint)

    Sets the tunnel endpoint.Endpoint.

  • withQuotaName(String quotaName)

    Specifies the quota name to use.

    MaxCompute supports two resource types: exclusive Data Transmission Service resource groups (subscription) Get the quota name as follows:

    • Exclusive Data Transmission Service resource group

      Log in to the MaxCompute console and select a region in the upper-left corner.

      In the left-side navigation pane, choose Manage Configurations > Quotas .

      View available quotas. For more information, see Compute resources - Quota management.

    • Log in to the MaxCompute console and select a region in the upper-left corner.

      In the left-side navigation pane, choose Manage Configurations > Tenants .

      On the Tenant Property tab, enable the Storage API Switch switch.

  • withCredentials(Credentials credentials)

    Specifies authentication information. For more information, see Credentials.

Credentials

• Object definition

  public class Credentials {
  
      public static Builder newBuilder() {
          return new Builder();
      }
  
      public static class Builder {
  
          public Builder withAccount(Account account);
  
          public Builder withAppAccount(AppAccount appAccount);
  
          public Builder withAppStsAccount(AppStsAccount appStsAccount);
  
          public Credentials build();
      }
  
  }

• Parameter descriptions

  • withAccount(Account account)

    Specifies the Odps Account object.

  • withAppAccount(AppAccount appAccount)

    Specifies the Odps appAccount object.

  • withAppStsAccount(AppStsAccount appStsAccount)

    Specifies the Odps appStsAccount object.

  • withRestOptions(RestOptions restOptions)

    Specifies network access configuration. RestOptions is defined as follows:

    public class RestOptions implements Serializable {
    
        public static Builder newBuilder() {
            return new RestOptions.Builder();
        }
    
        public static class Builder {
            public Builder witUserAgent(String userAgent);
            public Builder withConnectTimeout(int connectTimeout);
            public Builder withReadTimeout(int readTimeout);
            public RestOptions build();
        }
    }

    • witUserAgent(String userAgent): Specifies the userAgent information.

    • withConnectTimeout(int connectTimeout): Specifies the connection timeout for establishing the underlying network connection. Default: 10 seconds.

    • withReadTimeout(int readTimeout): Specifies the read timeout for the underlying network connection. Default: 120 seconds.

• DataSchema is defined as follows:

  public class DataSchema implements Serializable {
      
      List<Column> getColumns();
  
      List<String> getPartitionKeys();
  
      List<String> getColumnNames();
  
      List<TypeInfo> getColumnDataTypes();
  
      Optional<Column> getColumn(int columnIndex);
  
      Optional<Column> getColumn(String columnName);
  
  }

• Parameter descriptions

  • getColumns(): Gets column information for the table and partitions to read.

  • getPartitionKeys(): Gets partition column names to read.

  • getColumnNames(): Gets column names for the table and partitions to read.

  • getColumnDataTypes(): Gets column data types for the table and partitions to read.

  • getColumn(int columnIndex): Gets a column object by index. Returns empty if the index is out of range.

  • getColumn(String columnName): Gets a column object by name. If the column name columnName does not exist in the table, returns empty.columnName

• InputSplitAssigner is defined as follows:

  public interface InputSplitAssigner {
  
      int getSplitsCount();
  
      long getTotalRowCount();
  
      InputSplit getSplit(int index);
  
      InputSplit getSplitByRowOffset(long startIndex, long numRecord);
  }

• Parameter descriptions

  • getSplitsCount(): Gets the number of data partitions in the session.

    Note

    When SplitOptions is SplitByByteSize, this method returns a value greater than or equal to 0.

  • getTotalRowCount(): Gets the total number of data rows in the session.

    Note

    When SplitOptions is SplitByByteSize, this method returns a value greater than or equal to 0.

  • getSplit(int index): Gets the InputSplit for the specified partition Index. The index parameter ranges from [0,SplitsCount-1].

  • getSplitByRowOffset(long startIndex, long numRecord): Gets the corresponding InputSplit. Parameters are as follows:

  • startIndex: Starting row index for InputSplit data reading. Range: [0,RecordCount-1].

  • numRecord: Number of data rows for InputSplit to read.

• Examples

  // 1. If SplitOptions is SplitByByteSize
  
  TableBatchReadSession scan = ...;
  InputSplitAssigner assigner = scan.getInputSplitAssigner();
  int splitCount = assigner.getSplitsCount();
  for (int k = 0; k < splitCount; k++) {
      InputSplit split = assigner.getSplit(k);
      ...
  }
  
  // 2. If SplitOptions is SplitByRowOffset
  TableBatchReadSession scan = ...;
  InputSplitAssigner assigner = scan.getInputSplitAssigner();
  long rowCount = assigner.getTotalRowCount();
  long recordsPerSplit = 10000;
  for (long offset = 0; offset < numRecords; offset += recordsPerSplit) {
      recordsPerSplit = Math.min(recordsPerSplit, numRecords - offset);
      InputSplit split = assigner.getSplitByRowOffset(offset, recordsPerSplit);
      ...
  }

• ReaderOptions is defined as follows:

  public class ReaderOptions {
      
      public static ReaderOptions.Builder newBuilder() {
          return new Builder();
      }
  
      public static class Builder {
  
          public Builder withMaxBatchRowCount(int maxBatchRowCount);
  
          public Builder withMaxBatchRawSize(long batchRawSize);
  
          public Builder withCompressionCodec(CompressionCodec codec);
  
          public Builder withBufferAllocator(BufferAllocator allocator);
  
          public Builder withReuseBatch(boolean reuseBatch);
  
          public Builder withSettings(EnvironmentSettings settings);
  
          public ReaderOptions build();
      }
  
  }

• Parameter descriptions

  • withMaxBatchRowCount(int maxBatchRowCount)

    Specifies the maximum number of rows per batch returned by the server. The maxBatchRowCount parameter defaults to a maximum of 4096.

  • withMaxBatchRawSize(long batchRawSize)

    Specifies the maximum raw byte size per batch returned by the server.

  • withCompressionCodec(CompressionCodec codec)

    Specifies the data compression type. Only ZSTD and LZ4_FRAME are supported.

    Note

    • Transferring large amounts of uncompressed Arrow data directly can significantly increase transfer time due to network bandwidth limits.

    • If no compression type is specified, data is not compressed by default.

  • withBufferAllocator(BufferAllocator allocator)

    Specifies the memory allocator for reading Arrow data.

  • withReuseBatch(boolean reuseBatch)

    Specifies whether ArrowBatch memory can be reused.reuseBatch Values:

    • true (default): ArrowBatch memory can be reused.

    • false: ArrowBatch memory cannot be reused.

  • withSettings(EnvironmentSettings settings)

    Specifies the runtime environment information.