Use the select-object command to execute SQL statements on objects - Object Storage Service

Run a SQL statement against an object in a bucket and retrieve matching records — without downloading the full object.

Usage notes

By default, an Alibaba Cloud account has the permissions to run this command. RAM users and Security Token Service (STS) users require the oss:GetObject permission. For details, see Attach a custom policy to a RAM user.

Syntax

ossutil api select-object --bucket <bucket-name> --key <object-key> --select-request <value> [flags]

Parameters

Parameter	Type	Required	Description
`--bucket`	string	Yes	Name of the bucket.
`--key`	string	Yes	Full path of the object.
`--select-request`	string	Yes	The select request configuration, in XML or JSON format. Pass a file path prefixed with `file://` to load from a file, or pass the configuration inline as an escaped JSON string.

The select-object command corresponds to the SelectObject API operation. For all supported global flags, see Supported global command line options.

Quick start

The following example queries the first 100 lines of a CSV object and returns results in JSON format.

The SQL expression (select Year,StateAbbr,CityName,Short_Question_Text from ossobject) must be Base64-encoded before passing it to the Expression field.

1. Create the request configuration file (`select-request.json`):

{
  "Expression": "c2VsZWN0IFllYXIsU3RhdGVBYmJyLCBDaXR5TmFtZSwgU2hvcnRfUXVlc3Rpb25fVGV4dCBmcm9tIG9zc29iamVjdA==",
  "InputSerialization": {
    "CSV": {
      "FileHeaderInfo": "Use",
      "Range": "line-range=0-100"
    }
  },
  "OutputSerialization": {
    "JSON": {
      "RecordDelimiter": ","
    }
  }
}

The Expression value above is the Base64 encoding of: select Year,StateAbbr,CityName,Short_Question_Text from ossobject

2. Run the command:

ossutil api select-object \
  --bucket examplebucket \
  --key exampleobject \
  --select-request file://select-request.json

--select-request configuration

The --select-request option accepts configuration in XML or JSON format. If the value starts with file://, ossutil reads the configuration from the specified file.

XML format

<SelectRequest>
  <Expression>string</Expression>
  <InputSerialization>
    <CompressionType>string</CompressionType>
    <CSV>
      <Range>string</Range>
      <AllowQuotedRecordDelimiter>boolean</AllowQuotedRecordDelimiter>
      <FileHeaderInfo>string</FileHeaderInfo>
      <RecordDelimiter>string</RecordDelimiter>
      <FieldDelimiter>string</FieldDelimiter>
      <QuoteCharacter>string</QuoteCharacter>
      <CommentCharacter>string</CommentCharacter>
    </CSV>
    <JSON>
      <Range>string</Range>
      <ParseJsonNumberAsString>boolean</ParseJsonNumberAsString>
      <Type>string</Type>
    </JSON>
  </InputSerialization>
  <OutputSerialization>
    <CSV>
      <RecordDelimiter>string</RecordDelimiter>
      <FieldDelimiter>string</FieldDelimiter>
    </CSV>
    <JSON>
      <RecordDelimiter>string</RecordDelimiter>
    </JSON>
    <KeepAllColumns>boolean</KeepAllColumns>
    <OutputHeader>boolean</OutputHeader>
    <OutputRawData>boolean</OutputRawData>
    <EnablePayloadCrc>boolean</EnablePayloadCrc>
  </OutputSerialization>
  <Options>
    <MaxSkippedRecordsAllowed>integer</MaxSkippedRecordsAllowed>
    <SkipPartialDataRecord>boolean</SkipPartialDataRecord>
  </Options>
</SelectRequest>

JSON format

{
  "Expression": "string",
  "InputSerialization": {
    "CompressionType": "string",
    "CSV": {
      "FileHeaderInfo": "string",
      "RecordDelimiter": "string",
      "FieldDelimiter": "string",
      "QuoteCharacter": "string",
      "CommentCharacter": "string",
      "Range": "string",
      "AllowQuotedRecordDelimiter": boolean
    },
    "JSON": {
      "Range": "string",
      "ParseJsonNumberAsString": boolean,
      "Type": "string"
    }
  },
  "OutputSerialization": {
    "CSV": {
      "RecordDelimiter": "string",
      "FieldDelimiter": "string"
    },
    "JSON": {
      "RecordDelimiter": "string"
    },
    "KeepAllColumns": boolean,
    "OutputHeader": boolean,
    "OutputRawData": boolean,
    "EnablePayloadCrc": boolean
  },
  "Options": {
    "SkipPartialDataRecord": boolean,
    "MaxSkippedRecordsAllowed": integer
  }
}

Examples

All examples use examplebucket as the bucket name and exampleobject as the object key. The Expression field in each example is a Base64-encoded SQL statement.

Query a CSV object

The SQL expression is: select Year,StateAbbr,CityName,Short_Question_Text from ossobject

Using an XML configuration file:

Create select-request.xml:

<?xml version="1.0" encoding="UTF-8"?>
<SelectRequest>
    <Expression>c2VsZWN0IFllYXIsU3RhdGVBYmJyLCBDaXR5TmFtZSwgU2hvcnRfUXVlc3Rpb25fVGV4dCBmcm9tIG9zc29iamVjdA==</Expression>
    <InputSerialization>
        <CSV>
            <FileHeaderInfo>Use</FileHeaderInfo>
            <Range>line-range=0-100</Range>
        </CSV>
    </InputSerialization>
    <OutputSerialization>
        <JSON>
            <RecordDelimiter>,</RecordDelimiter>
        </JSON>
    </OutputSerialization>
</SelectRequest>

ossutil api select-object --bucket examplebucket --key exampleobject --select-request file://select-request.xml

Using a JSON configuration file:

Create select-request.json:

{
  "Expression": "c2VsZWN0IFllYXIsU3RhdGVBYmJyLCBDaXR5TmFtZSwgU2hvcnRfUXVlc3Rpb25fVGV4dCBmcm9tIG9zc29iamVjdA==",
  "InputSerialization": {
    "CSV": {
      "FileHeaderInfo": "Use",
      "Range": "line-range=0-100"
    }
  },
  "OutputSerialization": {
    "JSON": {
      "RecordDelimiter": ","
    }
  }
}

ossutil api select-object --bucket examplebucket --key exampleobject --select-request file://select-request.json

Using inline JSON:

ossutil api select-object --bucket examplebucket --key exampleobject --select-request "{\"Expression\":\"c2VsZWN0IFllYXIsU3RhdGVBYmJyLCBDaXR5TmFtZSwgU2hvcnRfUXVlc3Rpb25fVGV4dCBmcm9tIG9zc29iamVjdA==\",\"InputSerialization\":{\"CSV\":{\"FileHeaderInfo\":\"Use\",\"Range\":\"line-range=0-100\"}},\"OutputSerialization\":{\"JSON\":{\"RecordDelimiter\":\",\"}}}"

Query a JSON object

The SQL expression is: select * from ossobject.objects[*] where party = 'Democrat'

Using an XML configuration file:

Create select-request.xml:

<?xml version="1.0" encoding="UTF-8"?>
<SelectRequest>
    <Expression>c2VsZWN0ICogZnJvbSBvc3NvYmplY3Qub2JqZWN0c1sqXSB3aGVyZSBwYXJ0eSA9ICdEZW1vY3JhdCc=</Expression>
    <InputSerialization>
        <CSV>
            <SplitRange></SplitRange>
        </CSV>
        <JSON>
            <Type>DOCUMENT</Type>
            <SplitRange></SplitRange>
        </JSON>
    </InputSerialization>
    <OutputSerialization>
        <CSV></CSV>
        <JSON>
            <RecordDelimiter>LA==</RecordDelimiter>
        </JSON>
    </OutputSerialization>
    <Options></Options>
</SelectRequest>

ossutil api select-object --bucket examplebucket --key exampleobject --select-request file://select-request.xml

Using a JSON configuration file:

Create select-request.json:

{
  "Expression": "c2VsZWN0ICogZnJvbSBvc3NvYmplY3Qub2JqZWN0c1sqXSB3aGVyZSBwYXJ0eSA9ICdEZW1vY3JhdCc=",
  "InputSerialization": {
    "CSV": {},
    "JSON": {
      "Type": "DOCUMENT"
    }
  },
  "OutputSerialization": {
    "JSON": {
      "RecordDelimiter": "LA=="
    }
  }
}

ossutil api select-object --bucket examplebucket --key exampleobject --select-request file://select-request.json