Use create-select-object-meta to query metadata of CSV and JSON objects - Object Storage Service

Run create-select-object-meta to retrieve metadata from a CSV or JSON object in OSS—such as the total number of rows, columns, and splits. If the metadata does not exist, OSS scans the entire object, analyzes it, and saves the results. Subsequent calls return the cached metadata without rescanning.

Prerequisites

Before you begin, ensure that you have:

An Alibaba Cloud account with permission to manage objects, or a RAM user or Security Token Service (STS) identity with the oss:PutObject permission. For details, see Attach a custom policy to a RAM user

Syntax

ossutil api create-select-object-meta --bucket <value> --key <value> --select-meta-request <value> [flags]

Square brackets indicate optional parameters. Parameters without brackets are required.

Parameter	Type	Required	Description
`--bucket`	string	Yes	Name of the bucket
`--key`	string	Yes	Full path of the object
`--select-meta-request`	string	Yes	Request body for the CreateSelectObjectMeta operation, in XML or JSON format. Use the `file://` prefix to read the value from a file

This command maps to the CreateSelectObjectMeta API operation. For global command-line options, see Command-line options.

--select-meta-request

Pass the request body as an inline string or as a file reference using the file:// prefix.

Both XML and JSON formats are supported.

XML format

<SelectMetaRequest>
  <InputSerialization>
    <CompressionType>string</CompressionType>
    <CSV>
      <RecordDelimiter>string</RecordDelimiter>
      <FieldDelimiter>string</FieldDelimiter>
      <QuoteCharacter>string</QuoteCharacter>
      <CommentCharacter>string</CommentCharacter>
      <Range>string</Range>
      <AllowQuotedRecordDelimiter>boolean</AllowQuotedRecordDelimiter>
      <FileHeaderInfo>string</FileHeaderInfo>
    </CSV>
    <JSON>
      <Range>string</Range>
      <ParseJsonNumberAsString>boolean</ParseJsonNumberAsString>
      <Type>string</Type>
    </JSON>
  </InputSerialization>
  <OverwriteIfExists>boolean</OverwriteIfExists>
</SelectMetaRequest>

JSON format

{
  "InputSerialization": {
    "CompressionType": "string",
    "CSV": {
      "CommentCharacter": "string",
      "Range": "string",
      "AllowQuotedRecordDelimiter": boolean,
      "FileHeaderInfo": "string",
      "RecordDelimiter": "string",
      "FieldDelimiter": "string",
      "QuoteCharacter": "string"
    },
    "JSON": {
      "Type": "string",
      "Range": "string",
      "ParseJsonNumberAsString": boolean
    }
  },
  "OverwriteIfExists": boolean
}

Examples

CSV object

The following examples retrieve metadata from exampleobject in examplebucket. All three approaches pass equivalent CSV configuration.

Using an XML configuration file

Create select-meta-request.xml:

<?xml version="1.0"?>
<CsvMetaRequest>
    <InputSerialization>
        <CSV>
            <RecordDelimiter>Cg==</RecordDelimiter>
            <FieldDelimiter>LA==</FieldDelimiter>
            <QuoteCharacter>Ig==</QuoteCharacter>
        </CSV>
    </InputSerialization>
    <OverwriteIfExisting>false</OverwriteIfExisting>
</CsvMetaRequest>

ossutil api create-select-object-meta --bucket examplebucket --key exampleobject --select-meta-request file://select-meta-request.xml

Using a JSON configuration file

Create select-meta-request.json:

{
  "InputSerialization": {
    "CSV": {
      "RecordDelimiter": "Cg==",
      "FieldDelimiter": "LA==",
      "QuoteCharacter": "Ig=="
    }
  },
  "OverwriteIfExisting": "false"
}

ossutil api create-select-object-meta --bucket examplebucket --key exampleobject --select-meta-request file://select-meta-request.json

Inline JSON string

ossutil api create-select-object-meta --bucket examplebucket --key exampleobject --select-meta-request "{\"InputSerialization\":{\"CSV\":{\"RecordDelimiter\":\"Cg==\",\"FieldDelimiter\":\"LA==\",\"QuoteCharacter\":\"Ig==\"}},\"OverwriteIfExisting\":\"false\"}"

JSON object

The following examples retrieve metadata from a JSON Lines object. All three approaches pass equivalent JSON configuration.

Using an XML configuration file

Create select-request.xml:

<?xml version="1.0"?>
<JsonMetaRequest>
    <InputSerialization>
        <JSON>
            <Type>LINES</Type>
        </JSON>
    </InputSerialization>
    <OverwriteIfExisting>false</OverwriteIfExisting>
</JsonMetaRequest>

ossutil api create-select-object-meta --bucket examplebucket --key exampleobject --select-meta-request file://select-meta-request.xml

Using a JSON configuration file

Create select-request.json:

{
  "InputSerialization": {
    "JSON": {
      "Type": "LINES"
    }
  },
  "OverwriteIfExisting": "false"
}

ossutil api create-select-object-meta --bucket examplebucket --key exampleobject --select-meta-request file://select-meta-request.json

Inline JSON string

ossutil api create-select-object-meta --bucket examplebucket --key exampleobject --select-meta-request "{\"InputSerialization\":{\"JSON\":{\"Type\":\"LINES\"}},\"OverwriteIfExisting\":\"false\"}"