All Products
Search
Document Center

Data Lake Formation:Common data structures

Last Updated:May 24, 2022

Catalog

Catalog

Parameter

Type

Example

Description

CatalogId

String

1344371

The catalog ID of the metadatabase. Default value: the user ID of your Alibaba Cloud account.

CreateTime

Integer

1608863962

The time when the catalog was created. The value is in the UNIX timestamp format. Unit: seconds.

ColumnStatistics

ColumnStatisticsList

Parameter

Type

Example

Description

ColumnStatisticsDesc

Object

The description of field statistics.

ColumnStatisticsObjList

Array of ColumnStatisticsObj

The statistics on the field.

Engine

String

The engine.

IsStatsCompliant

Boolean

false

IsStatsCompliant

ColumnStatisticsObj

ColumnStatisticsObj

Parameter

Type

Example

Description

ColumnName

String

col1

The name of the field.

ColumnType

String

int

The data type of the field. Valid values include but are not limited to array, bigint, binary, boolean, char, date, decimal, double, float, int, interval, map, set, smallint, string, struct, timestamp, tinyint, union, and varchar.

ColumnStatisticsData

Object

The statistics on the field.

StatisticsData

String

{"numNulls":0,"numTrues":1,"numFalses":5}

The statistical results of the field.

StatisticsType

String

BOOLEAN_STATS

The type of statistics. Valid values include but are not limited to BOOLEAN_STATS, LONG_STATS, DOUBLE_STATS, STRING_STATS, BINARY_STATS, DECIMAL_STATS, and DATE_STATS.

Database

The information about a metadatabase.

Parameter

Type

Example

Description

CreateTime

Integer

1608707407

The time when the metadatabase was created. The value is in the UNIX timestamp format. Unit: seconds.

Description

String

this is a hive database

The description of the metadatabase.

LocationUri

String

/hdfs/hivedb/data

The storage path of the metadatabase, such as a Hadoop Distributed File System (HDFS) path.

Name

String

database_test

The name of the metadatabase.

OwnerName

String

zhangsan

The owner of the metadatabase.

OwnerType

String

USER

The type of the owner. Valid values: USER, ROLE, and GROUP.

Parameters

Parameters

The key-value pairs (KVPs) that define the parameters and attributes of the metadatabase.

Privileges

PrincipalPrivilegeSet

The permissions on the metadatabase. This parameter is optional.

UpdateTime

Integer

1608707407

The time when the metadatabase was updated. The value is in the UNIX timestamp format. Unit: seconds.

DatabaseInput

The input information about a metadatabase.

Parameter

Type

Example

Description

Description

String

this is a hive database

The description of the metadatabase.

LocationUri

String

/hdfs/hivedb/data

The storage path of the metadatabase, such as an HDFS path.

Name

String

name

The name of the metadatabase. The name can be up to 128 characters in length. The name does not support special characters other than underscores (_).

OwnerName

String

zhangsan

The owner of the metadatabase.

OwnerType

String

USER

The type of the owner. Valid values: USER, ROLE, and GROUP.

Parameters

Parameters

The KVPs that define the parameters and attributes of the metadatabase.

Privileges

PrincipalPrivilegeSet

The permissions on the metadatabase. This parameter is optional.

FieldSchema

FieldSchema

Parameter

Type

Example

Description

Comment

String

comment_day

The comments on the field.

Name

String

day

The name of the field.

Parameters

Parameters

The KVPs that define the attributes associated with the field.

Type

String

int

The data type of the field. Valid values include but are not limited to array, bigint, binary, boolean, char, date, decimal, double, float, int, interval, map, set, smallint, string, struct, timestamp, tinyint, union, and varchar.

Function

The information about a function.

Parameter

Type

Example

Description

CatalogId

String

1344371

The catalog ID of the metadatabase. Default value: the user ID of your Alibaba Cloud account.

ClassName

String

com.xxx.xxxxFunction

The name of the Java class to implement the function.

CreateTime

Integer

1608863962

The time when the function was created. The value is in the UNIX timestamp format. Unit: seconds.

DatabaseName

String

database_test

The name of the metadatabase.

FunctionName

String

function1

The name of the function.

FunctionType

String

JAVA

The type of the function. Set the value to JAVA.

OwnerName

String

zhangsan

The owner of the function.

OwnerType

String

USER

The type of the owner. Valid values: USER, ROLE, and GROUP.

ResourceUri

Array of ResourceUri

The resource uniform resource identifiers (URIs) of the function.

UpdateTime

Integer

1608863962

The time when the function was updated. The value is in the UNIX timestamp format. Unit: seconds.

FunctionInput

The input information about a function.

Parameter

Type

Example

Description

ClassName

String

com.xxx.xxxxFunction

The name of the Java class to implement the function.

FunctionName

String

function1

The name of the function.

FunctionType

String

JAVA

The type of the function. Set the value to JAVA.

OwnerName

String

zhangsan

The owner of the function.

OwnerType

String

USER

The type of the owner. Valid values: USER, ROLE, and GROUP.

ResourceUri

Array of ResourceUri

The resource URIs of the function.

ListFieldSchema

The information about table fields.

Parameter

Type

Example

Description

FieldSchema

FieldSchema

The definitions of table fields.

ListResourceUri

The list of resource URIs.

Parameter

Type

Example

Description

ResourceType

String

JAR

The type of resource. Valid values: JAR, FILE, and ARCHIVE.

Uri

String

hdfs:///tmp/jar1.jar

The URI of the resource.

LockObj

LockObj

Parameter

Type

Example

Description

CatalogId

String

1344371

The catalog ID of the metadatabase. Default value: the user ID of your Alibaba Cloud account.

DatabaseName

String

database_test

The name of the metadatabase.

PartitionName

String

dt

The name of the partition. This parameter is optional.

TableName

String

test_table

The name of the metadata table.

LockStatus

LockStatus

Parameter

Type

Example

Description

LockId

Long

1

The unique ID of the lock.

LockState

String

ACQUIRED

The status of the lock. Valid values: ACQUIRED and NOT_ACQUIRED.

Order

Order

Parameter

Type

Example

Description

Col

String

col1

The name of the field.

Order

Integer

1

A value of 1 indicates the ascending order, and a value of 0 indicates the descending order.

Parameters

Parameters

Parameter

Type

Example

Description

Partition

Partition

Parameter

Type

Example

Description

CreateTime

Integer

1608730999

The time when the partition was created. The value is in the UNIX timestamp format. Unit: seconds.

DatabaseName

String

database_test

The name of the metadatabase.

LastAccessTime

Integer

1608730999

The time when the partition was last accessed. The value is in the UNIX timestamp format. Unit: seconds.

LastAnalyzedTime

Integer

1608730999

The time when the last statistical operation was performed. The value is in the UNIX timestamp format. Unit: seconds.

Parameters

Map

The KVPs that are related to the partition.

Privileges

PrincipalPrivilegeSet

The configurations of permissions. This parameter can be left empty.

Sd

StorageDescriptor

The descriptor of the storage.

TableName

String

test_table_20201223

The name of the metadata table.

Values

Array of String

part1

The values in the partition.

PartitionInput

Partition

Parameter

Type

Example

Description

DatabaseName

String

database_test

The name of the metadatabase.

LastAccessTime

Integer

1608730999

The time when the partition was last accessed. The value is in the UNIX timestamp format. Unit: seconds.

LastAnalyzedTime

Integer

1608730999

The time when the last statistical operation was performed. The value is in the UNIX timestamp format. Unit: seconds.

Parameters

Map

The KVPs that are related to the partition.

Privileges

PrincipalPrivilegeSet

The configurations of permissions. This parameter can be left empty.

Sd

StorageDescriptor

The descriptor of the storage.

TableName

String

test_table_20201223

The name of the metadata table.

Values

Array of String

part1

The values in the partition.

PartitionValueList

PartitionValueList

Parameter

Type

Example

Description

Values

ListString

The values in the partition.

PrincipalPrivilegeSet

PrincipalPrivilegeSet Object

Parameter

Type

Example

Description

GroupPrivileges

Privileges

GroupPrivileges

RolePrivileges

Privileges

RolePrivileges

UserPrivileges

Privileges

UserPrivileges

PrivilegeGrantInfo

PrivilegeGrantInfo Object

Parameter

Type

Example

Description

CreateTime

Integer

1606226844

The time when the permission was granted. The value is in the UNIX timestamp format. Unit: seconds.

GrantOption

Boolean

true

GrantOption

Grantor

String

Grantor

Grantor

GrantorType

String

GrantorType

GrantorType

Privilege

String

Privilege

Privilege

Privileges

Privileges Object

Parameter

Type

Example

Description

ResourceUri

ResourceUri

Parameter

Type

Example

Description

ResourceType

String

JAR

The type of resource. Valid values: JAR, FILE, and ARCHIVE.

Uri

String

hdfs:///tmp/jar1.jar

The URI of the resource.

SerDeInfo

SerDeInfo

Parameter

Type

Example

Description

Name

String

name

The name of Serde.

SerializationLib

String

org.apache.hadoop.hive.serde2.OpenCSVSerde

The implementation class of Serde.

SkewedInfo

SkewedInfo

Parameter

Type

Example

Description

SkewedColNames

Array of String

col1

The names of the skewed fields.

SkewedColValueLocationMaps

Map

The mappings between skewed values and fields.

SkewedColValues

Array of Array

The values of the skewed fields.

StorageDescriptor

StorageDescriptor

Parameter

Type

Example

Description

BucketCols

ListString

The array of bucket fields.

Cols

Array of FieldSchema

All the fields in a table.

Compressed

Boolean

false

Specifies whether to enable compression.

InputFormat

String

org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat

The input format. For more information, see the description of the CreateTable operation.

Location

String

file:///tmp/table

The physical location of the table.

NumBuckets

Integer

10

The number of buckets. This parameter is required if the BucketCols parameter is specified.

OutputFormat

String

org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat

The output format. For more information, see the description of the CreateTable operation.

Parameters

Map

The mapping array of KVPs.

SerDeInfo

SerDeInfo

The information about serialization and deserialization.

SkewedInfo

SkewedInfo

The duplicate values or skewed values in a field.

SortCols

Array of Order

The sort fields.

StoredAsSubDirectories

Boolean

false

Specifies whether table data is stored in a subdirectory.

Table

Table

Parameter

Type

Example

Description

Cascade

Boolean

false

Specifies whether to perform cascade deletion.

CreateTime

Integer

1608707407

The time when the table was created. The value is in the UNIX timestamp format. Unit: seconds.

CreatedBy

String

zhangsan

The name of the user who created the table.

DatabaseName

String

database_test

The name of the metadatabase.

LastAccessTime

Integer

1608707407

The time when the table was last accessed. The value is in the UNIX timestamp format. Unit: seconds.

LastAnalyzedTime

Integer

1608707407

The time when the last statistical operation was performed on columns. The value is in the UNIX timestamp format. Unit: seconds.

Owner

String

zhangsan

The owner of the table.

OwnerType

String

USER

The type of the owner. Valid values: USER, ROLE, and GROUP.

Parameters

Parameters

The KVPs that define the attributes related to the table.

PartitionKeys

ListFieldSchema

The partition key columns of the partitioned table.

Privileges

PrincipalPrivilegeSet

The configurations of permissions.

Retention

Integer

30000

The retention time of the table.

RewriteEnabled

Boolean

true

Specifies whether the view can be used to rewrite a query.

Sd

StorageDescriptor

The descriptor of the storage, which contains physical storage information about this table.

TableName

String

table_name

The name of the table.

TableType

String

MANAGED_TABLE

The type of the metadata table, such as MANAGED_TABLE, EXTERNAL_TABLE, VIRTUAL_VIEW, INDEX_TABLE, or MATERIALIZED_VIEW.

Temporary

Boolean

false

Specifies whether the table is a temporary table.

UpdateTime

Integer

1608707407

The time when the table was updated. The value is in the UNIX timestamp format. Unit: seconds.

ViewExpandedText

String

''

The expanded text of the view.

ViewOriginalText

String

''

The original text of the view.

TableError

TableError

Parameter

Type

Example

Description

ErrorDetail

Object

The information about the error.

TableName

String

The name of the table in which the error occurs.

TableInput

Table

Parameter

Type

Example

Description

Cascade

Boolean

false

Specifies whether to perform cascade deletion.

CreatedBy

String

zhangsan

The name of the user who created the table.

DatabaseName

String

database_test

The name of the metadatabase.

LastAccessTime

Integer

1608707407

The time when the table was last accessed. The value is in the UNIX timestamp format. Unit: seconds.

LastAnalyzedTime

Integer

1608707407

The time when the last statistical operation was performed on columns. The value is in the UNIX timestamp format. Unit: seconds.

Owner

String

zhangsan

The owner of the table.

OwnerType

String

USER

The type of the owner. Valid values: USER, ROLE, and GROUP.

Parameters

Parameters

The KVPs that define the attributes related to the table.

PartitionKeys

ListFieldSchema

The partition key columns of the partitioned table. Only the partition keys of the basic types are supported. If you do not specify a partition key, you must leave the PartitionKeys parameter empty.

Privileges

PrincipalPrivilegeSet

The configurations of permissions. This parameter can be left empty.

Retention

Integer

30000

The retention time of the table.

RewriteEnabled

Boolean

true

Specifies whether the view can be used to rewrite a query.

Sd

StorageDescriptor

The descriptor of the storage, which contains physical storage information about this table.

TableName

String

table_name

The name of the table.

TableType

String

MANAGED_TABLE

The type of the metadata table, such as MANAGED_TABLE, EXTERNAL_TABLE, VIRTUAL_VIEW, INDEX_TABLE, or MATERIALIZED_VIEW.

Temporary

Boolean

false

Specifies whether the table is a temporary table.

ViewExpandedText

String

''

The expanded text of the view.

ViewOriginalText

String

''

The original text of the view.

TablePartitionColumnStatisticsRequest

TablePartitionColumnStatisticsRequest

Parameter

Type

Example

Description

CatalogId

String

The catalog ID of the metadatabase. Default value: the user ID of your Alibaba Cloud account.

ColumnNames

ListString

The names of fields.

DatabaseName

String

The name of the metadatabase.

PartitionNames

ListString

The names of partitions.

TableName

String

The name of the metadata table.

TableVersion

The schema of a table version.

Parameter

Type

Example

Description

Table

Table

The details about the metadata table.

VersionId

Integer

The number of the version.

UpdateTablePartitionColumnStatisticsRequest

UpdateTableColumnStatistics

Parameter

Type

Example

Description

CatalogId

String

1344371

The catalog ID of the metadatabase. Default value: the user ID of your Alibaba Cloud account.

ColumnStatisticsList

Array of ColumnStatistics

The list of field statistics.

DatabaseName

String

database_test

The name of the metadatabase.

Engine

String

The engine.

TableName

String

The name of the metadata table.

ValidWriteIdList

String

This parameter is not supported and can be left empty.

WriteId

String

This parameter is not supported and can be left empty.

IsStatsCompliant

Boolean

true

IsStatsCompliant