Appendix: Metadata versions and property descriptions - Dataphin

Different data source operations may or may not generate a new metadata version. The properties collected also vary by metadata object type. This appendix covers both.

Metadata versions

General description

Modifying the following information does not generate a new metadata version:
- Partition definition changes: Changes to the number of partitions, partition field types, and partition filter expressions.
- Property definition changes: Changes to custom properties.
- Collection policy configuration: When the collection task is configured to retain deleted metadata, and two consecutive collections only have field deletion changes, a new metadata version is not generated.
Modifying the following information generates a new metadata version:
- Table information changes: Including changes to table descriptions and metadata types.
- Table structure changes: Including modifications to partition fields, changes from non-partitioned tables to partitioned tables (whether it is a partitioned table), and view DDL changes.
- Field structure changes: Including adding/updating/deleting fields, changing field order, field types, and field key types.
- Field detail changes: Including changes to field default values, nullable status, and field descriptions.

Changes to specific properties of certain data sources also affect version generation, as shown below.

Data source type	Change method
Hologres	Changes to the partition table type, such as converting a parent table to a child table or vice versa, the name of the parent table, or the conditions of a child table.
StarRocks	Changes to the table type, materialized view refresh type, invalid status, sort fields, bucketing method, or bucketing fields. However, a change to the number of buckets does not generate a new version.
OceanBase, Greenplum	Partition field type changes.
Amazon Redshift	Changes to the table storage class, distribution key fields, sorting method, or sort fields.
SelectDB, Doris	Changes to the table storage class, bucketing method, bucketing fields, sort fields, or distribution fields.
Dameng (DM)	Table storage class changes.
DolphinDB	Changes to sort fields.
openGauss	Changes to the table storage class or partition field type.

Metadata properties

All data sources display the following information:

Table technical properties: Including description, number of table rows, storage class, and storage volume information.
Field list: Including the name (primary key fields are identified with a key symbol), data type, description, nullable status, and default value information of fields in the specified version.
Partition information: Includes partition fields and partition details. For partition fields, the field name, data type, and description are provided. For partition details, the partition name, record count, storage size, and creation time are provided.

Note

When the data source type is a relational database and the metadata type is a view, only description information can be viewed in technical properties.
Amazon Redshift collects data using system tables. Super administrators can access all row data (all metadata) by default. Regular users can only view their own data. To collect all data without using a super administrator account, grant usage permissions on pg_catalog to a specific account: GRANT USAGE ON SCHEMA ${schema, such as pg_catalog} TO ${username}.
Users configured in Dameng (DM) and Oracle data sources need select permissions on the relevant tables to collect metadata. To grant permissions, use: grant SOI to your_username;.
DolphinDB collects metadata by calling functions. The scope of collected data depends on the user account role specified in the data source configuration:
- Cluster administrator: Can collect tables from all databases.
- DB_OWNER, and accounts with DB_MANAGE or DB_READ permissions for specific databases: Can collect all tables from the corresponding databases.
- Other accounts: Can collect tables for which they have TABLE_READ permission.
SelectDB and Doris data sources can collect asynchronous materialized view metadata, which the system detects as tables. Synchronous materialized views cannot be collected at this time.

The following table lists the differences in collected properties by data source:

Data source type	Table technical properties	Field list	Partition information
MySQL, Amazon RDS for MySQL, TDSQL for MySQL	Cannot collect storage class.	The default value of view fields is displayed as the default value of source table fields.	Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
Microsoft SQL Server, Amazon RDS for SQL Server	Cannot collect storage class.	/	Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
AnalyticDB for MySQL 3.0	/	The default value of view fields is displayed as the default value of source table fields.	Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
OceanBase	Collects information about whether the table is partitioned, the creation time, the last update time (MySQL tenants only), and the last DDL update time (Oracle tenants only). The collected information varies by table type: View: Cannot collect whether it is a partitioned table, number of table rows, or storage volume information. Foreign table: Cannot collect number of table rows or storage volume information.	Regular fields display unique key, foreign key, and CHECK constraint information. Unique keys are identified with UNI (only displayed for Oracle tenant). Foreign keys are identified with FOR. CHECK constraints are identified with CHK (only displayed for Oracle tenant).	Partition field: Displays the partition type of the field, including Range partition, List partition, and Hash partition. Partition details: Cannot collect partition details. Note Only OceanBase Oracle tenant mode supports displaying partition information.
PolarDB-X (formerly DRDS)	/	The default value of view fields is displayed as the default value of source table fields.	Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
IBM DB2, Amazon RDS for DB2	/	Cannot collect default value information	Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
ClickHouse	Collects information about whether the table is partitioned, the table engine, and the last DDL update time. The collected information varies by table type: View: Cannot collect whether it is a partitioned table, number of table rows, storage class, or storage volume information. Materialized view: Cannot collect whether it is a partitioned table, number of table rows, or storage class information. Foreign table: Cannot collect storage volume information.	Displays sampling key and sort key information. Sampling keys are identified with SMP. Sort keys are identified with SOR.	Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
Hive (MySQL metadatabase, HMS metadata)	Collects information about whether the table is partitioned, whether it is a lake table, the lake table format, the table storage mode, the storage format, the creation time, and the last DDL update time. The collected information varies by table type: View: Cannot collect storage class, storage format, storage volume, whether it is a partitioned table, or number of table rows information. Foreign table: Cannot collect whether it is a partitioned table or number of table rows information.	Cannot collect nullable status and default value information.	Partition field: Partition field information is collected. Partition details: / Note For Iceberg tables, partition information cannot be collected.
Hologres	Collects information about whether the table is partitioned, the parent table name (partition child tables only), partition conditions (partition child tables only), the number of child tables (partition parent tables only), the storage format, the creation time, the last update time, the last DDL update time, and the lifecycle. The collected information varies by table type: View : Cannot collect lifecycle, storage volume, number of table rows, whether it is a partitioned table, last DDL update time, storage format, data update time, or creation time information. Materialized view : Cannot collect lifecycle, storage volume, number of table rows, whether it is a partitioned table, last DDL update time, storage format, or data update time information. Foreign table : Cannot collect lifecycle, storage volume, number of table rows, whether it is a partitioned table, data update time, or creation time information. Internal table as partitioned table : Cannot collect last data update time information.	/	Partition field: Collects the partition type, which is List. Partition details: Collects the partition value, partition child table, and partition condition.
StarRocks	Collects information about the table type (internal tables only), materialized view type (materialized views only), invalid status (materialized views only), whether the table is partitioned, whether it is a lake table, the lake table format, the storage format, the creation time, and the last update time. The collected information varies by table type: View: Cannot collect storage class, storage format, storage volume, or number of table rows information. Materialized view: Cannot collect storage class, storage format, or storage volume information. Foreign table: Cannot collect whether it is a partitioned table or number of table rows information.	Displays bucketing key and sort key information.	Partition field: Collects the partition type, such as Range, List, or Hash. Partition details: The partition filter expression is collected.
Greenplum	Collects information about whether the table is partitioned and the Location (foreign tables only). The collected information varies by table type: View: Cannot collect whether it is a partitioned table, number of table rows, storage class, or storage volume information. Materialized view: Cannot collect whether it is a partitioned table or storage class information. Foreign table: Cannot collect whether it is a partitioned table, number of table rows, or storage volume information.	Regular fields display unique key, foreign key, and CHECK constraint information. Unique keys are identified with UNI (only displayed for Oracle tenant). Foreign keys are identified with FOR. CHECK constraints are identified with CHK (only displayed for Oracle tenant).	Partition field: Collects partition types, such as value and range. Partition details: Cannot collect partition details.
Amazon Redshift	Collects information about whether the table is partitioned and the Location (foreign tables only). The number of rows and storage size cannot be collected. The collected information varies by table type: Internal table: Cannot collect whether it is a partitioned table or storage format information. View/Materialized view: Cannot collect whether it is a partitioned table, storage class, or storage format information.	Displays bucketing key and sort key information. If all returned sort key values are positive, it is a regular sort. If both positive and negative values are returned, it is an interlaced sort.	Only foreign tables display partition information. Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
SelectDB, Doris	Collects information about whether the table is partitioned, whether it is a lake table (only if the storage format is Hudi or Iceberg), the lake table format (lake tables only), the storage format, the table type, the creation time, and the last data update time. The differences in information displayed for different table types are as follows: Foreign table: Cannot collect whether it is a partitioned table, number of table rows, table type, storage volume, creation time, or last data update time information. View: Cannot collect whether it is a partitioned table, number of table rows, storage class, storage format, table type, storage volume, or last data update time information.	Regular fields display unique key, non-unique key, and aggregate key information. Unique keys are identified with UNI. Non-unique keys are identified with DUP. Aggregate keys are identified with AGG. Displays bucketing key and sort key information.	Partition field: Collects the partition type, such as list, range, or expression. Partition details: The last update time is collected.
Dameng (DM)	Collects information about whether the table is partitioned, the storage class, the storage format, the Location (foreign tables only), the creation time, and the last DDL update time. The differences in information displayed for different table types are as follows: Foreign table: Cannot collect whether it is a partitioned table, storage format, creation time, or last DDL update time information. View: Cannot collect whether it is a partitioned table, storage class, storage format, or last DDL update time information. Materialized view: Cannot collect whether it is a partitioned table or storage format information.	Regular fields display unique key, reference key, and CHECK constraint information. Unique keys are identified with UNI. Reference keys are identified with REF. CHECK constraints are identified with CHK.	Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
Elasticsearch	Only table objects (indices) support collecting document count and storage volume information.	Only table objects (indices) support collecting field name and data type information.	Partition field: Cannot collect partition field types. Partition details: Cannot collect partition details.
DolphinDB	Collects information about whether the table is partitioned, the storage engine, the storage format, and the creator. The storage class cannot be collected.	The nullability and default value cannot be collected. Displays sort key information.	Partition field: Collects the partition type, such as Value, Range, List, or Hash. Partition details: The creation time cannot be collected.
openGauss	Collects information about whether the table is partitioned, the storage format, the Location (foreign tables only), and the view DDL (views and materialized views only). The collected information varies by table type: Foreign table: The number of rows, storage size, and information about whether the table is partitioned cannot be collected. View: The number of rows, storage class, storage format, storage size, or information about whether the table is partitioned cannot be collected. Materialized view: The storage class, storage format, or information about whether the table is partitioned cannot be collected.	Regular fields display primary key, CHECK constraint, unique key, and trigger information. CHECK constraints are identified by CHK. Unique keys are identified by UNI.	Partition field: Collects nullability and default value information. Partition types include Range, Interval, List, Hash, Value. Partition details: The record count, storage size, and creation time cannot be collected.