Defining and using data standards

Field standards

A field standard normalizes a field's definition, including its name, data type, and value range. Standardizing fields with the same meaning prevents confusion from variations in names or data types. For example, you can create a field standard named member_id and associate it with relevant tables to ensure that all member ID fields are consistently defined.

Lookup tables

A lookup table defines the valid value range and allowed content for a field. For example, the values for a gender field can be restricted to Male, Female, and Unknown.

Measurement units

A measurement unit defines business-relevant units of measure, such as currency, object quantifiers, and time units. For example, the measurement unit for product quantity is pieces.

Naming dictionaries

A naming dictionary provides standardized translations for business terms, physical tables, and fields, including their root words and morphemes. It serves as your enterprise-level library for naming conventions. For example, the standard term for a company's annual revenue can be defined as Annual Gross Revenue.

Hierarchy

• When you create a field standard, you must place it under the root directory, a directory, or a standard set. The following items describe the hierarchy:

  • Root directory: The top-level directory. All other directories, standard sets, and standards must be placed in the root directory.

  • Directory: A folder used to store standards and standard sets, similar to a folder in an operating system.

  • Standard set: Similar to a directory, but can only contain standards.

• Field standards can have inheritance relationships. For example, both buyer ID standards and seller ID standards can inherit from a member ID standard.

Define a field standard

1. In the left navigation bar of the Data Standard page, click Field Standard to go to the Field Standard page.

2. In the directory tree on the left, right-click the target directory or standard set and select Create Standard.

   You can create directories or standard sets to organize your field standards as needed.

3. In the Create Standard dialog box, the key parameters are as follows.

   Parameter	Description
   Abbreviation	The name of the associated field in a logical model.
   Display Name	The display name of the associated field in a logical model.
   Length	A parameter related to the data type. For example, if you select the DECIMAL data type, the length corresponds to 20 in DECIMAL(20,4).
   Precision	A parameter related to the data type. For example, if you select the DECIMAL data type, the precision corresponds to 4 in DECIMAL(20,4).
   Not Null	Specifies whether a field that references this standard can be null. Null values are allowed by default.
   Default Value	The default value for the field if no value is provided. The value cannot exceed 2,048 characters in length.
   Parent Standard	You can select an existing standard as a parent standard to establish an inheritance relationship. This helps you better identify field associations. For example, both buyer ID and seller ID are types of member ID. Therefore, the parent standard for both the buyer ID standard and seller ID standard is the member ID standard.
   Referenced Lookup Table	You can reference an existing lookup table to constrain the value range of a field. For more information, see Data standards: Lookup tables.

   Important

   Before you can delete a field standard, you must remove all of its references.

Use a field standard

You can use field standards to define specific fields in logical models. This feature is supported for source tables, dimension tables, fact tables, aggregate tables, and application tables. For example, in the dimension table for member information dim_ec_con_member_df, you can associate the member ID field with the field standard member_id. In this case, the field name is inherited from the abbreviation of the field standard, the field display name is inherited from the display name of the field standard, and the Type and Not Empty properties are also directly inherited. For more information, see Configure dimension table fields.

Define a lookup table

1. In the left navigation pane on the Data Standard page, click Lookup Table to go to the Lookup Table page.

2. Right-click the created directory name, and click Create Lookup Table.

   You can create directories to organize your lookup tables as needed.

3. In the Create Lookup Table dialog box, configure the following parameters and add enumerated values.

   For example, set the ID to gender, the display name to Gender, and the name to gender. The enumerated values are shown in the following table.

   Code value	Code name	Name	Description
   0	Unknown	unknown	Gender not specified
   1	Male	male	Male
   2	Female	female	Female

   Important

   Before you can delete a lookup table, you must remove all of its references.

Publish a lookup table

On the details page of the lookup table, click Publish in the upper-right corner to materialize the lookup table as a physical table or a materialized view.

Use a lookup table

You can use lookup tables to define specific fields in logical models. This feature is supported for source tables, dimension tables, and fact tables. For example, in the dimension table for member information dim_ec_con_member_df, you can associate the gender field with the lookup table gender. In this case, the field name is inherited from the ID of the lookup table, and the field display name is inherited from the display name of the lookup table. For more information, see Configure dimension table fields.

Associating a lookup table can automatically unify the names of a field used across multiple tables.

Apply standards based on lookup tables

When a logical model fact table or dimension table is published as a physical table, you can generate a quality rule for a specific field that is associated with a lookup table. You can then use this rule to create a quality monitoring task that applies the standard to the physical table. For more information, see Data labeling.

An example of a generated quality rule is as follows: The rule name is Field Rule-Lookup Table-(gender), the quality dimension is Consistency, the scope is Field-level(gender), and the rule template is Number of rows with mismatched enumerated values is 0. For the monitoring threshold, !=0 indicates an exception and =0 indicates normal. The severity is Weak Rule.

Define a measurement unit

1. On the Data Standard page, in the left navigation bar, click Measurement Unit to enter the Measurement Unit page.

2. On the Measurement Unit page, right-click the target measurement category, and select Create Measurement Unit.

3. In the Create Measurement Unit dialog box that appears, configure the parameters to complete the setup.

   For example, set the abbreviation to m, the name to meter, and the display name to Meter.

Define a naming dictionary

1. On the Data Standard page, in the left navigation bar, click Naming Dictionary to go to the Naming Dictionary page.

2. Click Create. In the Create Naming Dictionary dialog box, configure the parameters and click OK.

   For example, set the display name to Engine, the name to engine, and the abbreviation to eng..

Use a naming dictionary

You can use a naming dictionary to check whether table names comply with the naming conventions for data warehouse layers. This feature is supported for source tables, dimension tables, fact tables, aggregate tables, and application tables. For example, if a naming dictionary entry with the abbreviation trade does not exist, the table name dwd_trade_order does not comply with the naming convention for the DWD layer of the fact data table.

On the Checker Management page, click Add Rule. Select Table Name as the rule type. You can then define the rule by using the fixed text dwd and selections from the Root Word Abbreviation and Business Process Abbreviation dropdowns to match your naming convention.

To use this feature, you can configure it in the data warehouse layer checker. For more information, see Configure data warehouse layer checkers and Use checkers.

Batch import standards

If you have a large number of data standards to create, you can import them in batches. DataWorks provides an import template that you can download and fill in for batch imports.

1. On the Data Standard page, in the left navigation bar, click Naming Dictionary to go to the Naming Dictionary page.

   The details pages for field standards and lookup tables also provide Import and Export buttons.

2. Click Import, and on the Import page, select the Import Type.

3. In the Template Preview section, click Download Template and fill in the required fields.

4. Click Next Step. On the Data Import tab, upload and preview the data file.

   Note

   • Import Mode: If an object with the same name as the imported file already exists in DataWorks, you can choose to skip the object or overwrite it with the content of the imported file.

   • Only files in the .xlsx format are supported for batch import. You can import up to 30,000 records at a time. The file size cannot exceed 10 MB.

5. On the Complete tab, you can view the import result details. Click View More Details in the details list to go to the edit page, where you can perform additional operations. If the import status is Failed, you need to resolve the error based on the error details and re-import the data.

Batch export standards

If you need to reuse data standards across workspaces, you can use the export feature to export data standards in batches. You can click the export button on the details pages for field standards, lookup tables, or naming dictionaries.