DataWorks:Materialize a logical model

Conditions

Before you publish a model, the following conditions must be met:

• Target engine: You can publish to MaxCompute, Hologres, EMR Hive, CDH, and MySQL engines.

• Publish type:

  • When you publish a model as a view or materialized view, the target engine must be MaxCompute.

  • When you publish a model as a materialized view, the Publish Mode must be set to Delete and Recreate.

• Environment:

  • In a basic mode workspace, you can publish only to the production environment.

  • In a standard mode workspace, you can publish to the development or production environment.

• Resource group:

  • When the target engine is EMR Hive or CDH, you must specify an exclusive resource group for scheduling or a Serverless resource group to run the publish task.

Publish and materialize a model

You can publish and materialize dimension tables, fact tables, aggregate tables, and application tables directly to a data source instance. This makes them available to compute engines for data development and analysis. DataWorks supports materialization to MaxCompute, Hologres, EMR Hive, CDH, and MySQL.

1. Publish the model.

   After a model is created, you can click the icon in the menu bar at the top of the table details page. In the Publish dialog box that appears, configure the key parameters.

   Parameter	Description
   Publish Type	Sets the form of the materialized model. Physical Table: Creates a physical table in the target engine. View: Creates a view in the target engine, which supports only the MaxCompute data source type. Materialized view: Creates a materialized view in the target DPI engine. This feature is available only for the MaxCompute data source type. A materialized view is a pre-computation object that stores the results of time-consuming operations, such as JOIN and AGGREGATE. The results can then be directly reused in queries to avoid these costly operations and accelerate query execution. For more information, see Materialized view operations.
   Data Source Type	Select one of the following compute engines: MaxCompute, EMR Hive, Hologres, CDH, or MySQL.
   Effective Environment	You can publish to the Development or Production environment. If you use a DataWorks workspace in basic mode, you can publish the table only to the production environment. If you use a DataWorks workspace in standard mode, you can publish the table to the production or development environment.
   Publish Mode	Defines how the model is published to the target engine. Incremental Publish: Updates only the changed parts of the current model to the target DPI engine. We recommend that you use this mode for published models to retain historical data and reduce the impact on online services. Delete and Recreate: Deletes the existing model with the same name from the target engine and then recreates it. Note: This operation deletes and recreates the table, resulting in the loss of historical data. For the first publish, both modes have the same effect.
   Automatic Rule Generation Based on Field Standard	Specifies whether to automatically generate quality rules after a model is published. For more information, see Data Tagging. Yes: After a successful publish to the production environment, automatically generates quality rules for fields that are primary keys, non-null, or associated with lookup tables. No: Does not automatically generate quality rules.
   Resource Group	A resource group is required to materialize a model table as a physical table for E-MapReduce or CDH. Ensure network connectivity between the selected resource group and the data source. Note You can use only an exclusive resource group for scheduling to publish physical tables for E-MapReduce or CDH. If you do not have a suitable exclusive resource group for scheduling, create one. For more information, see Use an exclusive resource group for scheduling.
   DDL	DataWorks automatically generates an SQL statement to publish the model as a materialized view based on the model configuration. The publishing operation can succeed only if the SELECT statement is successfully executed. Therefore, you must manually modify the AS SELECT statement to make it an executable SQL statement. Note This parameter is required only when the Publish Type parameter is set to Materialized View. For more information about DDL operations on materialized views, see Materialized view operations.

2. After you complete the configuration, click Publish. The system publishes the model to the corresponding Development or Production environment and materializes it to the selected engine.

   • You can view the current publishing progress and related logs in the Publish dialog box.

   • You can repeat the publish operation to publish the model to different environments.

3. Optional: After the publish is complete, click the icon in the menu bar of the table details page to view the result and details of the latest publish operation.

   The Publishing Status can be Succeeded, Failed, or Publishing. The following figure shows the logs for the Succeeded status.

   

   Note

   • If the Publishing Status is Failed, you must locate and troubleshoot the error based on the specific error message.

   • If No publishing logs found. is displayed in the Publishing Logs dialog box, this means that the model has never been published.

4. View publish records

   After a model is successfully published, you can go to the model's edit page and click Publishing Record in the navigation bar on the right to view its publish history.

Generate an ETL code frame (model development)

In DataWorks Intelligent Data Modeling, you can use the model development feature to generate an extract, transform, and load (ETL) code frame for a published model table, such as a dimension table or a fact table.

1. In DataWorks Intelligent Data Modeling, double-click a published model table to go to its details page.

2. Click Develop Model in the Actions column.

3. In the Develop Model dialog box, select Create DataStudio Node or Associate DataStudio Node, configure the relevant parameters, and then click OK.

   1. Select Create DataStudio Node.

      

      The system automatically creates a DataStudio node and populates the node name based on the table name.

      • If the model table is published to a MaxCompute engine, an ODPS SQL node is created.

      • If the model table is published to an EMR Hive engine, an EMR Hive node is created.

      After you confirm the configuration, click Go to DataStudio. The system automatically opens a dialog box in Data Development to create a node. Click OK to create the node and generate the ETL code

   2. Select Associate DataStudio Node

      

      Associates the code frame with an existing DataStudio node of the same engine type. After a successful binding, you can click the node name to go to the corresponding node page to continue development.

Billing description

The model publishing feature is free of charge. However, you are charged for the computing and storage resources used during the materialization process:

Target engine fees: After a model is materialized, the storage resources it occupies and the computing resources it consumes in the target engine, such as MaxCompute or Hologres, are billed according to that product's billing rules.