A warehouse project allows you to develop and manage data in a data warehouse. When you create a warehouse project, you must select a data warehouse engine and bind the warehouse project to a database.
A data warehouse engine is a database instance in Data Management Service (DMS). Currently, you can create a warehouse project based on one of the following database engines: AnalyticDB for MySQL V3.0, AnalyticDB for PostgreSQL, and ApsaraDB RDS for MySQL. The database instances used to create warehouse projects must be managed in the Secure Collaboration mode. For more information about control modes, see Control modes.
The data integration feature aggregates data from different databases and allows you to analyze the data in a data warehouse. Data integration is completed through real-time synchronization or batch synchronization.
The data synchronization feature of Data Transmission Service (DTS) is used to synchronize data to a target database in real time. This is the basis for building a real-time data warehouse. For more information, see Data synchronization.
The data migration feature of DTS is used to synchronize a large amount of data at a time to a target database. You can configure a scheduling policy to synchronize incremental data periodically, for example, on a daily basis.
Internal tables are objects in a data warehouse engine. They are processed, stored, and exchanged during the data warehouse development process. Internal tables can be associated with themes to be marked with business labels or layers to be managed hierarchically in a data warehouse.
Foreign tables are mapped to tables in other systems outside a data warehouse engine. You must define a foreign table in Data Warehouse Developer before you can use data in the foreign table. If you have stored a small amount of data in an external system, you can define a foreign table to import the data from the external system and query the data jointly with the data in internal tables.
ThemeA theme is a label used to vertically divide data domains based on business conditions. Currently, data warehouses in DMS support two levels of themes. For example, the level-1 theme Log can be divided into three level-2 themes: Exposure, Browsing, and Click-through.
LayerBy default, a data warehouse in DMS has three layers from bottom up: operational data store (ODS), common data model (CDM), and application data store (ADS). You can also customize the layers.
- The ODS layer stores raw data in the data warehouse. The data structure is basically consistent with that in the source system. The ODS layer serves as the data staging area of the data warehouse.
- The CDM layer consists of the dimension (DIM) layer, data warehouse detail (DWD) layer, and data warehouse summary (DWS) layer. The CDM layer processes and integrates the data of the ODS layer to define conformed dimensions, create reusable detailed fact tables for analysis and statistics, and aggregate common metrics.
- The ADS layer stores personalized statistical metrics of data products. It processes the data of the CDM and ODS layers.
Folders are used to classify internal tables and task nodes.
The definitions of other terms, such as task flow, variable, and Operation Center, are the same as those in Task orchestration.