All Products
Search
Document Center

Data Warehouse Developer overview

Last Updated: Aug 28, 2020

Enterprises store a lot of valuable data, such as transaction data and customer information, in databases. They need a scheme to easily analyze and process the data and convert the data into business intelligence. Data Warehouse Developer provided by Data Management Service (DMS) is such a scheme. It offers an all-in-one development platform to integrate, process, and visualize data, and mine the value of data. DMS Data Warehouse Developer uses databases as the computing engine and integrates a variety of tools and services in the database ecosystem, such as Data Transmission Service (DTS) and Data Lake Analytics (DLA). It allows you to easily build, develop, and manage data warehouses. DMS Data Warehouse Developer provides the following features:

  • Free selection of data warehouse engines: To develop data warehouses, you can select appropriate database engines, such as AnalyticDB, DLA, PolarDB, and Relational Database Service (RDS), based on your enterprise scale, data volume, and timeliness requirements. DMS provides the same features for all these database engines.
  • Two development modes: DMS provides the task orchestration and data warehouse development modes to meet different data warehouse development requirements. In the task orchestration mode, you do not need to specialize in data warehousing. You only need to focus on the business logic and know how to create task flows and edit SQL statements. The data warehouse development mode applies only to professional data warehouse developers. This mode provides various features such as theme management, hierarchical management, production publish, multi-person collaboration, data map, and data quality control, some of which are being developed. It offers a professional data warehouse development scheme to enterprises.
  • Offline and real-time data warehouses: You can use batch synchronization and run scheduled tasks to easily develop offline data warehouses in DMS. In addition, DMS is integrated with Alibaba Cloud DTS and AnalyticDB. You can use DTS to synchronize data in real time and use AnalyticDB as the computing engine to easily build real-time data warehouses. In this way, you can produce and consume data in DMS in real time.
  • Integrated global data management: Based on the unified database management and permission control capabilities, DMS manages your online databases such as online transaction processing (OLTP) databases and offline databases such as online analytical processing (OLAP) databases in a uniform manner. This guarantees data security. DMS can also track data lineage and analyze the impacts throughout the data lifecycle.

The following figure shows the process of developing a data warehouse in DMS.
Data Warehouse Developer  overview

  1. Data warehouse engine selection: Purchase or select an appropriate database engine, such as PolarDB, RDS, or AnalyticDB, based on your business needs and use it as the data warehouse engine.
  2. Data integration: Synchronize data to be analyzed from an external data store such as a transaction database to the data warehouse.
  3. Data processing: Create tables and a task flow and configure the scheduling policy to process data in the data warehouse.
  4. Data application: Provide consumption channels for processed data. For example, create API operations to consume the data or present the data in charts.
  5. Data governance: Manage data throughout the data warehouse development process. For example, monitor the data quality, track data lineage, and analyze data impacts.