Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog DataWorks Data Analysis Module

DataWorks Data Analysis Module

Part 2 of this 10-part series explains data analysis and the DataWorks data analysis module.

By Zhang Huarui, Product Manager of DataWorks

This article is a part of the One-stop Big Data Development and Governance DataWorks Use Collection.

1. What Is Data Analysis?

What is data analysis? The figure below depicts the relationship between data, information, and intelligence. We collect data from an external environment, either a software environment, such as log data in a business system, or a hardware environment, such as collecting sensor data. The value density of the collected raw data is usually relatively low. More valuable information will be obtained after processing and extraction. Further analysis and production of the information will result in intelligence.

In general, data analysis is the process of exploring, cleaning, transforming, and modeling data to uncover valuable information and support decision-making with conclusions. Data analysis can make decision-making more scientific and help enterprises operate more effectively.

1

Data analysis includes the following steps:

1.  Requirements Analysis: You should clarify the requirements first, such as what are the core issues of the business, what decisions need to be made, and what information needs to be obtained to support this decision.

2.  Data Collection: Once the requirements are identified, data collection can be carried out. However, you may encounter problems during this process. The first problem is that the expected data may not be stored at the technical level, or it may be stored but scattered in different locations in the data warehouse. Data collection needs to solve these problems first.

2

3.  Data Cleaning: The collected data is incomplete, repeated, incorrect, and null. Data cleaning is the process of preventing and correcting these errors.

4.  Exploration and Analysis: Exploration and analysis refer to analyzing data with analysis tools. This process may also involve the application of machine learning algorithms.

5.  Visualization Presentation: When data analysis is completed, the results are presented to the demand side in a visualization way.

Data analysis includes local data analysis and online data analysis.

The most common local data analysis method is to use Excel. However, since data analysts need to repeat their analysis weekly, monthly, or quarterly, the first problem of using Excel is its low efficiency. Moreover, poor performance is another issue when using Excel for data analysis. Performance becomes a bottleneck when there is a large amount of data to analyze. Excel is limited in the number of rows it can display and the performance of personal computers. Data isolation will occur when using Excel for data analysis. The value of analysis on a single data table is limited, and enterprises pay more attention to the results of analysis by pulling through data from various business systems. The last problem is security risks. The enterprise's data is placed on the personal computer of a local employee, which makes it difficult to control the rights in the sharing and downloading process.

3

Online data analysis can solve the preceding problems. Online data analysis tools can refresh data to avoid repeated operations. For example, there is a partition table of a user profile in the data analysis module of DataWorks. If you perform a pivot operation on this partition table in September and want to do this operation again in October, you can update the data by copying the configuration in September to the partition in October.

In addition to the great computing power of the computing engine, online data analysis can perform efficient data analysis on large amounts of data. At the same time, it can fetch analysis from databases of different business systems to break down barriers. The DataWorks data analysis module also allows you to export the analysis results into a MaxCompute table or directly share the analysis results with others, so data flow among different systems and people.

Using the online data analysis module allows you to carry out data analysis and share the data without downloading data locally, which is very important. In this process, the authority is controllable to ensure data safety.

2. DataWorks Data Analysis

DataWorks users often encounter the following questions:

  • Question 1: I have a table that exists in MaxCompute, EMR, or RDS, and I want to perform further statistical analysis on the query results. What should I do?
  • Question 2: How do I know if there is any dirty data in someone else's table? Can I only run SQL? Can DataWorks perform pivot analysis?
  • Question 3: I want to edit a dimension table manually, but I can't write SQL. What should I do?

These problems can be solved using the DataWorks analysis module.

(1) DataWorks Data Analysis Module

The following figure shows how various DataWorks modules work together to fulfill data analysis requirements.

4

Firstly, data collection is carried out by searching the tables needed for data analysis on the data map. Then, you can view and understand the data by table details, data preview, and data lineage. After that, you need to apply for the query permission of this table or a field. Then, you can go to DataStudio for further processing or go to the data analysis module to use the function of query from data source in data analysis. You can take the numbers from the data source, put them into the table, and carry out data profiling and pivot analysis in the form of a table.

In general, data analysis is suitable for quick data insight analysis, online editing, and data visualization modules. The three functions of the data analysis module include spreadsheets, dimension tables, and reports.

(2) Spreadsheet

The spreadsheet is the core function of the data analysis module, which provides personal space for data fetching, exploration, and sharing. It takes the spreadsheet as the main body and supports the common table functions, allowing you to get started quickly.

The spreadsheet has the following functions:

  • Import data from local/data sources
  • Personal query workbench
  • Data Profiling
  • Data Pivot
  • Analysis results can be solidified into MaxCompute tables or shared

(3) Spreadsheet – Function of Query from Data Sources

The function of query from data sources supports a wide range of data source types. When a data source is added, you can query the field structure, generate a data preview, or generate query statements by double-clicking or dragging. Also, you can click run to generate data query results. Its editor is intelligent and provides functions, such as completion, highlighting prompts, and intelligent error correction. The query code can be saved, and the running history can be traced back.

5

(4) Spreadsheet – Data Profiling Function

When you get the data, you can use the data profiling function to gain insight into the data. This function counts the distribution of the values of each field to help you know about the distribution of data. It also supports overview mode and verbose mode. In addition, it supports data screening and multi-level drill down, which can help you determine whether these data need to be cleaned and clarify and determine the cleaning purposes.

6

(5) Spreadsheet – Pivot Function

The Pivot function is a common function for data analysts. Spreadsheets support local pivot and data source pivot. You can pivot the data of the spreadsheets or pivot the full data of the data sources. The dimensions of the pivot functionality support customized sorting, numeric types can be grouped, and the pivot configuration supports copying.

7

(6) Spreadsheets – Saving and Sharing

The saving and sharing of spreadsheets support the template function. You can save the spreadsheet as a template first and create a new one directly from the saved one. You can also solidify the analysis results into a MaxCompute table, generate create table statements, and insert data into it. When you want to share a spreadsheet, you can assign a person or permission.

(7) Dimension Table

Dimension table is a simple and efficient table editing tool. In daily work, the operation department often needs to maintain a MaxCompute table online. Generally, they need to ask a research and development colleague to create a new MaxCompute production table. Every time the data is updated, it is necessary to ask research and development colleagues to repeat the process of creating a new production table. This process involves many people but is inefficient.

After using the dimension table, the operation department can create a MaxCompute production table by themselves visually. They can name and describe the fields and click OK to generate an online table. After the table is generated, you can import local data into the table or write data on it directly. You can write or modify data directly in the dimension table, shortening the entire operation process and improving work efficiency.

8

(8) Report

A report is a visualization tool that provides a variety of report components. You can drag the components to complete a report. After the build is finished, you can share the entire report with others.

9

MaxCompute Big Data DataWorks Data analysis Data Visualization Data Governance Data Development MaxCompute Table
0 0 0
Share on

Read previous post:

An Introduction and Best Practice of DataWorks Data Security Module

Read next post:

An Introduction to DataWorks O&M Center and Mobile Edition

Alibaba Cloud Community

872 posts | 198 followers

You may also like

Comments

Alibaba Cloud Community

872 posts | 198 followers

Related Products

More Posts by Alibaba Cloud Community

See All