All Products
Search

This Product

    Dataphin:Data Development Overview

    Document Center

    Dataphin:Data Development Overview

    Last Updated:Jan 21, 2025

    Data development encompasses code development and standard modeling. Code development involves constructing compute tasks, including SQL, Shell, Python, MR, and Spark tasks. Standard modeling entails building logical data models based on business requirements.

    Prerequisites

    To engage in real-time development, you must have purchased the real-time development value-added service and activated the real-time development module for the current tenant.

    Feature introduction

    • Standard Modeling: This process involves designing data models from a business perspective, ensuring data standardization, normalization, and automation. It helps avoid inconsistencies in metric calculations and visualizes data warehouse model construction. Upon submission of a logical table, an intelligence data engine scheduling task is automatically generated to facilitate data production.

      Note

      The Standard Modeling feature is available only in projects associated with the data section.

    • Data Processing: This involves constructing complex data models, synchronization tasks, and various code tasks through coding.

    • Ad Hoc Query: Enables thematic queries for business data, abstracting away the technical complexities of the physical model and providing query services from a business standpoint using the logical model.

    • Dual Development Modes: Dataphin supports both Basic and Dev-Prod modes for data development:

      • Basic projects associated with the Basic data section enable standard modeling. Conversely, Basic projects linked to the Prod data section lack standard modeling capabilities, offering only data processing and ad hoc query functionalities.

      • Dev projects support standard modeling, data processing, and ad hoc query features. Prod projects support standard modeling and data processing.

    • Intelligent Editor: The Dataphin code editor offers basic code highlighting, intelligent code hinting, and permission verification, enhancing the coding experience and reducing development costs. For editor usage instructions, see Enhance editor code efficiency.

    Data development entry

    On the Dataphin home page, select the top menu bar Development to access the data Development page by default.

    Data development interface introduction

    image

    Area

    Description

    ①Global Search and Code Search

    • Global Search: Use the global search box to find physical tables, logical tables, meta tables, standard definitions, functions, resources, or templates by entering relevant keywords. The Development and Asset tabs allow for quick object searches.

      image

      • ①Project Switch: Toggle between Dev and Prod project objects in a Dev-Prod environment to query specific objects.

      • ②Perspective Switch: Dataphin supports object viewing from development and asset perspectives, each offering visibility into different object types.

        • Development Perspective: Includes physical tables, logical tables, meta tables, standard definitions, functions, resources, or templates.

        • Asset Perspective: Comprises physical tables, logical tables, and meta tables.

      • ③Object list: Displays the currently matched objects for you. Supports global and project perspective switching. It also supports viewing all or a specific object type. Click on the Object Name and the system will automatically locate the object.

    • Code Search: Click the gageg icon to open the Code Search box and quickly locate compute tasks containing the specified code. The following figure is an example:

      image

      • ①Code Search Input Box: Enter keywords related to compute tasks. Use the image icon to switch to a text input box for multiple codes, or the image icon to access the Advanced Search page for more detailed searches. For further information, see Enhance advanced search efficiency.

      • ②Search Result List: Shows compute tasks matching the code. Clicking a task reveals its details.

      • ③Task Match Details: Displays code matches within compute tasks, including line numbers and match counts.

    Note

    • Code search is limited to Submitted, Developing, and Published statuses.

    • A maximum of 50 tasks can be matched per code logic. Code search is project-specific.

    • Search functionality applies to code submitted post-version upgrade (July 14, 2020).

    ②Project and Environment

    • Project: By default, the name of the currently active project is displayed. Click the test icon to choose a project classification (Prod, Dev, or Basic) and name from the dropdown list, or use the search box to find projects quickly.

    • Environment: Switch between project environments, including Production and Development, by clicking the tstt icon.

    Note

    Basic projects default to the production environment and do not differentiate between environments.

    ③Feature Items

    Dataphin's data development section includes modules for standard modeling, data processing, recently opened, ad hoc query, and run records.

    • Standard Modeling: This module covers logical tables, including logical dimension tables, logical fact tables, atomic metrics, business filters, metrics, and logical aggregate tables. For more information, see Data standardization and modeling.

    • Data Processing: Focuses on compute task-related features for data processing, such as table management, compute tasks, compute task templates, resources, and functions.

      • Table Management: Manages offline physical tables and real-time compute tables used in compute task development.

        • Offline physical table: This feature allows you to set up and manage offline physical tables that are essential for developing compute tasks. For more information, see Create offline physical table.

        • Real-time Compute Meta Table: Manages meta tables and mirror tables used in real-time task development.

          • Meta table: Through Data Management, a meta table serves as a cross-storage type table. It facilitates the creation and management of input tables, output tables, and dimension tables essential for developing real-time tasks. For more information, see Create and manage meta tables.

          • Image table: Primarily used for the development of real-time tasks that integrate streaming and batch processing. The image table maps streaming tables and offline tables. When developing real-time computing tasks that integrate streaming and batch processing, referencing the image table will support sync operations for streaming tables and offline tables. For more information, see create image table.

      • Compute tasks: This feature allows for the creation of compute tasks in SQL, MR, Jar, Shell, Python, and Virtual formats. For more information, see Computing Tasks.

      • Compute Task Templates: Facilitates the creation of templates for offline and real-time compute tasks, enhancing development efficiency. For more information, see:

        • For creating an offline compute template, see this guide.

        • To create a real-time compute template, see this guide.

      • Resources: This section is dedicated to storing and managing files essential for code development, such as JAR, JSON, Python, and other types of resource files. For more information, see Upload resources and references.

      • Functions: Showcases built-in functions and user-defined functions supported by Dataphin's compute engine. Function information includes names, types, command formats, and descriptions. The displayed functions depend on the compute source bound to the project.

        • Built-in Functions: Varies with the compute engine in use. Refer to the system-displayed functions for specifics.

        • User-defined functions: You can create custom functions by uploading Jar resources or using other methods. For more information, see Create user-defined functions.

    • Recently opened: Lists the computing nodes, functions, tables, and other objects you have accessed recently.

    • Recycle Bin: Objects that are deleted from the data development section are moved to the recycle bin. Within the recycle bin, you have the option to either restore or permanently delete these objects. For more information, see managing recycle bin resources

    • Ad hoc query: Ad hoc queries allow for the customization and execution of query statements, along with the downloading of data, tailored to the current business needs. For more information, see Query and download data.

    • Run Records: Run records maintain a history of the past 15 days for activities such as ad hoc queries, compute task executions, logical data table previews, derived metric smoke tests, asset data previews, and OpenAPI data queries. For additional details, see how to view and manage run records.

    ④Object List Directory

    The object list directory displays all objects created or built-in within the Dataphin data development section.

    ⑤Data Development Welcome Page

    This section outlines the essential workflow and development tools for data development in Dataphin. To create specific objects, click on the path points or tool blocks, then click the image icon labeled 'new'.