O DataWorks fornece um conjunto abrangente de operações de API para gerenciamento de metadados. Este guia demonstra como consultar tabelas, obter detalhes de tabelas e, por fim, gerencie metadados com a API do DataWorks.
Pré-requisitos
Antes de começar, revise os seguintes recursos:
1. Listar tabelas
Esta seção descreve como usar a API para consultar a lista de tabelas em um projeto ou schema do MaxCompute, com suporte a consultas paginadas. O procedimento principal é o seguinte:
Backend: Use o MetaServiceProxy para consultar detalhes da tabela
-
Para processar os parâmetros do frontend, crie um método ListTables no MetaServiceProxy para enviar solicitações à API ListTables para obter as informações da lista de tabelas de metadados.
-
Parâmetros de consulta suportados: ID da entidade pai, nome (correspondência aproximada), comentário (correspondência aproximada), tipo, parâmetros de ordenação e parâmetros de paginação.
NotaPara mais detalhes, consulte Metadata entity-related concept description. O ID da entidade pai pode ter dois formatos.
O ID da entidade pai é o ID do projeto MaxCompute. Formato:
maxcompute-project:${Alibaba Cloud account ID}::{projectName}O ID da entidade pai é o ID do schema do MaxCompute. Formato:
maxcompute-schema:${Alibaba Cloud account ID}::{projectName}:{schemaName}
Os resultados da consulta incluem: Número total de tabelas de dados, informações de paginação e, para cada tabela de dados: ID, ID da entidade pai, nome da tabela, comentário, nome do banco de dados, nome do schema, tipo, lista de campos de partição, hora de criação e hora de modificação.
-
Obtenha o ID da conta Alibaba Cloud: Independentemente de usar um usuário RAM ou uma conta Alibaba Cloud, você pode visualizar o ID da conta Alibaba Cloud no canto superior direito da página.
Faça login no console do DataWorks com sua conta Alibaba Cloud ou usuário RAM.
-
Passe o mouse sobre a foto do perfil no canto superior direito e visualize o ID da conta Alibaba Cloud.
Conta Alibaba Cloud: O Account ID é o ID da conta Alibaba Cloud que você precisa obter.
Usuário RAM: Visualize diretamente o Alibaba Cloud Account ID.
-
Código de exemplo:
/** * @author dataworks demo */ @Service public class MetaServiceProxy { @Autowired private DataWorksOpenApiClient dataWorksOpenApiClient; /** * DataWorks OpenAPI : ListTables * * @param listTablesDto */ public ListTablesResponseBodyPagingInfo listTables(ListTablesDto listTablesDto) { try { Client client = dataWorksOpenApiClient.createClient(); ListTablesRequest listTablesRequest = new ListTablesRequest(); // Parent entity ID (see metadata entity concepts) listTablesRequest.setParentMetaEntityId(listTablesDto.getParentMetaEntityId()); // Table name, supports fuzzy matching listTablesRequest.setName(listTablesDto.getName()); // Table comment, supports fuzzy matching listTablesRequest.setComment(listTablesDto.getComment()); // Table type, returns all types by default listTablesRequest.setTableTypes(listTablesDto.getTableTypes()); // Sort method, CreateTime (default) / ModifyTime / Name / TableType listTablesRequest.setSortBy(listTablesDto.getSortBy()); // Sort direction, supports Asc (default) / Desc listTablesRequest.setOrder(listTablesDto.getOrder()); // Page number, default is 1 listTablesRequest.setPageNumber(listTablesDto.getPageNumber()); // Page size, default is 10, maximum 100 listTablesRequest.setPageSize(listTablesDto.getPageSize()); ListTablesResponse response = client.listTables(listTablesRequest); // Get the total number of tables that meet the requirements System.out.println(response.getBody().getPagingInfo().getTotalCount()); for (Table table : response.getBody().getPagingInfo().getTables()) { // Table ID System.out.println(table.getId()); // Parent entity ID System.out.println(table.getParentMetaEntityId()); // Table name System.out.println(table.getName()); // Table comment System.out.println(table.getComment()); // Database/MaxCompute project name System.out.println(table.getId().split(":")[3]); // Schema name System.out.println(table.getId().split(":")[4]); // Table type System.out.println(table.getTableType()); // Table creation time System.out.println(table.getCreateTime()); // Table modification time System.out.println(table.getModifyTime()); // Partition field list of the table System.out.println(table.getPartitionKeys()); } return response.getBody().getPagingInfo(); } catch (TeaException error) { // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } catch (Exception _error) { TeaException error = new TeaException(_error.getMessage(), _error); // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } return null; } }
-
-
Adicione um método
listTablesno MetaRestController como ponto de entrada para o acesso do frontend./** * Custom metadata platform using the DataWorks API * * @author dataworks demo */ @RestController @RequestMapping("/meta") public class MetaRestController { @Autowired private MetaServiceProxy metaService; /** * Query table data by page * * @param listTablesDto * @return {@link ListTablesResponseBodyPagingInfo} */ @CrossOrigin(origins = "http://localhost:8080") @GetMapping("/listTables") public ListTablesResponseBodyPagingInfo listTables(ListTablesDto listTablesDto) { System.out.println("listTablesDto = " + listTablesDto); return metaService.listTables(listTablesDto); } }
Frontend: Exibir informações de metadados da tabela
import React from "react";
import cn from "classnames";
import {
Table,
Form,
Field,
Input,
Button,
Pagination,
Dialog,
Card,
Grid,
Select,
} from "@alifd/next";
import type { TableListInput, TableEntity } from "../services/meta";
import * as services from "../services";
import DetailContent from "./detailContent";
import LineageContent from "./lineageContent";
import classes from "../styles/app.module.css";
import moment from "moment";
export interface Props {}
const { Column } = Table;
const { Item } = Form;
const { Header, Content } = Card;
const { Row, Col } = Grid;
const { Option } = Select;
const App: React.FunctionComponent<Props> = () => {
const field = Field.useField();
const [datasource, setDatasource] = React.useState<TableEntity[]>([]);
const [loading, setLoading] = React.useState<boolean>(false);
const [total, setTotal] = React.useState<number>(0);
const [current, setCurrent] = React.useState<number>(1);
const onSubmit = React.useCallback(
(pageNumber: number = 1) => {
field.validate(async (errors, values) => {
if (errors) {
return;
}
setLoading(true);
try {
const response = await services.meta.getTableList({
pageNumber,
...values,
} as TableListInput);
setDatasource(response.tables);
setCurrent(pageNumber);
setTotal(response.totalCount);
} catch (e) {
throw e;
} finally {
setLoading(false);
}
});
},
[field]
);
const onViewDetail = React.useCallback((record: TableEntity) => {
Dialog.show({
title: "Data Table Details",
content: <DetailContent item={record} />,
shouldUpdatePosition: true,
footerActions: ["ok"],
});
}, []);
const onViewLineage = React.useCallback((record: TableEntity) => {
Dialog.show({
title: "Table Lineage Relationship",
content: <LineageContent item={record} />,
shouldUpdatePosition: true,
footerActions: ["ok"],
});
}, []);
React.useEffect(() => {
field.setValue("dataSourceType", "odps");
}, []);
return (
<div className={cn(classes.appWrapper)}>
<Card free hasBorder={false}>
<Header title="Metadata Table Management Scenario Demo (MaxCompute)" />
<Content style={{ marginTop: 24 }}>
<Form field={field} colon fullWidth>
<Row gutter="1">
<Col>
<Item
label="MaxCompute Project/Schema ID"
name="parentMetaEntityId"
required
help="ID"
>
<Input />
</Item>
</Col>
</Row>
<Row gutter="4">
<Col>
<Item label="Name" name="name">
<Input placeholder="Enter table name, fuzzy matching" />
</Item>
</Col>
<Col>
<Item label="Comment" name="comment">
<Input placeholder="Enter table comment, fuzzy matching" />
</Item>
</Col>
<Col>
<Item label="Type" name="tableTypes">
<Select mode="multiple" style={{ width: "100%" }}>
<Option value="TABLE">Table</Option>
<Option value="VIEW">View</Option>
<Option value="MATERIALIZED_VIEW">Materialized View</Option>
</Select>
</Item>
</Col>
<Col>
<Item label="Sort Method" name="sortBy">
<Select defaultValue="CreateTime" style={{ width: "100%" }}>
<Option value="CreateTime">Creation Time</Option>
<Option value="ModifyTime">Modification Time</Option>
<Option value="Name">Name</Option>
<Option value="TableType">Type</Option>
</Select>
</Item>
</Col>
<Col>
<Item label="Sort Direction" name="order">
<Select defaultValue="Asc" style={{ width: "100%" }}>
<Option value="Asc">Ascending</Option>
<Option value="Desc">Descending</Option>
</Select>
</Item>
</Col>
</Row>
<div
className={cn(
classes.searchPanelButtonWrapper,
classes.buttonGroup
)}
>
<Button type="primary" onClick={() => onSubmit()}>
Query
</Button>
</div>
</Form>
<div>
<Table
dataSource={datasource}
loading={loading}
className={cn(classes.tableWrapper)}
emptyContent={
<div className={cn(classes.noDataWrapper)}>No Data</div>
}
>
<Column
title="Project Name"
dataIndex="id"
cell={(value) => {
const parts = value.split(":");
return parts.length > 1 ? parts[parts.length - 3] : "";
}}
/>
<Column
title="schema"
dataIndex="id"
cell={(value) => {
const parts = value.split(":");
return parts.length > 1 ? parts[parts.length - 2] : "";
}}
/>
<Column title="Table Name" dataIndex="name" />
<Column title="Table Comment" dataIndex="comment" />
<Column title="Table Type" dataIndex="tableType" />
<Column
title="Partition Status"
dataIndex="partitionKeys"
cell={(value) => {
return value != null && value.length > 0 ? "Yes" : "No";
}}
/>
<Column
title="Creation Time"
dataIndex="createTime"
cell={(value) => {
return moment(value).format("YYYY-MM-DD HH:mm:ss");
}}
/>
<Column
title="Modification Time"
dataIndex="modifyTime"
cell={(value) => {
return moment(value).format("YYYY-MM-DD HH:mm:ss");
}}
/>
<Column
title="Operation"
width={150}
cell={(value, index, record) => (
<div className={cn(classes.buttonGroup)}>
<Button
type="primary"
onClick={() => onViewDetail(record)}
text
>
View Details
</Button>
<Button
type="primary"
onClick={() => onViewLineage(record)}
text
>
View Table Lineage
</Button>
</div>
)}
/>
</Table>
<Pagination
current={current}
total={total}
onChange={onSubmit}
showJump={false}
className={cn(classes.tablePaginationWrapper)}
/>
</div>
</Content>
</Card>
</div>
);
};
export default App;
Após concluir o desenvolvimento do código acima, implante e execute o código do projeto. Em seguida, insira o ID do projeto MaxCompute e outros parâmetros para consultar e obter a lista de todas as tabelas do espaço do projeto, com suporte a consultas paginadas.
2. Consultar detalhes da tabela
A seção a seguir combina três operações de API: GetTable, ListColumns e ListPartitions nos metadados para consultar os detalhes da tabela, e usa a API UpdateColumnBusinessMetadata para atualizar a descrição de negócios dos campos. O fluxo de trabalho é o seguinte.
Backend: Use o MetaServiceProxy para consultar detalhes da tabela
-
Para chamar as operações, crie os métodos
getTable,listColumnselistPartitionsno MetaServiceProxy para obter as informações básicas, de campos e de partições da tabela. Além disso, crie o métodoupdateColumnBusinessMetadatae atualize a descrição de negócios dos campos./** * @author dataworks demo */ @Service public class MetaServiceProxy { @Autowired private DataWorksOpenApiClient dataWorksOpenApiClient; /** * DataWorks OpenAPI : getTableDto * * @param getTableDto */ public Table getTable(GetTableDto getTableDto) { try { Client client = dataWorksOpenApiClient.createClient(); GetTableRequest getTableRequest = new GetTableRequest(); // Table ID getTableRequest.setId(getTableDto.getId()); // Whether to return business metadata getTableRequest.setIncludeBusinessMetadata(getTableDto.getIncludeBusinessMetadata()); GetTableResponse response = client.getTable(getTableRequest); // Corresponding data table Table table = response.getBody().getTable(); // Table ID System.out.println(table.getId()); // Parent entity ID System.out.println(table.getParentMetaEntityId()); // Table name System.out.println(table.getName()); // Table comment System.out.println(table.getComment()); // Database/MaxCompute project name System.out.println(table.getId().split(":")[3]); // Schema name System.out.println(table.getId().split(":")[4]); // Table type System.out.println(table.getTableType()); // Table creation time System.out.println(table.getCreateTime()); // Table modification time System.out.println(table.getModifyTime()); // Check if table is partitioned System.out.println(table.getPartitionKeys() != null && !table.getPartitionKeys().isEmpty()); // Technical metadata of the table TableTechnicalMetadata technicalMetadata = table.getTechnicalMetadata(); // Table owner (name) System.out.println(technicalMetadata.getOwner()); // Whether it is a compressed table System.out.println(technicalMetadata.getCompressed()); // Input format (HMS, DLF and other types support) System.out.println(technicalMetadata.getInputFormat()); // Output format (HMS, DLF and other types support) System.out.println(technicalMetadata.getOutputFormat()); // Class used for table serialization (HMS, DLF and other types support) System.out.println(technicalMetadata.getSerializationLibrary()); // Table storage location (HMS, DLF and other types support) System.out.println(technicalMetadata.getLocation()); // Other parameters Map<String, String> parameters = technicalMetadata.getParameters(); // Last DDL update time (millisecond-level timestamp), only MaxCompute type supports System.out.println(parameters.get("lastDDLTime")); // Lifecycle (unit: days), only MaxCompute type supports System.out.println(parameters.get("lifecycle")); // Storage size (unit: bytes), not real-time, only MaxCompute type supports System.out.println(parameters.get("dataSize")); // Business metadata of the table TableBusinessMetadata businessMetadata = table.getBusinessMetadata(); if (businessMetadata != null) { // Usage instructions System.out.println(businessMetadata.getReadme()); // Custom tag information List<TableBusinessMetadataTags> tags = businessMetadata.getTags(); if (tags != null && !tags.isEmpty()) { for (TableBusinessMetadataTags tag : tags) { System.out.println(tag.getKey() + ":" + tag.getValue()); } } // Upstream production tasks List<TableBusinessMetadataUpstreamTasks> upstreamTasks = businessMetadata.getUpstreamTasks(); if (upstreamTasks != null && !upstreamTasks.isEmpty()) { for (TableBusinessMetadataUpstreamTasks upstreamTask : upstreamTasks) { // Task ID and task name, task details can be obtained through GetTask API System.out.println(upstreamTask.getId() + ":" + upstreamTask.getName()); } } // Categories List<List<TableBusinessMetadataCategories>> categories = businessMetadata.getCategories(); if (categories != null && !categories.isEmpty()) { // Traverse multi-level categories for (List<TableBusinessMetadataCategories> category : categories) { // Output a single multi-level category System.out.println( category.stream() .map(TableBusinessMetadataCategories::getName) .collect(Collectors.joining("->")) ); } } // Extension information, only MaxCompute type supports TableBusinessMetadataExtension extension = businessMetadata.getExtension(); if (extension != null) { // Project ID System.out.println(extension.getProjectId()); // Environment (Prod: production / Dev: development) System.out.println(extension.getEnvType()); // Favorite count System.out.println(extension.getFavorCount()); // Read count System.out.println(extension.getReadCount()); // View count System.out.println(extension.getViewCount()); } } return table; } catch (TeaException error) { // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } catch (Exception _error) { TeaException error = new TeaException(_error.getMessage(), _error); // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } return null; } /** * DataWorks OpenAPI : ListColumns * * @param listColumnsDto */ public ListColumnsResponseBodyPagingInfo listColumns(ListColumnsDto listColumnsDto) { try { Client client = dataWorksOpenApiClient.createClient(); ListColumnsRequest listColumnsRequest = new ListColumnsRequest(); // Table ID (from ListTables API) listColumnsRequest.setTableId(listColumnsDto.getTableId()); // Field name, supports fuzzy matching listColumnsRequest.setName(listColumnsDto.getName()); // Field comment, supports word segmentation matching listColumnsRequest.setComment(listColumnsDto.getComment()); // Sort method, supports Name / Position (default) listColumnsRequest.setSortBy(listColumnsDto.getSortBy()); // Sort direction, supports Asc (default) / Desc listColumnsRequest.setOrder(listColumnsDto.getOrder()); // Page number, default is 1 listColumnsRequest.setPageNumber(listColumnsDto.getPageNumber()); // Page size, default is 10, maximum 100 listColumnsRequest.setPageSize(listColumnsDto.getPageSize()); ListColumnsResponse response = client.listColumns(listColumnsRequest); // Get the total number of fields that meet the requirements System.out.println(response.getBody().getRequestId()); System.out.println(response.getBody().getPagingInfo().getTotalCount()); for (Column column : response.getBody().getPagingInfo().getColumns()) { // Field ID System.out.println(column.getId()); // Field name System.out.println(column.getName()); // Field comment System.out.println(column.getComment()); // Field type System.out.println(column.getType()); // Field position System.out.println(column.getPosition()); // Whether it is a partition field System.out.println(column.getPartitionKey()); // Whether it is a primary key, only MaxCompute type supports System.out.println(column.getPrimaryKey()); // Whether it is a foreign key, only MaxCompute type supports System.out.println(column.getForeignKey()); // Business description for the field (supported by MaxCompute, HMS, and DLF) if (column.getBusinessMetadata() != null) { System.out.println(column.getBusinessMetadata().getDescription()); } } return response.getBody().getPagingInfo(); } catch (TeaException error) { // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } catch (Exception _error) { TeaException error = new TeaException(_error.getMessage(), _error); // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } return null; } /** * DataWorks OpenAPI : UpdateColumnBusinessMetadata * * @param updateColumnBusinessMetadataDto */ public boolean updateColumnBusinessMetadata(UpdateColumnBusinessMetadataDto updateColumnBusinessMetadataDto) { try { // Currently supports MaxCompute, DLF, HMS (EMR cluster) types Client client = dataWorksOpenApiClient.createClient(); UpdateColumnBusinessMetadataRequest updateColumnBusinessMetadataRequest = new UpdateColumnBusinessMetadataRequest(); // Field ID updateColumnBusinessMetadataRequest.setId(updateColumnBusinessMetadataDto.getId()); // Field business description updateColumnBusinessMetadataRequest.setDescription(updateColumnBusinessMetadataDto.getDescription()); UpdateColumnBusinessMetadataResponse response = client.updateColumnBusinessMetadata(updateColumnBusinessMetadataRequest); System.out.println(response.getBody().getRequestId()); // Whether the update is successful return response.getBody().getSuccess(); } catch (TeaException error) { // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } catch (Exception _error) { TeaException error = new TeaException(_error.getMessage(), _error); // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } return false; } /** * DataWorks OpenAPI : ListPartitions * * @param listPartitionsDto */ public ListPartitionsResponseBodyPagingInfo listPartitions(ListPartitionsDto listPartitionsDto) { try { // Currently supports MaxCompute type and HMS (EMR cluster) type Client client = dataWorksOpenApiClient.createClient(); ListPartitionsRequest listPartitionsRequest = new ListPartitionsRequest(); // Table ID (from ListTables API) listPartitionsRequest.setTableId(listPartitionsDto.getTableId()); // Partition name, supports fuzzy matching listPartitionsRequest.setName(listPartitionsDto.getName()); // Sort method, supports Name (HMS method default, MaxCompute type supports) / CreateTime (MaxCompute type default) / ModifyTime (MaxCompute type supports) / RecordCount (MaxCompute type supports) / DataSize (MaxCompute type supports) listPartitionsRequest.setSortBy(listPartitionsDto.getSortBy()); // Sort direction, supports Asc (default) / Desc listPartitionsRequest.setOrder(listPartitionsDto.getOrder()); // Page number, default is 1 listPartitionsRequest.setPageNumber(listPartitionsDto.getPageNumber()); // Page size, default is 10, maximum 100 listPartitionsRequest.setPageSize(listPartitionsDto.getPageSize()); ListPartitionsResponse response = client.listPartitions(listPartitionsRequest); // Get the total number of partitions that meet the requirements System.out.println(response.getBody().getPagingInfo().getTotalCount()); for (Partition partition : response.getBody().getPagingInfo().getPartitionList()) { // Table ID System.out.println(partition.getTableId()); // Partition name System.out.println(partition.getName()); // Creation time (millisecond-level timestamp) System.out.println(partition.getCreateTime()); // Modification time (millisecond-level timestamp) System.out.println(partition.getModifyTime()); // Partition record count, only MaxCompute type supports System.out.println(partition.getRecordCount()); // Partition storage size (unit: bytes), only MaxCompute type supports System.out.println(partition.getDataSize()); } return response.getBody().getPagingInfo(); } catch (TeaException error) { // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } catch (Exception _error) { TeaException error = new TeaException(_error.getMessage(), _error); // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } return null; } } -
Forneça quatro métodos no MetaRestController para chamadas do frontend:
getTable,listColumns,listPartitionseupdateColumnBusinessMetadata./** * Custom metadata platform using the DataWorks API * * @author dataworks demo */ @RestController @RequestMapping("/meta") public class MetaRestController { @Autowired private MetaServiceProxy metaService; /** * Get table details * * @param getTableDto * @return {@link Table} */ @CrossOrigin(origins = "http://localhost:8080") @GetMapping("/getTable") public Table getTable(GetTableDto getTableDto) { System.out.println("getTableDto = " + getTableDto); return metaService.getTable(getTableDto); } /** * List columns with pagination * * @param listColumnsDto * @return {@link ListColumnsResponseBodyPagingInfo} */ @CrossOrigin(origins = "http://localhost:8080") @GetMapping("/listColumns") public ListColumnsResponseBodyPagingInfo listColumns(ListColumnsDto listColumnsDto) { System.out.println("listColumnsDto = " + listColumnsDto); return metaService.listColumns(listColumnsDto); } /** * Update column business metadata * * @param updateColumnBusinessMetadataDto * @return {@link Boolean} */ @CrossOrigin(origins = "http://localhost:8080") @PostMapping("/updateColumnBusinessMetadata") public Boolean updateColumnBusinessMetadata(@RequestBody UpdateColumnBusinessMetadataDto updateColumnBusinessMetadataDto) { System.out.println("updateColumnBusinessMetadataDto = " + updateColumnBusinessMetadataDto); return metaService.updateColumnBusinessMetadata(updateColumnBusinessMetadataDto); } /** * Query partition data by page * * @param listPartitionsDto * @return {@link ListPartitionsResponseBodyPagingInfo} */ @CrossOrigin(origins = "http://localhost:8080") @GetMapping("/listPartitions") public ListPartitionsResponseBodyPagingInfo listPartitions(ListPartitionsDto listPartitionsDto) { System.out.println("listPartitionsDto = " + listPartitionsDto); return metaService.listPartitions(listPartitionsDto); } }
Frontend: Exibir informações básicas, de campos e de partições da tabela
import React from "react";
import moment from "moment";
import {
Form,
Grid,
Table,
Pagination,
Tag,
Field,
Input,
Button,
Select,
Dialog,
} from "@alifd/next";
import cn from "classnames";
import * as services from "../services";
import {
type ListColumnsInput,
type TableColumn,
type TableEntity,
type TablePartition,
} from "../services/meta";
import classes from "../styles/detailContent.module.css";
export interface Props {
item: TableEntity;
}
const formItemLayout = {
labelCol: {
fixedSpan: 6,
},
wrapperCol: {
span: 18,
},
labelTextAlign: "left" as const,
colon: true,
className: cn(classes.formItemWrapper),
};
const { Row, Col } = Grid;
const { Item } = Form;
const { Column } = Table;
const { Option } = Select;
const DetailContent: React.FunctionComponent<Props> = (props) => {
let columnsPageSize = 10;
let partitionsPageSize = 5;
const listColumnsField = Field.useField();
const listPartitionsField = Field.useField();
const labelAlign = "top";
const [detail, setDetail] = React.useState<Partial<TableEntity>>({});
const [columns, setColumns] = React.useState<Partial<TableColumn[]>>([]);
const [partitions, setPartitions] = React.useState<Partial<TablePartition[]>>(
[]
);
const [columnsTotal, setColumnsTotal] = React.useState<number>(0);
const [partitionTotal, setPartitionTotal] = React.useState<number>(0);
const [editingKey, setEditingKey] = React.useState<string>("");
const [editingDescription, setEditingDescription] =
React.useState<string>("");
const [updateColumnDialogVisible, setUpdateColumnDialogVisible] =
React.useState<boolean>(false);
const onUpdateColumnDialogOpen = () => {
setUpdateColumnDialogVisible(true);
};
const onUpdateColumnDialogOk = () => {
handleColumnDescriptionSave();
onUpdateColumnDialogClose();
};
const onUpdateColumnDialogClose = () => {
setUpdateColumnDialogVisible(false);
};
const handleEditColumnDescription = async (record: TableColumn) => {
setEditingKey(record.id);
setEditingDescription(record.businessMetadata?.description);
onUpdateColumnDialogOpen();
};
const handleColumnDesrciptionChange = async (e: string) => {
setEditingDescription(e);
};
const handleColumnDescriptionSave = async () => {
try {
const res = await services.meta.updateColumnDescription({
id: editingKey,
description: editingDescription,
});
if (!res) {
console.log("Request update failed");
}
} catch (e) {
console.error("Update failed", e);
} finally {
listColumns();
setEditingKey("");
}
};
const getTableDetail = React.useCallback(async () => {
const response = await services.meta.getTableDetail({
id: props.item.id,
includeBusinessMetadata: true,
});
setDetail(response);
}, [props.item.id]);
const listColumns = React.useCallback(
async (pageNumber: number = 1) => {
listColumnsField.setValue("tableId", props.item.id);
listColumnsField.setValue("pageSize", columnsPageSize);
listColumnsField.validate(async (errors, values) => {
if (errors) {
return;
}
try {
const response = await services.meta.getMetaTableColumns({
pageNumber,
...values,
} as ListColumnsInput);
setColumns(response.columns);
setColumnsTotal(response.totalCount);
} catch (e) {
throw e;
} finally {
}
});
},
[listColumnsField]
);
const listPartitions = React.useCallback(
async (pageNumber: number = 1) => {
listPartitionsField.setValue("tableId", props.item.id);
listPartitionsField.setValue("pageSize", partitionsPageSize);
listPartitionsField.validate(async (errors, values) => {
if (errors) {
return;
}
try {
const response = await services.meta.getTablePartition({
pageNumber,
...values,
} as ListColumnsInput);
setPartitions(response.partitionList);
setPartitionTotal(response.totalCount);
} catch (e) {
throw e;
} finally {
}
});
},
[listPartitionsField]
);
React.useEffect(() => {
if (props.item.id) {
getTableDetail();
listColumns();
listPartitions();
}
}, [props.item.id]);
return (
<div className={cn(classes.detailContentWrapper)}>
<Dialog
v2
title="Update Field Business Metadata"
visible={updateColumnDialogVisible}
onOk={onUpdateColumnDialogOk}
onClose={onUpdateColumnDialogClose}
>
<Row>
<Col>
<Item
{...formItemLayout}
labelAlign={labelAlign}
label="Field Business Description"
>
<Input
value={editingDescription}
onChange={(e) => handleColumnDesrciptionChange(e.toString())}
></Input>
</Item>
</Col>
</Row>
</Dialog>
<Form labelTextAlign="left">
<Row>
<Col>
<Item {...formItemLayout} label="Table ID">
<span className={cn(classes.formContentWrapper)}>
{detail.id}
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="MaxCompute Project">
<span className={cn(classes.formContentWrapper)}>
{detail?.id?.split(":").slice(-3, -2)[0]}
</span>
</Item>
</Col>
<Col>
<Item {...formItemLayout} label="schema Name">
<span className={cn(classes.formContentWrapper)}>
{detail?.id?.split(":").slice(-2, -1)[0]}
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Table Name">
<span className={cn(classes.formContentWrapper)}>
{detail.name}
</span>
</Item>
</Col>
<Col>
<Item {...formItemLayout} label="Comment">
<span className={cn(classes.formContentWrapper)}>
{detail.comment}
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Table Type">
<span className={cn(classes.formContentWrapper)}>
{detail.tableType}
</span>
</Item>
</Col>
<Col>
<Item {...formItemLayout} label="Is Compressed Table">
<span className={cn(classes.formContentWrapper)}>
{detail.technicalMetadata?.compressed ? "Yes" : "No"}
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Workspace ID">
<span className={cn(classes.formContentWrapper)}>
{detail.businessMetadata?.extension?.projectId}
</span>
</Item>
</Col>
<Col>
<Item {...formItemLayout} label="Environment">
<span className={cn(classes.formContentWrapper)}>
{detail.businessMetadata?.extension?.envType === "Dev"
? "Development"
: "Production"}
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Table Lifecycle">
<span className={cn(classes.formContentWrapper)}>
{detail.technicalMetadata?.parameters["lifecycle"]
? `${detail.technicalMetadata?.parameters["lifecycle"]} days`
: "Permanent"}
</span>
</Item>
</Col>
<Col>
<Item {...formItemLayout} label="Storage Space">
<span
className={cn(classes.formContentWrapper)}
>{`${detail.technicalMetadata?.parameters["dataSize"]} Bytes`}</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="DDL Last Change Time">
<span className={cn(classes.formContentWrapper)}>
{moment(
Number(detail.technicalMetadata?.parameters["lastDDLTime"])
).format("YYYY-MM-DD HH:mm:ss")}
</span>
</Item>
</Col>
<Col>
<Item {...formItemLayout} label="Partition Status">
<span className={cn(classes.formContentWrapper)}>
{detail.partitionKeys != null && detail.partitionKeys.length > 0
? "Yes"
: "No"}
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Tags">
<div className={cn(classes.formContentWrapper)}>
{detail.businessMetadata?.tags?.map((tag, index) => (
<span key={index} className={cn(classes.tag)}>
<Tag type="primary" size="small">
{tag.key + (tag.value ? ":" + tag.value : "")}
</Tag>
</span>
))}
</div>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Categories">
<span className={cn(classes.formContentWrapper)}>
{detail.businessMetadata?.categories?.map((category, index) => (
<span key={index} className={cn(classes.tag)}>
<Tag type="normal" color="gray" size="small">
{category.map((obj) => obj.name).join("->")}
</Tag>
</span>
))}
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Upstream Production Tasks">
<span className={cn(classes.formContentWrapper)}>
{detail.businessMetadata?.upstreamTasks?.map((task, index) => (
<span key={index} className={cn(classes.tag)}>
<Tag type="primary" size="small">
{task.name + " (" + task.id + ")"}
</Tag>
</span>
))}
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="View Count">
<span className={cn(classes.formContentWrapper)}>
<Tag type="primary" size="small">
{detail.businessMetadata?.extension?.viewCount}
</Tag>
</span>
</Item>
</Col>
<Col>
<Item {...formItemLayout} label="Read Count">
<span className={cn(classes.formContentWrapper)}>
<Tag type="primary" size="small">
{detail.businessMetadata?.extension?.readCount}
</Tag>
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Favorite Count">
<span className={cn(classes.formContentWrapper)}>
<Tag type="primary" size="small">
{detail.businessMetadata?.extension?.favorCount}
</Tag>
</span>
</Item>
</Col>
</Row>
<Row>
<Col>
<Item {...formItemLayout} label="Creation Time">
<span className={cn(classes.formContentWrapper)}>
{moment(detail.createTime).format("YYYY-MM-DD HH:mm:ss")}
</span>
</Item>
</Col>
<Col>
<Item {...formItemLayout} label="Modification Time">
<span className={cn(classes.formContentWrapper)}>
{moment(detail.modifyTime).format("YYYY-MM-DD HH:mm:ss")}
</span>
</Item>
</Col>
</Row>
<br />
<Item label="Field Details" colon>
<Form
field={listColumnsField}
colon
fullWidth
style={{ marginTop: 10, marginBottom: 10 }}
>
<Row gutter="4">
<Col>
<Item
{...formItemLayout}
labelAlign={labelAlign}
label="Name"
name="name"
>
<Input placeholder="Enter field name, fuzzy matching" />
</Item>
</Col>
<Col>
<Item
{...formItemLayout}
labelAlign={labelAlign}
label="Comment"
name="comment"
>
<Input placeholder="Enter field comment, fuzzy matching" />
</Item>
</Col>
<Col>
<Item
{...formItemLayout}
labelAlign={labelAlign}
label="Sort Method"
name="sortBy"
>
<Select defaultValue="Position" style={{ width: "100%" }}>
<Option value="Position">Position</Option>
<Option value="Name">Name</Option>
</Select>
</Item>
</Col>
<Col>
<Item
{...formItemLayout}
labelAlign={labelAlign}
label="Sort Direction"
name="order"
>
<Select defaultValue="Asc" style={{ width: "100%" }}>
<Option value="Asc">Ascending</Option>
<Option value="Desc">Descending</Option>
</Select>
</Item>
</Col>
<Col>
<div
className={cn(
classes.searchPanelButtonWrapper,
classes.buttonGroup
)}
style={{ marginTop: "20px", textAlign: "right" }}
>
<Button type="primary" onClick={() => listColumns()}>
Query
</Button>
</div>
</Col>
</Row>
</Form>
<Table dataSource={columns}>
<Column title="Name" dataIndex="name" />
<Column title="Type" dataIndex="type" />
<Column title="Comment" dataIndex="comment" />
<Column
title="Business Description"
dataIndex="businessMetadata"
cell={(value, index, record) => (
<div
style={{
display: "flex",
justifyContent: "space-between",
alignItems: "center",
}}
>
<span>{value?.description}</span>
<Button
onClick={() => handleEditColumnDescription(record)}
size="small"
type="primary"
text
style={{ marginLeft: 8 }}
>
Edit
</Button>
</div>
)}
/>
<Column
title="Is Primary Key"
dataIndex="primaryKey"
cell={(value) => (value ? "Yes" : "")}
/>
<Column
title="Is Foreign Key"
dataIndex="foreignKey"
cell={(value) => (value ? "Yes" : "")}
/>
<Column
title="Is Partition Field"
dataIndex="partitionKey"
cell={(value) => (value ? "Yes" : "")}
/>
<Column title="Position" dataIndex="position" />
</Table>
<Pagination
total={columnsTotal}
pageSize={columnsPageSize}
onChange={listColumns}
showJump={false}
className={cn(classes.tablePaginationWrapper)}
/>
</Item>
<br />
<Item label="Partition Details" style={{ marginBottom: 32 }} colon>
<Form
field={listPartitionsField}
colon
fullWidth
style={{ marginTop: 10, marginBottom: 10 }}
>
<Row gutter="4">
<Col>
<Item
{...formItemLayout}
labelAlign={labelAlign}
label="Name"
name="name"
>
<Input placeholder="Enter partition name (partial matches allowed)" />
</Item>
</Col>
<Col>
<Item
{...formItemLayout}
labelAlign={labelAlign}
label="Sort Method"
name="sortBy"
>
<Select defaultValue="CreateTime" style={{ width: "100%" }}>
<Option value="CreateTime">Creation Time</Option>
<Option value="ModifyTime">Modification Time</Option>
<Option value="Name">Partition Name</Option>
<Option value="RecordCount">Partition Record Count</Option>
<Option value="DataSize">Partition Size</Option>
</Select>
</Item>
</Col>
<Col>
<Item
{...formItemLayout}
labelAlign={labelAlign}
label="Sort Direction"
name="order"
>
<Select defaultValue="Asc" style={{ width: "100%" }}>
<Option value="Asc">Ascending</Option>
<Option value="Desc">Descending</Option>
</Select>
</Item>
</Col>
<Col>
<div
className={cn(
classes.searchPanelButtonWrapper,
classes.buttonGroup
)}
style={{ marginTop: "20px", textAlign: "right" }}
>
<Button type="primary" onClick={() => listPartitions()}>
Query
</Button>
</div>
</Col>
</Row>
</Form>
<Table dataSource={partitions} emptyContent={<span>Non-partitioned Table</span>}>
<Column title="Name" dataIndex="name" />
<Column
title="Creation Time"
dataIndex="createTime"
cell={(value) => moment(value).format("YYYY-MM-DD HH:mm:ss")}
/>
<Column
title="Modification Time"
dataIndex="modifyTime"
cell={(value) => moment(value).format("YYYY-MM-DD HH:mm:ss")}
/>
<Column
title="Partition Size"
dataIndex="dataSize"
cell={(value) => `${value} bytes`}
/>
<Column title="Partition Record Count" dataIndex="recordCount" />
</Table>
<Pagination
total={partitionTotal}
pageSize={partitionsPageSize}
onChange={listPartitions}
showJump={false}
className={cn(classes.tablePaginationWrapper)}
/>
</Item>
</Form>
</div>
);
};
export default DetailContent;
Após concluir o desenvolvimento do código acima, implante e execute o código do projeto localmente. Para operações de implantação e execução, consulte Operações comuns: Implantação e execução local.
3. Consultar a linhagem da tabela
Quando a definição de uma tabela muda, use a API do DataWorks para realizar a análise de linhagem de tarefas upstream e downstream e encontrar os nós correspondentes à tabela.
Use a API ListLineages para consultar a linhagem upstream e downstream das tabelas de dados.
Backend: Use o MetaServiceProxy para consultar a linhagem da tabela
-
Para chamar a API, crie o método
listTableLineagesnoMetaServiceProxye obtenha as informações de entidades e relacionamentos de linhagem upstream e downstream da tabela./** * @author dataworks demo */ @Service public class MetaServiceProxy { @Autowired private DataWorksOpenApiClient dataWorksOpenApiClient; /** * DataWorks OpenAPI : ListLineages * * @param listTableLineagesDto */ public ListLineagesResponseBodyPagingInfo listTableLineages(ListTableLineagesDto listTableLineagesDto) { try { Client client = dataWorksOpenApiClient.createClient(); ListLineagesRequest listLineagesRequest = new ListLineagesRequest(); // Lineage query direction, whether it is upstream boolean needUpstream = "up".equalsIgnoreCase(listTableLineagesDto.getDirection()); // Table ID (from ListTables API) // Partition name, supports fuzzy matching if (needUpstream) { listLineagesRequest.setDstEntityId(listTableLineagesDto.getTableId()); listLineagesRequest.setSrcEntityName(listTableLineagesDto.getName()); } else { listLineagesRequest.setSrcEntityId(listTableLineagesDto.getTableId()); listLineagesRequest.setDstEntityName(listTableLineagesDto.getName()); } // Sort method, supports Name listLineagesRequest.setSortBy(listTableLineagesDto.getSortBy()); // Sort direction, supports Asc (default) / Desc listLineagesRequest.setOrder(listTableLineagesDto.getOrder()); // Page number, default is 1 listLineagesRequest.setPageNumber(listTableLineagesDto.getPageNumber()); // Page size, default is 10, maximum 100 listLineagesRequest.setPageSize(listTableLineagesDto.getPageSize()); // Whether to return specific lineage relationship information listLineagesRequest.setNeedAttachRelationship(true); ListLineagesResponse response = client.listLineages(listLineagesRequest); // Get the total number of lineage entities that meet the requirements System.out.println(response.getBody().getPagingInfo().getTotalCount()); if (response.getBody().getPagingInfo().getLineages() == null) { response.getBody().getPagingInfo().setLineages(Collections.emptyList()); } if (response.getBody().getPagingInfo().getLineages() != null) { for (ListLineagesResponseBodyPagingInfoLineages lineage : response.getBody().getPagingInfo().getLineages()) { LineageEntity entity; // Get the corresponding lineage entity if (needUpstream) { entity = lineage.getSrcEntity(); } else { entity = lineage.getDstEntity(); } // Entity ID System.out.println(entity.getId()); // Entity name System.out.println(entity.getName()); // Entity attribute information System.out.println(entity.getAttributes()); // Lineage relationship list between entities List<LineageRelationship> relationships = lineage.getRelationships(); if (relationships != null) { relationships.forEach(relationship -> { // Lineage relationship ID System.out.println(relationship.getId()); // Creation time System.out.println(relationship.getCreateTime()); // Task corresponding to the lineage relationship LineageTask task = relationship.getTask(); if (task != null) { // Task ID, for DataWorks tasks, task details can be obtained through GetTask System.out.println(task.getId()); // Task type System.out.println(task.getType()); // Task attribute information Map<String, String> attributes = task.getAttributes(); if (attributes != null) { // For DataWorks tasks, task and instance attribute information can be obtained from it // Project ID System.out.println(attributes.get("projectId")); // Task ID String taskId = attributes.containsKey("taskId") ? attributes.get("taskId") : attributes.get("nodeId"); System.out.println(taskId); // Instance ID String taskInstanceId = attributes.containsKey("taskInstanceId") ? attributes.get("taskInstanceId") : attributes.get("jobId"); System.out.println(taskInstanceId); } } }); } } return response.getBody().getPagingInfo(); } } catch (TeaException error) { // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } catch (Exception _error) { TeaException error = new TeaException(_error.getMessage(), _error); // For demonstration only. Handle exceptions properly in production code. // Error message System.out.println(error.getMessage()); // Diagnostic address System.out.println(error.getData().get("Recommend")); com.aliyun.teautil.Common.assertAsString(error.message); } return null; } } -
Forneça o método
listTableLineagesnoMetaRestControllerpara chamadas do frontend./** * Custom metadata platform using the DataWorks API * * @author dataworks demo */ @RestController @RequestMapping("/meta") public class MetaRestController { @Autowired private MetaServiceProxy metaService; /** * Query the lineage relationship of the table * * @param listTableLineagesDto * @return */ @CrossOrigin(origins = "http://localhost:8080") @GetMapping("/listTableLineages") public ListLineagesResponseBodyPagingInfo listTableLineages(ListTableLineagesDto listTableLineagesDto) { System.out.println("listTableLineagesDto = " + listTableLineagesDto); return metaService.listTableLineages(listTableLineagesDto); } }
Frontend: Exibir a linhagem da tabela
import React from 'react';
import Graphin, { Behaviors } from '@antv/graphin';
import type { IUserNode, IUserEdge, GraphinData, GraphEvent } from '@antv/graphin';
import cn from 'classnames';
import * as services from '../services';
import type { TableEntity, ListTableLineageOutput, LineageEntity } from '../services/meta';
import classes from '../styles/lineageContent.module.css';
import '@antv/graphin/dist/index.css';
export interface Props {
item: TableEntity;
}
function transNode(entity: LineageEntity | TableEntity, direction?: string): IUserNode {
return {
id: entity.id,
label: entity.name,
data: {
...entity,
direction,
},
style: {
label: {
value: entity.name,
}
},
}
}
function transEdge(source: LineageEntity | TableEntity, target: LineageEntity | TableEntity): IUserEdge {
return {
source: source.id,
target: target.id,
style: {
keyshape: {
lineDash: [8, 4],
lineWidth: 2,
},
animate: {
type: 'line-dash',
repeat: true,
},
}
};
}
function parse(
source: LineageEntity | TableEntity,
data: ListTableLineageOutput,
direction: string,
): [IUserNode[], IUserEdge[]] {
const nodes: IUserNode[] = [];
const edges: IUserEdge[] = [];
data.lineages.forEach((lineageRelationship) => {
if (direction === 'down') {
nodes.push(transNode(lineageRelationship.dstEntity, direction));
edges.push(transEdge(source, lineageRelationship.dstEntity));
} else {
nodes.push(transNode(lineageRelationship.srcEntity, direction));
edges.push(transEdge(lineageRelationship.srcEntity, source));
}
});
return [nodes, edges];
}
function mergeNodes(prev: IUserNode[], next: IUserNode[]) {
const result: IUserNode[] = prev.slice();
next.forEach((item) => {
const hasValue = prev.findIndex(i => i.id === item.id) >= 0;
!hasValue && result.push(item);
});
return result;
}
function mergeEdges(source: IUserEdge[], target: IUserEdge[]) {
const result: IUserEdge[] = source.slice();
target.forEach((item) => {
const hasValue = source.findIndex(i => i.target === item.target && i.source === item.source) >= 0;
!hasValue && result.push(item);
});
return result;
}
const { ActivateRelations, DragNode, ZoomCanvas } = Behaviors;
const LineageContent: React.FunctionComponent = (props) => {
const ref = React.useRef();
const [data, setData] = React.useState({ nodes: [], edges: [] });
const getTableLineage = async (
collection: GraphinData,
target: TableEntity | LineageEntity,
direction: string,
) => {
if (!direction) {
return collection;
}
const response = await services.meta.getTableLineage({
direction,
tableId: target.id,
});
const [nodes, edges] = parse(target, response, direction);
collection.nodes = mergeNodes(collection.nodes!, nodes);
collection.edges = mergeEdges(collection.edges!, edges);
return collection;
};
const init = async () => {
let nextData = Object.assign({}, data, { nodes: [transNode(props.item)] });
nextData = await getTableLineage(nextData, props.item, 'up');
nextData = await getTableLineage(nextData, props.item, 'down');
setData(nextData);
ref.current!.graph.fitCenter();
};
React.useEffect(() => {
ref.current?.graph && init();
}, [
ref.current?.graph,
]);
React.useEffect(() => {
const graph = ref.current?.graph;
const event = async (event: GraphEvent) => {
const source = event.item?.get('model').data;
let nextData = Object.assign({}, data);
nextData = await getTableLineage(nextData, source, source.direction);
setData(nextData);
};
graph?.on('node:click', event);
return () => {
graph?.off('node:click', event);
};
}, [
data,
ref.current?.graph,
]);
return (
}
layout={{
type: 'dagre',
rankdir: 'LR',
align: 'DL',
nodesep: 10,
ranksep: 40,
}}
>
);
};
export default LineageContent;
Após concluir o desenvolvimento do código acima, implante e execute o código do projeto localmente. Para operações de implantação e execução, consulte Operações comuns: Implantação e execução local.
4. Encontrar tarefas correspondentes à tabela
Quando a definição de uma tabela muda, realize a análise de linhagem de tarefas downstream por meio da API do DataWorks, OpenData e assinatura de mensagens para encontrar os nós correspondentes à tabela. As operações específicas são as seguintes.
Atualmente, as mensagens suportam alterações de tabela, alterações de tarefa, entre outras. Usuários da Enterprise Edition podem se conectar a mensagens de alteração de tabela. Ao receber uma mensagem de alteração de tabela, visualize o relacionamento de linhagem da tabela.
Tarefas de linhagem
Use a API ListLineages para visualizar a linhagem da tabela e obter o ID da tarefa do DataWorks (e o ID da instância da tarefa) a partir da tarefa no relacionamento de linhagem.
Com base no ID da tarefa obtido, use a API GetTask para obter os detalhes da tarefa.
Tarefas de produção
Operações comuns: Implantação e execução local
-
Prepare as dependências.
Prepare os seguintes ambientes de dependência: Java 8 ou superior, ferramenta de compilação Maven, ambiente Node e ferramenta pnpm. Execute os seguintes comandos para verificar se possui esses ambientes:
npm -v // If the installation is successful, the version is displayed in the command output. If the installation fails, an error that indicates no command is available is reported. java -version // If the installation is successful, the version is displayed in the command output. If the installation fails, an error that indicates no command is available is reported. pnpm -v // If the installation is successful, the version is displayed in the command output. If the installation fails, an error that indicates no command is available is reported. -
Primeiro, baixe o código do projeto e execute o seguinte comando:
Link para o código do projeto: meta-api-demo.zip.
pnpm i -
Localize o arquivo
application.propertiesno caminho backend/src/main/resources do projeto de exemplo e modifique os parâmetros principais do arquivo.-
Inicialmente, defina
api.access-key-ideapi.access-key-secretcomo o AccessKey ID e o AccessKey Secret da sua conta Alibaba Cloud.NotaPara obter o AccessKey, consulte AccessKey pairs e outras informações relacionadas à sua conta Alibaba Cloud.
Em seguida, defina
api.region-ideapi.endpointcom as informações de região da API a ser chamada.
Outros parâmetros podem ser modificados conforme necessário. O exemplo de preenchimento após a modificação é o seguinte:
## AccessKey Id of the RAM user api.access-key-id=xxx ## AccessKey Secret of the account api.access-key-secret=xxx ## RegionId of the DataWorks service: cn-shanghai/cn-hangzhou api.region-id=cn-hangzhou ## Product of the DataWorks OpenAPI, default is dataworks-public api.product=dataworks-public ## Endpoint of the DataWorks OpenAPI ## Public access: dataworks.${regionId}.aliyuncs.com ## VPC access: dataworks-vpc.${regionId}.aliyuncs.com api.endpoint=dataworks.cn-hangzhou.aliyuncs.com ## Network environment: false for public access, true for VPC api.vpc-env=false -
-
Execute o seguinte comando no diretório raiz do projeto para executar o código de exemplo:
npm run dev