Before you synchronize data from an Oracle data source to Kafka, you can refer to the operations in this topic to configure the network, whitelists, and permissions for the data sources to implement data synchronization.
Prerequisites
- Prepare data sources: An Oracle data source and a Kafka data source are prepared.
- Plan and prepare resources: An exclusive resource group for Data Integration is purchased and configured. For more information, see Plan and configure resources.
- Evaluate and plan the network environment: Before you perform data integration, connect
data sources to exclusive resource groups for Data Integration based on your business
requirements. After data sources and exclusive resource groups for Data Integration
are connected, you can refer to the operations in this topic to configure access settings
such as vSwitches and whitelists.
- If data sources and exclusive resource groups for Data Integration reside in the same region and virtual private cloud (VPC), they are automatically connected.
- If data sources and exclusive resource groups for Data Integration reside in different network environments, you must connect data sources and resource groups by using methods such as a VPN gateway.
Background information
- Configure whitelists for data sources
If the source and destination data sources and the exclusive resource group for Data Integration reside in the same VPC, you must add the CIDR block of the exclusive resource group for Data Integration to the whitelists of the data sources. This ensures that the exclusive resource group for Data Integration can be used to access the data sources.
- Create an account and grant permissions the account
You must create an account that can be used to access the data sources, read data from the source, and write data to the destination during the data synchronization process.
- Check whether the version of the Oracle data source is available for creating real-time
sync nodes of Data Integration in DataWorks
Data synchronization from an Oracle data source in real time is implemented based on the Oracle LogMiner utility that is used to analyze log files. Only the Oracle
10g
,11g
,12c non-CDB
,18c non-CDB
, and19c non-CDB
databases are supported by real-time sync nodes in DataWorks. The Oracle12c CDB
,18c CDB
, and19c CDB
databases are not supported. An Oracle database of Oracle 12c or later versions can act as a container database (CDB) to host multiple pluggable databases (PDBs).- You can execute one of the following statements to view the version of the Oracle
database:
- Statement 1:
select * from v$version;
- Statement 2:
select version from v$instance;
- Statement 1:
- If the version of the Oracle database is
12c
,18c
, or19c
, you can execute the following statement to check whether the database can act as aCDB
. If the Oracle database can act as aCDB
, this Oracle database is not supported by real-time sync nodes of Data Integration in DataWorks.select name,cdb,open_mode,con_id from v$database;
Note You must change the version of the Oracle database that is not supported by real-time sync nodes of Data Integration in DataWorks. Otherwise, you cannot synchronize data from Oracle in real time by using Data Integration. - You can execute one of the following statements to view the version of the Oracle
database:
- Enable the generation of database-level archived log files and redo log files and
enable database-level supplemental logging
You must enable the generation of database-level archived log files and redo log files, and database-level supplemental logging for an Oracle data source.
- Archived log files: Oracle archives all redo log files as archived log files that are used to restore the database in the event of a failure.
- Redo log files: Oracle uses redo log files to ensure that database transactions can be re-executed. This way, data can be recovered in the case of a failure such as a power outage.
- Supplemental logging: Supplemental logging is used to supplement the information recorded
in redo log files. In Oracle, a redo log file is used to record the values of the
fields that are modified. Supplemental logging is used to supplement the change history
in the redo log file. This ensures that the redo log file contains complete information
that describes data changes. If operations such as data recovery and data synchronization
are performed, you can view complete statements and data updates. Some features of
the Oracle database can be better implemented after supplemental logging is enabled.
Therefore, you must enable supplemental logging for the database.
For example, if you do not enable supplemental logging, after you execute the UPDATE statement, the redo log file records only the values of the fields that are modified when the UPDATE statement is executed. If you enable supplemental logging, the redo log file records the values of fields before and after a modification. The conditions that are used to modify destination fields are also recorded. When an exception such as a power outage occurs in the database, you can recover data based on the modification details.
We recommend that you enable supplemental logging for primary key columns or unique index columns.- After you enable supplemental logging for primary key columns, the columns that compose a primary key are recorded in logs if the database is updated.
- After you enable supplemental logging for unique index columns, the columns that compose a unique key or bitmap index are recorded in logs if a column is modified.
Before you synchronize data from an Oracle database in real time by using Data Integration in DataWorks, make sure that the generation of database-level archived log files and supplemental logging are enabled for the Oracle database. To check whether the generation of database-level archived log files and supplementary logging are enabled for the Oracle database, execute the following SQL statement:select log_mode, supplemental_log_data_pk, supplemental_log_data_ui from v$database;
- The returned value ARCHIVELOG of the
log_mode
parameter indicates that the generation of database-level archived log files is enabled for the Oracle database. If the value ARCHIVELOG is not returned, you must enable supplemental logging for the Oracle database. - The returned value YES of the
supplemental_log_data_pk
andsupplemental_log_data_ui
parameters indicates that supplemental logging is enabled for the Oracle database. If the returned value is FALSE, you must enable the generation of database-level archived log files for the Oracle database.
- Check character encoding formats
You must make sure that the Oracle database contains only the character encoding formats that are supported by Data Integration to prevent data synchronization failures. The following encoding formats are supported by Data Integration: UTF-8, AL32UTF8, AL16UTF16, and ZHS16GBK.
- Check data types
You must make sure that the Oracle database contains only the data types that are supported by Data Integration to prevent data synchronization failures. The following data types are not supported by Data Integration for real-time data synchronization: LONG, BFILE, LONG RAW, and NCLOB.
Limits
- You can configure the supplemental logging feature only in a primary Oracle database. Supplemental logging can be enabled for a primary or secondary database.
- The following encoding formats are supported by Data Integration: UTF-8, AL32UTF8, AL16UTF16, and ZHS16GBK.
- The following data types are not supported by Data Integration for real-time data synchronization: LONG, BFILE, LONG RAW, and NCLOB.
- Only the Oracle
10g
,11g
,12c non-CDB
,18c non-CDB
, and19c non-CDB
databases are supported by real-time sync nodes in DataWorks. The Oracle12c CDB
,18c CDB
, and19c CDB
databases are not supported. An Oracle database of Oracle 12c or later versions can act as a CDB to host multiple PDBs.
Usage notes
- Real-time sync nodes of Data Integration can subscribe to online redo log files only in primary Oracle databases. Real-time sync nodes of Data Integration can subscribe to only archived log files in secondary Oracle databases. If you have a high requirement on the timeliness of your real-time sync nodes, we recommend that you subscribe to real-time incremental change logs in primary Oracle databases. If you subscribe to archived log files in a secondary Oracle database, the minimum latency between the generation and the receiving of archived logs is determined by the interval at which Oracle automatically switches between two archived log files. Therefore, the timeliness of real-time sync nodes cannot be ensured.
- We recommend that you retain the archived log files of your Oracle database for three days. When a large amount of data is written to an Oracle database, the speed of real-time data synchronization may be slower than that of log generation. This provides you sufficient time to trace data when issues occur during data synchronization. You can analyze archive log files to troubleshoot issues and recover data.
- Real-time sync nodes of Data Integration cannot perform
truncate
operations on tables without a primary key in an Oracle database. A real-time sync node analyzes logs of a table without a primary key by performing alogminer
operation. In this process, the real-time sync node usesrowid
to find rows.Truncate
operations modify therowid
values in a table. This causes the failure of real-time sync nodes. - When you run real-time sync nodes on a DataWorks server with the configurations of
24 vCPUs and 192 GiB memory
, the utilization of a single CPU core on an Oracle database server varies based on the status of logs in the database. If most logs are notupdate
logs and the sync nodes can process about 30 to 50 thousand data records every second, the utilization of a single CPU core can be 25% to 35%. If most logs areupdate
logs, the DataWorks server encounters a performance bottleneck during real-time synchronization, and the utilization of a single CPU core on the Oracle database server is only 1% to 5%.
Procedure
What to do next
After data sources are configured, the source data source, destination data source, and exclusive resource group for Data Integration are connected. Then, the exclusive resource group for Data Integration can be used to access the data sources. You can add the source data source and destination data source to DataWorks, and associate them with a data synchronization solution when you create the solution.
For more information about how to add the source and destination data sources, see Add data sources.