The MongoDB input widget reads data from a MongoDB data source. To synchronize MongoDB data to another data source, configure the MongoDB input widget first, then set up the target data source.
Prerequisites
-
A MongoDB data source is created. For more information, see Create a MongoDB Data Source.
-
The account used to configure the MongoDB input widget has read-through permission for the data source. If not, request permission. For more information, see Request Data Source Permission.
Procedure
-
Select Development > Data Integration from the top menu bar on the Dataphin home page.
-
On the integration page, select Project from the top menu bar (select an environment if using the Dev-Prod pattern).
-
In the left-side navigation pane, click on the Batch Pipeline. From the Batch Pipeline list, select the offline pipeline you want to develop to access its configuration page.
-
To open the Component Library panel, click Component Library located in the upper right corner of the page.
-
In the Component Library panel's left-side navigation pane, select Input. Then, locate the MongoDB component in the input widget list on the right and drag it onto the canvas.
-
To configure the MongoDB input widget, click the
icon on the widget card, which opens the MongoDB Input Configuration dialog box. -
Configure the necessary parameters in the MongoDB Input Configuration dialog box.
Parameter
Description
Step Name
Enter a name for the widget. The naming conventions are as follows:
-
Can only contain Chinese characters, letters, underscores (_), and numbers.
-
Cannot exceed 64 characters.
Datasource
The drop-down list displays all MongoDB-type data sources in Dataphin, regardless of whether you have read-through permission. Click the
icon to copy the current data source name.-
For data sources without read-through permission, click Request after the data source to request read-through permission. For more information, see Request Data Source Permission.
-
If no MongoDB-type data source exists, click Create Data Source in the drop-down list to create a data source. For more information, see Create a MongoDB Data Source.
Table
Enter table name keywords to search, or enter the exact table name and click Exact Search. After you select a table, the system automatically detects the table status. Click the
icon to copy the name of the currently selected table. Input Filter (optional)
Enter a MongoDB filter statement to filter input data. For example, to retrieve data where age is greater than 23
{"age":{"$gt":23}}.Convert Document And Array To JSON Format
By default, Yes is selected and the system converts Document and Array formats to JSON for output. Select No to retain the original format.
Output Fields
Displays the output fields. You can manually add output fields as follows:
-
Batch Add Output Fields.
-
Click Batch Add to configure fields in JSON or Text format.
NoteAfter batch addition is completed, clicking confirm will overwrite the configured field information.
-
JSON Format:
// Example: [{ "name": "Chinese", "type": "string" }, { "name": "sid", "type": "string" }, { "name": "user_id", "type": "string" }] -
TEXT Format:
// Example: Chinese,string sid,string user_id,string
-
-
-
Create New Output Field: Click +create New Output Field and fill in Column and select Type according to the page prompts. After completing the configuration, click the
icon to save. -
Manage Output Fields: You can perform the following operations on added fields.
-
Click the Actions column
icon to edit a field. -
Click the Actions column's
icon to delete a field.
-
-
-
After you configure the properties for the MongoDB Input Widget, click Confirm.