Salesforce data source - DataWorks - Alibaba Cloud Documentation Center

Salesforce provides on-demand software for customer relationship management (CRM), including contact management, product catalogs, order management, opportunity management, and sales management. DataWorks data integration supports reading from Salesforce data sources. This topic describes how to use Salesforce as a data source.

Supported field types

Field type	DataWorks type
address	STRING
anyType	STRING
base64	BYTES
boolean	BOOL
combobox	STRING
complexvalue	STRING
currency	DOUBLE
date	DATE
datetime	DATE
double	DOUBLE
email	STRING
encryptedstring	STRING
id	STRING
int	LONG
json	STRING
long	LONG
multipicklist	STRING
percent	DOUBLE
phone	STRING
picklist	STRING
reference	STRING
string	STRING
textarea	STRING
time	DATE
url	STRING
geolocation	STRING

Add a data source

Before you develop a synchronization task in DataWorks, you must add the required data source to DataWorks by following the instructions in Data source management. You can view parameter descriptions in the DataWorks console to understand the meanings of the parameters when you add a data source.

You can create a Salesforce data source in one of the following ways:

Salesforce official: Log on to the Salesforce website to automatically obtain the access URL and create a data source.

Customize: If you want to set dedicated connection credentials for DataWorks, you can configure the Salesforce data source in Customize mode. You need to log in to Salesforce and create a Connected App to obtain security credentials (Consumer Key and Consumer Secret), which grants connection authorization to DataWorks. The configuration procedure is as follows:

Custom configuration

Create a Connected App

Go to the app creation page.
1. Go to your Salesforce system URL and log on.
2. In the upper-right corner, click , and then click App Manager in the left-side navigation pane.
3. On the Lightning Experience App Manager page, click New Connected App.

Configure the Connected App.

Key parameters:

Step	Description
1	Enter a Connected App Name, API Name, and Contact Email that comply with Salesforce standards.
2	Select Enable OAuth Settings. For Callback URL, enter `https://bff-cn-shanghai.data.aliyun.com/di/oauth/callback/index.html`.
3	In Selected OAuth Scopes, add the following scopes: Access Connect REST API resources (chatter api) Access the identity URL service (id, profile, email, address, phone) Access unique user identifiers (openid) Manage user data via APIs (api) Perform requests at any time (refresh token, offline_access)
4	Clear the Require Proof Key for Code Exchange (PKCE) Extension for Supported Authorization Flows checkbox. Select the Require Secret for Web Server Flow checkbox. Select the Require Secret for Refresh Token Flow checkbox.

View the Consumer Key and Secret of the Connected App.
1. In the App Manager list, find your app. Click the arrow icon next to the app name and select View.
2. On the app details page, find the Consumer Key and Secret section.
  
  Click Manage Consumer Details to retrieve the values.
3. Copy the Consumer Key and Consumer Secret values for later use.

Configure the data source parameters

Log on to the DataWorks console. In the target region, click Data Integration > Data Integration in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Integration.
In the left-side navigation pane, click Data Source to go to the data source management page.

Click Add Data Source, then find and select Salesforce. On the Add Salesforce Data Source page, set Data Source Type to Customize.

Key parameters:

Parameter	Description
Login URL	Enter `https://<YourSalesforceDomain>/services/oauth2/authorize`.
Authentication URL	Enter `https://<YourSalesforceDomain>/services/oauth2/token`.
Consumer Key and Consumer Secret	Enter the Consumer Key and Consumer Secret from the Connected App details.

Click Log in to Salesforce. On the authorization page, enter your username and password, and then click Allow.

Important

Salesforce is a third-party service. Ensure that the VPC for your resource group can connect to the Salesforce platform. Otherwise, you cannot create the data source.

Data synchronization tasks

For information about the entry point for and the procedure of configuring a synchronization task, see the following configuration guides.

Single-table offline synchronization task

See Configuring in wizard mode and Configuring in script mode for instructions.
The Appendix: Script demo and parameter descriptions provides a complete list of parameters and a script demo for script mode.

Appendix: Script demos and parameters

Configure a batch synchronization task by using the code editor

If you want to configure a batch synchronization task by using the code editor, you must configure the related parameters in the script based on the unified script format requirements. For more information, see Script mode configuration. The following information describes the parameters that you must configure for data sources when you configure a batch synchronization task by using the code editor.

Reader script demos

Example 1: Query a Salesforce object

{
  "type":"job",
  "version":"2.0",
  "steps":[
    {
      "stepType":"salesforce",
      "parameter":{
        "datasource":"",
        "serviceType": "sobject",
        "table": "Account",
        "beginDateTime": "20230817184200",
        "endDateTime": "20231017184200",
        "where": "",
        "column": [
          {
            "type": "STRING",
            "name": "Id"
          },
          {
            "type": "STRING",
            "name": "Name"
          },
          {
            "type": "BOOL",
            "name": "IsDeleted"
          },
          {
            "type": "DATE",
            "name": "CreatedDate"
          }
        ]
      },
      "name":"Reader",
      "category":"reader"
    },
    {
      "stepType":"stream",
      "parameter":{},
      "name":"Writer",
      "category":"writer"
    }
  ],
  "setting":{
    "errorLimit":{
      "record":"0"
    },
    "speed":{
      "throttle":true,
      "concurrent":1,
      "mbps":"12"
    }
  },
  "order":{
    "hops":[
      {
        "from":"Reader",
        "to":"Writer"
      }
    ]
  }
}

Example 2: Query using Bulk API 1.0

{
  "type":"job",
  "version":"2.0",
  "steps":[
    {
      "stepType":"salesforce",
      "parameter":{
        "datasource":"",
        "serviceType": "bulk1",
        "table": "Account",
        "beginDateTime": "20230817184200",
        "endDateTime": "20231017184200",
        "where": "",
        "blockCompoundColumn":true,
        "bulkQueryJobTimeoutSeconds":86400,
        "column": [
          {
            "type": "STRING",
            "name": "Id"
          },
          {
            "type": "STRING",
            "name": "Name"
          },
          {
            "type": "BOOL",
            "name": "IsDeleted"
          },
          {
            "type": "DATE",
            "name": "CreatedDate"
          }
        ]
      },
      "name":"Reader",
      "category":"reader"
    },
    {
      "stepType":"stream",
      "parameter":{
        "print": true
      },
      "name":"Writer",
      "category":"writer"
    }
  ],
  "setting":{
    "errorLimit":{
      "record":"0"
    },
    "speed":{
      "concurrent":1
    }
  },
  "order":{
    "hops":[
      {
        "from":"Reader",
        "to":"Writer"
      }
    ]
  }
}

Example 3: Query using Bulk API 2.0

{
  "type":"job",
  "version":"2.0",
  "steps":[
    {
      "stepType":"salesforce",
      "parameter":{
        "datasource":"",
        "serviceType": "bulk2",
        "table": "Account",
        "beginDateTime": "20230817184200",
        "endDateTime": "20231017184200",
        "where": "",
        "blockCompoundColumn":true,
        "bulkQueryJobTimeoutSeconds":86400,
        "column": [
          {
            "type": "STRING",
            "name": "Id"
          },
          {
            "type": "STRING",
            "name": "Name"
          },
          {
            "type": "BOOL",
            "name": "IsDeleted"
          },
          {
            "type": "DATE",
            "name": "CreatedDate"
          }
        ]
      },
      "name":"Reader",
      "category":"reader"
    },
    {
      "stepType":"stream",
      "parameter":{},
      "name":"Writer",
      "category":"writer"
    }
  ],
  "setting":{
    "errorLimit":{
      "record":"0"
    },
    "speed":{
      "throttle":true,
      "concurrent":1,
      "mbps":"12"
    }
  },
  "order":{
    "hops":[
      {
        "from":"Reader",
        "to":"Writer"
      }
    ]
  }
}

Example 4: Query using a SOQL statement

{
  "type":"job",
  "version":"2.0",
  "steps":[
    {
      "stepType":"salesforce",
      "parameter":{
        "datasource":"",
        "serviceType": "query",
        "query": "select Id, Name, IsDeleted, CreatedDate from Account where Name!='Aliyun' ",
        "column": [
          {
            "type": "STRING",
            "name": "Id"
          },
          {
            "type": "STRING",
            "name": "Name"
          },
          {
            "type": "BOOL",
            "name": "IsDeleted"
          },
          {
            "type": "DATE",
            "name": "CreatedDate"
          }
        ]
      },
      "name":"Reader",
      "category":"reader"
    },
    {
      "stepType":"stream",
      "parameter":{},
      "name":"Writer",
      "category":"writer"
    }
  ],
  "setting":{
    "errorLimit":{
      "record":"0"
    },
    "speed":{
      "throttle":true,
      "concurrent":1,
      "mbps":"12"
    }
  },
  "order":{
    "hops":[
      {
        "from":"Reader",
        "to":"Writer"
      }
    ]
  }
}

Reader script parameters

Parameter	Required	Description	Default
datasource	Yes	The name of the data source. In script mode, you can add a data source. The value of this parameter must match the name of the added data source.	None
serviceType	No	The sync mode. Valid values: `sobject`: Queries a Salesforce object. `query`: Queries data by using a SOQL statement. `bulk1`: Queries a Salesforce object using Bulk API 1.0. `bulk2`: Queries a Salesforce object using Bulk API 2.0. Important The `bulk1` and `bulk2` modes do not support compound data types, such as `address` and `geolocation`. The `bulk2` mode does not support distributed tasks. Performance may vary between `bulk1` and `bulk2`. Test both modes on your Salesforce object to select the optimal one.	sobject
table	Yes	The Salesforce object, which is similar to a table name, such as `Account`, `Case`, or `Group`. Required if `serviceType` is `sobject`, `bulk1`, or `bulk2`.	None
beginDateTime	No	The start and end times for data extraction. This parameter is used when `serviceType` is set to `sobject`, `bulk1`, or `bulk2`. Filters data based on the last modified time of the Salesforce object. The last modified time field is automatically identified in the following order of priority: `SystemModstamp` > `LastModifiedDate` > `CreatedDate`. The time range is a left-closed, right-open interval. A time string in the `yyyymmddhhmmss` format. You can use this parameter with scheduling parameters to perform incremental extraction.	None
endDateTime	No		None
splitPk	No	The sharding key. This parameter is used when `serviceType` is set to `sobject`. Specifies a field to use as a sharding key for data sharding. This enables concurrent tasks and improves data synchronization efficiency. Only `datetime`, `int`, and `long` data types are supported. Using other types causes the job to fail.	None
blockCompoundColumn	No	Specifies the behavior when a compound data type is encountered. This parameter is used when `serviceType` is set to `bulk1` or `bulk2`. Valid values: `true`: The job fails. You must remove the column mapping for the compound data type and run the job again. `false`: The compound data type is read as NULL.	true
bulkQueryJobTimeoutSeconds	No	The timeout period for batch data preparation, in seconds. This parameter is used when `serviceType` is set to `bulk1` or `bulk2`. The Salesforce server prepares batch data before reading begins. The job fails if this preparation exceeds the specified timeout.	86400
batchSize	No	The batch size, which is the number of records to download per batch. This parameter is used when `serviceType` is set to `bulk1` or `bulk2`. For optimal performance, set this value slightly higher than the batch size determined by Salesforce's data preparation task. Because data is streamed, increasing this value does not increase memory usage. This is an advanced parameter and is not supported in wizard mode.	300000
where	No	The filter condition. This parameter is used when `serviceType` is set to `sobject`, `bulk1`, or `bulk2`. In actual business scenarios, you can filter data. For example, you can specify `Name != 'Aliyun'`. If this parameter is empty, the system synchronizes full data. The value must be a valid SOQL WHERE clause. For example, `limit 10` is not valid because `LIMIT` is a separate clause in SOQL.	None
query	No	The query statement. This parameter is used when `serviceType` is set to `query`. When the `where` parameter is insufficient, use `query` to provide a complete SOQL query statement. If specified, this parameter overrides the `table`, `column`, `beginDateTime`, `endDateTime`, `where`, and `splitPk` parameters. Example: `select Id, Name, IsDeleted from Account where Name!='Aliyun'`. This is an advanced parameter and is not supported in wizard mode.	None
queryAll	No	Specifies whether to include deleted records in the query. This parameter is used when `serviceType` is set to `sobject` or `query`. If set to `true`, results include deleted records, which can be identified by the `IsDeleted` field.	false
column	Yes	The columns that you want to synchronize. Specify the columns as a JSON array. Supports column pruning, which allows you to select a subset of columns to export. Supports column reordering, which allows you to export columns in an order that is different from the table schema. You can configure constants. Example: `[ { "name": "Id", "type": "STRING" }, { "name": "Name", "type": "STRING" }, { "name": "'123'", "type": "LONG" }, { "name": "'abc'", "type": "STRING" } ]` Note `Id` and `Name` are regular column names. `'123'` is an integer constant. Note that it must be enclosed in single quotation marks. `'abc'` is a string constant. Note that it must be enclosed in single quotation marks. You must explicitly specify all columns to synchronize as a non-empty array.	None
connectTimeoutSeconds	No	The timeout for establishing an HTTP connection, in seconds. The job fails if the connection is not established within this time. This is an advanced parameter and is not supported in wizard mode.	30
socketTimeoutSeconds	No	The HTTP response timeout, in seconds. The job fails if the time between receiving consecutive data packets exceeds this value. This is an advanced parameter and is not supported in wizard mode.	600
retryIntervalSeconds	No	The retry interval, in seconds. This is an advanced parameter and is not supported in wizard mode.	60
retryTimes	No	The number of retry attempts. This is an advanced parameter and is not supported in wizard mode.	3