E-MapReduce:Access Databricks data through Delta Sharing in Unity Catalog

Create a Share

• Log in to your Databricks workspace. In the navigation pane on the left, go to Catalog > Delta Sharing > Shared by me.

• Click Share data to create a new Share. A Share is a logical container used to organize and manage the tables that you want to share.

Add assets

• In the new Share, enter a Share name and then add data assets.

• Select one or more tables that you want to share externally.

  Serverless Spark uses the configured Share name and the corresponding tables to access the shared data.

Create a recipient

• If you have not created a Recipient, you can create a Recipient from the drop-down list. You can also return to the Share page and click Add recipient to create a new data Recipient.

  

Configure the recipient and generate a credential file

• Generate a credential file that contains an access token for the Recipient.

  

  • Recipient type: Select Open. The Open type is used for open sharing with any client that supports the Delta Sharing protocol, such as EMR Serverless Spark. It generates a portable credential file. The Databricks type is used for internal sharing between Databricks platforms.

    Note

    Make sure that the Recipient type is set to Open. If the Open option is not selectable, go to Organization > View Delta Sharing Settings in the upper-right corner and enable External delta sharing.

  • Authentication method: Select Token.

  • Token lifecycle: Set the validity period for the Token. The maximum period is 365 days.

• After the process is complete, an Activation link is generated.

Download the credential file

• Open the Activation link from the previous step in a browser. The system prompts you to download a credential file with a .share extension.

Upload the credential file to OSS

• Upload the .share credential file that you downloaded in the previous step to your OSS bucket.

• Record the full OSS path of the file, for example, oss://your-bucket/path/to/credential.share.

  Note

  Security recommendations:

  • Store the credential file in a bucket with private permissions.

  • Make sure that the RAM role used to run the Spark task has read permissions for the OSS file.

Configure public network access

You must prepare a virtual private cloud (VPC) that has public network access and configure a compatible vSwitch for Serverless Spark. This setup allows Serverless Spark to access Databricks Delta Sharing over the public network. Alibaba Cloud provides NAT Gateway to enable public network access for a VPC. For more information, see NAT Gateway. For a list of zones and vSwitches that Serverless Spark supports, see List of supported zones and vSwitches.

Add network connectivity

When you add network connectivity in the workspace, select the VPC that you configured for public network access and its corresponding vSwitch. For more information, see Add network connectivity.

Prepare and upload the SQL file

• Prepare an SQL file to write the shared dataset to DLF.

  -- Create a temporary table
  -- Replace the credential file path and the share dataset
  -- demo_share is the name of the Share created earlier
  -- default.alitable is the shared library and table
  CREATE TEMPORARY TABLE dbc_delta_sharing USING deltaSharing LOCATION 'oss://your-bucket/path/to/credential.share#demo_share.default.alitable';
  SELECT * FROM dbc_delta_sharing limit 10;
  
  -- Create a database and a table
  CREATE database if NOT EXISTS demo_day_ss_dlf;
  DROP TABLE if EXISTS demo_day_ss_dlf.dw_songs_ss_dlf;
  
  CREATE TABLE demo_day_ss_dlf.dw_songs_ss_dlf SELECT * FROM dbc_delta_sharing limit 10;
  SELECT * FROM demo_day_ss_dlf.dw_songs_ss_dlf limit 10;

• Upload the SQL file to the Serverless Spark workspace using the file management feature. For more information, see Manage files.

Add a data catalog

In the Add Data Catalog dialog box, select DLF Data Catalog. For more information about data catalogs, see Manage data catalogs.

Create a Spark task