Configure private connectivity to Azure Databricks

This page provides instructions for configuring front-end private connectivity, which secures the connection between users and their Azure Databricks workspaces.

• To enable back-end private connectivity to Azure Databricks, see Azure Private Link concepts.
• To connect from the serverless compute plane to Azure resources, see Configure private connectivity to Azure resources.

Why choose a front-end connection?

Whether you use serverless or classic compute, users must connect to Azure Databricks. Organizations choose to connect to Azure Databricks for several reasons, including the following:

• Enhanced security: By restricting all access to private endpoints and disabling public access, you minimize the attack surface and verify all user interactions with Azure Databricks occur over a secure, private network.
• Compliance requirements: Many organizations have strict compliance mandates that require all data and management plane traffic to remain in their private network boundaries, even for SaaS services like Azure Databricks.
• Simplified network architecture (for specific use cases): If you are only using serverless compute, or your primary interaction with Azure Databricks is through the web UI or REST APIs, and you have no immediate need for private connectivity to data sources from Azure Databricks (which requires a back-end connection), a front-end-only setup simplifies the overall network design.
• Data exfiltration prevention: By preventing public access and forcing all traffic through private endpoints, you reduce the risk of data exfiltration, ensuring traffic can only be accessed from authenticated network environments.

Connectivity model

You can configure private connectivity in one of two ways:

• No public access: This configuration disables all public access to the workspace. All user traffic must originate from a VNet connected through a private endpoint. This model is required for full traffic privatization. For full traffic privatization you also need a back-end Private Link connection. See Azure Private Link concepts.
• Hybrid access: Private Link is active, but public access remains enabled with context-based ingress controls and IP access lists. Context-based ingress controls let you restrict access based on identity, request type, and network source. This allow you to safely permit access from trusted public sources (for example, static corporate IPs), while still usingPrivate Link for for private connectivity.

This guide explains how to implement the recommended No Public Access model. We achieve this using a standard hub-and-spoke network topology.

Architecture overview

This model uses the transit VNet:

• Transit VNet: This is your central virtual network containing all the private endpoints needed for client access to workspaces and browser authentication. Your browser authentication workspace is also connected to this VNet.

Before you begin

Review the following prerequisites and recommendations:

Requirements

• The workspace is on the Premium plan.
• You have an Azure Databricks workspace deployed with VNet injection. See Deploy Azure Databricks in your Azure virtual network (VNet injection)
• You must have Azure permissions to create private endpoints and manage DNS records.

Networking configuration

• A transit VNet configured for the following:
  • It acts as the primary transit point for all user/client traffic connecting to your Azure network.
  • It provides centralized connectivity for on-premises or other external networks.
  • It manages shared services and contains the primary route for outbound internet traffic (egress).
• A dedicated subnet must exist within your workspace VNet specifically for private endpoints. If one doesn't exist, create it.
• Your private DNS zones are managed by Azure DNS.

Best practices

Azure Databricks recommends the following for a resilient and manageable setup:

• Architecture: Your network must follow the Microsoft-recommended hub-spoke architecture. See Hub-spoke network topology in Azure.
• Isolated authentication workspace: For improved resilience, create a separate browser authentication workspace inside your transit VNet. This dedicated workspace should host the browser-authentication private endpoint, preventing a single point of failure if other workspaces are deleted. See Step 2: Create a browser_authentication private endpoint.

Configure private connectivity for an existing workspace

Before you begin, you must stop all compute resources like clusters, pools, or classic SQL warehouses. No workspace compute resources can be running, or the upgrade attempt fails. Azure Databricks recommends planning the timing of the upgrade for downtime.

1. On the Workspaces page, select Compute.
2. Select each active compute cluster and in the top-right, click Terminate.

Step 1: Verify VNet-injected workspace with public access enabled

1. Go to your Azure Databricks workspace in the Azure portal.
2. In the workspace overview section, verify your Azure Databricks workspace uses your own virtual network:
   1. Under Settings, select the Networking tab. Confirm the following settings:
      1. Secure cluster connectivity (No Public IP) is enabled.
      2. Allow Public Network Access is Enabled.

Step 2: Create databricks_ui_api private endpoint

1. From your workspace's Networking tab, select Private endpoint connections.
2. Click Private endpoint.
3. Select the resource group for the endpoint, provide a name like my-workspace-fe-pe. Verify the region matches your workspace.
4. Click Next: Resource.
5. Set Target sub-resource to databricks_ui_api.
6. Click Next: Virtual Network.
7. Select your transit VNet. Your transit VNet is a separate, pre-existing VNet in your network architecture that manages and secures egress traffic, often containing a central firewall.
8. Select the subnet that hosts the private endpoints.
9. Click Next and verify Integrate with private DNS zone is set to Yes. It should automatically select the privatelink.azuredatabricks.net zone.

Step 2: Create a browser_authentication private endpoint

Create a private endpoint for browser authentication to support SSO over your private network path. Azure Databricks recommends hosting this endpoint on a dedicated private web auth workspace.

Create a resource group

1. In the Azure portal, navigate to and select Resource groups.
2. Click + Create.
3. Provide a Name for the resource group, such as web-auth-rg-eastus.
4. For Region, select the same Azure region where your production Databricks workspaces are deployed.
5. Click Review + create, then Create.

Create a VNet

1. In the Azure portal, search for and select Virtual networks.
2. Click + Create.
3. On the Basics tab, select the Resource group you just created and give the VNet a descriptive Name, like web-auth-vnet-eastus.
4. Verify the Region matches your resource group.
5. On the IP Addresses tab, define an IP address space for the VNet, for example, 10.20.0.0/16. You are also prompted to create an initial subnet.
6. Select Review + create, then Create.

Create and secure the private web auth workspace

1. In the Azure portal, search for and select Azure Databricks. Click + Create.
2. On the Basics tab, configure the following:
   1. Select the Resource group you just created.
   2. Give the workspace a descriptive name, like WEB_AUTH_DO_NOT_DELETE_<region>.
   3. Select the same region as your resource group and VNet.
3. Click Next:Networking and configure the following:
   1. Deploy Azure Databricks workspace with Secure Cluster Connectivity (No Public IP): Select Yes.
   2. Deploy Azure Databricks workspace in your own Virtual Network (VNet): Select Yes.
   3. Virtual Network: Select the VNet you just created. You are prompted to define subnet ranges.
   4. Public Network Access: Select Disabled.
   5. Required NSG Rules: Select NoAzureDatabricksRules.
4. Click Review + create, then Create.

Once the workspace is created, you must protect it from accidental deletion.

1. In the Azure portal, navigate to the workspace you just created.
2. Go to Settings and select Locks.
3. Click + Add.
4. Set the Lock type to Delete and provide a descriptive Lock name.
5. Click OK.

Step 4: Configure and verify DNS

After you deploy the private endpoints, you must verify that DNS correctly resolves Azure Databricks URLs to their new private IP addresses.

1. Verify private DNS zone records:
   1. In the Azure portal, search for and navigate to the Private DNS zone named privatelink.azuredatabricks.net.
   2. Verify the following A records exist and point to the private IP addresses of your endpoints:
      1. Workspace UI/API record:
         • Name: Your unique workspace ID, like adb-xxxxxxxxxxxxxxxx.x
         • Value: The private IP address of your databricks_ui_api private endpoint.
      2. Browser authentication record:
         • Name: Choose a descriptive name like pl-auth.<your_region>.
         • Value: The private IP address of your browser_authentication private endpoint.

Step 5: Verify private network access

Confirm that you can access the workspace through your private network connection.

From a connected network

If your on-premises network is already connected to your Azure VNet, by VPN or ExpressRoute, the test is simple:

• From your computer, open a web browser and go directly to your Azure Databricks workspace URL to log in. A successful login confirms your private connection is working.

Using a test VM

If you cannot access the workspace VNet from your current ___location, create a temporary virtual machine (a "jump box") to test from:

1. Create a VM: In the Azure portal, create a Windows virtual machine. Place it in a subnet using the same transit VNet where you configured your front-end private endpoint.
2. Connect to the VM: Use a Remote Desktop client to connect to your new VM.
3. Test from the VM: After you connect to the VM, open a web browser, go to the Azure portal, and find your Azure Databricks workspace.
4. Launch Workspace: Click Launch Workspace. A successful login confirms that access from within your private VNet is working correctly.

Verify DNS with nslookup

1. Connect to a virtual machine inside your configured VNet or to your on-premises network via VPN or Azure ExpressRoute. Your machine must be able to use Azure's private DNS.
2. Open a command prompt or terminal and use nslookup to verify DNS resolution.

# Verify the workspace URL resolves to a private IP
nslookup adb-xxxxxxxxxxxxxxxx.x.azuredatabricks.net

# Expected output:
# Server:  <your-dns-server>
# Address: <your-dns-server-ip>
#
# Name:    adb-xxxxxxxxxxxxxxxx.x.privatelink.azuredatabricks.net
# Address: 10.10.1.4  <-- This should be the private IP of your 'databricks_ui_api' endpoint
# Aliases: adb-xxxxxxxxxxxxxxxx.x.azuredatabricks.net

Custom DNS configuration

When using a private front-end endpoint with your own custom DNS, you must verify that both the workspace URL and the SSO (single sign-on) authentication URLs resolve correctly to the private endpoint's IP address.

Recommended: Conditional forwarding

The most reliable method is to configure your DNS server to forward queries for all Databricks domains to Azure's internal DNS.

1. Set up conditional forwarding for the following domains to your Azure DNS server:
   • *.azuredatabricks.net
   • *.privatelink.azuredatabricks.net
   • *.databricksapps.com
2. Verify your VNet is linked to the Azure Private DNS Zone.

This allows Azure to automatically resolve all necessary hostnames, including SSO and workspace URLs, to your private endpoint's IP address.

Alternative: Manual A Records

If conditional forwarding is not an option, you must manually create DNS A records.

1. Workspace URL: Create an A record mapping your per-workspace URL, such as adb-1111111111111.15.azuredatabricks.net, to the private endpoint IP address.
2. SSO Authentication URL: Create an A record mapping the regional SSO URL, such as westus.pl-auth.azuredatabricks.net, to the same private endpoint IP address.

Some Azure regions use multiple control plane instances for SSO. You might need to create several A records for authentication. Contact your Azure Databricks account team for the complete list of domains for your region.