Azure Private Link concepts

Azure Private Link creates a private, secure connection to your Azure Databricks workspace, ensuring your network traffic isn't exposed to the public internet.

There are three types of Private Link connections:

• Front-end (User-to-workspace): Secures the connection from users and their tools, like the web app, APIs, or BI tools, to your Azure Databricks workspace.
• Back-end (Compute-to-control plane): Secures the connection from your Azure Databricks clusters to the core Azure Databricks services they need to function.
• Web authentication: Enables user login to the web app through single sign-on (SSO) when using a private network. It creates a special private endpoint to handle authentication callbacks from Microsoft Entra ID, which otherwise fail in a private environment. Web authentication is only necessary with front-end private connectivity.

Architecture overview

The architecture described in this document represents a standard implementation pattern for Azure Private Link. It is intended to serve as a foundational framework, not a universal solution, as the optimal configuration depends entirely on your network topology. Key factors that might require you to adapt this model include:

• Existing hub-and-spoke or transit VNet designs.
• Custom DNS forwarding and resolution policies.
• Specific firewall and Network Security Group (NSG) rules.
• Cross-region or multi-cloud connectivity requirements.

Required VNets

Private connectivity uses two distinct virtual networks.

• Transit VNet: This virtual network functions as a central hub for user connectivity. It contains the front-end private endpoints required for client access to workspaces and for browser-based SSO authentication.
• Workspace VNet: This is a virtual network you create specifically to host your Azure Databricks workspace and back-end private endpoint.

Subnet allocation and sizing

Plan subnets in each VNet to support private connectivity and deployments.

• Transit VNet subnets:

  • Private endpoint subnet: Allocates IP addresses for all front-end private endpoints.
  • Browser authentication workspace subnets: Two dedicated subnets, a host or public subnet and a container or private subnet, are recommended for deploying the browser authentication workspace.

• Workspace VNet subnets:

  • Workspace subnets: Two subnets, a host or public subnet and a container or private subnet, are required for the Azure Databricks workspace deployment itself.
  • Back-end private endpoint subnet: An additional subnet is required to host the private endpoint for back-end private connectivity.

Sizing depends on your individual implementation needs, but you can use the following as a guide:

Azure Databricks private endpoints

Azure Databricks uses two distinct types of private endpoints to fully privatize traffic. Understand their different roles to implement them correctly.

• Workspace endpoint (databricks_ui_api): This is the primary private endpoint for securing core traffic to and from your workspace. It handles connections for both the front-end and back-end.
• Web authentication endpoint (browser_authentication): This is a specialized, additional endpoint required only to make Single Sign-On (SSO) work for web browser logins over a private connection. It is required for front-end and end-to-end connectivity.

For the web authentication endpoint, note the following:

• SSO callback requirement: When using Private Link for client access, this endpoint is mandatory for user web login. It securely handles the SSO authentication callbacks from Microsoft Entra ID that otherwise are blocked on a private network. Azure Databricks automatically configures the necessary private DNS records when you create this endpoint. This process doesn't affect REST API authentication.
• Deployment rule: Only one browser_authentication endpoint can exist per Azure region and private DNS zone. This single endpoint serves all workspaces in that region that share the same DNS configuration.
• Production best practice: To prevent outages, create a dedicated "private web auth workspace" in each production region. Its sole purpose is to host this critical endpoint. Disable "Public network access" for this workspace and verify no other front-end private endpoints are created for it. If this host workspace is deleted, web login fails for all other workspaces in the region.
• Alternative configuration: For simpler deployments, you can host the endpoint on an existing workspace instead of creating a dedicated one. This is suitable for non-production environments or if you are confident you only have one workspace in the region. However, be aware that deleting the host workspace immediately breaks authentication for any other workspaces that depend on it.

Front-end private connectivity

The following diagram illustrates the network flow for front-end private connectivity. Front-end connectivity uses a databricks_ui_api private endpoint and a browser_authentication private endpoint over the transit VNet to secure the connection from users and their tools to their Azure Databricks workspace.

See Configure private connectivity to Azure Databricks.

Back-end private connectivity

The following diagram illustrates the network flow for back-end private connectivity. Back-end private connectivity uses a databricks_ui_api private endpoint over the workspace VNet to secure the connection from your Azure Databricks clusters to the core Azure Databricks services they need to function.

See Configure back-end private connectivity to Azure Databricks.

Key considerations

Before you configure private connectivity, keep the following in mind:

• If you have a Network Security Groups policy enabled on the private endpoint, you must allow ports 443, 6666, 3306, and 8443-8451 for Inbound Security Rules in the network security group on the subnet where the private endpoint is deployed.
• To create a connection between your network and the Azure portal and its services, you might need to add Azure portal URLs to your allowlist. See Allow the Azure portal URLs on your firewall or proxy server.

Choose the right Private Link implementation

Use this guide to determine which implementation best fits your needs.