Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Microsoft Fabric Data Engineering and Data Science experiences operate on a fully managed Spark compute platform. This platform is designed to deliver unparalleled speed and efficiency. It includes starter pools and custom pools.
A Fabric environment contains a collection of configurations, including Spark compute properties, that you can use to configure the Spark session after they're attached to notebooks and Spark jobs. With an environment, you have a flexible way to customize compute configurations for running your Spark jobs.
In an environment, you use the Compute section to configure the Spark session-level properties to customize the memory and cores of executors based on workload requirements. The Spark properties set via spark.conf.set control application-level parameters. They aren't related to environment variables.
Configure settings
As a workspace admin, you can enable or disable compute customizations.
On the Workspace settings pane, select the Data Engineering/Science section.
On the Pool tab, turn the Customize compute configurations for items toggle to On.
You can also delegate members and contributors to change the default session-level compute configurations in a Fabric environment by enabling this setting.
If you disable this option on the Workspace settings pane, the Compute section of the environment is disabled. The default pool compute configurations for the workspace are used for running Spark jobs.
Customize session-level compute properties in an environment
As a user, you can select a pool for the environment from the list of pools available in the Fabric workspace. The Fabric workspace admin creates the default starter pool and custom pools.
After you select a pool in the Compute section, you can tune the cores and memory for the executors within the bounds of the node sizes and limits of the selected pool.
For example, say that you want to select a custom pool with a large node size, which is 16 Spark vCores, as the environment pool.
In the Compute section, under Environment pool, use the Spark driver core dropdown to choose either 4, 8, or 16, based on your job-level requirement.
To allocate memory to drivers and executors, under Spark executor memory, select 28 g, 56 g, or 112 g. All are within the bounds of a large node memory limit.
For more information about Spark compute sizes and their cores or memory options, see What is Spark compute in Fabric?.