Microsoft Fabric Custom Live Pools: Setup, Scheduling and Tuning

1. How Microsoft Fabric custom live pools work

Starting a Spark session can take far longer than the notebook code itself. Fabric may need to provision compute, attach the environment, and make the required libraries available. That overhead becomes obvious when engineers keep opening notebooks, when scheduled notebooks run in a narrow window, or when a pipeline has lots of short notebook activities.

A custom live pool lets Fabric prepare compatible clusters ahead of a configured schedule. The cluster is hydrated with the attached environment and kept available through the active window, so a new notebook session can use warm capacity instead of waiting for a fresh cluster.

What it does, and what it doesn’t

Keep this distinction in mind when you analyse performance. If a 20-minute notebook spends three minutes waiting for startup, a correctly configured live pool can cut that wait. It won’t turn the notebook’s 20 minutes of processing into 17. Measure startup and execution separately.

2. Is a custom live pool the right option?

Fabric offers several ways to run Spark, and a live pool isn’t the best choice for every notebook. Pick it when predictable startup during a known window matters more than leaving the default configuration alone.

Microsoft’s documentation lists custom live pools as supporting interactive notebooks, scheduled notebooks, and notebook runs triggered by pipelines. Spark job definitions are not listed as supported in the current release. Check the latest official support matrix before choosing this route for a Spark job definition.

3. Prerequisites and permissions

Confirm each requirement before you touch the workspace. Most failed setups trace back to an unavailable paid capacity, no custom Spark pool, an unpublished environment, or missing permission to customise compute.

• Paid Fabric capacity: the workspace must be assigned to an active paid Fabric SKU. Fabric trial capacity is not supported for custom live pools.
• Existing custom Spark pool: a workspace administrator creates the underlying custom Spark pool. A starter pool can’t be the base for a custom live pool.
• Fabric environment: create or select an environment, configure the Spark runtime and libraries you need, and publish it successfully.
• Workspace role: an Admin can manage the custom Spark pool and live pool configuration. A Member can configure live pool settings for an existing pool only if an administrator has enabled the relevant item-level compute customisation setting.
• Active schedule: each live pool needs a schedule before it can be published.
• Operational owner: decide who owns environment publishing, schedule changes, capacity reviews, and failed-run triage.

4. Step-by-step: configure a custom live pool

Portal labels can change as Fabric evolves, but the documented flow is: create a custom Spark pool, attach it to an environment, turn on live pool compute and a schedule, publish the environment, and check that hydration has finished.

Step 1: Create the underlying custom Spark pool

1. Open the Fabric workspace and select Workspace settings.
2. Expand Data Engineering/Science, then open Spark settings.
3. Open the Pool tab. From the workspace default-pool selection, choose the option to create a new pool.
4. Give the pool a descriptive name, such as team-notebook-live-pool.
5. Select a node family and node size based on the workload you plan to run, and configure autoscale to match the Spark pool options and your team’s needs.
6. Save the custom pool. This step only creates the compute pool. The live and warm behaviour is set through the environment next.

Name pools after ownership and purpose, for example finance-notebooks or etl-notebooks-prod. Avoid names that capture a temporary experiment or one person’s name.

Step 2: Create or select a Fabric environment

1. Open an existing Fabric environment, or create one from the workspace’s New item menu.
2. Choose the Spark runtime and add the Python, Java, or other supported libraries your notebooks need.
3. For shared or scheduled work, use a stable, versioned dependency set. Avoid unpinned packages that can resolve differently from one publish to the next.
4. Choose the library publish mode on purpose. The next section explains why Full mode matters for the fastest session start.
5. Save and publish the environment, and resolve any dependency or build errors before you configure the live pool.

Step 3: Enable live pool compute for the environment

1. In the environment, open the Compute section.
2. Select the custom Spark pool you created in Step 1.
3. Under live pool compute, activate the live pool option.
4. Enable the live pool schedule, and set a recurring schedule or a defined window that matches when the workload really runs.
5. Set the start and end times, time zone, cluster count, idle timeout, and reactivation behaviour using the options in your portal.
6. Save the compute settings, then publish the environment.

Step 4: Attach the environment to the notebook

1. Open the notebook that should use the live pool.
2. Choose the environment you configured and confirm it is the one you intended.
3. Make sure the notebook’s libraries and Spark compute settings line up with that environment. A different configuration may need a different session and won’t benefit from a pool set up for another environment.
4. Save the notebook and run a test inside the intended schedule window.

Step 5: Confirm that clusters are hydrated

1. Open the Fabric Monitoring hub.
2. Find the published environment, open its context menu, and select View details.
3. Expand the Live pool status section.
4. Check the pool status, available clusters, busy clusters, and next schedule.
5. If you want to test warm-start behaviour, run the test notebook only when a compatible cluster is available.

A useful operating view tells Active, Hydrating, Idle, and Stopped apart rather than treating every non-error state as ready. For a session to take the fast path, the signal that matters is that a compatible hydrated cluster is actually available.

5. Full vs Quick library publishing: the choice that affects startup

An environment decides which libraries are available to the Spark session. The publish mode changes when those libraries are installed, so it can change the startup experience even when the live pool has already prepared the compute.

For the fastest and most reproducible live pool startup, Microsoft’s guidance is to use Full publishing mode for the libraries the notebook needs. Quick mode still benefits from warm compute, but libraries that have to install at session start can eat into the advantage.

Library update runbook

1. Change the library or version in the environment.
2. Publish the environment and make sure the publish completes successfully.
3. Refresh the hydrated clusters manually if the portal offers that action, or wait for the next hydration cycle.
4. Check the pool status and run a smoke test against the new environment.
5. Only then release scheduled notebooks that depend on the new package version.

A common mistake is to publish a new environment version and assume every hydrated cluster instantly has it. Existing clusters aren’t automatically refreshed on every library change, so the test has to verify the environment state the session actually used.

6. Schedule planning: make the cluster warm before the notebook starts

The schedule defines when Fabric keeps clusters hydrated and can reactivate them. It does not force a cluster to be ready at the exact start time. Hydration takes time, especially after a fresh environment publish or when new compute has to be created.

Base the schedule on measured demand, not on the team’s full business hours by default. If production notebook work starts at 08:00, schedule the pool early enough to allow hydration. If interactive use normally ends at 16:00, keeping the pool scheduled until midnight just to cover a rare manual request is wasteful. An off-window notebook can still run through standard provisioning, but nobody should promise it the live pool start time.

Schedule design checklist

• Time zone: pick the zone explicitly, and account for daylight-saving changes if the schedule follows local business time.
• Hydration lead time: start ahead of the first deadline. Use Microsoft’s recommended 60 to 90 minute lead for a critical 08:00 start, then adjust from what you observe.
• Peak overlap: estimate the number of concurrent notebook sessions, not just the number of notebooks in the workspace.
• Idle timeout: balance a long warm idle period against cluster churn and extra allocated time.
• End date and holidays: check that a recurring schedule keeps running over the period you need and covers business-critical runs.
• Change window: after publishing an environment update, allow time for fresh hydration and check status before the next time-sensitive run.

7. Cluster sizing and concurrency: start from evidence

There are two separate sizing choices: the compute size of the Spark pool, and the maximum number of clusters the live pool may keep hydrated. Don’t mistake the second for unlimited autoscaling. The live pool does not grow beyond the maximum you configure.

Microsoft’s configuration guide gives workload-based starting points, such as smaller configurations for exploration and larger ones for demanding jobs. They are starting points, not universal sizing rules. Profile your own notebook and capacity before you standardise a cluster size.

A practical sizing experiment

1. Pick one representative notebook that is safe to test and runs often enough to measure.
2. Record its current startup delay separately from its execution time.
3. Choose a conservative live pool maximum from the simultaneous notebook demand you expect in the scheduled window.
4. Test with one session, then with realistic peak concurrency. Record available and busy cluster counts.
5. If demand exceeds the available warm clusters, decide whether to raise the maximum or accept on-demand fallback, based on the business deadline and cost.
6. Look at Spark resource usage before you increase node size. If processing is slow after the session starts, fix the query plan, skew, shuffle, partitions, or data layout. That is not a startup problem.

8. Cost controls: reduce waste without losing the startup benefit

Custom live pools trade some allocated warm-compute time for predictable session startup. Clusters stay warm only within the schedule, and billing applies while clusters are allocated. When the schedule ends or an idle cluster is deactivated, that cluster is deallocated and billing for that allocation stops. Your overall Fabric bill still depends on capacity, workloads, the billing model, and whatever else shares the capacity.

There is no single monthly price for a custom live pool that fits every workspace. Capacity SKU, region, reservation or pay-as-you-go terms, schedule length, pool size, concurrency, and utilisation all matter. Use the current Fabric pricing page and your capacity metrics, and don’t assume a five-second startup target comes free.

A simple operational cost model

Does the saved wait justify the warm time? A quick break-even check

Before you commit, put rough numbers on both sides. You don’t need exact pricing for this. You need to know whether the waiting you remove is worth the warm hours you add.

A made-up example to show the shape, using illustrative numbers only: if a team starts 40 notebook sessions a day and each cold start costs about three minutes, that is around two hours of waiting per day that a warm start would mostly remove. If those same 40 sessions are overnight jobs with no deadline, the two hours cost nobody anything, and a live pool is probably not worth it. The answer depends on who is waiting and what the deadline is, not on the technology.

Four ways to avoid paying for readiness you don’t need

1. Shorten the schedule sensibly. Keep the warm window around real demand, with enough lead time for hydration. Don’t trim the lead time on critical runs to save a few minutes without testing.
2. Set the cluster maximum from evidence. Use the busy and available cluster metrics at peak. Don’t pre-warm more concurrent sessions than the team really uses.
3. Choose a sensible idle timeout. A long timeout reduces repeated hydration but can leave unused clusters allocated. A very short one can make clusters churn and rehydrate more often.
4. Keep different requirements apart, but don’t multiply pools. Don’t create overlapping pools for slightly different configurations unless a library or runtime boundary, or a concurrency need, really calls for it.

Review usage after a representative business cycle. If the live pool is idle most of the time, try a tighter schedule or a lower cluster cap. If it is saturated at deadline time again and again, work out how many start-time targets you missed before you raise the cap. A good configuration balances wait time, utilisation, and deadline reliability, and doesn’t push any one of them to the extreme.

9. Validate startup performance instead of assuming it

Don’t publish a claimed speed-up from one run. Set a repeatable baseline and compare like with like. Environment libraries, schedule state, cluster availability, concurrency, capacity pressure, and notebook logic can all move the result.

Suggested measurement plan

For each test, do several runs under comparable conditions and compare medians as well as outliers. Wherever your monitoring allows, keep three measures apart: session startup time, notebook execution time, and end-to-end pipeline duration. The pipeline figure includes orchestration and queueing, so don’t present it as pure Spark startup time.

Notebook smoke test

Once the environment is attached and the session starts, use a small deterministic test to confirm the Spark session is available and the expected runtime can run. It checks basic function. It does not measure how long hydration took.

from datetime import datetime, timezone

print("Execution timestamp (UTC):", datetime.now(timezone.utc).isoformat())
print("Spark version:", spark.version)

sample = spark.range(0, 10000).selectExpr(
    "id",
    "id % 7 AS bucket"
)

print("Rows:", sample.count())
sample.groupBy("bucket").count().orderBy("bucket").show()

Measure the real wait from a pipeline

A timer inside the notebook can’t see the time spent waiting for the session, because it only starts once the session exists. A workable trick for pipeline-triggered notebooks is to pass the pipeline’s trigger time in as a parameter, then compare it with the time the first cell actually runs. The gap covers queueing plus session start, which is the wait your users feel.

In the notebook, mark the first cell as a parameters cell:

# Parameters cell
trigger_time = ""

In the pipeline’s notebook activity, set the trigger_time parameter to the dynamic expression @pipeline().TriggerTime. Then add this in the next cell:

from datetime import datetime, timezone

started = datetime.now(timezone.utc)

if trigger_time:
    triggered = datetime.fromisoformat(trigger_time.replace("Z", "+00:00"))
    wait_seconds = (started - triggered).total_seconds()
    print("Seconds from pipeline trigger to first cell:", round(wait_seconds, 1))
else:
    print("No trigger_time supplied, skipping wait measurement")

Log that number for every run, and you get a real before and after picture across days instead of a single lucky test. It also includes any time the activity spent queued, so treat it as an upper bound on session start, not a precise measure of it. Compare it with the Monitoring hub timings to see how much of the gap is startup.

10. Troubleshooting common custom live pool problems

11. A rollout plan that works

Turning this on for a whole workspace at once is how teams end up with a big bill and no proof it helped. A small pilot gives you evidence and keeps the cost contained. Here is a plan I would follow.

1. Pick one team and one environment. Choose a group that runs notebooks daily and complains about startup. Don’t pick a team whose pain is slow queries.
2. Baseline for a week first. Log the wait from trigger to first cell, the notebook duration, and the number of sessions per day, before you change anything.
3. Pin and publish in Full mode. Fix the dependency versions, publish, and confirm the environment is Ready.
4. Start with a short schedule and a low cluster cap. Cover the real peak hours, with the hydration lead time, and no more.
5. Run for two weeks and compare. Use the same measures as the baseline, and note the days that missed the warm path and why.
6. Adjust, then decide. Tighten or widen the window, raise or lower the cap, and write down the result in a few lines. If the gain is small for the cost, stop. That is a legitimate outcome.
7. Document it as a runbook. Because configuration is portal-only, record who owns it, how to change the schedule, how to republish after a library change, and how to check status. Keep a change log.

12. Limitations and production-readiness checklist

Before you rely on a custom live pool for a business-critical notebook, write down the current limitations and who owns operations. Some are easy to miss, because a pool can look enabled long before it can actually serve a warm session.

• Notebook sessions only: interactive notebooks, scheduled notebooks, and pipeline-triggered notebook runs are documented as supported. Spark job definitions aren’t supported in the current release.
• Paid capacity: Fabric trial capacities aren’t supported for custom live pools.
• Warm start is conditional: the roughly 5-second start applies after hydration completes and only when a compatible cluster is available.
• Schedules are mandatory: a live pool can’t be published without one, and the fast path isn’t promised outside the active window.
• Library changes need care: republish the environment and make sure hydrated clusters have refreshed before you rely on the changed package set.
• Warm clusters are capped: set and monitor the maximum cluster count from measured concurrency.
• Portal administration: the dedicated documentation says custom live pool configuration is done through the Fabric portal, not environment public APIs or CI/CD pipelines.
• Security and access: follow workspace-role policies, environment governance, approved dependency processes, and any tenant networking restrictions that affect Spark provisioning.

Production checklist to copy into your runbook

13. Frequently asked questions

14. Official Microsoft documentation and further reading

Use the official references below for portal steps, prerequisites, supported workloads, and operational details. If Microsoft changes a label or requirement, these pages take precedence over any saved tutorial.

• Custom live pools in Microsoft Fabric: overview, supported workloads, licensing, and limitations
• Configure custom live pools: portal steps, schedules, monitoring, sizing, and troubleshooting
• Create custom Spark pools in Fabric: base pool setup and node sizing
• Create, configure, and use a Fabric environment: runtime, libraries, and publishing
• Apache Spark compute for Fabric Data Engineering and Data Science
• Fabric Spark capacity and cluster planning
• Fabric September 2026 feature summary: release status and data engineering updates
• Microsoft Fabric pricing: verify current capacity rates and billing options