Fabric Runtime 2.0: Critical Spark 4.1 Migration Guide
Runtime 2.0 is Fabric’s biggest Spark platform jump to date — six major components upgraded at once, a six-times-faster execution engine, and a hard default-runtime deadline in late September 2026. This guide covers what actually changed, how to enable it, the breaking change most teams will hit first, and a migration checklist verified against Microsoft Learn.
Fabric Runtime 2.0 is Microsoft Fabric’s generally available Apache Spark runtime, built on Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, Scala 2.13, R 4.5.2, and Azure Linux 3.0. It succeeds Runtime 1.3 (Spark 3.5) and ships with the Native Execution Engine, which Microsoft benchmarks show running up to six times faster than open-source Spark on TPC-DS workloads — with no code changes required. It is GA but not yet the default; that switch happens in late September 2026.
Six Major Versions, One Runtime
Most Fabric runtime upgrades are incremental — a point release here, a library bump there. Runtime 2.0 is not that. It moves six core components forward simultaneously, several of them across major-version boundaries.
| Component | Runtime 1.3 | Runtime 2.0 |
|---|---|---|
| Apache Spark | 3.5 | 4.1 |
| Delta Lake | 3.2 | 4.2 |
| Python | 3.11 | 3.13 |
| Java | 11 | 21 |
| Scala | 2.12 | 2.13 |
| Operating System | — | Azure Linux 3.0 (Mariner 3.0) |
| R | — | 4.5.2 |
Why This Matters for Planning
Each row in that table would normally justify its own change-management cycle. Together, they mean a Runtime 2.0 migration touches Spark SQL behavior, PySpark library compatibility, JVM garbage collection characteristics, and Delta table protocol versions all at once. Treat this as a platform migration, not a version bump.
Not Default Yet
Existing workspaces stay on Runtime 1.3 until you opt in. Nothing changes for current production workloads unless you explicitly switch.
GA, Production-Ready
Runtime 2.0 is fully supported for production use today — this is not a preview you’re testing at your own risk.
September 2026 Default
New workspaces and environment items switch to Runtime 2.0 by default in late September 2026. Plan validation before that date.
Two Enablement Scopes
Set it workspace-wide, or scope it to a single Environment item so specific notebooks pick it up without affecting everything else.
How to Enable Runtime 2.0
You can enable Runtime 2.0 at the workspace level, which makes it the default for every Spark workload in that workspace, or scope it to a single Environment item, which overrides the workspace default for just the notebooks or Spark job definitions attached to it.
Workspace-Level (applies to everything)
- Open Workspace settingsNavigate to your Fabric workspace and select Workspace settings.
- Go to Data Engineering/Science → Spark settingsSelect the Data Engineering/Science tab, then Spark settings.
- Open the Environment tabInside Spark settings, select the Environment tab.
- Select Runtime version 2.0Under the Runtime version dropdown, choose 2.0 (Spark 4.1, Delta 4.2) and save.
- ConfirmRuntime 2.0 is now the default for all Spark workloads in this workspace going forward.
Environment-Level (scoped to specific notebooks)
- Create or open an Environment itemThis lets you target Runtime 2.0 at specific notebooks or Spark job definitions without changing the workspace default.
- Select Runtime 2.0, save, publishUnder the Runtime dropdown, choose 2.0 (Spark 4.1, Delta 4.2), then Save and Publish.
- Attach itUse this Environment item with the specific Notebook or Spark Job Definition you want to run on Runtime 2.0.
Environment-scoped enablement is the safer starting point for production workspaces. It lets you validate Runtime 2.0 against a single pipeline or a low-risk notebook first, without changing behavior for anything else in the workspace, before committing to the workspace-wide default.
Native Execution Engine: The Headline Feature
Runtime 2.0 ships with the Native Execution Engine, a vectorized C++ execution path (built on Apache Gluten and Velox) that runs supported Spark operators directly against lakehouse infrastructure instead of the JVM. When an operator isn’t yet supported natively, execution automatically falls back to standard JVM-based Spark — so there’s no correctness cliff, only a performance one.
Up to 6x Faster
Microsoft’s TPC-DS benchmarks at scale factor 1000 using Delta tables show up to six times faster performance versus open-source Spark.
~83% Compute-Cost Savings
On a fixed-size Fabric cluster, that performance gain translates to roughly 83% lower compute cost for the same workload.
Zero Code Changes
The engine integrates seamlessly — existing Spark code runs unmodified. Unsupported operators fall back to JVM Spark automatically.
Vectorized CSV Parsing
CSV ingestion and query workloads get an accelerated vectorized parser today. Vectorized JSON parsing and Structured Streaming support are planned.
How to Turn It On
Enable the Native Execution Engine at the environment level so every job and notebook attached to that environment automatically inherits the performance gain — no per-notebook configuration needed.
Data Layout Optimization That Ships With It
- Z-ordering: co-locates related data within Delta table files by specified columns, improving filtered-query performance.
- Liquid Clustering: a flexible clustering approach that keeps data layout optimized automatically, without manual maintenance jobs.
- Parallel Delta snapshot loading: the native engine loads Delta table snapshots in parallel, cutting query startup time on large tables.
What’s New in Apache Spark 4.1
Spark 4.0 was the first release in the 4.x line; Spark 4.1, what Runtime 2.0 ships, builds on that foundation. The headline changes split across three areas.
Spark SQL
- VARIANT data type support
- SQL user-defined functions
- Session variables
- Pipe syntax
- String collation support
PySpark
- Native plotting API
- New Python Data Source API
- Python UDTF support
- Unified profiling for PySpark UDFs
Structured Streaming
- Arbitrary State API v2: more flexible state management for stateful streaming jobs.
- State Data Source: makes debugging streaming state substantially easier.
Deprecation Notice
SparkR is deprecated in Spark 4.x and might be removed in a future version. If any workloads still depend on SparkR, plan a migration path to PySpark or Scala before that removal lands.
Delta Lake 4.2 and Interoperability Limits
Delta Lake 4.2 continues the project’s push toward format interoperability, easier authoring, and better performance. It rolls up the changes introduced across Delta Lake 3.3, 4.0, and 4.1 as well.
Experimental Outside Spark Experiences
Delta Lake 4.x-specific features are experimental and only work inside Spark experiences — notebooks and Spark job definitions. If the same Delta tables need to be consumed by other Fabric workloads (the SQL analytics endpoint, Warehouse, Power BI Direct Lake), do not enable those 4.x-specific features yet. Check Delta Lake table format interoperability before designing around them.
This is the detail most migration writeups skip. Upgrading the runtime doesn’t mean every downstream Fabric engine understands every new Delta protocol feature yet. Test cross-workload reads — not just Spark reads — before relying on Delta Lake 4.2 features in a shared lakehouse.
Compute Management: Resource Profiles & Custom Live Pools
Resource Profiles
Predefined resource allocations for Spark sessions, matched to workload requirements — a straightforward way to control cost without hand-tuning executor settings per job.
Custom Live Pools Preview
Dedicated, pre-warmed Spark pools that reduce session startup time. Available in preview for Runtime 2.0 workloads — useful for interactive notebook development where cold-start latency hurts iteration speed.
The Breaking Change Most Teams Will Hit
Microsoft is actively rolling out an update to Runtime 2.0 that introduces a breaking change for anyone using Environment items with Python and wheel libraries. The Python 3.13 upgrade is the root cause.
What You’ll See
- A Spark deprecation warning referencing setting
-deprecationor-replay -deprecation, sourced from SparkCoreService, or - A
LibraryManagementErrorstating that an upgrade to the base Spark Python environment has been detected and the environment needs to be republished
The Fix
- Remove all librariesOpen the affected Environment item and remove every library, including wheel packages.
- PublishPublish the Environment with no libraries attached.
- Re-add all librariesAdd the same libraries back to the Environment item.
- Publish againPublishing a second time rebuilds the environment against the updated Python 3.13 runtime and resolves the error.
This isn’t a one-time migration step — it’s the required rebuild pattern any time the underlying Spark Python environment shifts under a published Environment item. Bookmark this sequence; it’s worth keeping as a runbook entry, not just a one-off fix.
Migration Checklist
A practical order of operations for validating Runtime 2.0 before the September 2026 default switch.
| Step | What to Check | Why |
|---|---|---|
| 1. Inventory environments | List every Environment item using Python/wheel libraries | These are guaranteed to need the republish fix above |
| 2. Scope-test first | Enable Runtime 2.0 on one Environment item, not the whole workspace | Limits blast radius while validating behavior |
| 3. Run the SQL migration guides | Cross-check Spark SQL, Structured Streaming, PySpark, and MLlib migration guides against your codebase | Catches behavioral differences beyond the Python library issue |
| 4. Test cross-workload Delta reads | Confirm SQL endpoint, Warehouse, and Power BI still read your Delta tables correctly | Delta Lake 4.x features are experimental outside Spark experiences |
| 5. Enable the Native Execution Engine | Turn it on at the environment level and compare job runtimes | This is where most of the cost and speed benefit comes from |
| 6. Set a workspace-wide date | Decide your own cutover date ahead of the September 2026 default | Avoids being forced into an unplanned migration on new workspaces |
VS Code Support
The VS Code extension for Fabric Spark supports Runtime 2.0 for both notebook and Spark job definition development, so local development workflows don’t need a separate migration path.
Frequently Asked Questions
Official References & Related Guides
⚠️ Accuracy Disclaimer
This guide is verified against Microsoft Learn — Runtime 2.0 in Fabric (last updated August 17, 2026) and the official Fabric Runtime 2.0 GA announcement. Runtime rollout status, default-runtime timelines, and known issues change frequently during active rollouts. Always verify current status in Microsoft Learn before planning a production migration. UIG Data Lab is an independent publication, not affiliated with or endorsed by Microsoft Corporation.
