Fabric Data Engineering Interview Questions
40 practitioner-level questions across Spark compute, Notebooks, optimization, Lakehouse architecture, and troubleshooting. Verified against Runtime 1.3 / Runtime 2.0 documentation, FabCon 2026, and the June 2026 Feature Summary.
spark.conf.set("spark.native.enabled", "true")Ask whether the Native Execution Engine is on. Check Spark Advisor alerts in the notebook cell output — fallback warnings mean unsupported operators are running on JVM. This is the highest-impact, zero-code-change fix available. Name it before any partition tuning or join strategy. Source: Microsoft Learn · Runtime 1.3 docs
Senior DE interviews test production debugging over syntax knowledge. Highest signal questions: the Native Execution Engine and when it falls back to JVM, the Runtime 1.3 vs Runtime 2.0 differences, Autoscale Billing vs capacity billing for Spark, Liquid Clustering vs Z-Order for write-heavy tables, and the correct OOM diagnosis flow (Driver vs Executor). Know that Runtime 1.2 was deprecated March 31 2026 — mentioning this unprompted signals you actively track the platform.
Spark Compute & Pools
How Fabric allocates compute — cost implications, cold start behavior, and the 2025 Autoscale Billing model that changes how you size production workloads.
Q1
Starter Pools are pre-provisioned, Microsoft-managed shared compute that initialize in under 30 seconds. Azure Synapse Dedicated Spark Pools took 3–5 minutes to cold-start because they provisioned VMs on demand. The tradeoff: Starter Pools are shared, so they’re appropriate for dev/test and light ad-hoc work, not production ETL with resource isolation requirements.
Fabric keeps a warm pool of pre-allocated nodes under the hood. When you trigger a Spark session, you’re not waiting for VM provisioning — you’re waiting for session initialization only. This is the “Starter Pool” contract: fast start, shared resource, configurable Time-To-Live (TTL) for session keep-alive.
Whether you understand the architectural shift from dedicated VM pools to pre-warmed shared compute and can articulate the specific tradeoff (start speed vs isolation) rather than just saying “it’s faster.”
Q2
Starter Pools for dev, exploration, and lightweight notebooks. Custom Pools for production ETL where you need defined node size, auto-scale limits, pinned library versions, and resource isolation from other workspace users.
| Dimension | Starter Pool | Custom Pool |
|---|---|---|
| Start time | 5–30 seconds | 2–5 minutes (cold) |
| Node type | Microsoft-managed | You define (Small → XXL, Memory Optimized) |
| Library pinning | Limited | Full via Fabric Environment |
| Resource isolation | Shared, no guarantee | Dedicated for the session |
| Best for | Exploration, dev/test, light notebooks | Production ETL, ML training, TB-scale processing |
Whether you default to Custom Pools for everything (over-engineering) or Starter Pools for everything (under-isolating production). The correct answer acknowledges both have a place and you choose based on workload SLA requirements.
Q3
Single Node runs Driver and Executor on the same VM — eliminates network shuffle overhead and reduces cost significantly. Hard limit: everything must fit in one machine’s memory. Not viable for datasets approaching the node’s RAM ceiling.
The practical sweet spot: datasets under ~50–80 GB (depending on node size), ML model training on moderate data, and notebooks where network I/O would dominate over compute. When a job spills to disk on a Single Node because data exceeds memory, switching to a multi-node pool usually recovers more time than it costs in overhead.
Whether you know the hard constraint is single-machine RAM, not just “it’s for small jobs.” Candidates who can name the threshold where shuffling across nodes becomes cheaper than OOM/spill on a single node are demonstrating real production judgment.
Q4
High Concurrency lets multiple Notebook sessions share a single Spark session and its resources. Reduces startup overhead for parallel light queries. But it provides no workload isolation — one heavy job can starve all others sharing that session.
In an interactive analytics scenario where 10 analysts run small queries against the same Gold tables, High Concurrency saves spin-up time. In a production pipeline where a nightly ETL job runs a heavy join at the same time another session triggers a large window function, High Concurrency turns one session’s execution plan into everyone’s bottleneck. Use dedicated Spark Job Definitions (SJDs) for production, not High Concurrency Notebooks.
Whether you understand the difference between resource sharing and resource isolation. The phrase “no workload isolation guarantee” is the precise technical answer that signals production experience.
Q5
Cold Start: session initializes from scratch, VMs allocated, library Environment loaded. Custom Pools: 2–5 minutes. Warm Start: session reused within the TTL window. Starter Pools: 5–30 seconds because nodes are pre-warmed. TTL is configurable — set it higher for interactive analyst workspaces, lower to release capacity for cost control.
For scheduled Spark Job Definitions in production, Cold Start time is baked into the job’s SLA. Size your pipeline schedule accordingly or use Autoscale Billing to get dedicated compute without the shared-pool cold-start variability.
Whether you know that TTL is the lever that controls warm start availability — not just “use Starter Pools.” A good answer mentions that high TTL extends warm availability but keeps the session consuming capacity even when idle.
Q6
Autoscale Billing (GA August 2025) runs Spark jobs on dedicated serverless compute billed only for job runtime — no idle cost. It operates outside the shared F-SKU capacity pool, eliminating contention with Power BI and other Fabric workloads. Enable from Fabric Capacity Settings.
The decision point: base F-SKU capacity is 24/7 cost, suitable for predictable, sustained workloads. Autoscale Billing bills per-second for actual job compute, suitable for bursty, unpredictable, or infrequent batch jobs. For a daily ETL that runs 2 hours a night, Autoscale Billing is nearly always cheaper than over-provisioning base capacity to absorb that burst.
Separately from Autoscale Billing, Fabric now allows Spark jobs to burst up to 3× their base CU allocation by default. Capacity admins can disable this (“Disable Job-level Bursting” in Admin Portal → Capacity Settings → Spark Compute) to prefer concurrency over per-job throughput.
Whether you know this option exists and can articulate when it’s cheaper. Candidates who only know “resize the F-SKU” for cost issues are missing the 2025 billing model that resolves the over-provisioning problem for batch workloads.
Notebooks & Development
The daily developer experience — MSSparkUtils, Environments, VS Code integration, and the Runtime versioning model that changed significantly in 2025–2026.
Q7
Runtime 1.3 (Spark 3.5, Delta Lake 3.2, Python 3.11) is the current GA production runtime. Runtime 2.0 (Spark 4.1, Delta Lake 4.1, Python 3.13) is in Public Preview. Runtime 1.2 was deprecated March 31, 2026 — workloads still on it run unsupported.
The versioning convention: Runtime major version = Spark major version. Runtime 1.x = Spark 3.x, Runtime 2.x = Spark 4.x. Always use the latest GA runtime for new production workloads. Runtime 1.3 enters Long-Term Support (LTS) October 1, 2026 through March 2027 — if you’re on it, you can stay safely for that window.
| Runtime | Spark | Delta Lake | Python | Status |
|---|---|---|---|---|
| 1.2 | 3.4.x | 2.4 | 3.10 | Deprecated Mar 2026 |
| 1.3 | 3.5 | 3.2 | 3.11 | GA — use for production |
| 2.0 | 4.1 | 4.1 | 3.13 | Public Preview |
The Python 3.13 upgrade in Runtime 2.0 requires rebuilding every Environment that has libraries. Re-add all libraries and select Publish to rebuild them against the new runtime before switching production workloads. Also: SparkR is deprecated in Spark 4.x.
Whether you know the deprecation status of Runtime 1.2 and what “Public Preview” means for production readiness. Mentioning the LTS window for 1.3 unprompted signals you plan for platform lifecycle, not just features.
Q8
The Native Execution Engine offloads supported Spark operators from JVM to a vectorized C++ execution path via Apache Gluten and Velox (Runtime 1.3+). Up to 6× faster on TPC-DS benchmarks vs open-source Spark, ~83% compute cost savings. No code changes required. Toggle with spark.native.enabled=true.
# Enable for current session spark.conf.set("spark.native.enabled", "true") # Or set at Environment level for all jobs in the workspace # Workspace Settings → Data Engineering/Science → Spark → Native Execution Engine # Spark Advisor alerts you in the notebook cell output # when an operator falls back to JVM (unsupported operators auto-fallback)
The performance comparison with Databricks has shifted. Databricks Photon is still proprietary and optimized specifically for Databricks. But Fabric’s Native Execution Engine uses the open Velox framework and closes the gap significantly. For standard SQL and Delta Lake analytics workloads, Fabric with the Native Execution Engine is now competitive on throughput. Databricks still leads on managed infrastructure features and Spark 4.x feature availability.
Whether you know this feature exists — and whether you give a nuanced current comparison rather than the outdated “Databricks Photon is always faster” answer. Knowing the ~83% cost savings benchmark signals you’ve read the current documentation.
Q9
MSSparkUtils (also accessible as notebookutils — the canonical alias since Runtime 1.2) is a Fabric-specific utility library that extends PySpark with Fabric-aware operations: file system access, secrets retrieval, notebook chaining, and credentials management.
# File system — mount and browse OneLake notebookutils.fs.ls("Files/") notebookutils.fs.cp("Files/raw/data.csv", "Files/archive/") # Secrets — never hardcode credentials secret = notebookutils.credentials.getSecret("my-keyvault", "db-password") # Notebook chaining — run child notebooks with parameters result = notebookutils.notebook.run( "Silver_Transform", timeout_seconds=600, arguments={"env": "prod", "date": "2026-06-01"} )
Whether you know notebookutils is the current alias (not mssparkutils in newer docs), and whether you immediately go to secrets retrieval for credentials — not environment variables or hardcoded strings.
Q10
%pip install in Fabric restarts the Python interpreter mid-notebook — any variables defined before that cell are lost. It also adds installation time to every run and creates version inconsistency across runs if PyPI releases a new version. Fabric Environments solve all three problems.
A Fabric Environment is a versioned, publishable artifact where you define PyPI/Conda dependencies once. Attaching an Environment to a workspace or Spark Job Definition means: (1) libraries are baked in at startup — no mid-run restarts, (2) version is pinned and consistent across all runs, (3) startup time is faster because dependencies are cached.
If you use %pip install pandas==2.1.0 and then reference a DataFrame defined before that cell, the DataFrame no longer exists — the interpreter restarted. This is a common source of confusing “NameError: df is not defined” errors in Fabric notebooks that don’t occur on Databricks where %pip install doesn’t restart the interpreter.
Whether you know the Fabric-specific behavior of %pip install that differs from Databricks. This is a real gotcha that surfaces in production. Candidates who only say “use Environments for production” without explaining why reveal surface knowledge.
Q11
Use Azure Key Vault linked to the Fabric workspace via Cloud Connections. Retrieve at runtime with notebookutils.credentials.getSecret(vault_url, secret_name). Hardcoded secrets appear in notebook cell output, version control history, and Spark logs — all accessible to workspace members.
The risk beyond the obvious: Fabric Notebooks synced to Git (DevOps/GitHub) will commit cell output if not explicitly cleared before push. A password in a notebook’s output cell persists in Git history even after the cell is edited. The cleanup process is non-trivial. Secrets management via Key Vault prevents this failure mode entirely.
Security hygiene under pressure. The strongest answer includes the Git history risk — most candidates stop at “don’t hardcode.” The interviewer wants to see you understand persistence vectors, not just the surface rule.
Q12
Fabric Notebook Public APIs (GA, FabCon March 2026) allow external systems — Azure DevOps pipelines, GitHub Actions, orchestrators — to programmatically trigger notebook runs, retrieve run status, and pass parameters without the Fabric portal. CI/CD pipelines can now trigger notebooks directly from code.
Before Notebook Public APIs, triggering a Fabric notebook from external CI/CD required a Spark Job Definition wrapper or a Data Pipeline trigger — indirect and less flexible. Now you can POST to the API with parameters, poll for completion, and handle the response in the same way you’d call any REST API from a pipeline. This enables proper notebook-as-a-job patterns in multi-stage deployment workflows.
Whether you know this capability exists in 2026 — and whether you can articulate why it matters (external orchestration, CI/CD integration, GitOps patterns). Candidates who still describe only DevOps Git sync without mentioning the APIs reveal they haven’t tracked the platform recently.
Optimization & Performance
Where senior engineers are separated from junior ones. V-Order, Z-Order, Liquid Clustering (new in Delta 4.x), shuffle tuning, join strategies, and Delta MERGE optimization.
Q13
V-Order is a write-time sorting and compression optimization that makes Parquet files efficient for the VertiPaq engine used by Power BI Direct Lake. It adds ~15% write overhead but delivers 2–3× read speedup for BI queries. It doesn’t belong on write-heavy staging or Bronze tables where the overhead is pure cost with no read benefit.
| Layer | V-Order | Reason |
|---|---|---|
| Bronze / Staging | Disable | Write-heavy ingestion — 15% overhead wasted before data is transformed |
| Silver / Transform | Optional | Enable only if analysts query Silver directly in Power BI |
| Gold / Serving | Enable | Read-heavy, BI-facing — speedup justifies write overhead on every OPTIMIZE |
Whether you understand V-Order as a write-time cost / read-time benefit tradeoff that should be applied selectively, not globally. The layer-specific recommendation demonstrates real architecture thinking.
Q14
Frequent streaming or micro-batch inserts create thousands of small Parquet files, degrading scan performance because each file requires a separate metadata read. Fix options: Auto-Compaction (default on in Fabric), manual OPTIMIZE, coalesce() before writing, or Liquid Clustering (Delta 4.x, Runtime 2.0 Preview).
# Option 1: Manual OPTIMIZE (compacts + optionally Z-Orders) spark.sql("OPTIMIZE sales_gold ZORDER BY (customer_id, order_date)") # Option 2: coalesce before writing — prevents small files at source df.coalesce(8).write.format("delta").mode("append").save(table_path) # Option 3: Liquid Clustering (Runtime 2.0, Delta 4.x — incremental, self-managing) spark.sql(""" CREATE TABLE sales_gold CLUSTER BY (customer_id, order_date) USING DELTA LOCATION '/Tables/sales_gold' """)
Whether you know three distinct solutions and can recommend the right one per scenario. Liquid Clustering on Runtime 2.0 is the most forward-looking answer for new table creation; manual OPTIMIZE is still the right answer for Runtime 1.3 production tables.
Q15
V-Order: BI read performance for VertiPaq/Direct Lake. Z-Order: data skipping for Spark SQL filtered reads on specific columns. Liquid Clustering (Delta 4.x, Runtime 2.0): a flexible, incremental, self-managing replacement for Z-Order + static partitioning on tables with evolving query patterns or high write frequency.
| Technique | Purpose | Maintenance | Best for |
|---|---|---|---|
| V-Order | VertiPaq read compression | Applied at write time | Gold tables used in Power BI Direct Lake |
| Z-Order | Co-locate related rows for data skipping | Manual OPTIMIZE required | Large fact tables with known filter columns in Spark SQL, Runtime 1.3 |
| Liquid Clustering | Flexible, incremental data layout | Self-managing (no manual OPTIMIZE) | New tables on Runtime 2.0 with evolving query patterns or heavy writes |
You can combine V-Order with either Z-Order or Liquid Clustering — they operate at different layers. V-Order is a Parquet encoding optimization; Z-Order/Liquid Clustering are data layout strategies.
Whether you know Liquid Clustering is available in Runtime 2.0 and how it replaces the Z-Order + static partition combination for new table design. This is the clearest signal of whether a candidate is following Delta Lake 4.x development.
Q16
Default is 200 — appropriate for small jobs, catastrophic for TB-scale. Target 100–200 MB per partition: divide your largest shuffle stage’s write size (visible in Spark UI → Stages → Shuffle Write) by 128 MB for a starting count. AQE (enabled by default in Fabric) will coalesce small partitions at runtime.
spark.conf.set("spark.sql.shuffle.partitions", "800") # start here for 100GB shuffle spark.conf.set("spark.sql.adaptive.enabled", "true") # on by default in Fabric spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "134217728") # 128 MB
For full coverage on this topic — AQE limitations (coalesce() blocks it, LEFT OUTER JOIN only optimizes left-side skew) and Storage Partition Join — see our Spark Shuffle Optimization guide.
Whether you know the formula for deriving the partition count from actual data size rather than guessing a magic number. Mentioning AQE as a runtime correction layer alongside static tuning signals mature understanding.
Q17
Three tools: (1) AQE’s built-in skew join handling (fires when a partition is >skewedPartitionFactor × median AND >256 MB), (2) Broadcast join for small dimension tables, (3) Manual salting when 1–2 keys dominate and AQE isn’t enough.
AQE’s skew join handling works by splitting oversized partitions at runtime and optionally replicating the matching rows from the other side. But it has limits: for LEFT OUTER JOINs it only handles left-side skew, and it doesn’t fire at all if you used coalesce() (no shuffle boundary = no AQE stats). Past roughly 5–10× key concentration, manual salting produces cleaner results.
The progression from “enable AQE” to “AQE has limits” to “salting when AQE fails.” Candidates who stop at step one haven’t debugged production skew. Candidates who jump straight to salting haven’t used AQE.
Q18
cache() is shorthand for persist(StorageLevel.MEMORY_AND_DISK). If the DataFrame fits in executor memory it’s stored there; if not, it spills to disk automatically. Use persist(StorageLevel.DISK_ONLY) explicitly when you know the DataFrame won’t fit in memory to avoid the overhead of attempting memory storage first.
Critical production rule: always call df.unpersist() when you’re done with a cached DataFrame. Fabric Spark sessions can persist for hours (TTL), and cached DataFrames that are no longer needed compete with shuffle buffers for executor memory — a common cause of OOM errors mid-pipeline that are difficult to trace.
Whether you know unpersist discipline — not just when to cache. Candidates who mention unpersist proactively have seen memory issues caused by forgotten caches in production.
Q19
Broadcast the small table when joining a large fact table with a small dimension (lookup) table. Spark sends the small table to every executor, eliminating a shuffle of the large table entirely. AQE can promote a Sort-Merge Join to Broadcast if the filtered small side is under autoBroadcastJoinThreshold — but that still occurs after the first shuffle already ran.
from pyspark.sql.functions import broadcast # Force broadcast of the dimension table — skips the shuffle entirely df_result = df_large.join( broadcast(df_small), on="customer_id" ) # Explicit hint is better than relying on AQE dynamic promotion: # AQE promotes AFTER the first shuffle already ran. The hint avoids it entirely.
Whether you know the difference between static broadcast hints and AQE dynamic broadcast promotion. The explicit hint is always faster when you know the small side is stable — AQE’s promotion is better only when you don’t know in advance.
Q20
Three levers: (1) Ensure the MERGE’s match condition references the partition column of the target table so Spark can skip non-matching partitions. (2) Run OPTIMIZE before the MERGE on large tables to compact files and reduce the number of Parquet files that need scanning. (3) Filter the source to only the rows that could match before the MERGE.
from delta.tables import DeltaTable # Run OPTIMIZE first if target hasn't been compacted recently spark.sql("OPTIMIZE sales_silver WHERE order_date >= '2026-01-01'") # Pre-filter source to only rows that can possibly match source_filtered = source_df.filter("order_date >= '2026-01-01'") DeltaTable.forName(spark, "sales_silver").\ alias("target").\ merge(source_filtered.alias("src"), "target.order_id = src.order_id AND target.order_date = src.order_date").\ whenMatchedUpdateAll().\ whenNotMatchedInsertAll().\ execute()
Whether you know the MERGE match condition must reference the partition key to enable partition pruning. A MERGE on an unpartitioned column against a 50B-row table does a full scan — a common performance disaster in production CDC pipelines.
Lakehouse Architecture
Delta Lake table design, the Direct Lake integration between Spark and Power BI, VACUUM, partitioning strategy, and schema evolution.
Q21
Managed (Tables folder): Spark controls both metadata and physical Parquet files. DROP TABLE deletes the data. Unmanaged (Files folder or external path): Spark controls only metadata. DROP TABLE removes the schema entry but physical files survive. Use unmanaged for Bronze — accidental DROP shouldn’t destroy raw data.
Data lifecycle governance — specifically whether you know the Bronze-layer rule: never use managed tables for raw ingestion because a pipeline failure that triggers a DROP + recreate would destroy the source-of-truth data.
Q22
Delta Lake is Fabric’s native format — all Lakehouse and Warehouse tables, Direct Lake, and Mirroring land in Delta Parquet. Iceberg is accessed via OneLake’s Iceberg REST Catalog endpoint or shortcuts, letting Snowflake, Trino, and Databricks query Fabric data without rewriting it. From Runtime 2.0, Delta Lake 4.1 adds Liquid Clustering and improved open-format interoperability.
The practical pattern in 2026: Fabric engineers write in Delta, and Iceberg compatibility is solved at the OneLake layer (Delta UniForm / REST Catalog) rather than converting tables. You don’t choose Delta or Iceberg in Fabric — you write Delta and expose it as Iceberg to external consumers via OneLake’s virtualization layer.
Whether you understand that the choice isn’t “Delta vs Iceberg” — it’s “Delta natively, Iceberg via OneLake interoperability layer.” Candidates who say “we’d use Iceberg for Snowflake compatibility” reveal they haven’t used OneLake’s REST Catalog yet.
Q23
Spark writes to a Delta table update the Delta transaction log. Direct Lake reads the log to determine which Parquet files to load. Excessive small files from Spark writes slow down “framing” (the process where Direct Lake loads file metadata). V-Order on Gold tables improves Direct Lake read performance after framing.
The 2026 addition: Direct Lake on OneLake (GA March 2026) bypasses the SQL Analytics Endpoint entirely and reads Delta files directly. Unlike the original Direct Lake variant (which can fall back to DirectQuery silently), Direct Lake on OneLake has no DirectQuery fallback — it fails explicitly if guardrails are exceeded. This changes the Spark engineer’s responsibility: if Gold table files aren’t well-compacted (OPTIMIZE + V-Order), Power BI reports fail outright rather than degrading silently.
Whether you know about the two Direct Lake variants and understand the data engineer’s responsibility for Gold table quality now that the no-fallback mode can cause explicit report failures. A strong answer connects Spark write patterns to Power BI runtime behavior.
Q24
VACUUM removes Parquet files that are no longer referenced by the Delta transaction log. After a DELETE, the row is logically removed but the physical Parquet file persists until VACUUM runs. For GDPR “right to erasure,” you must run DELETE followed by VACUUM to achieve physical deletion.
# Step 1: Logical delete spark.sql("DELETE FROM customers WHERE customer_id = '12345'") # Step 2: Physical deletion via VACUUM # Default retention = 7 days. Override for compliance (with care). spark.sql("VACUUM customers RETAIN 0 HOURS") # Warning: disabling retention check required if period < 7 days # spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")
Running VACUUM with a short retention period (0 hours) removes all historical Parquet files. You can no longer Time Travel to versions before the VACUUM point. For GDPR tables, this is acceptable — for audit tables, it’s catastrophic. Maintain separate retention policies per table type.
Whether you know GDPR deletion requires two steps and that the retention period on VACUUM matters. Candidates who say “just run DELETE” reveal they haven’t thought about physical data lifecycle.
Q25
Don’t partition tables under 10 GB. For larger tables: low cardinality (date at year/month level, not day for small tables), matches your most common filter pattern, avoids high-cardinality columns (UserID, GUID) which create millions of tiny directories. On Runtime 2.0, consider Liquid Clustering instead — it avoids the static partition problem entirely.
- Good partition columns: Year, YearMonth, Region, Product Category — bounded, relatively low-cardinality values that appear in WHERE clauses
- Bad partition columns: DateTime (creates millions of partitions), UserID, OrderID — unbounded high-cardinality values that fragment the table
- The alternative: On Runtime 2.0 (Delta 4.x), use
CLUSTER BY(Liquid Clustering) instead of static PARTITION BY. It adapts to query patterns over time and doesn’t require knowing your filter columns at table creation time.
Whether you can articulate the cardinality rule specifically and name Liquid Clustering as the modern alternative. The “don’t partition under 10 GB” rule is also a common trap — over-partitioning small tables is more harmful than not partitioning at all.
Q26
.option("mergeSchema", "true") allows Spark to add new columns automatically when writing. Schema enforcement (the default) rejects writes that don’t match the existing schema. Type changes that aren’t widening (e.g., int → long is OK; string → int is not) are rejected even with mergeSchema.
# ✓ Safe: add a new nullable column df_new.write.format("delta").option("mergeSchema", "true").\ mode("append").save(table_path) # ✓ Safe: widen type (int → long) spark.sql("ALTER TABLE orders CHANGE COLUMN amount amount BIGINT") # ✗ Unsafe: rename a column (old readers break; use overwriteSchema) # ✗ Unsafe: change string → int (Delta rejects — incompatible widening)
Whether you know the difference between additive schema changes (mergeSchema) and type narrowing (which Delta rejects). Candidates who say “just use mergeSchema for everything” haven’t hit a type incompatibility in production.
Troubleshooting
Where senior engineers are separated from everyone else. Use the STAR method for failure scenarios. The 2025-introduced JobInsight library changes how you diagnose Spark jobs programmatically.
Q27
Py4JJavaError is a wrapper — the real error is in the Java stack trace buried in the exception message. Common root causes: file path doesn’t exist (permission or shortcut broken), schema mismatch between DataFrame and Delta table, file locked by concurrent write, or a Java class version incompatibility after a library upgrade.
Diagnosis process: read the full .getMessage() output, not just the Python surface error. Search for “Caused by:” in the stack trace — that’s the actual exception. The most common production case: a Delta table is being written by one Spark session while another tries to read the same path with an incompatible schema, triggering a ProtocolChangedException wrapped in Py4JJavaError.
If the error says “Not a Delta table” or “Path not found” — it’s a shortcut/path issue. If it says “AnalysisException” — it’s a schema issue. If it says “ConcurrentWriteException” — it’s a locking issue. Read the Caused by: line first.
Whether you know to read the Java stack trace inside the Py4J wrapper rather than treating it as a single generic error. Candidates who say “Py4JJavaError means something went wrong in the JVM” without drilling into the root cause reveal they’ve only seen this error, not debugged it.
Q28
OOM errors are different depending on where they occur. Driver OOM: caused by pulling data to the driver (collect(), toPandas(), large broadcast tables). Fix: write to disk, don’t collect. Executor OOM: caused by large shuffle partitions, skewed joins, or forgotten cached DataFrames competing with shuffle buffers. Fix: raise shuffle partitions, fix skew, unpersist unused DataFrames.
| OOM Type | Symptom | Root Cause | Fix |
|---|---|---|---|
| Driver OOM | Error in driver log, job dies immediately | collect(), toPandas(), oversized broadcast | Write to Delta instead of collecting; sample before toPandas; check broadcast table size |
| Executor OOM | Error in executor log, tasks fail and retry | Shuffle partitions too large, skewed keys, uncached DataFrames filling memory | Raise shuffle.partitions, fix skew, df.unpersist() |
| Executor overhead OOM | “Container killed” message | High JVM object churn (UDFs, large JSON parsing) | Increase spark.executor.memoryOverhead; switch from Python UDFs to Spark SQL functions |
Fabric runs on a managed Spark infrastructure — it doesn’t use YARN. If you see memory kill messages in Fabric, the terminology in logs may differ from “Container killed by YARN” which is specific to YARN-based clusters. The underlying cause (executor overhead exceeded) is the same; fix with spark.executor.memoryOverhead.
Whether you can distinguish three types of OOM immediately — not just say “increase memory.” The “Fabric doesn’t use YARN” correction is important: candidates who cite YARN-specific error messages without this context are showing Databricks/Hadoop knowledge but not Fabric-specific knowledge.
Q29
Four causes: (1) Shortcut broken — source file deleted or renamed in ADLS/S3. (2) Permission error — the Notebook’s identity lacks read access to the path (delegated auth in Shortcuts). (3) Incorrect path in code — case-sensitive, check for typos. (4) Delta table hasn’t been registered in the Lakehouse catalog.
Diagnosis order: run notebookutils.fs.ls("path") to verify the path exists before any Spark operation. If ls fails, it’s a path or permission issue. If ls succeeds but Spark fails, it’s a catalog or Delta metadata issue — try spark.read.format("delta").load("path") with an explicit path instead of table name to isolate.
Whether you know notebookutils.fs.ls() as the first diagnostic step rather than immediately modifying Spark code. Using the right tool at the right layer separates experienced engineers from those who debug blindly.
Q30
Spill occurs when executor RAM fills during a shuffle — Spark writes temporary data to local disk. This is 10–100× slower than memory. A job that “works” but spills for 6 hours when it should finish in 20 minutes is a common production situation that looks like a “slow job” but is actually a configuration failure.
Spill appears in the Spark UI as “Spill (Memory)” and “Spill (Disk)” columns in the Stages tab. Any value above 0 warrants investigation. The fix: raise spark.sql.shuffle.partitions so each partition is smaller and fits in executor RAM, or switch to a Memory Optimized node type if the workload is inherently memory-intensive.
Whether you know where to look in the Spark UI for spill evidence (the Spill columns in Stages) and can explain why spill is a configuration failure, not just an expected slowdown.
Q31
JobInsight (GA August 2025) is a Java-based library built into Fabric Notebooks that returns Spark application execution data — queries, jobs, stages, tasks, executors — as Spark Datasets you can query with PySpark or SQL. Replaces manual Spark UI clicking with programmatic, scriptable analysis.
from jobinsight import JobInsight ji = JobInsight(spark) # Get all stages of the last Spark application stages = ji.get_stages() stages.filter("spill_memory_bytes > 0").show() # Get task-level detail for a specific stage tasks = ji.get_tasks(stage_id=12) tasks.orderBy("duration", ascending=False).show(20) # Export event log to OneLake for long-term storage or offline diagnostics ji.export_event_log("Files/spark-logs/")
Whether you know this exists — it’s a significant productivity improvement for production debugging that was added in August 2025. Candidates who still describe Spark diagnosis only through the Spark UI are missing a purpose-built tool that makes the process scriptable and reproducible.
Q32
Use the Fabric Capacity Metrics App (available from the Admin Portal). It shows CU consumption by workload type, workspace, and time, with 30-minute granularity. For Spark specifically, look for the “Spark Engine” item type to identify which jobs are consuming the most capacity.
Key patterns to watch: a sustained CU spike (not a burst) that continues for hours means a job is either stuck in a retry loop or genuinely larger than anticipated. A spike during business hours that competes with Power BI semantic model refreshes indicates a capacity sizing issue — move heavy batch jobs to off-peak hours or use Autoscale Billing to move them off the shared capacity pool.
Whether you can connect Spark behavior to capacity cost — not just “fix the Spark code” but understand when the right fix is “schedule differently” or “switch to Autoscale Billing.”
Scenarios & Architecture Decisions – Fabric Data Engineering Interview Questions
Production pattern design — streaming checkpoints, CI/CD, agentic automation, GDPR, unit testing, and the recurring Spark-vs-SQL-vs-Dataflow decision.
Q33
Specify a checkpoint location in the Files folder of a Lakehouse (not the Tables folder — managed table paths can be affected by schema changes). The checkpoint must survive job restarts; if it’s deleted, the stream starts from the beginning of the source.
query = (df_stream .writeStream .format("delta") .outputMode("append") .option("checkpointLocation", "Files/checkpoints/events_stream") .trigger(processingTime="5 minutes") .toTable("events_silver") )
2026 alternative for CDC streams: The Mirrored Database Change Feed connector (June 2026) provides a fully managed path from CDC sources to Real-Time Intelligence destinations — no custom Spark streaming notebook required. Evaluate this for new CDC pipelines before writing Structured Streaming code.
Whether you know the checkpoint must be in an unmanaged, persistent location — not a managed Delta table path. Mentioning the 2026 Change Feed connector alternative signals awareness of when to not write Spark code.
Q34
In Fabric 2026, agentic data engineering specifically refers to: Fabric Data Agents for NL2SQL query generation over Lakehouses and Warehouses (enhanced June 2026 with a new accuracy-focused runtime), the Data Engineering Agent that builds Dataform-style pipelines autonomously, and Fabric remote MCP (public preview) that lets external AI agents trigger Fabric operations programmatically.
The concrete engineering application: rather than writing a bespoke notebook to poll a Mirrored Change Feed for incremental updates, the June 2026 Change Feed connector handles this automatically. Rather than debugging SQL generation manually, Fabric Data Agents’ new NL2SQL engine can ask clarifying questions when intent is ambiguous, explore the data schema, and explain query steps — making the error surface more debuggable.
Whether you can describe specific 2026 capabilities rather than a vague “AI will automate pipelines” statement. Naming the NL2SQL engine update (June 2026), the Change Feed connector, and remote MCP demonstrates current platform knowledge.
Q35
Three components: (1) Git integration (Azure DevOps or GitHub) for version control, (2) Deployment Pipelines for workspace promotion (Dev → Test → Prod), (3) Fabric Notebook Public APIs (GA March 2026) for external orchestrators to trigger notebook runs programmatically in each stage.
Best practices: parameterize notebooks so the same code runs against Dev Lakehouse vs Prod Lakehouse based on a parameter, not hardcoded paths. Clear notebook cell output before committing to avoid secrets in Git history. Use the Fabric local MCP (GA) for GitHub Copilot to interact with Fabric resources from the terminal during development.
Whether you know about all three layers (Git, Deployment Pipelines, Public APIs) and specifically that the Public APIs enabling external CI/CD triggers reached GA in 2026. This is what makes Fabric Notebooks first-class citizens in a GitHub Actions or Azure Pipelines workflow.
Q36
DELETE removes a row from the Delta log (logical deletion). The physical Parquet file persists until VACUUM runs. For GDPR right-to-erasure, you must: (1) DELETE the row, (2) run VACUUM with a retention period that expires the historical version containing the deleted data. Only then is the data physically absent from storage.
Additional consideration for event data: if the same user’s PII appears in Bronze (raw), Silver (cleaned), and Gold (aggregated) tables, you must cascade the deletion through all three layers. A deletion in Gold that leaves PII in Bronze raw files is not GDPR-compliant. Build a “Right-to-Erasure” notebook that targets all three layers in the correct order: Gold first (to avoid breaking BI), then Silver, then Bronze.
Whether you understand the Bronze-Silver-Gold cascade requirement. Candidates who only discuss the DELETE + VACUUM pattern for a single table haven’t thought through the cross-layer data lineage implications.
Q37
Extract transformation logic into Python modules (not notebook cells). Test those modules locally in VS Code using pytest with a local SparkSession. Deploy tested modules to a Fabric Environment. The notebook becomes an orchestrator that calls tested functions — not the location of business logic.
In-notebook testing with pytest works but requires the full Spark session to be running, which means tests are slow and depend on Fabric infrastructure. Local testing with small synthetic DataFrames (a few hundred rows) is faster, more isolated, and compatible with CI/CD pipelines that don’t have Fabric access.
Whether you know the fundamental principle: logic should live in modules, notebooks should be thin orchestrators. Candidates who describe only in-notebook testing reveal they haven’t separated concerns in their codebases.
Q38
Spark for heavy ETL, complex transformations, Python-native ML, and unstructured data. T-SQL Warehouse for BI serving, enforcing row-level security, and schema-on-write governance. Dataflow Gen2 for simple transformations by non-engineers without Spark or SQL skills.
| Tool | Use when | Avoid when |
|---|---|---|
| Spark (Notebook / SJD) | TB-scale ETL, Python ML libraries, complex joins, streaming, unstructured data | Simple SQL aggregations, BI serving with RLS requirements, no-code scenarios |
| T-SQL Warehouse | Serving Gold layer to Power BI, enforcing column/row security, strict schema governance | Transforming raw Bronze data, ML workloads, streaming ingestion |
| Dataflow Gen2 | Simple ETL by citizen developers, Power Query-familiar users, low-volume transformations | TB-scale, Python-native requirements, real-time processing |
Whether you know Dataflow Gen2 as a third option and can explain why an architect would choose it (citizen developer accessibility) rather than treating everything as a Spark-vs-SQL binary.
Q39
Bronze: append-only (never overwrite raw). Silver: MERGE for deduplication and SCD. Gold: overwrite (rebuild from Silver on each run) for aggregated tables, or MERGE for incremental Gold updates. Each layer uses different Delta write modes for different reasons.
# Bronze — append only, unmanaged, never reprocess raw df_raw.write.format("delta").mode("append").save("Files/bronze/events") # Silver — MERGE for deduplication + schema enforcement DeltaTable.forName(spark, "events_silver").merge( df_cleaned.alias("src"), "t.event_id = src.event_id" ).whenMatchedUpdateAll().whenNotMatchedInsertAll().execute() # Gold — overwrite for aggregated business metrics (full rebuild) df_metrics.write.format("delta").mode("overwrite").saveAsTable("sales_metrics_gold")
Whether you know each layer uses a different write semantic for a principled reason, not just different syntax. The Bronze append-only rule and the Gold overwrite-vs-MERGE decision are the two most commonly misunderstood in production deployments.
Q40
Five-layer approach: (1) Native Execution Engine if not enabled (up to 6× speedup = 83% cost reduction, no code changes), (2) Autoscale Billing for batch jobs that don’t need 24/7 capacity, (3) Spark Advisor + JobInsight to find spill and skew, (4) V-Order and OPTIMIZE on Gold tables to reduce query time, (5) Move heavy batch jobs off-peak to avoid bursting into expensive throttle territory.
Priority order: Enable the Native Execution Engine first — it’s the highest-impact, lowest-effort change and requires zero code changes. Check if it’s already on by looking for the Spark Advisor fallback alerts in recent notebooks. If not enabled, turning it on is a single config change.
Next, audit which jobs run on base F-SKU capacity vs Autoscale Billing. Batch jobs that run infrequently are prime candidates for Autoscale — you pay only for the hours they run, not for the capacity to run them. For a job that runs 2 hours nightly, Autoscale Billing on most SKU sizes is cheaper than 24/7 F-SKU allocation within the first week.
Whether the Native Execution Engine is your first recommendation — most candidates jump to code-level optimizations without checking if the execution engine is enabled. Starting there signals you know the 2025-2026 platform capabilities, not just generic Spark tuning.
Architecture done. Data Engineering done. T-SQL, Warehouse design, and query optimization are next.
Start Warehouse Questions →
