Ultimate Info Guide · Databricks · Interview preparation
Databricks Interview Questions for Data Engineers: 60 Real-World Scenarios
Prepare for a Databricks data engineering interview by working through production decisions, not memorized definitions. This guide covers Spark and PySpark, Delta Lake, Auto Loader, Structured Streaming, Lakeflow, Unity Catalog, compute, troubleshooting and system design — with practical answer frameworks and code drills.
Start here
How to answer a Databricks scenario question like a working engineer
Advertisements
Many interview pages give a definition and stop. In a real technical conversation, a definition is only the starting point. When someone says “our pipeline is slow”, the interviewer wants to know how you find the bottleneck, what evidence you inspect, what you change and how you know the fix worked.
A useful opening sentence: “Before I change compute, I would confirm whether the delay is in source discovery, shuffle, stateful processing or the sink. I would compare the failing run with a healthy baseline and use the task metrics to identify the largest difference.” That answer shows an investigation plan rather than a guess.
A production Databricks pipeline — what the interviewer expects you to connect
The diagram is conceptual: orchestration and governance span all layers rather than happening only after Gold. A production design also includes compute selection, deployment, alerting and recovery.
How interview depth usually changes by level
| Level | What to demonstrate | What separates a strong answer |
|---|---|---|
| Junior / entry | Basic Spark, SQL, Delta tables, notebooks, medallion layers and simple ingestion. | Correct concepts, readable code, careful null and duplicate handling, and ability to explain the result. |
| Mid-level | Auto Loader, MERGE, Lakeflow Jobs, schema evolution, Unity Catalog, performance diagnosis. | Can make a design choice, discuss failure recovery, estimate trade-offs and show the validation plan. |
| Senior | Streaming recovery, CDC, concurrency, governance architecture, cost, CI/CD and platform design. | Can reason under incomplete requirements, make workloads safe to replay and tie technical metrics to business SLAs. |
Interview formats vary by employer and role. This is a preparation map, not a guaranteed sequence of interview rounds.
U.S. interview preparation
U.S.-based data engineering interview videos
For this U.S.-targeted guide, the video picks below come from U.S.-based creators and focus on interview patterns, coding exercises and big-tech preparation. They are intentionally broader than Databricks alone: U.S. data-engineering interviews commonly test SQL, Python, data modeling, Spark reasoning and system design alongside the platform named in the job description. Use these videos for interview technique, then use the Databricks-specific question bank on this page to practice Delta Lake, Auto Loader, Lakeflow, Unity Catalog and production operations.
Top 10+ Data Engineer Interview Questions and Answers
Jay Feng walks through common data-engineering interview question types and example solutions. Use this as a broad warm-up for SQL, data modeling, pipeline reasoning and how to explain a solution. Then apply the same structure to Databricks-specific scenarios in our question bank.
Cracking the FAANG Interview as a Data Engineer
Seattle Data Guy shares a U.S. big-tech interview-preparation perspective, including why preparation can span SQL, Hadoop or Spark fundamentals, system design and long interview loops. The video is older, so treat it as strategy—not a current promise of how Amazon, Meta or another employer runs interviews today.
Amazon-Style Data Engineer Coding Question: Remove Duplicate Products
A worked e-commerce data problem framed around a question reported at companies such as Amazon and Booking.com. It is useful for practicing requirement clarification, duplicate definitions, normalization trade-offs and follow-up questions. Pair it with our Delta MERGE and deduplication scenarios; the video itself is not a Databricks-specific exercise.
60 original questions
Databricks interview questions and answers
Search the question bank or filter by level. Each answer includes a short spoken response, the implementation reasoning, what the interviewer is testing and a common weak answer to avoid. Expand the questions most relevant to the role you are applying for.
60 questions shown. Open a question to read the complete answer.
INTERVIEW QUESTION BANK
Platform foundations
8 original questions. Open each answer for the reasoning, evidence, follow-up and trap to avoid.
Q01How would you explain the difference between Apache Spark and Databricks?Junior
Spark is the distributed processing engine; Databricks is the managed data and AI platform built around Spark and other services.
What a practitioner should explain
Separate the engine from the platform. Spark provides distributed DataFrame, SQL, streaming and execution primitives. Databricks adds workspace collaboration, managed compute, Delta Lake integration, Unity Catalog governance, Lakeflow orchestration and platform operations. The exact runtime and available features depend on cloud, workspace configuration and compute type.
One concrete split: Spark = execution engine, Databricks = governance, compute, orchestration around it.
Likely follow-up: Where would you investigate a slow Spark stage, and where would you inspect a failed Lakeflow Job?
Common weak answer: Do not say Databricks and Spark are interchangeable products.
Q02A team says its lakehouse is just a data lake with folders named Bronze, Silver and Gold. What is missing from that design?Mid
The layer names are only an organization pattern. A lakehouse also needs table reliability, governance, quality contracts, lineage, compute choices and operational controls.
What a practitioner should explain
Describe the data contract at each layer. Bronze preserves source fidelity and ingestion metadata. Silver applies typing, deduplication, standardization and quality rules. Gold publishes business-grain datasets and aggregates. Delta transactions, Unity Catalog permissions, pipeline dependencies, monitoring and replay strategy make the design operational.
Names the contract per layer (what is kept, rejected, replayed) rather than reciting Bronze/Silver/Gold.
Likely follow-up: Ask how a malformed record is retained, how a late correction is applied and who can read each layer.
Common weak answer: Do not treat medallion architecture as three mandatory folders or claim every workload needs exactly three layers.
Q03A platform architect asks you to draw the control plane and compute plane. What would you explain?Mid
The control plane provides workspace and orchestration services; the compute plane is where data processing runs. The exact network arrangement differs between classic and serverless compute.
What a practitioner should explain
Explain responsibility rather than overgeneralizing network diagrams. Workspace APIs, job definitions, permissions and orchestration are control-plane concerns. Spark execution and data access happen in the compute plane. For classic compute, customer cloud resources and network configuration matter. Serverless compute is managed differently, so use the cloud-specific architecture documentation before making private-network or data-path claims.
Talks about who owns what, and flags that serverless differs, instead of drawing one fixed diagram.
Likely follow-up: How does the answer change for serverless compute or a private network deployment?
Common weak answer: Do not claim all customer data always passes through one central Databricks server.
Q04When would you choose a Databricks SQL warehouse instead of a notebook job?Junior
Use a SQL warehouse for interactive SQL analytics, BI connectivity and SQL workloads. Use a Lakeflow Job for scheduled or event-driven orchestration of tasks.
What a practitioner should explain
A SQL warehouse is a compute endpoint optimized for SQL workloads and tools such as dashboards and BI clients. A job can coordinate notebooks, Python files, SQL, pipelines and dependent tasks. A notebook can still run interactive SQL, but that does not make it the best production scheduler.
Picks by workload shape: ad hoc SQL and BI versus multi-step scheduled work with dependencies.
Likely follow-up: What would you choose for a nightly multi-step ingest, transform, quality-check and publish workflow?
Common weak answer: Do not use notebooks as a substitute for a repeatable production orchestration design.
Q05What belongs in a Unity Catalog three-level name?Junior
A catalog, schema and object, for example `main.silver.orders`.
What a practitioner should explain
The hierarchy is `catalog.schema.object`. Catalogs are broad governance boundaries, schemas group related assets, and the object can be a table, view, volume, function or another supported securable. Unity Catalog also governs external locations, credentials, connections and other objects.
Writes `catalog.schema.object` correctly and knows catalogs are governance boundaries, not environments by default.
Likely follow-up: What permissions must a reader have to query a table in that hierarchy?
Common weak answer: Do not confuse a workspace name with a catalog or schema.
Q06A team wants to store raw JSON files and query curated records. Would you use a table, a volume or both?Mid
Use a volume for governed file storage and a table for structured, queryable records. They solve different problems.
What a practitioner should explain
A Unity Catalog volume is appropriate for non-tabular files such as raw JSON, CSV, images or model artifacts. A table is appropriate after parsing and structuring records for SQL or DataFrame use. A common design lands source files in a governed volume or external location, then writes Delta tables for Bronze, Silver and Gold.
Separates file storage (volume) from queryable records (table) and lands raw data before parsing.
Likely follow-up: How would you grant file access without granting broad access to the whole storage account?
Common weak answer: Do not call a volume a table or assume a table permission automatically grants arbitrary file-path access.
Q07An analyst has SELECT on a table but still receives a permission error. What do you check?Mid
Check the parent `USE CATALOG` and `USE SCHEMA` privileges as well as the table-level `SELECT` grant.
What a practitioner should explain
For a normal table read, the principal generally needs usage on the parent catalog, usage on the parent schema and SELECT on the table. Also confirm the principal is provisioned at the account level, group membership has propagated, the correct workspace is attached to the metastore, and the object name is correct.
Checks USE CATALOG and USE SCHEMA first, then group sync, then the exact error text. Never reaches for ALL PRIVILEGES.
Likely follow-up: How do you diagnose this without granting ALL PRIVILEGES?
Common weak answer: Do not solve access failures by granting broad admin or ALL PRIVILEGES access.
Q08The same notebook works on classic compute but fails on serverless. How do you approach it?Mid
Check serverless-specific API, language, library, networking and Spark configuration support before changing the business logic.
What a practitioner should explain
Serverless notebook and job compute uses Spark Connect APIs; RDD APIs are not supported, and Scala and R notebooks are not supported in serverless notebooks. The Spark UI and Spark logs are not available in the same way; use the query profile and available client logs. Check whether the code depends on DBFS mounts, unsupported Spark configurations, init scripts or compute-scoped libraries.
Reads the unsupported-feature list (RDD, mounts, init scripts) before touching business logic.
Likely follow-up: What would you use to inspect a slow SQL query on serverless?
Common weak answer: Do not assume every classic Spark API or cluster-level setting is supported on serverless.
INTERVIEW QUESTION BANK
Spark and PySpark
10 original questions. Open each answer for the reasoning, evidence, follow-up and trap to avoid.
Q09A DataFrame transformation runs without an error but no output is produced. Why?Junior
Spark transformations are lazy. They build a logical plan; an action or a write triggers execution.
What a practitioner should explain
Examples of transformations include `select`, `filter`, `withColumn` and `join`. Actions include `count`, `collect`, `show` and writes. Explain that a notebook cell can define a DataFrame without executing the full distributed computation. Avoid calling `collect` on large datasets just to force execution.
Can say which line triggered execution and why `collect()` is the wrong way to prove it.
Likely follow-up: Which action would you use to validate a small sample without pulling the full table to the driver?
Common weak answer: Do not describe every DataFrame method as an action.
Q10A join creates a stage with a large shuffle. What does that tell you?Mid
The join requires data movement between partitions. Inspect the plan, join strategy, input sizes, partition distribution and skew before tuning.
What a practitioner should explain
A shuffle commonly occurs for joins, aggregations and repartitioning. It writes and redistributes intermediate data. Check whether the join keys are appropriate, whether filters can be pushed earlier, whether one side is small enough to broadcast, whether partition counts are sensible and whether the data is skewed.
Reads the plan and stage metrics before proposing a fix, and knows some shuffles are unavoidable.
Likely follow-up: What evidence would you capture before and after changing the join?
Common weak answer: Do not assume every shuffle is avoidable; some data movement is required by the operation.
Q11One task in a 500-task stage runs for 35 minutes while the others finish in 2 minutes. How do you investigate?Senior
Treat it as a straggler or skew signal. Inspect task metrics and key distribution, then identify whether one or a few partitions contain disproportionate work.
What a practitioner should explain
Compare task input size, shuffle read and write, spill, executor time and records processed. Check for hot keys, uneven partitioning, a slow external source, garbage collection or a failing executor. If skew is confirmed, consider pre-aggregation, salting for suitable joins or grouping, AQE skew handling, or changing the data model.
Compares the slow task to its siblings (input size, spill, shuffle read) before blaming skew.
Likely follow-up: How would you prove the fix did not simply move the bottleneck elsewhere?
Common weak answer: Do not immediately increase every executor or partition count without checking the stage metrics.
Q12A large fact table joins to a small country lookup table. What join strategy might help?Mid
A broadcast join can avoid shuffling the large side when the lookup table is genuinely small enough to fit safely in executor memory.
What a practitioner should explain
Use a broadcast hint only after checking the size and plan. Example: `fact.join(F.broadcast(country_dim), "country_code", "left")`. Broadcast is not automatically safer for a dimension that has grown unexpectedly. Check the executed plan and memory pressure after the change.
Mentions the size threshold, checks the plan actually shows a broadcast, and knows when it fails (OOM).
Likely follow-up: When might a broadcast join make performance worse?
Common weak answer: Do not broadcast a table just because it is called a dimension table.
Q13What is Adaptive Query Execution, and should you turn it on manually?Senior
AQE re-optimizes eligible query plans during execution using runtime statistics. In Databricks it is enabled by default for supported workloads.
What a practitioner should explain
AQE can coalesce post-shuffle partitions, change certain sort-merge joins to broadcast joins, handle skewed shuffle partitions and propagate empty relations. Compare the initial and final executed plan. `EXPLAIN` shows a pre-execution plan and does not demonstrate runtime AQE changes. Tune settings only when evidence shows the defaults do not fit.
Knows AQE is on by default in current runtimes and what it changes at runtime: coalescing, skew, join switching.
Likely follow-up: Why might AQE not convert a join to a broadcast join even when one input looks small?
Common weak answer: Do not claim AQE fixes every slow query or that it automatically rewrites streaming queries.
Q14When would you use `repartition` and when would you use `coalesce`?Junior
Use repartition when you need to redistribute data and can accept a shuffle; use coalesce mainly to reduce partitions with less data movement.
What a practitioner should explain
`repartition(n)` can increase or decrease partitions and redistributes rows. `coalesce(n)` reduces partitions without a full shuffle in the common case, which can leave uneven partition sizes. For a write, choose based on output file size, upstream parallelism and data distribution—not a fixed rule.
Repartition = full shuffle to rebalance or increase. Coalesce = narrow, reduce only, may leave skew.
Likely follow-up: Why can coalesce to a very small number make a large write slow?
Common weak answer: Do not state that coalesce always balances data.
Q15A PySpark pipeline uses a Python UDF for a simple string cleanup. What would you review?Mid
Prefer built-in Spark SQL functions for standard transformations because they can be optimized and avoid Python serialization overhead.
What a practitioner should explain
Check whether the logic can be expressed with functions such as `trim`, `lower`, `regexp_replace`, `when`, `coalesce` or higher-order functions. A UDF may still be justified for logic unavailable in built-ins, but measure it and consider the supported UDF type and execution environment.
Tries built-in functions first, then pandas UDF, and can explain the serialization cost of a row-at-a-time UDF.
Likely follow-up: What is the trade-off between a Python UDF and a Pandas UDF?
Common weak answer: Do not say UDFs are always bad; explain the reason and measure the workload.
Q16A team caches every intermediate DataFrame because it believes caching always speeds up Spark. What do you recommend?Mid
Cache only when a dataset is reused enough to justify the memory and maintenance cost. Measure before and after.
What a practitioner should explain
Caching can help when an expensive result is reused multiple times. It can hurt by occupying executor memory, evicting useful data or adding materialization overhead. Also distinguish Databricks disk caching from DataFrame cache or persist. On serverless, some cache APIs differ; check current limitations.
Cache only what is reused and expensive; knows memory pressure and eviction can make it slower.
Likely follow-up: How would you decide whether an intermediate DataFrame is worth caching?
Common weak answer: Do not confuse local disk caching of files with DataFrame persistence.
Q17A query plan shows a sort-merge join, but the run becomes slow after data growth. What do you inspect?Senior
Inspect actual data sizes, table statistics, filters, join keys, skew, shuffle size and the final executed plan.
What a practitioner should explain
Use `EXPLAIN FORMATTED` to understand the planned operations, then inspect the Spark UI on classic compute or Query Profile where available. Check whether statistics are stale, whether filters are selective, whether a broadcast is feasible, and whether AQE changed the join or partition layout at runtime.
Looks at the data growth shape: new skew, broadcast no longer fitting, spilled sort, stale statistics.
Likely follow-up: What can `EXPLAIN` not tell you about AQE?
Common weak answer: Do not present the estimated plan as proof of what the runtime executed.
Q18A job writes thousands of tiny files to a Delta table every few minutes. How do you respond?Mid
Measure file counts and sizes, then address write frequency, partitioning, optimized writes, auto compaction, OPTIMIZE or predictive optimization where applicable.
What a practitioner should explain
Small files increase metadata and scheduling overhead. First ask why the job emits many files: too many partitions, tiny micro-batches, high-cardinality partition columns or frequent small writes. Avoid blindly repartitioning to one file. For Unity Catalog managed tables, review predictive optimization before scheduling duplicate maintenance jobs.
Fixes at the writer first (batching, optimized writes), then compaction. OPTIMIZE alone is a treadmill.
Likely follow-up: Would you partition the table by customer_id to fix this? Why or why not?
Common weak answer: Do not prescribe one file per table or a high-cardinality partition key.
INTERVIEW QUESTION BANK
Delta Lake and table reliability
11 original questions. Open each answer for the reasoning, evidence, follow-up and trap to avoid.
Q19What does Delta Lake add beyond Parquet files?Junior
Parquet stores columnar data; Delta Lake adds a transaction log and table protocol for reliable table operations such as ACID transactions, schema enforcement and versioned reads.
What a practitioner should explain
Explain that a Delta table typically consists of data files plus transaction-log metadata. The log determines which files make up each committed table version. Features such as MERGE, time travel and concurrent writes depend on the protocol and enabled table features, not on Parquet alone.
Names the transaction log as the thing Parquet lacks, and ties it to concrete features.
Likely follow-up: Why should a reader not list the storage folder and treat every Parquet file as current table data?
Common weak answer: Do not claim the transaction log is just a Hive metastore catalog.
Q20Two users query a Delta table while a write is in progress. Can a reader see half of the commit?Mid
A Delta reader uses a consistent table version from the transaction log; an uncommitted write is not part of that version.
What a practitioner should explain
The writer prepares data files and commits table actions through the transaction log. Readers select a consistent snapshot. Explain the difference between files physically present in storage and files active in the committed table version. Keep cloud storage and table protocol details accurate for the deployment.
Explains snapshot isolation via the log: files on storage are not the same as files in the table version.
Likely follow-up: How are old versions represented and what can eventually remove their underlying files?
Common weak answer: Do not recommend manually deleting data files from the table directory.
Q21A MERGE fails because multiple source rows match one target row. What is the correct fix?Mid
Deduplicate or aggregate the source to one deterministic row per target key before MERGE.
What a practitioner should explain
Use a window such as `ROW_NUMBER()` ordered by a trustworthy source sequence or update timestamp, then filter to rank 1. The business rule must define which event wins. Databricks Runtime 16.0 and later evaluates `WHEN MATCHED` conditions when detecting duplicate matches; 15.4 LTS and below considers only the ON clause. Do not rely on runtime-specific duplicate handling instead of clean source data.
Gets the source to one deterministic row per key. Retrying or `SELECT DISTINCT` is the weak answer.
Likely follow-up: What if two source events have the same timestamp?
Common weak answer: Do not arbitrarily drop duplicates without a deterministic tie-breaker.
Q22A Delta MERGE succeeds on the first run but a retry appears to double-count records. How do you design the write to be idempotent?Senior
Define a stable business key and source ordering, deduplicate the incoming batch and use a deterministic MERGE or replace-by-partition strategy.
What a practitioner should explain
A retry must produce the same target state as the first successful execution. Store a batch ID or source sequence when useful, avoid append-only writes for mutable entities, and validate the sink semantics. For append workloads, deduplicate with durable event identifiers or use a transactional pattern that records processed batches.
Defines what happens on replay: same input, same final state, with a stable key and ordering column.
Likely follow-up: How do you handle a late-arriving update that is older than the current target record?
Common weak answer: Do not equate job retries with exactly-once business outcomes.
Q23Two concurrent jobs update overlapping data and one receives a concurrency conflict. What do you do?Senior
Treat it as a transaction conflict. Identify overlapping read and write sets, table features and operation patterns, then reduce unnecessary overlap or serialize conflicting work.
What a practitioner should explain
Delta uses optimistic concurrency control. Conflict behaviour depends on the operations, table configuration, isolation and supported concurrency features. Check the exact exception and operation history. Consider disjoint write predicates, better batch boundaries, row-level concurrency where supported, or a controlled retry after re-reading current state.
Reads the actual exception and operation history, then narrows predicates or serializes writers.
Likely follow-up: Why is blindly retrying every transaction error unsafe?
Common weak answer: Do not assume all Delta tables have identical concurrency behaviour or that every conflict is transient.
Q24What is the difference between time travel and RESTORE?Mid
Time travel reads an earlier table version; RESTORE creates a new current version whose contents correspond to an earlier version.
What a practitioner should explain
Time travel is for querying historical state, for example `VERSION AS OF`. RESTORE is an operational rollback action recorded as a new commit. Both depend on the required log and data files still being available. A historical version can become unreadable after retention cleanup removes needed files.
Time travel reads, RESTORE writes a new commit. Both fail once files are vacuumed.
Likely follow-up: Would RESTORE undo an external side effect performed by a downstream API?
Common weak answer: Do not say time travel itself rolls back a table.
Q25A developer proposes setting VACUUM retention to zero hours to reclaim storage immediately. What do you say?Senior
Do not do that as a routine cleanup. Confirm retention safety, concurrent readers, streams and time-travel requirements before vacuuming.
What a practitioner should explain
VACUUM removes unreferenced data files older than the configured retention threshold. The commonly documented default deleted-file retention is seven days, and Databricks has a safety check against unsafe retention. Lowering retention can break readers or streams that still need files and can remove historical data needed for time travel.
Refuses the zero-hour idea with a reason: running streams, long readers and time travel all depend on retention.
Likely follow-up: What is the difference between OPTIMIZE and VACUUM?
Common weak answer: Do not describe VACUUM as table compaction.
Q26A source adds a new column. Should you enable schema evolution for every table?Mid
No. Choose a schema contract deliberately. Permit additive evolution only where the downstream contract can accept it; otherwise quarantine or reject unexpected fields.
What a practitioner should explain
Schema enforcement protects the target schema. Schema evolution allows approved changes to be incorporated, such as new columns, using supported write options or table settings. Auto Loader has separate schema-evolution behaviour from Delta write schema evolution. Keep source schema state and target table schema state distinct.
Separates source schema from target contract and has a defined path for unexpected columns.
Likely follow-up: How would you alert on a type change rather than a new nullable column?
Common weak answer: Do not switch on `mergeSchema` globally without reviewing downstream consumers.
Q27A new table has growing query patterns and a high-cardinality customer filter. How would you choose its layout?Senior
Evaluate liquid clustering first for new Databricks Delta tables, then measure actual query patterns and clustering results.
What a practitioner should explain
Databricks recommends liquid clustering over traditional partitioning and Z-ORDER for new tables in its current guidance. Clustering keys can evolve without rewriting all data at once. It is not compatible with partitioning or Z-ORDER on the same table. Workload filters, column statistics and table size still matter; do not cluster on every column.
Starts from query filters and table size. Mentions liquid clustering and its limits, not cargo-cult Z-ORDER.
Likely follow-up: How would you migrate an existing partitioned or Z-ORDER table?
Common weak answer: Do not recommend Z-ORDER as the universal default for new tables.
Q28A downstream system needs to process only inserts, updates and deletes since the last run. What Delta feature might help?Senior
Use Change Data Feed when it is enabled and supported for the table, and design downstream consumption around its retention and version semantics.
What a practitioner should explain
CDF exposes row-level changes between table versions, including change type and commit metadata. Confirm it is enabled for the relevant table and that the downstream process checkpoints the last processed version. Do not confuse CDF with reading the full latest snapshot.
Mentions enabling CDF ahead of time, checkpointing the last version, and its retention limits.
Likely follow-up: What happens if the downstream consumer falls behind beyond the retained history?
Common weak answer: Do not assume CDF is enabled automatically on every table or retained forever.
Q59A team runs VACUUM and later cannot time travel to a version from last week. What happened?Mid
VACUUM can remove unreferenced data files older than the retention threshold. Time travel needs both usable transaction-log history and the data files referenced by the selected version.
What a practitioner should explain
The documented default VACUUM retention threshold is seven days, and Databricks strongly recommends retaining at least seven days. Do not confuse deleted-file retention with transaction-log retention. Before reducing the threshold, account for long-running jobs and streams that may still need files; a short threshold can delete uncommitted files. VACUUM is not a backup.
Connects VACUUM retention to the time-travel window and says how to set retention deliberately.
Likely follow-up: How do transaction-log retention and deleted-file retention differ?
Common weak answer: Do not promise that every version shown in history remains queryable after its data files have been vacuumed.
INTERVIEW QUESTION BANK
Ingestion and streaming
10 original questions. Open each answer for the reasoning, evidence, follow-up and trap to avoid.
Q29When would you choose Auto Loader over COPY INTO?Junior
Auto Loader is suited to incremental, scalable file ingestion with schema inference and evolution. COPY INTO is a SQL-oriented, idempotent file-loading command for simpler file-ingestion patterns.
What a practitioner should explain
Choose based on source characteristics, arrival pattern, state, schema management and orchestration. Auto Loader tracks discovered files and uses streaming checkpoints and schema state. COPY INTO can be run repeatedly and incrementally loads new files from a location. Neither choice removes the need to handle bad records, permissions, source retention and replay.
Chooses on scale, schema handling and discovery mechanism, not on which is newer.
Likely follow-up: What would you use for a continuously arriving folder with changing JSON schema?
Common weak answer: Do not say Auto Loader is always superior or that COPY INTO is only a one-time load.
Q30Auto Loader discovers a new column and the stream stops. Is that necessarily a data-loss incident?Mid
Not necessarily. With the default add-new-columns evolution mode, Auto Loader can update schema state and raise an UnknownFieldException so the stream can restart with the evolved schema.
What a practitioner should explain
Explain the selected `cloudFiles.schemaEvolutionMode`. With `addNewColumns`, new columns are appended to the inferred schema and the stream stops; configure Lakeflow Jobs restart behaviour so it can restart. `rescue` keeps unexpected fields in the rescued-data column without evolving the table schema. Use a strict mode when a schema contract must fail.
Treats it as expected schema-evolution behaviour, checks rescued data, then restarts under a defined policy.
Likely follow-up: What is the role of `cloudFiles.schemaLocation` and how is it different from a checkpoint?
Common weak answer: Do not treat schema evolution as a silent, zero-downtime change in every mode.
Q31A streaming query restarts from the beginning after a deployment. What do you investigate?Mid
Check whether its checkpoint path changed, was deleted, became inaccessible or was reused by another query.
What a practitioner should explain
A checkpoint stores source offsets, committed micro-batches, state and query metadata. Each streaming query needs its own durable checkpoint location. Deleting it or changing to a new location starts a fresh query identity and may replay data depending on source and sink.
Suspects checkpoint location first (changed path, wiped, per-environment config) before blaming Spark.
Likely follow-up: How would you safely change stateful streaming logic when the state schema changes?
Common weak answer: Do not delete a checkpoint as a general troubleshooting shortcut.
Q32Can you promise end-to-end exactly-once processing just because Structured Streaming uses checkpoints?Senior
No. Checkpoints support recovery and track offsets and commits, but end-to-end guarantees depend on the source, sink and how writes are implemented.
What a practitioner should explain
Explain source replay, checkpoint recovery and sink transactionality separately. A transactional Delta sink can offer strong guarantees under supported conditions. For external sinks, design idempotent writes or deduplicate by a stable event key. Be explicit about what “exactly once” means for this application.
Separates engine guarantees from sink semantics. Exactly-once needs a replayable source and an idempotent or transactional sink.
Likely follow-up: What failure can still create duplicate business effects after a successful external API call but before the checkpoint commit?
Common weak answer: Do not promise universal exactly-once semantics across arbitrary external systems.
Q33A stream receives events up to two hours late. What does a watermark do?Mid
A watermark bounds how long the engine retains state for certain stateful operations and defines the late-data threshold the query is designed to handle.
What a practitioner should explain
Use event time, not arrival time, for event-time windows. Choose a delay threshold based on observed lateness and the business SLA. Records within the threshold are processed; records beyond it are not guaranteed to be processed for stateful operations. Longer watermarks retain more state and can increase memory and latency.
Explains the watermark as a state-cleanup and lateness bound, not a filter on arrival time.
Likely follow-up: How would you prove two hours is an acceptable threshold?
Common weak answer: Do not claim a watermark guarantees processing of every late event forever.
Q34A consumer needs current events within seconds, but a daily analytics pipeline can tolerate minutes. Would you use the same streaming mode?Senior
Not automatically. Match the processing mode to the latency SLA and operational complexity.
What a practitioner should explain
For analytical ETL, micro-batch Structured Streaming or triggered pipeline processing may be sufficient. Databricks also provides real-time mode for specific low-latency use cases with source, sink, runtime and operator requirements. Validate those requirements before promising sub-second latency.
Matches trigger and latency to each consumer and splits workloads when SLAs differ by orders of magnitude.
Likely follow-up: What is the operational cost of a much tighter trigger?
Common weak answer: Do not call all Structured Streaming “real time”.
Q35The same file is delivered twice by an upstream system. How do you prevent double counting?Mid
Use durable source tracking and stable record identifiers, then make downstream writes idempotent.
What a practitioner should explain
Auto Loader tracks files using checkpoint state, but business-level duplicates can still exist across different filenames or upstream replays. Deduplicate by a source event ID or natural key and sequence, retain ingestion metadata, and use MERGE or another deterministic sink pattern.
Uses a durable file or event identifier and decides where dedup lives: ingest, Silver, or MERGE.
Likely follow-up: What if the file is renamed but contains the same business records?
Common weak answer: Do not assume file-level discovery automatically removes semantic duplicates.
Q36The streaming backlog grows during peak traffic. What metrics do you inspect first?Senior
Compare input rate, processed rate, batch duration, trigger interval, state size, source lag and sink throughput.
What a practitioner should explain
Break the pipeline into source, transformation, stateful operations and sink. Check whether batch processing time regularly exceeds the interval, whether one partition is skewed, whether state grows because a watermark is absent or too long, and whether the destination throttles. Scale only after identifying the constrained stage.
Looks at input rate versus processing rate, batch duration, state size and source partitions, in that order.
Likely follow-up: How do you distinguish source lag from slow transformation?
Common weak answer: Do not solve a slow sink by increasing source ingestion parallelism.
Q37A serverless streaming job fails with an unsupported trigger. What do you check?Senior
Check the serverless trigger support matrix. For serverless notebooks and jobs, AvailableNow is recommended; continuous and processing-time triggers are not supported there.
What a practitioner should explain
Serverless compute has different Structured Streaming constraints from classic compute. For continuous streaming workloads, consider Lakeflow pipelines in continuous mode or the supported architecture for the required latency. Do not confuse Lakeflow Job continuous scheduling with a Spark `processingTime` trigger.
Checks the serverless streaming trigger documentation instead of guessing, and offers an alternative.
Likely follow-up: How does a Lakeflow pipeline processing mode differ from a Jobs scheduling mode?
Common weak answer: Do not mix the word “continuous” across the Jobs UI and Structured Streaming APIs.
Q60A streaming pipeline writes to an external API and creates duplicates after a restart. Does the checkpoint guarantee exactly-once side effects?Senior
No. A checkpoint tracks query progress and state, but end-to-end exactly-once behaviour also depends on a replayable source and an idempotent or transactional sink.
What a practitioner should explain
A failed micro-batch can be replayed after restart. If the external API accepted the request but the query did not record the batch as committed, the side effect can happen again. Use a deterministic idempotency key, an upsert by event identifier, or a sink with an appropriate transaction protocol. Do not assume a foreachBatch callback becomes transactional because the query has a checkpoint.
Checkpoints do not undo an HTTP call. Needs idempotency keys or an outbox pattern at the sink.
Likely follow-up: How would you choose an idempotency key when the same business event can arrive more than once?
Common weak answer: Do not claim that a checkpoint alone makes arbitrary external API calls exactly once.
INTERVIEW QUESTION BANK
Lakeflow, orchestration and delivery
8 original questions. Open each answer for the reasoning, evidence, follow-up and trap to avoid.
Q38What is the difference between Lakeflow Jobs and Lakeflow pipelines?Junior
Jobs orchestrate tasks and dependencies; Lakeflow pipelines provide a declarative framework for building and operating data transformations.
What a practitioner should explain
A job is a DAG that can combine notebooks, Python, SQL, pipeline tasks and other work. Lakeflow pipelines define data flows and dependencies and add pipeline capabilities such as expectations, AUTO CDC and an event log. A job can orchestrate a pipeline, so they are complementary rather than competitors.
Jobs orchestrate tasks; pipelines declare datasets and let the engine manage dependencies and updates.
Likely follow-up: Would you use a pipeline to coordinate a third-party API call and a file export?
Common weak answer: Do not treat a job and a pipeline as two names for the same feature.
Q39A multi-task job has three independent source ingestions and one consolidation task. How should the graph run?Mid
Run the independent ingestions in parallel and make consolidation depend on all required upstream tasks.
What a practitioner should explain
Lakeflow Jobs uses a DAG. Upstream tasks run before downstream tasks, and independent tasks can run in parallel. Use If/else or For each tasks only when the business control flow needs them. Keep task boundaries observable and avoid splitting a single tightly coupled transformation into excessive jobs.
Fan-out tasks in parallel, one consolidation task depending on all three, with sensible retry and failure policy.
Likely follow-up: How would you add a notification or cleanup step that must run even when an upstream task fails?
Common weak answer: Do not add unnecessary sequential dependencies that remove safe parallelism.
Q40A failed job can be repaired. Why is that not a substitute for idempotent code?Senior
A repair reruns unsuccessful tasks from the beginning; if a task partially wrote data before failing, the repaired run can repeat those side effects.
What a practitioner should explain
Use deterministic MERGE, overwrite of a well-defined partition, transaction keys or an external idempotency token. Inspect which tasks and downstream dependencies will rerun before pressing Repair. Successful upstream tasks are generally retained in the repair flow, but failed tasks rerun with current settings.
Repair reruns tasks, it does not make them safe to rerun. Idempotency is a code property.
Likely follow-up: What records could be duplicated if the task appended before it timed out?
Common weak answer: Do not say Lakeflow Jobs automatically makes task side effects idempotent.
Q41When should a job use a schedule, file-arrival trigger or table-update trigger?Mid
Use a schedule for time-based SLAs, file arrival when a landing location receives new files, and table update when upstream table changes should trigger downstream work.
What a practitioner should explain
Pick the trigger that reflects the actual dependency. File-arrival triggers watch supported locations and have service-specific configuration. Table update triggers are data-driven. Avoid repeated empty scans when no new data is expected, but consider trigger latency, quotas and operational behaviour.
Picks the trigger from what actually signals readiness: clock, file landing or upstream table change.
Likely follow-up: How would you prevent two upstream events from causing overlapping downstream work?
Common weak answer: Do not choose a trigger solely because it is the newest option.
Q42A pipeline should quarantine invalid records but continue processing valid ones. How would you implement data quality?Mid
Define explicit expectations and decide per rule whether to retain, drop or fail the update. Keep rejected records observable.
What a practitioner should explain
Use pipeline expectations for supported Lakeflow pipelines, with rule behaviour chosen to match severity. A non-null key may justify failing an update; a correctable optional attribute may be quarantined or flagged. Track rejected counts and make the rule part of the data contract.
Uses expectations or quarantine tables with an explicit policy (warn, drop, fail) per rule, and counts what was dropped.
Likely follow-up: How will you detect a sudden rise in quarantined rows?
Common weak answer: Do not silently drop records without a metric or audit trail.
Q43A team manually exports notebooks and job settings to promote dev to production. What would you recommend?Mid
Use source control with Declarative Automation Bundles to define and deploy workspace resources consistently across targets.
What a practitioner should explain
Bundles (formerly Databricks Asset Bundles) describe source files, resources, target-specific configuration and tests. Validate the bundle, then deploy it to a named target. Keep secrets outside source files and use an appropriate service principal or workload identity for automation.
Moves to version-controlled config and automated deployment, not manual export/import.
Likely follow-up: What does `databricks bundle validate` prove and what does it not prove?
Common weak answer: Do not treat a successful configuration validation as proof that production data processing will succeed.
Q44A developer says Databricks Repos disappeared from a deployment process. What is the current term?Mid
The current UI terminology is Git folders, formerly Databricks Repos.
What a practitioner should explain
Git folders support source-control workflows such as branch switching, committing, pushing and opening pull requests. Declarative Automation Bundles address deployment of project resources. The CLI command family still uses `databricks bundle`.
Knows the current name is Git folders and that older docs and teammates may still say Repos.
Likely follow-up: Which tool would you use for day-to-day branch work and which for promoting a job definition?
Common weak answer: Do not mix Git collaboration with bundle deployment.
Q45How would you separate dev, test and prod without maintaining three copies of a notebook?Senior
Keep one source-controlled project and use bundle targets, variables and environment-specific configuration.
What a practitioner should explain
Use target overrides for workspace host, catalog, schema, service principal and resource configuration. Promote the same code through environments after validation and tests. Ensure production writes use production identities and storage locations; do not hard-code an environment into transformation code.
Same code across environments with parameterized catalog or schema, deployed by pipeline.
Likely follow-up: How do you stop a development job from accidentally writing into the production catalog?
Common weak answer: Do not solve environment separation by manually cloning workspaces and editing notebooks.
INTERVIEW QUESTION BANK
Governance, compute and operations
7 original questions. Open each answer for the reasoning, evidence, follow-up and trap to avoid.
Q46A developer asks for ALL PRIVILEGES because SELECT does not work. What is your least-privilege response?Mid
Grant the required usage privileges on catalog and schema plus SELECT on the table, preferably through a group.
What a practitioner should explain
Use account-level identity provisioning and groups aligned to the identity provider. Grant `USE CATALOG`, `USE SCHEMA` and the specific object privilege. Reserve MANAGE, ownership and broad administrative roles for a small set of responsible groups. Verify effective grants with SHOW GRANTS or Catalog Explorer.
Asks what the principal needs to do, grants the minimum on the parent objects and the table, nothing wider.
Likely follow-up: What is the difference between BROWSE and SELECT?
Common weak answer: Do not use ALL PRIVILEGES as a debugging shortcut.
Q47A production job is owned by an engineer who is leaving the company. What should change?Senior
Run production automation with a service principal or workload identity, not a personal user identity.
What a practitioner should explain
Provision a non-human principal through the account identity model, assign only the required workspace and data privileges, manage credentials securely, and assign job/resource ownership to an operational group. Document rotation and break-glass procedures.
Moves ownership to a service principal or group and checks run-as, secrets and notifications.
Likely follow-up: What breaks if the only owner is deactivated?
Common weak answer: Do not embed a personal PAT or password in a notebook.
Q48How should notebooks retrieve credentials for an external database?Mid
Use the platform secret-management integration or a managed identity/service credential pattern supported by the source and compute.
What a practitioner should explain
Do not store passwords or tokens in notebook source, Git, job parameters or logs. Restrict secret-read access, use least privilege and rotate credentials. Where supported, prefer service credentials or workload identity federation rather than manually managed long-lived secrets.
Uses a secret scope or equivalent, never hard-coded values, and limits who can read the secret.
Likely follow-up: How would you avoid leaking a secret through `display`, exception text or debug logging?
Common weak answer: Do not assume a secret is safe merely because it is stored in a secret scope.
Q49When would you choose serverless instead of classic compute?Mid
Prefer serverless for supported workloads when reduced infrastructure management is valuable; choose classic when you need capabilities or controls that serverless does not support.
What a practitioner should explain
Compare language and API support, networking, libraries, init scripts, Spark configuration, observability and workload duration. Serverless notebooks and jobs use Spark Connect, do not support RDD APIs and do not expose the classic Spark UI. Check the current limitations for the selected cloud and workload.
Weighs startup time, ops burden and cost model against control, library and network needs.
Likely follow-up: What is the best way to troubleshoot a slow serverless SQL query?
Common weak answer: Do not assume serverless is always cheaper or compatible with every Spark application.
Q50The monthly Databricks bill rises 40% while volume rises only 5%. What is your investigation plan?Senior
Attribute spend by workspace, job, warehouse, SKU, owner and time period before changing compute.
What a practitioner should explain
Use system billing tables where available, job and warehouse usage metrics, run duration, concurrency, retries, idle time and query history. Separate workload growth from inefficient scans, repeated full refreshes, skew, oversized compute, unnecessary refreshes and runaway development sessions. Compare cost per unit of business output, not only total DBUs.
Breaks cost down by workload, tag and SKU before touching a single cluster setting.
Likely follow-up: What would you change if a high-cost warehouse has low query utilization?
Common weak answer: Do not respond by reducing all compute sizes without measuring workload performance and SLA impact.
Q51How do you explain managed versus external tables to a platform team?Mid
Unity Catalog manages the lifecycle of managed-table data; external tables keep the data files at a location the customer manages.
What a practitioner should explain
Managed tables are the default and recommended choice for most Databricks use cases because the platform manages file lifecycle and can apply managed-table optimizations. External tables are useful when existing files must remain at a customer-controlled location or be accessed by other systems. Dropping the metadata for an external table does not delete the external files.
Explains who owns the files and what DROP does to them, then ties it to governance.
Likely follow-up: Would you choose an external table simply to avoid vendor lock-in?
Common weak answer: Do not claim managed tables mean Databricks owns the customer's cloud data.
Q52A dashboard user sees a row-filtered result, but a notebook user sees more rows. What do you investigate?Senior
Trace the principal, effective permissions, query path, row-filter or mask policy, and whether both tools are using the same governed object.
What a practitioner should explain
Check whether the filter or mask is attached to the table or view being queried, whether both paths resolve to the same catalog object, and whether the users map to different groups or service principals. Test with the actual principal, not only an administrator. Review audit and lineage evidence.
Compares the compute type and identity in each path, since policy enforcement can differ.
Likely follow-up: How would you prove the policy is applied consistently to a new BI connection?
Common weak answer: Do not assume that two tools using the same storage path are using the same governed securable.
INTERVIEW QUESTION BANK
Production design and communication
6 original questions. Open each answer for the reasoning, evidence, follow-up and trap to avoid.
Q53Design a pipeline for daily order files, with late files, duplicate orders and a changing source schema. What is your approach?Senior
Land files immutably, ingest incrementally, preserve source metadata, validate schema, deduplicate by business key and sequence, and publish curated Delta tables through governed jobs or pipelines.
What a practitioner should explain
A solid design separates file discovery from business deduplication. Auto Loader can incrementally process arriving files with schema and checkpoint state. Bronze retains source payload and ingestion metadata. Silver casts and validates fields, rescues or quarantines unexpected schema, then keeps the deterministic latest record per order. Gold publishes business aggregates. Orchestration must restart safely and alert on rejected data or backlog.
Walks ingest, dedup, late files, schema contract, replay and monitoring in a sentence each.
Likely follow-up: What would you do if the source changes `order_total` from decimal to a string?
Common weak answer: Do not silently cast malformed values to zero or drop the source file.
Q54A source sends CDC updates out of order. How do you maintain a correct SCD Type 2 customer dimension?Senior
Use a stable business key and a source sequence or event ordering field, and use a CDC-capable pipeline pattern where supported.
What a practitioner should explain
The design must retain effective dates, current-row status and prior versions. Late updates must be ordered by the source sequence, not simply file arrival time. Databricks Lakeflow pipelines support AUTO CDC patterns for SCD Type 1 and Type 2. Validate initial snapshots, deletes, duplicate sequence values and backfills before using the feature in production.
Orders by source sequence, not arrival, and handles late events by re-closing intervals.
Likely follow-up: What if two changes for one customer share the same sequence value?
Common weak answer: Do not use ingestion timestamp as a substitute for authoritative change ordering when the source provides a sequence.
Q55A backfill must reprocess six months of data while daily loads continue. How do you keep the result safe?Senior
Define non-overlapping ownership or a controlled write strategy, use deterministic batch boundaries and idempotent writes, and validate before switching consumers.
What a practitioner should explain
Options include isolated backfill tables followed by controlled promotion, partition-scoped replacement for immutable date slices, or ordered MERGE for mutable data. Coordinate with live writes, monitor concurrency conflicts, and keep a record of source snapshots and processing code version. Do not reuse a streaming checkpoint for an unrelated query.
Isolates backfill output, uses deterministic overwrites by range, and validates before swap.
Likely follow-up: How do you prevent the backfill from overwriting newer live records?
Common weak answer: Do not run an unrestricted overwrite against the entire production table.
Q56A pipeline run is green, but the business reports revenue is 8% too high. What is your incident response?Senior
Check data correctness independently of job status: grain, join cardinality, duplicate keys, late/replayed events, currency rules and aggregation logic.
What a practitioner should explain
Start with a reproducible affected partition and trace source-to-target row counts and monetary totals. Inspect duplicate source keys before MERGE, many-to-many joins, replay after retries, timestamp cutoffs, soft deletes and null/default handling. Quarantine the affected output, communicate impact and correct the data through a controlled idempotent backfill.
Stops the bleeding, scopes the wrong number, finds the join or dedup fault, then adds a reconciliation check.
Likely follow-up: What metrics would you add to prevent recurrence?
Common weak answer: Do not assume a successful task status is proof of business correctness.
Q57A data platform must process 2 TB per day with a 30-minute freshness SLA and sensitive customer attributes. What design decisions matter?Senior
Start with the SLA, source characteristics, data contracts, security classification, concurrency and recovery objectives; then design ingestion, compute and governance around those constraints.
What a practitioner should explain
Estimate peak throughput and arrival patterns, separate file ingestion and stateful streaming needs, define Bronze/Silver/Gold contracts, choose a supported compute mode, use Unity Catalog groups and column controls for sensitive fields, and implement service-principal job execution. Measure p95 freshness, backlog, failure rate, DBU per TB, data quality and recovery time.
Quantifies the SLA, partitions the problem, names the masking and access design, and states the cost trade-off.
Likely follow-up: Which requirements would you clarify before choosing a cluster size?
Common weak answer: Do not size compute from daily volume alone; peak rate, transformations, joins, state and SLA matter.
Q58Tell me about a production incident without giving a vague “we optimized Spark” answer.Mid
Use a structured incident story: business impact, baseline, evidence, root cause, fix, measured result and prevention.
What a practitioner should explain
Give the interviewer a concrete workload, data volume, start-to-finish duration, failure or freshness symptom, and one or two metrics. Explain the alternatives considered and why the chosen fix worked. Separate your own contribution from the team's work and mention what you would change next time.
Tells a real incident: symptom, evidence, root cause, change, measured result.
Likely follow-up: What metric would demonstrate that the improvement mattered to the business?
Common weak answer: Do not invent percentages or claim an improvement without a measurement method.
Hands-on practice
Four SQL and PySpark coding drills
Do these without looking at the solution first. In an interview, state the expected grain, duplicates, null behaviour and data-volume assumptions before typing. The examples use Unity Catalog three-part names; replace catalog and schema names for your workspace.
Drill 1 — Keep the latest source change per order before MERGE
Scenario: a source batch contains multiple changes for the same order. The target must receive only the most recent deterministic event.
WITH ranked AS (
SELECT
order_id,
order_status,
source_sequence,
source_updated_at,
ROW_NUMBER() OVER (
PARTITION BY order_id
ORDER BY source_sequence DESC, source_updated_at DESC
) AS row_num
FROM main.staging.order_changes
)
SELECT order_id, order_status, source_sequence, source_updated_at
FROM ranked
WHERE row_num = 1;What to explain: the ordering columns must define a deterministic winner. If two events can share both values, add a source event ID or another authoritative tie-breaker. Then MERGE the deduplicated source. A Delta MERGE expects no more than one source row to match a target row.
Drill 2 — Use a broadcast join only for a genuinely small dimension
Scenario: a large order fact table joins to a small country lookup. Check its size and plan before using a hint.
from pyspark.sql import functions as F
from pyspark.sql.functions import broadcast
orders = spark.table("main.silver.orders")
countries = spark.table("main.reference.countries")
result = (
orders
.filter(F.col("order_date") >= F.lit("2026-01-01"))
.join(broadcast(countries), "country_code", "left")
)
result.explain("formatted")What to explain: the broadcast table must be small enough for safe distribution to executors. Confirm the join strategy in the plan and validate runtime memory, shuffle and duration. A hint is not a substitute for checking data growth.
Drill 3 — Incrementally ingest files with Auto Loader
Scenario: JSON files arrive continuously. You need schema state and a durable checkpoint. The example uses a Unity Catalog volume for paths.
source = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "/Volumes/main/landing/ops/orders_schema")
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")
.load("/Volumes/main/landing/ops/orders")
)
(
source.writeStream
.option("checkpointLocation", "/Volumes/main/landing/ops/checkpoints/orders")
.toTable("main.bronze.orders")
)What to explain: schema location and checkpoint location have different jobs. With add-new-columns evolution, the stream can stop with an UnknownFieldException after updating the schema state; configure restart behaviour through Lakeflow Jobs. Each distinct ingestion stream needs its own checkpoint.
Drill 4 — Bound deduplication state with a watermark
Scenario: events may be duplicated, and the business can tolerate a defined lateness window. This API is available in supported Databricks Runtime versions.
deduplicated = (
events
.withWatermark("event_time", "2 hours")
.dropDuplicatesWithinWatermark(["event_id"])
)What to explain: the watermark bounds state and establishes the late-data threshold. Events within the threshold are processed; events later than it are not guaranteed to be processed by the stateful operation. Choose two hours from observed event lateness and the business requirement, not by habit.
Architecture round
Three system-design case studies interviewers can use
Advertisements
CASE 01 E-commerce order lakehouse with unreliable files
Prompt: order files land throughout the day from multiple regions. Files can be duplicated, arrive late and add optional fields. Analysts need trusted revenue tables within 30 minutes.
Strong design: retain raw files with source metadata; use Auto Loader for incremental file discovery; write Bronze first; validate schema and record quality; deduplicate by order ID plus authoritative source sequence; use Delta tables for Silver and Gold; orchestrate with Lakeflow Jobs or a declarative pipeline; and publish data under Unity Catalog governance.
Trade-off to discuss: a 30-minute SLA may be met with triggered micro-batches; continuous low-latency processing adds operational complexity and should be justified by the workload.
CASE 02 Customer CDC with SCD Type 2 history
Prompt: a source database emits customer inserts, updates and deletes. Business users need both the current customer profile and historical values as of any date.
Strong design: ingest changes with a source sequence or log position; preserve the source event; apply changes in deterministic order; use a CDC-capable Lakeflow pipeline pattern such as AUTO CDC for SCD Type 2 where supported; and validate initial snapshots, deletes, duplicate sequence numbers and late events.
Trade-off to discuss: clarify whether history is source-of-truth CDC history or business-valid-time history. Those are not always the same requirement.
CASE 03 Slow pipeline with rising cost
Prompt: a daily transformation has moved from 22 minutes to 70 minutes, while the cloud bill has increased. The business wants the 30-minute SLA restored.
Strong investigation: compare recent and baseline job runs; locate the slow task and stage; compare input bytes, shuffle read/write, spill, task-time distribution and output file sizes; inspect joins and filters; check skew, stale statistics and source or sink throttling; then choose a targeted fix.
Trade-off to discuss: a larger compute shape may reduce elapsed time but raise cost. A successful optimization should improve the agreed SLA and cost metric, not only one benchmark run.
Avoid outdated interview answers
Current Databricks terminology to know
Interviewers may accept legacy names when discussing older projects, but you should recognize current product names and explain the transition accurately.
| Older wording | Current wording | How to speak about it |
|---|---|---|
| Databricks Workflows | Lakeflow Jobs | Say Lakeflow Jobs, while acknowledging older projects and screenshots may say Workflows. |
| Delta Live Tables / DLT | Lakeflow Spark Declarative Pipelines / Lakeflow pipelines | Recognize the legacy DLT name and use the current name for new work. Existing code paths may continue to be supported; check the migration guidance before changing code. |
| Databricks Asset Bundles / DABs | Declarative Automation Bundles | Use the current name; the CLI command family continues to use `databricks bundle`. |
| Databricks Repos | Git folders | Describe the workspace Git integration using current terminology. |
Focused revision
A practical 7-day Databricks interview preparation plan
| Day | Focus | Deliverable |
|---|---|---|
| 1 | Platform fundamentals, Unity Catalog, compute choices | Explain a governed Bronze-to-Gold design in three minutes. |
| 2 | Spark execution, joins, shuffles, skew, AQE | Diagnose one deliberately slow join using a plan and task metrics. |
| 3 | Delta transactions, MERGE, time travel, maintenance | Write a deduplicated MERGE source and explain safe recovery. |
| 4 | Auto Loader, schema evolution, checkpoints, watermarks | Ingest a file set twice and test a new column and a late event. |
| 5 | Lakeflow Jobs, pipelines, retries and CI/CD | Build a multi-task DAG and explain how a repair run can repeat side effects. |
| 6 | SQL and PySpark coding drills | Complete all four drills without copy-pasting the solution. |
| 7 | System design and mock interview | Whiteboard one case study and tell one measured incident story. |
For exam-focused study, the official Data Engineer Associate exam and your dedicated practice simulator are separate resources. The current exam blueprint is not a replacement for a job interview syllabus.
What weakens an answer
Common Databricks interview mistakes
Primary sources
Official Databricks documentation to verify before an interview
Use official documentation for feature behaviour, version requirements and cloud-specific limits. These pages are more authoritative than a generic question list.
- Auto Loader overview and configuration — file discovery, schema location, evolution and rescued data.
- Upsert into a Delta table using MERGE — insert, update, delete and duplicate-source handling.
- Structured Streaming checkpoints — offsets, commits, state and query identity.
- Watermarks in Structured Streaming — state cleanup and late-data thresholds.
- Production considerations for Structured Streaming — job deployment, recovery and latency choices.
- Adaptive Query Execution — runtime query re-optimization and skew handling.
- Databricks optimization recommendations — file layout, predictive optimization and query performance.
- Liquid clustering — modern table layout guidance.
- Lakeflow Jobs — task graphs, dependencies, triggers and run management.
- Troubleshoot and repair job failures — repair behaviour and duplicate side-effect risk.
- Apache Spark Declarative Pipelines — declarative pipeline concepts and capabilities.
- Declarative Automation Bundles — project structure, targets and deployment.
- Unity Catalog privileges reference — securable objects and grants.
- Unity Catalog best practices — groups, service principals and least privilege.
- Serverless compute limitations — language, API, streaming and troubleshooting constraints.
- Unity Catalog table types — managed, external and foreign tables.
The documentation links above use the AWS documentation path for the concepts cited. Azure and Google Cloud workspaces can have cloud-specific differences; use the corresponding cloud documentation for network, identity, storage and service-specific behaviour.
Quick answers
Frequently asked questions
What are the most important Databricks interview topics for data engineers?
Prioritize Spark execution and performance, SQL and PySpark, Delta Lake transactions and MERGE, ingestion with Auto Loader, Structured Streaming recovery, Lakeflow Jobs and pipelines, Unity Catalog permissions, compute choices and production troubleshooting. The exact mix depends on seniority and the job description.
Are these actual Databricks company interview questions?
No. These are original practice questions built around documented platform behaviour and common data-engineering responsibilities. They are not leaked questions and are not represented as questions asked by Databricks or any named employer.
Is this the same as the Databricks Data Engineer Associate exam?
No. The certification exam is a proctored multiple-choice assessment with an official exam guide. Job interviews are open-ended and often include coding, design, debugging and discussion of your own projects. Use the separate certification practice page for exam preparation.
Should I prepare SQL or PySpark first?
Prepare both. SQL is often used for relational transformations, deduplication, aggregations and window functions. PySpark is common for DataFrame transformations, ingestion and pipeline logic. The role description and interview format should determine how you split preparation time.
How many questions should I practice before an interview?
Depth is more useful than memorizing a large number. Work through the scenarios relevant to the job, write the coding drills, and practice explaining one incident and one end-to-end design. Be ready for follow-up constraints that change the best answer.
Do I need hands-on Databricks experience?
Hands-on work is strongly helpful because many questions ask you to diagnose jobs, reason about Spark plans, inspect Delta history, or configure governance. If you cannot access a workspace, use official tutorials and be clear about what you have personally implemented versus what you have studied.
Are older DLT and Workflows tutorials still useful?
They can explain concepts, but verify current terminology and APIs. Lakeflow Jobs is the current name for Workflows, and Lakeflow Spark Declarative Pipelines is the current product family associated with the former Delta Live Tables terminology. Check migration documentation before changing existing code.
Can I use Databricks Free Edition to practice?
It is useful for learning many platform basics, but it has compute, concurrency, language and feature limitations. It is not a production environment. Review the current Free Edition limitations and do not assume every classic compute or serverless capability is available.
How do I make an interview answer sound experienced?
State the workload, symptom, evidence, decision, trade-off and validation. Use concrete metrics only when you measured them. A thoughtful investigation plan is stronger than claiming that you always increase cluster size, repartition the data or turn on a feature.
Hands-on
SQL and PySpark coding drills
Type these from memory first, then compare. Interviewers care less about syntax than about the choices you narrate: the key, the ordering column, what happens on retry.
Drill 1 · Latest row per key, then idempotent MERGE
Prompt: a batch can contain several updates for one customer. Apply only the newest to main.silver.customers, safely on retry.
MERGE INTO main.silver.customers AS t
USING (
SELECT * EXCEPT (rn) FROM (
SELECT *, ROW_NUMBER() OVER (
PARTITION BY customer_id
ORDER BY source_seq DESC, updated_at DESC) AS rn
FROM main.bronze.customer_updates_batch
) WHERE rn = 1
) AS s
ON t.customer_id = s.customer_id
WHEN MATCHED AND s.source_seq > t.source_seq THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;Say out loud: the guard s.source_seq > t.source_seq stops a replayed older batch from overwriting newer data.
Drill 2 · Auto Loader into Bronze with a checkpoint
Prompt: ingest JSON landing in a volume, keep source metadata, survive restarts.
from pyspark.sql import functions as F
(spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "/Volumes/main/raw/_schemas/orders")
.load("/Volumes/main/raw/landing/orders")
.withColumn("_source_file", F.col("_metadata.file_path"))
.withColumn("_ingested_at", F.current_timestamp())
.writeStream
.option("checkpointLocation", "/Volumes/main/raw/_checkpoints/orders_bronze")
.trigger(availableNow=True)
.toTable("main.bronze.orders"))Follow-ups to expect: what changes the checkpoint path would cause, where rescued data goes, and why the schema location and checkpoint are separate.
Drill 3 · Prove skew, then choose a fix
Prompt: a join on customer_id has one 35-minute task. Show the evidence first.
from pyspark.sql import functions as F
# 1) Is the key hot?
(orders.groupBy("customer_id").count()
.orderBy(F.desc("count")).show(10))
# 2) Small side? Broadcast it explicitly.
joined = orders.join(F.broadcast(dim_country), "country_code")
# 3) Check what actually ran
joined.explain("formatted")Only after the counts show a hot key do you discuss salting, pre-aggregation or AQE skew handling.
Drill 4 · Top N per group and a running total
Prompt: top 3 products by revenue per category, plus cumulative revenue by day.
SELECT * FROM (
SELECT category, product_id, SUM(revenue) AS rev,
DENSE_RANK() OVER (PARTITION BY category ORDER BY SUM(revenue) DESC) AS rnk
FROM main.gold.sales
GROUP BY category, product_id
) WHERE rnk <= 3;
SELECT order_date, SUM(revenue) AS day_rev,
SUM(SUM(revenue)) OVER (ORDER BY order_date) AS running_rev
FROM main.gold.sales
GROUP BY order_date;Be ready to explain why DENSE_RANK versus ROW_NUMBER changes how ties behave.
Architecture
System-design case studies
Case 1 · Daily order files with late files, duplicates and drift
Clarify: file volume, arrival window, key, whether late files can change history. Design: Auto Loader to Bronze with source file and ingest time; Silver deduplicates on order key plus source sequence and MERGEs; schema changes land in rescued data and trigger an alert, not silent evolution; Gold rebuilds by date range so a late file is a deterministic re-run. Prove: row-count and revenue reconciliation per file against the source manifest.
Case 2 · 2 TB/day, 30-minute freshness, sensitive attributes
Clarify: what “fresh” means and for whom, peak versus average rate. Design: incremental ingestion with a trigger that matches the SLA; clustering on the dominant filter; sensitive columns governed in Unity Catalog with masks or views and least-privilege groups; separate compute for ingest and BI so one cannot starve the other. Trade-off to state: cost of always-on streaming versus frequent micro-batches against the SLA actually required.
Case 3 · Green run, revenue 8% too high
Order of work: confirm the number and the baseline, bisect by layer (Gold, Silver, Bronze) and by day, check join fan-out and duplicate keys, compare to source totals, fix and backfill into a staging table, then reconcile before swapping. Prevention: a row-count and sum check as a gating task, plus uniqueness expectations on join keys.
Vocabulary
Current Databricks terminology
Names change. Using the current term, and knowing the old one, signals you work with the platform now.
| You may hear | Current idea | Say in an interview |
|---|---|---|
| Workflows | Lakeflow Jobs | “Orchestrated with Jobs, with task dependencies and repair runs.” |
| Delta Live Tables / DLT | Lakeflow pipelines (declarative) | “Declared datasets with expectations; the engine manages dependencies.” |
| Repos | Git folders | “Code lives in Git folders and deploys through CI.” |
| Partition plus Z-ORDER | Liquid clustering for new tables | “I would evaluate clustering keys from real query filters.” |
| DBFS mounts | Unity Catalog volumes and external locations | “Governed file access instead of mount points.” |
Product names move quickly. Confirm anything you plan to quote against the official documentation before an interview.
Plan
7-day focused preparation plan
Avoid
Common interview mistakes
Answering with a definition and stopping. Follow with how you would verify it. Tuning before measuring. Resizing the cluster before reading stage metrics is the classic miss. Claiming exactly-once for the whole pipeline. It depends on the sink. Fixing permissions with broad grants. Skipping retries and replay. If you cannot say what a rerun does, the design is unfinished. Memorising feature names. Say what problem each one solves and what it costs.
Sources
Official documentation
FAQ
Frequently asked questions
How should I use these 60 questions?
Do not memorise them. Pick a question, answer aloud in 30 seconds, then add how you would verify and what could go wrong.
Are these real Databricks interview questions?
No. They are original practice scenarios modelled on common data-engineering problems, not leaked or employer-specific questions.
Is this enough for the certification exam?
No. Certification is mostly multiple choice. This page is for open-ended interviews; see the Associate exam guide linked below.
How much code will I be asked to write?
It varies by employer. Expect SQL window functions and MERGE logic in most loops, and PySpark transformations in many.
Continue learning
Related Databricks resources on Ultimate Info Guide
For broader platform context, compare Microsoft Fabric vs Databricks. For workload planning, see the Databricks Cost Calculator.
Editorial note: This is an independent Ultimate Info Guide learning resource, not sponsored by or affiliated with Databricks. Platform behaviours and names can change. The official documentation links are the source of truth for current feature support, runtime compatibility and cloud-specific limits. Last terminology and documentation review: 9 November 2026.
