1. Read the plan and the slow stage together

Record the actual Spark version, attached Fabric Environment, workload/data version, executor resources and competing work. Use the SQL/DataFrame executed plan and the linked Spark UI stages, not elapsed time alone.

Compare shuffle read/write, records, task duration, spill and GC distributions. A ‘three times median’ rule is a useful inspection prompt, not Spark’s formal skew definition. AQE’s skew thresholds use configured size and median-factor conditions, and support depends on the operator/join.

2. Choose a starting value, not an alleged optimum

Divide a representative stage’s shuffle bytes by a candidate partition size, then check whether the resulting tasks balance memory, parallelism and scheduling overhead. Compressed bytes do not equal in-memory row/aggregation state. A 128 MiB target is an experiment, not a safety guarantee.

import math

# Measured representative stage size; replace this example.
shuffle_bytes = 100 * 1024**3
candidate_partition_bytes = 128 * 1024**2
initial_partitions = max(
    1, math.ceil(shuffle_bytes / candidate_partition_bytes)
)
print("Candidate initial partitions:", initial_partitions)  # 800

There is no universal 5,000-8,000 partition ceiling or mandatory 2-5 tasks per core rule. A large workload can justify more tasks; a small workload can suffer from excess scheduling. Increasing hash partition count alone does not distribute one hot join key across different keys.

Too many tiny files, input partitions and shuffle partitions are related but different concepts. Do not use one configuration to explain every stage or output layout.

3. Inspect active AQE settings and the final plan

Fabric manages runtime defaults; do not import a Databricks threshold or assume every setting is mutable. Inspect your actual session before changing it:

keys = [
    "spark.sql.shuffle.partitions",
    "spark.sql.adaptive.enabled",
    "spark.sql.adaptive.coalescePartitions.enabled",
    "spark.sql.adaptive.advisoryPartitionSizeInBytes",
    "spark.sql.adaptive.coalescePartitions.parallelismFirst",
    "spark.sql.adaptive.skewJoin.enabled",
    "spark.sql.autoBroadcastJoinThreshold",
]
for key in keys:
    print(key, spark.conf.get(key))

# Inspect the actual query you are investigating.
# result_df is your existing transformed DataFrame.
result_df.explain("formatted")

In an approved test session, set a measured initial partition count before constructing/executing the relevant workload. AQE can coalesce small shuffle partitions, change eligible joins and handle eligible skew. Its advisory size is not a hard per-task cap; other settings, including parallelism preference, influence coalescing. Inspect the final executed adaptive plan after a representative action.

coalesce(n) usually reduces partitions without a full reshuffle and can concentrate work. It does not globally disable AQE for every other exchange in the query. repartition(n) adds a shuffle, which is not automatically beneficial; repartitioning by the same hot key still keeps that key together.

4. Fix the join or data shape before forcing more partitions

• Reduce work first: project/filter early where semantics permit, avoid unintended many-to-many expansion and remove unused expensive transformations.
• Broadcast: a genuinely small build side may avoid shuffling the large side for the join. It must fit the applicable executor/driver memory and join-type constraints. A hint can override size-based planning; it is not safe merely because a file is below 100 MB.
• Skew: inspect hot-key counts and whether AQE handled the eligible join. A single-key partition is not categorically impossible for AQE to split; operator support, map-output blocks and replication costs matter.
• Storage Partition Join: requires a compatible data-source implementation that reports usable partitioning and a supported runtime/plan. Matching directory partition columns on arbitrary Delta tables does not prove SPJ works. Iceberg-specific settings are not a generic Fabric Delta repair.
• Plan evidence: no Exchange alone does not prove SPJ; broadcast or another strategy can also avoid an exchange. Inspect scans, distribution requirements and the selected join.

Sort-merge, hash and nested-loop strategies have different supported conditions. A non-equi join is not automatically an erroneous Cartesian product; verify the predicate and cardinality before redesigning it.

5. If manual salting is justified, preserve the original join semantics

Salting can distribute an evidenced hot key across buckets while replicating matching build-side rows. It increases work/storage and needs special care for outer joins, null semantics, duplicate keys and aggregates. Raising a salt bucket count is not a substitute for checking those costs.

This small PySpark inner-join example uses only in-memory sample data and verifies multiset equivalence. It is not a recommended bucket count for production and performs no writes.

from pyspark.sql import functions as F

fact = spark.createDataFrame(
    [(101, 1, 10), (102, 1, 20), (103, 2, 30)],
    ["row_id", "key", "amount"],
)
dim = spark.createDataFrame([(1, "A"), (2, "B")], ["key", "label"])
buckets = 4

baseline = fact.join(dim, "key", "inner").select(
    "row_id", "key", "amount", "label"
)
left = fact.withColumn(
    "salt", F.pmod(F.xxhash64("row_id"), F.lit(buckets))
)
right = dim.crossJoin(spark.range(buckets).toDF("salt"))
salted = left.join(right, ["key", "salt"], "inner").select(
    "row_id", "key", "amount", "label"
)
assert salted.exceptAll(baseline).limit(1).count() == 0
assert baseline.exceptAll(salted).limit(1).count() == 0

Do not replace an ordinary join with an aggregation and call the result identical. For a real optimization, preserve the intended columns, duplicates and null behavior; compare key-level/business totals as well as row counts. Replicating an entire non-small dimension can be worse than the original shuffle.

6. Measure the fix under comparable conditions

1. Save the original settings and plan. Change one hypothesis-driven setting or transformation in an isolated test.
2. Use the same data, executor resources and comparable cache/concurrency state. Perform an action that actually evaluates the expensive branch.
3. Compare final plans, stage times, task distributions, spill, GC, executor failures and end-to-end time, not one attractive task metric.
4. Reconcile output schema, duplicates, null semantics and business results.
5. Evaluate compute/capacity use as well as elapsed time. More parallel work is not automatically cheaper.
6. Repeat after realistic data growth and restore ineffective changes. Do not present an unexecuted example as an internal benchmark.

For OOM, distinguish executor heap, driver collection, Python/native memory, large broadcasts and oversized aggregation state. Compaction can improve file/scan efficiency, but OPTIMIZE does not remove logical rows or inherently reduce every later shuffle’s byte volume.

Common questions

Official references and review scope

Checked against the following Microsoft documentation on 2026-10-30. This is an independent diagnostic guide, not a Microsoft-certified repair. The examples have not been executed in your Fabric tenant; validate the applicable identity, schema, runtime and governance requirements before production changes.

• Fabric Spark monitoring and performance guidance
• Fabric Spark memory and executor troubleshooting
• Fabric runtime migration and compatibility
• Apache Spark SQL: AQE, skew and compatible V2 Storage Partition Join

Continue with a relevant guide

Practice Fabric interview scenarios