How Spark runs your code in Microsoft Fabric
You write a few lines of PySpark, SQL, Scala or R in a Fabric notebook. A few seconds later a table with hundreds of millions of rows has been read, joined, summed and written back to OneLake. This canvas follows one query through every step: how Spark plans it, where each language actually runs, how the work is split across machines, how memory and Delta files shape it, which Fabric settings change the outcome, and what it costs in capacity. A playground at the end lets you rebuild the query yourself.
Fabric Runtime 2.0 (default): Spark 4.1 · Delta Lake 4.2 · Java 21 · Scala 2.13 · Python 3.13 · R 4.5. Each section starts in plain language; open the “Deep dive” rows for the engineering detail.
used in every section
Total 2025 revenue per region, written to a new Delta table revenue_by_region.
sales · Delta fact table
400 million rows · 48 GB, 24 columns, about 1,600 Parquet files
4 years of orders (2022-2025)
stores · Delta dimension table
2,000 rows · about 1 MB, 6 columns
store_id → region, country, city …
0 of 10 sections done
You build pipelines and need to size, tune and debug them.
sections done
0 / 10
quiz correct
0 / 10
Click a section to jump to it. Each section ends with three takeaways, a one-question check (0 answered so far) and an “I've got this” box that fills these bars. Unfamiliar word? See the glossary at the bottom.
Section 1
From cell to result: what happens when you press Run
Spark is lazy. While you write transformations it only builds a description of the work. When you ask for a result, it optimizes that description, splits it into pieces and spreads them over the machines in your session.
Step 1 of 18 · click a box to jump
PySpark version
sales = spark.read.table("sales")Reading a table is a transformation. Spark looks up the table's schema in the Lakehouse catalog and records a plan step. No files are read.
step 1
Relation sales [24 columns]
Read from the bottom up: data flows from the relation at the bottom to the node at the top.
Transformations return a new DataFrame and only extend the plan: select, filter, withColumn, join, groupBy, union, repartition. Actions return something to you or write somewhere, and each one starts at least one job: count, show, collect, toPandas, write, saveAsTable, display. Calling two actions on the same DataFrame runs the whole plan twice unless you cache it.
The driver holds the plan, the catalog lookups and the scheduler. It never touches the 48 GB itself; executors read the files directly from OneLake. That is why collect() and toPandas() on big results are dangerous: they pull everything back onto the single driver machine.
Before Catalyst optimizes anything, the analyzer resolves names against the catalog (the Lakehouse tables and their Delta logs) and fails early on typos, missing columns or type mismatches. That is why an error about an unknown column appears immediately, while a bad file only shows up once the action runs.
Key takeaways
• Transformations only build a plan; an action (write, show, count, collect) runs it.
• The driver plans and schedules; executors do the reading and computing.
• AQE re-plans between stages using the real sizes, not estimates.
Check yourself: A cell runs sales = spark.read.table("sales").filter(...) and nothing else. How much data does Spark read?
Section 2
Four languages, one engine
PySpark, Spark SQL, Scala and SparkR are different front doors to the same engine. They all build the same plan, and the heavy work runs in the same executors. The language matters mainly when you write your own row-level logic.
Notebook cell magic: %%pyspark
from pyspark.sql import functions as F
sales25 = (spark.read.table("sales")
.filter((F.col("order_date") >= "2025-01-01")
& (F.col("order_date") < "2026-01-01")))
result = (sales25
.join(spark.read.table("stores"), "store_id")
.groupBy("region")
.agg(F.sum("amount").alias("revenue")))
result.write.mode("overwrite").saveAsTable("revenue_by_region")Where this code runs
Your Python runs in a Python process on the driver. Every DataFrame call is forwarded to the JVM through Py4J, where it only adds a step to the plan. No sales rows ever pass through Python.
Switch languages: the physical plan in Section 3 is identical for all four. Spark SQL and the DataFrame APIs share one optimizer.
Highlighted parts are in use for the selected language and custom logic below. Simplified: real sessions have many executor nodes, each running exactly one executor.
clean up region names
Suppose region names need trimming and upper-casing before the group-by. There are several ways to write that, and they travel very different paths.
F.upper(F.trim(F.col("region")))Shows up in explain() as: ProjectExecTransformer [upper(trim(region))]
Stays inside the engine. With the native engine on, it runs as vectorized C++; otherwise as generated JVM code.
Optimizer can see inside: Yes: it can reorder, push down and vectorize it
Data leaves the JVM: None
Relative time for the clean-up step (illustrative, built-in = 1×)
Built-in function
1.0×
SQL UDF
1.1×
Scala UDF
1.5×
Python UDF (row by row)
6.0×
Python UDF (Arrow)
2.4×
pandas UDF
1.8×
Illustrative ratios for a simple string function on 100 million rows; real numbers depend on the function and data. Pattern from Spark and Fabric guidance: built-ins first, then pandas or Arrow UDFs, then row-by-row Python UDFs last.
| Language | Where your code runs | Good for | Fabric Runtime 2.0 notes |
|---|---|---|---|
| PySpark | Python process on the driver, plan built over Py4J | Most data engineering, orchestration, notebookutils, ML | Python 3.13; check that your packages have 3.13 wheels |
| Spark SQL | Parsed directly in the JVM driver | Transformations, Delta maintenance (OPTIMIZE, VACUUM), analysts | ANSI mode is off by default in Fabric (spark.sql.ansi.enabled=false) |
| Scala | Compiled and run in the JVM driver | JVM libraries, typed Datasets, custom connectors, fast UDFs | Scala 2.13: JARs built for 2.12 must be recompiled |
| SparkR | R process on the driver, over a socket | Existing R code | Deprecated in Spark 4.x; plan a move to PySpark or SQL |
Py4J is a local gateway between the Python process and the driver JVM. A call like df.filter(...) sends a small command, and the JVM returns a reference to a new Java DataFrame object. Only commands and small results cross the bridge, which is why plan-building code costs the same in every language.
All languages in one notebook share the same Spark session, but not their variables. A Python DataFrame is invisible to a Scala cell. Share data through a temporary view or a Lakehouse table:
%%pyspark
result.createOrReplaceTempView("revenue")
%%sql
SELECT * FROM revenue ORDER BY revenue DESC
%%spark
val top3 = spark.table("revenue").orderBy(desc("revenue")).limit(3)When data must come into Python (toPandas, a small lookup, a chart), Spark uses Arrow to transfer it in columns. Keep it small: everything lands in the driver's memory.
Fabric does not offer Spark Connect. Code running outside Fabric submits work through the Livy API, or queries through the Fabric Spark JDBC (jdbc:fabricspark://), ODBC and ADO.NET drivers.
Key takeaways
• PySpark, SQL, Scala and SparkR build the same plan and run at the same speed for DataFrame code.
• Language matters when your own code touches every row: prefer built-ins, then pandas or Arrow UDFs.
• Share data between languages through temp views or tables, not variables.
Check yourself: When is PySpark noticeably slower than Scala?
Section 3
The optimizer: less reading, less moving
Catalyst, Spark's optimizer, rewrites your plan before anything runs. The biggest wins come from not reading data at all and not moving it between machines. Turn the rules off to see what each one saves on the 48 GB sales table.
The date filter is handed to the Parquet reader, which skips row groups whose min/max order_date can't match.
Parquet stores each column separately, so reading only store_id, order_date and amount skips 21 of 24 columns.
The Delta log keeps min/max values per file. Files whose order_date range is outside 2025 are never opened.
stores is tiny, so each executor gets a copy and the 2025 sales rows never cross the network for the join.
accent bar = current rules
Data read from OneLake
current
1.7 GB
all rules off
48 GB
Files opened
current
400
all rules off
1,600
Data shuffled over the network
current
31 KB
all rules off
12 GB
Model: sales is 48 GB in 1,600 files covering 2022-2025, so 2025 is about a quarter of the files. Times use the same model as the cluster simulator in Section 4, for 8 Medium executors (64 cores) plus a driver node: 72 vCores, billed at 2 vCores per CU. All rules off: 15 s and 544 CU-s. Illustrative, not a benchmark.
WHERE year(order_date) = 2025 looks equivalent, but the min/max statistics are on order_date itself, not on year(order_date). Spark may then have to open every file. A plain range on the raw column always lets Delta skip files:
-- may read all 1,600 files WHERE year(order_date) = 2025 -- skips files outside 2025 WHERE order_date >= '2025-01-01' AND order_date < '2026-01-01'
Keep the types matching too. In this example order_date is a DATE. If it were a timestamp, compare it with a timestamp or cast it explicitly, for example CAST(order_date AS DATE) >= DATE '2025-01-01'. Fabric's native engine may not accelerate a filter that compares a timestamp with a string.
Read the plan yourself: explain()
result.explain() (or EXPLAIN in SQL) prints the physical plan. Read it from the bottom up. Compare the same query with the native engine on, off, and with a Python UDF added.
runs in the native engine (Velox)
runs in the JVM
boundary: data changes format or process
Every operator ends in Transformer or starts with Native or Columnar, so the whole query runs in Velox on columnar batches.
Simplified output: real plans also show expression IDs, file counts and statistics. Operator names from Fabric's native execution engine documentation.
Analysis resolves table and column names against the catalog. Logical optimization applies rule batches until the plan stops changing: constant folding, predicate pushdown, column pruning, simplifying filters, and rewriting subqueries as joins. Physical planning picks algorithms: broadcast or sort-merge join, hash or sort aggregate. Code generation (JVM) or translation to native operators (Velox, through Gluten) produces the code that executors run.
Static planning relies on statistics such as table size, and those can be wrong after a filter or join. That is why Adaptive Query Execution exists: it re-runs physical planning between stages using the real sizes, as you saw in Section 1.
Useful variants: df.explain("formatted") splits the plan into a short tree plus per-operator details; df.explain("cost") adds size estimates. In the Spark UI, the SQL / DataFrame tab shows the same plan with row counts and timings after the run.
Key takeaways
• The cheapest data is the data you never read: pruning, pushdown and file skipping.
• Filter on the raw column with a range, and keep the literal's type matching the column.
• Read explain() from the bottom up; Transformer nodes mean native execution.
Check yourself: Which filter lets Delta skip the files outside 2025?
Section 4
Stages, tasks and the cluster
Spark splits each stage into tasks, one per slice of data. Each executor core runs one task at a time. If there are more tasks than cores, they run in waves, and the slowest task decides when the stage ends.
The running example as stages. A broadcast doesn't create a stage boundary in the main chain; a shuffle does.
illustrative model
Node size
Nodes in the session
5 (1 driver + 4)
Data read by stage 2
24 GB
Adaptive Query Execution
What it costs
Session left idle after the job (until timeout)
20 min (default timeout)
Price per CU-hour (pay-as-you-go)
$0.18 (East US list)
job
281 CU-s
idle
6.67 CU-h
CU-seconds = vCores ÷ 2 × seconds, for every node in the session. An interactive session keeps billing until it times out (20 minutes by default) or you stop it; on a capacity this shows up as usage rather than a separate bill. $0.18 per CU-hour is the East US pay-as-you-go list price (Oct 2026); reservations are about 41% cheaper. Check your region and agreement.
read stage: one task per 128 MB
shuffle stage (e.g. a sort-merge join on store_id)
tasks for the skewed key
dashed line = stage boundary · one thin row per core
AQE merged the shuffle into 116 partitions of about 64 MB instead of 200 fixed ones.
Model: about 1.6 s per 128 MB read task, about 0.8 s per 64 MB shuffle task plus scheduling overhead; the shuffle stage handles 30% of the read volume. Each node runs exactly one executor, the driver uses its own node, and in single-node mode driver and executor share one node at half the resources each.
Starter pool limits by capacity
Starter pools use Medium nodes and start in about 5 seconds. Two Spark vCores equal one capacity unit, and bursting allows up to 3× the base vCores (F64: 128 vCores, bursting to 384).
| Capacity | Spark vCores | Default max nodes | Max nodes |
|---|---|---|---|
| F2 | 4 | 1 | 1 |
| F4 | 8 | 1 | 1 |
| F8 | 16 | 2 | 2 |
| F16 | 32 | 3 | 4 |
| F32 | 64 | 8 | 8 |
| F64 | 128 | 10 | 16 |
Source: Microsoft Learn, “Configure starter pools in Fabric”.
Read tasks are sized by spark.sql.files.maxPartitionBytes (128 MB by default). Shuffle stages start with spark.sql.shuffle.partitions (200 by default); AQE then coalesces small partitions toward spark.sql.adaptive.advisoryPartitionSizeInBytes (64 MB by default).
Aim for task counts that are a few times the number of cores. Far fewer tasks than cores leaves cores idle; thousands of tiny tasks spend more time on scheduling than on work. Waves matter less than stragglers: one task at 10× the median holds the whole stage.
AQE's skew handling (spark.sql.adaptive.skewJoin.enabled) splits oversized partitions in sort-merge joins. It does not split skewed group-by keys; for those, salt the key or pre-aggregate. Broadcast joins avoid the problem entirely when the other side is small.
Pools autoscale between their minimum and maximum nodes, and dynamic allocation adds or releases executors within a session. Sessions expire after 20 minutes of inactivity by default. Executor memory, cores and counts can only be set when the session starts: in the environment, or with %%configure as the first cell.
Key takeaways
• One task per partition; each core runs one task at a time, in waves.
• The slowest task ends the stage, so skew hurts more than size.
• You pay for every vCore in the session while it is alive, busy or idle.
Check yourself: 4 Medium executors (8 cores each) run a stage with 100 tasks of equal size. How many waves?
Section 5
Joins: copy the small table or shuffle both
To join, matching rows must sit on the same machine. Spark either copies the small table to every executor (broadcast) or regroups both tables by the join key (shuffle). The size of the smaller side decides, and AQE can change its mind at runtime.
Size of the stores (dimension) table
1 MB
spark.sql.autoBroadcastJoinThreshold
10 MB (default)
Adaptive Query Execution
stores is estimated at 1 MB, at or under the 10 MB threshold, so Spark plans a broadcast. Each of the 8 executors gets a full copy and joins its own slice of sales locally.
moved (this plan)
9 MB
moved (sort-merge)
1.8 GB
Bytes sent between machines. Broadcast ≈ dimension size × (8 executors + driver); sort-merge ≈ 2025 sales after pruning (1.8 GB) + dimension.
Simplified to four executors. Highlighted arrows carry data over the network; dashed or plain arrows stay on the same machine.
Spark has three main equi-join strategies. A broadcast hash join builds a hash table from the small side on every executor. A shuffled hash join shuffles both sides and builds a hash table per partition, with no sort. A sort-merge join shuffles and sorts both sides, then merges them; it is the safe default for two large tables because it can spill to disk.
Hints override the threshold when you know better than the statistics: SELECT /*+ BROADCAST(st) */ ... in SQL or sales.join(F.broadcast(stores), "store_id") in PySpark. Spark refuses to broadcast tables over 8 GB, and big broadcasts cost driver memory, so raise the threshold with care.
Fabric autotune (spark.ms.autotune.enabled) can learn a better broadcast threshold, shuffle partition count and max partition bytes for queries that run repeatedly. It is off by default.
Key takeaways
• Small dimension: broadcast it and the big table never moves.
• Two big tables: both sides shuffle by the join key.
• AQE can switch to a broadcast at runtime once it sees the real size.
Check yourself: stores is 1 MB and sales is 48 GB. Which join does Spark choose by default?
Section 6
Memory, spill and caching
Each executor splits its memory between the cores running tasks. When a task's slice of data is bigger than its share, Spark writes the overflow to local disk (spill) and reads it back. Far bigger, and the task can run out of memory.
Node size (8 executor nodes)
Operation in the shuffle stage
Data entering the stage
200 GB
spark.sql.shuffle.partitions
200 (default)
AQE coalescing (merges small partitions toward 64 MB)
one executor
overhead 8.0 GB
reserved 0.3 GB
user 22.3 GB
execution + storage 33.4 GB
Each task's working set fits in its share of execution memory. No spill.
Estimated stage time: 52 s in 4 waves on 64 cores.
AQE only merges partitions; it can't split a normal partition that is too large. With big shuffles, raise spark.sql.shuffle.partitions (or let Fabric autotune do it).
Simplified model of Spark's unified memory: about 12.5% of the node for overhead, 300 MB reserved, then spark.memory.fraction = 0.6 of the rest for execution and storage, shared by the cores. Working-set factors (1.3× to 2.5× the partition size) stand for rows expanding in memory. Illustrative, not a sizing tool.
Reusing a DataFrame: cache, persist or write?
Say the 2025 sales joined with stores (1.7 GB on disk, about 5 GB in memory) feeds several reports in the same notebook. Each action recomputes it unless you keep it somewhere.
Times the DataFrame is used
3×
| Strategy | Total time | Executor memory | Survives | Note |
|---|---|---|---|---|
Do nothing (recompute) df = sales25.join(stores, "store_id") | 27 s | none | Nothing to keep | Every action re-reads and re-joins from OneLake. |
df.cache() df.cache(); df.count() | 12 s | 5.1 GB | This session only; evicted under memory pressure | MEMORY_AND_DISK by default. Fastest reuse, but it competes with execution memory. |
df.persist(DISK_ONLY) df.persist(StorageLevel.DISK_ONLY) | 17 s | none | This session only; lost if an executor is removed | Keeps executor memory free, at the cost of local disk reads. |
Write a Delta table df.write.mode("overwrite").saveAsTable("sales25_enriched") | 20 s | none | Permanent; other notebooks and Power BI can read it | Columnar, native-engine friendly, and file skipping works on it. |
Illustrative times for 8 Medium executors. Caching pays off from the second use; a Delta table wins when other sessions also need the data. Call unpersist() when you are done with a cache.
Spark divides executor memory into reserved (JVM internals), user (your objects, UDF data structures), storage (cached blocks, broadcast tables) and execution (shuffles, joins, sorts, aggregations). Storage and execution share one pool set by spark.memory.fraction; execution can evict cached blocks down to spark.memory.storageFraction, but not the other way round.
The native engine keeps its columnar batches in off-heap memory, outside the JVM heap, so part of the node's memory is set aside for it. That is one reason Fabric sizes executor memory below the node size.
In the Spark UI, open a stage and look at Spill (Memory) and Spill (Disk) in the task summary. A large disk spill on a few tasks means skew; a large spill on every task means too few partitions. Driver out-of-memory errors almost always come from collect(), toPandas() or broadcasting something too large. Setting spark.task.cpus to 2 gives each task twice the memory, at the cost of parallelism.
Key takeaways
• Each task gets a slice of executor memory; partitions larger than that spill to disk.
• More, smaller partitions or bigger nodes fix most spills.
• Cache only what you reuse in the same session; write a table for anything else.
Check yourself: A partition is several times larger than the memory its task can use. What usually happens first?
Section 7
Delta Lake under the hood
A Lakehouse table is a folder of ordinary Parquet files plus a _delta_log folder. The log says which files make up the current version, and keeps statistics that let Spark skip files without opening them. How files are laid out decides how much a query or an update has to touch.
The min/max dates of most files fall outside 2025, so Spark never opens them.
30 days of appends
Appends per day
96
Rows per append
20,000
files in the table
11,520
without any fix
11,520
full scan time
35 s
with 1 GB files
3.6 s
Table size 1.7 GB, average file 154 KB. Spark counts at least 4 MB of “open cost” per file (spark.sql.files.openCostInBytes), and every file adds an entry to list and read in the log.
Model: each append writes from 4 tasks unless Optimize Write bins them into one file; about 120 bytes per row compressed. Illustrative.
MERGE INTO sales
Rows changed by the MERGE
100,000
Where the changed rows are
MERGE INTO sales t USING corrections c ON t.order_id = c.order_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *
files touched
1,600 of 1,600
data written
28 MB
copy-on-write
47 GB
With deletion vectors, Delta writes a tiny file marking the old rows in each touched file, plus the new rows. Unchanged rows stay where they are. Scattered changes touch a file for almost every row, so even a few thousand rows hit most of the table.
Model: 1,600 files of about 30 MB and 250,000 rows each. OPTIMIZE later rewrites files where over 5% of rows are marked deleted. Illustrative.
They solve different problems and combine. Z-order and liquid clustering decide which rows go into which file, so filters and merges touch fewer files. V-Order decides how a file is encoded inside (sorting, row-group sizes, dictionaries) so Power BI and SQL read it faster. Liquid clustering is the newer, incremental option for new tables; Z-order is applied by OPTIMIZE ... ZORDER BY.
Every commit leaves the old files in place so that time travel (SELECT ... VERSION AS OF 12) works. VACUUM deletes files no longer referenced by versions within the retention window (7 days by default). Until you vacuum, rewrites from OPTIMIZE and MERGE keep costing OneLake storage.
Every 10 commits Delta writes a checkpoint, a Parquet snapshot of the table state, so readers don't replay thousands of JSON files. DESCRIBE HISTORY sales shows each commit with metrics such as files added, removed and deletion vectors written.
Key takeaways
• A Delta table is Parquet files plus a log that records which files are current and their min/max stats.
• Many small files cost more than their size: compact with Optimize Write, auto compaction or OPTIMIZE.
• Deletion vectors make small updates cheap; clustering makes them touch fewer files.
Check yourself: A MERGE updates 1,000 rows spread over 500 files. What do deletion vectors avoid?
Section 8
Fabric's Native Execution Engine, settings and capacity
Fabric can run Spark's operators in a C++ engine (Velox, through Apache Gluten) that processes whole columns at a time. When an operator isn't supported, that piece falls back to the JVM, and data has to be converted between the two formats at each switch.
Native execution engine (spark.native.enabled)
Fabric runtime
Source format
What does the query contain?
Relative speed is an illustrative model. Microsoft reports up to 6× faster than open-source Spark on TPC-DS at scale factor 1000 on Delta (another Microsoft page cites up to 4× on TPC-DS 1 TB). Python UDFs run with Arrow batches from the native engine; Microsoft measured vectorized Python UDFs up to 5.76× faster, but scalar ones only about 1.08×.
read top to bottom
Scan sales
NativeFileScan
Filter 2025
FilterExecTransformer
Broadcast join with stores
BroadcastHashJoinExecTransformer
Project columns
ProjectExecTransformer
Partial sum
HashAggregateTransformer
Shuffle by region
ColumnarExchange
Final sum
HashAggregateTransformer
Fabric settings that change how your code runs
| Setting | What it does | When to use it | How to set it |
|---|---|---|---|
| Native execution engine | Runs supported operators in Velox (C++, columnar) | Almost always; check for fallbacks | Environment → Acceleration, or spark.native.enabled in %%configure |
| Autotune | Learns shuffle partitions, broadcast threshold and max partition bytes per query | Jobs that run repeatedly on similar data | spark.ms.autotune.enabled=true (off by default) |
| Resource profile | Presets for write-heavy or read-heavy workloads | writeHeavy (default) for ingestion; readHeavyForPBI for Power BI tables; readHeavyForSpark for Spark readers | spark.fabric.resourceProfile |
| V-Order | Sorts and compresses Parquet for Fabric readers | Direct Lake tables: 40-60% faster queries, about 10% for the SQL endpoint, little gain for Spark; writes about 15% slower on average (up to about 33%) | On in readHeavyForPBI; table or session property |
| Optimize write / auto compaction | Fewer, larger files at write time | Frequent small appends and merges | spark.databricks.delta.optimizeWrite.enabled, spark.databricks.delta.autoCompact.enabled |
| High concurrency mode | Several notebooks share one session | Many short notebooks, pipelines with notebook activities | Workspace settings → Data Engineering |
| Starter vs custom pool | Medium nodes, about 5 s start, vs your own sizes (3-5 min start unless kept live) | Starter for most work; custom for big shuffles or special libraries | Workspace settings or environment |
| %%configure | Driver and executor memory, cores, executor count | Session-level resources; must be the first cell | %%configure -f { "conf": { ... } } |
spark.conf.set only changes spark.sql.* settings at runtime. Executor and driver settings take effect only at session start.
Capacity: bursting, smoothing and Autoscale Billing
On a Fabric capacity, Spark gets 2 vCores per CU and may burst to about 3× that for a single job (more on the smallest SKUs). Fabric then smooths the usage over time: background jobs such as scheduled notebooks and pipelines over 24 hours, interactive runs over 5 to 64 minutes. Throttling starts only when smoothed usage stays above the capacity.
Capacity
Job size (vCores)
72
Job duration
30 min
Run type
Autoscale Billing for Spark (serverless, pay-as-you-go)
raw usage while the job runs
smoothed charge per 30-second timepoint
capacity size (dashed)
The job bursts far above the capacity while it runs, but the smoothed charge is a small slice of each timepoint, leaving room for other workloads.
Cost of starting small: a Python notebook uses 1 CU at its default 2 vCores; a single-node Spark session on an 8-vCore starter node uses 4 CU; a starter pool after its first scale-up uses 8 CU or more. Burst limits from Microsoft Learn's Spark concurrency table; smoothing windows from the Fabric throttling policy. Model is simplified.
Do you need Spark at all? Python kernel vs Spark kernel
| Python notebook (Python kernel) | Spark notebook (Spark kernel) | |
|---|---|---|
| Machine | One node: 2 vCores, 16 GB by default, configurable up to 64 vCores | Starter pool: 8-vCore nodes with autoscale; single-node mode available |
| Engines | DuckDB, Polars, pandas, scikit-learn preinstalled | Spark with the native engine |
| Languages | Python 3.10, 3.11 or 3.12 | PySpark, Spark SQL, Scala, SparkR (Python 3.13 on Runtime 2.0) |
| Start time | About 5 s | About 5 s on a starter pool; 3-5 min on a custom pool |
| Full Delta Lake features, V-Order | No | Yes |
| Good for | Data under about 1-2 GB, API calls, light transforms, orchestration | From about 10 GB, big joins and shuffles, Delta maintenance at scale |
| Capacity use | 1 CU at the default 2 vCores | Single node on 8 vCores: 4 CU; starter pool after scale-up: 8 CU or more |
Catalyst still does all the planning. Gluten then walks the physical plan, replaces each supported operator with a Transformer node, and hands that part of the plan to Velox (Meta's open-source C++ engine), which runs it on columnar batches with vectorized instructions and off-heap memory.
Where an operator or expression is unsupported, Gluten leaves it on the JVM and inserts conversion nodes on both sides. Runtime 2.0 enables cost-based fallback, which keeps a whole segment on the JVM when switching back and forth would cost more than it saves.
Current limits to know: Structured Streaming is not supported yet; JSON and XML reads are not accelerated (Runtime 2.0 adds a vectorized CSV reader); ANSI mode falls back on Runtime 1.3 but runs natively on Runtime 2.0; comparing a date or timestamp column with a value of a different type (for example a string) can stop the filter from being accelerated, so cast explicitly; and results of round() or the order inside collect_list can differ slightly. Python UDFs, Scala UDFs and complex types (arrays, maps, structs) are now supported; libraries needing arbitrary Python object serialization and deeply nested types may still fall back. Some older Microsoft best-practice pages still say UDFs always fall back, so check explain() on your own workload.
Key takeaways
• The native engine runs most operators in C++; check for fallbacks in explain() or Spark Advisor.
• Resource profiles and V-Order should follow who reads the table.
• Capacity is shared: bursting and smoothing absorb peaks, Autoscale Billing moves Spark off the capacity.
Check yourself: Which of these runs on the JVM instead of Fabric's native engine?
Section 9
Query playground: build the query, read the plan
Put the pieces together. Change how the example query is written and see the code, the simulated explain() output, the work Spark does and what it costs. It starts in a deliberately bad state; try to make it fast.
Language
Source
Filter
Columns
Size of stores
Custom logic on region
Native engine
Your code
from pyspark.sql import functions as F
@F.udf("string")
def clean(r):
return r.strip().upper()
sales25 = (spark.read.schema(sales_schema).json("Files/raw/sales_json/")
.select("store_id", "order_date", "amount")
.filter(F.year("order_date") == 2025))
result = (sales25
.join(spark.read.table("stores"), "store_id")
.withColumn("region", clean("region"))
.groupBy("region")
.agg(F.sum("amount").alias("revenue")))
result.write.mode("overwrite").saveAsTable("revenue_by_region")Simulated explain() output
native engine
JVM
boundary (conversion or Python)
accent = your query
data read
192 GB
best
1.7 GB
data shuffled
1.8 GB
best
31 KB
run time
30 s
best
2.5 s
Same model as Sections 3 and 4: 8 Medium executors plus a driver (72 vCores). Text formats are modeled as 2.5× (CSV) and 4× (JSON) the Parquet size. Illustrative, not a benchmark.
What Spark would tell you
Key takeaways
• Format, filter style, join size and UDFs change the plan far more than the language does.
• Every warning in the playground maps to a line you can see in explain().
• Cost follows time × vCores, so read less and shuffle less to save money.
Check yourself: What drives the capacity units a Spark job consumes?
Section 10
Choosing, avoiding pitfalls, and checking your work
Answer four questions to get a starting point for a new pipeline, then check it against the common mistakes below.
How much data per run?
Custom logic beyond SQL functions?
What does the team know best?
Who reads the output table?
Notebook and compute
Spark notebook on the starter pool (Medium nodes).
Language
PySpark with DataFrame functions. Keep heavy logic in built-in functions so it stays in the engine.
Custom logic
None needed. Built-in functions and SQL keep everything native and visible to the optimizer.
Settings
• Native execution engine on; check explain() for fallbacks.
• readHeavyForPBI resource profile (V-Order) for tables read through Direct Lake.
• Defaults (AQE on, 128 MB file splits) are usually fine.
Pitfalls checklist
| Pitfall | Why it hurts | Do this instead |
|---|---|---|
| collect() or toPandas() on a big DataFrame | Every row lands in the single driver's memory | Aggregate or limit first; write results to a table |
| Row-by-row Python UDFs | Every row is pickled to a Python worker and back; the optimizer can't see inside | Built-in functions, then pandas UDFs or useArrow=True |
| Reading JSON or CSV with schema inference | An extra pass over the data, and JSON isn't accelerated by the native engine | Give an explicit schema; land raw files into Delta once |
| Many small files | Thousands of tiny tasks and slow file listing | Optimize write, auto compaction, or a scheduled OPTIMIZE |
| year(col) = 2025 style filters | Hides the column from Delta file skipping | Filter on a range of the raw column |
| Comparing a timestamp column with a string literal | The native engine may not accelerate a filter whose two sides differ in type | Cast explicitly, e.g. CAST(order_date AS DATE) >= DATE '2025-01-01' |
| Shuffle partitions left at 200 for huge shuffles | AQE only merges partitions, so each task gets gigabytes and spills | Raise spark.sql.shuffle.partitions or enable autotune |
| Sessions left running | An idle session keeps billing its vCores until the 20-minute timeout | Stop the session, or use high concurrency and pipelines |
| Executor settings set with spark.conf.set | Silently ignored after the session has started | %%configure as the first cell, or the environment |
| Calling several actions on the same DataFrame | Each action re-runs the whole plan from the files | Write an intermediate table, or cache() and unpersist() |
How to check what Spark actually did
Before running: confirm filters appear under PushedFilters, only the needed columns are read, the join is a BroadcastHashJoin where expected, and operators end in Transformer.
After running: in the Stages tab, compare median and max task time to spot skew, and look at shuffle read and write sizes. The SQL tab shows rows per operator.
The notebook's Spark Advisor flags skew, native-engine fallbacks and common anti-patterns. The Monitoring hub keeps history for scheduled runs.
correct
0
missed
0
not answered
10
Unanswered: Sections 1, 2, 3, 4, 5, 6, 7, 8, 9, 10.
Key takeaways
• Start small: Python notebook or single-node Spark, then scale out.
• Avoid collect, row-by-row UDFs, schema inference on big files and small files.
• Verify with explain(), the Spark UI and Spark Advisor.
Check yourself: Your data is 500 MB and the logic is simple. What is a sensible starting point?
Glossary
%%configure
A first-cell magic that sets session resources such as driver and executor size.
Action
An operation that returns or writes a result (show, count, collect, write) and starts a job.
AQE (Adaptive Query Execution)
Re-planning between stages using real statistics: coalescing partitions, switching joins, splitting skew.
Autoscale Billing for Spark
Pay-as-you-go serverless Spark that doesn't consume the Fabric capacity.
Broadcast hash join
Copying a small table to every executor so the large table doesn't move.
Bursting and smoothing
Using up to 3× the capacity's vCores now, with the usage averaged over later time windows.
Cache / persist
Keeping a computed DataFrame in executor memory or disk for reuse within the session.
Catalyst
Spark's optimizer. It rewrites the plan with rules such as pushdown and pruning.
Column pruning
Reading only the columns the query needs from a columnar format.
CU (capacity unit)
Fabric's unit of compute. One CU equals 2 Spark vCores.
Deletion vector
A small file marking rows as deleted so a data file doesn't have to be rewritten.
Delta log (_delta_log)
The JSON commits and checkpoints that define which Parquet files make up each table version.
Driver
The process that holds the plan, talks to the catalog and schedules tasks. Runs on its own node.
Executor
A worker process on a node that runs tasks. Fabric runs exactly one executor per node.
Fallback
An operator the native engine can't run, executed on the JVM with format conversions around it.
File skipping (data skipping)
Using per-file min/max statistics in the Delta log to avoid opening files.
High concurrency mode
Several notebooks sharing one Spark session to save start-up time and capacity.
Job
All the work started by one action.
Liquid clustering / Z-order
Ways to co-locate rows with similar key values so filters and merges touch fewer files.
Logical plan
What to compute, as a tree of relational steps, before any algorithm is chosen.
Native Execution Engine (NEE)
Fabric's C++ engine (Velox through Apache Gluten) that runs supported operators on columnar batches.
OPTIMIZE
A Delta command that compacts small files into large ones, optionally with Z-order or V-Order.
pandas UDF / Arrow
A vectorized UDF that receives Arrow column batches as pandas Series instead of single rows.
Partition
A slice of the data. Reads use about 128 MB per partition; shuffles start with 200.
Physical plan
How to compute it: the concrete operators and join algorithms. Printed by explain().
Predicate pushdown
Handing filters to the file reader so it can skip row groups.
Py4J
The bridge PySpark uses to call the JVM driver from Python.
Resource profile
A preset of Spark and Delta settings: writeHeavy (default), readHeavyForSpark, readHeavyForPBI.
Shuffle
Regrouping data by key across executors over the network. Usually the most expensive step.
Skew
One key or partition holding far more data than the others, creating a straggler task.
Sort-merge join
Shuffling both tables by the join key, sorting, then merging. Safe for two large tables.
Spill
Writing intermediate data to local disk when a task runs out of execution memory.
Stage
Part of a job that runs without moving data between machines. Stages are cut at shuffles.
Starter pool
Pre-warmed Medium nodes that start a session in about 5 seconds.
Task
One unit of work: one stage applied to one partition, run by one core.
Transformation
An operation that returns a new DataFrame and only extends the plan (filter, join, groupBy).
UDF
A user-defined function. Python UDFs run in Python worker processes next to each executor.
V-Order
A Fabric write-time Parquet layout that speeds up Power BI Direct Lake and SQL reads.
Wave
One round of tasks when there are more tasks than cores.
Sources
Fabric facts are from Microsoft Learn, re-checked on November 5, 2026. The $0.18 per CU-hour default is the East US pay-as-you-go list price (October 2026). Where Microsoft's pages disagree (for example on native-engine UDF support, or 4× versus 6× speed-ups), the canvas follows the newer, more specific page and says so. Numbers in the simulators are illustrative models, not benchmarks.