Bulk reload is an orchestration problem, not a bigger VM wish
InsertAll of multi-million ledger rows on one driver is a known failure mode. The notebook looks simple. The driver dies. Someone asks for a bigger cluster. The next reload dies later, with a larger bill.
Treat bulk reload as orchestration: batched parallel page ingest, progress monitoring, and entity filters for operable reruns. Separate one-time rebuild paths from incremental watermarks. Publish the pattern to prod only after batch-level observability exists. OOM here is a design smell in notebook API usage, not proof that "Fabric is flaky."
Figure 1. Multi-million InsertAll on one driver is an orchestration smell. Batch ingest belongs in an operable pipeline. Source: Microsoft Learn: copy data activity.
Figure 2. Batch with progress and entity filters, then promote the pattern like any other scheduled pipeline. Source: Microsoft Learn: pipeline overview.
I learned this on full ledger reloads that worked for mid-size entities and collapsed on the big ones. The API and Spark were fine. The driver was asked to hold a universe. Once we batched with progress and filters, reloads became dull operations instead of weekend heroes.
The core idea
Throughput for large entities comes from controlled batches and visibility, not from hoping the driver can materialize everything at once.
Parallelism without batch boundaries still OOMs; it just fails faster. Progress monitoring turns "is it stuck?" into a number. Entity filters let you rerun the failed slice without replaying the estate. Incremental watermarks are not a rebuild strategy; keep the paths separate. Observability is a ship gate for the pattern, not a nice-to-have dashboard later.
A model that stays explainable
1. Ban whole-entity driver collects for large ledgers
If the pattern requires the driver to assemble the full entity before write, it will fail at scale. Design for page or key-range batches that write incrementally. Simplicity that only works on small tenants is a trap.
2. Batch with explicit size and concurrency knobs
Choose batch sizes you can tune per entity. Cap concurrency so you do not DDoS the source or your own capacity. Document defaults and the dials operators may turn. Magic numbers buried in a cell become folklore under stress.
3. Emit progress per batch
Log batch id, row counts, duration, and success/fail. Operators need a trail when a six-hour reload stops at 70%. Progress also feeds SLAs and capacity planning. Silent loops are not enterprise ingestion.
4. Entity filters for operable reruns
Support rerunning one company, one entity, or one key range without truncating everything. Full restart as the only recovery path guarantees fatigue and shortcuts. Filters turn incidents into finite work.
5. Separate rebuild from incremental
Watermark-based incremental paths optimize for steady state. Rebuild paths optimize for correctness after schema or logic changes. Mixing them produces half-applied states that are hard to explain. Name the modes in the notebook and in the pipeline.
6. Require batch-level observability before prod
Ship to production when you can answer which batch failed and why, not when a happy-path demo finished once. Metrics and structured logs are part of the ingest contract. Without them, every OOM becomes a platform complaint.
Failure modes I design against
Bigger VM as architecture. Cost up, failure delayed.
Unbounded parallelism. Source throttling and cluster chaos.
No progress. Humans cancel jobs that were still working (or babysit forever).
Rerun-all-only recovery. Engineers skip reloads; drift grows.
Incremental pretending to be rebuild. Missing history, unclear watermarks.
Prod pattern without batch metrics. Support cannot localize failure.
What good batch telemetry looks like
Each batch should leave a row or log line you could build a small table from: entity, filter, batch key, attempt, rows in, rows written, duration, error class. Aggregate for a run summary. On failure, operators replay from the failed batch key, not from zero, unless integrity requires a clean rebuild. If integrity requires clean rebuild, say so in the runbook. Ambiguous recovery is how two engineers choose opposite strategies at 2 a.m.
Source courtesy is part of reliability
Business Central and similar APIs have throttling realities. Batch design must respect them: backoff, concurrency caps, and clear error classes for throttle versus hard fail. Treating throttle as a generic Spark error trains people to enlarge clusters. Treating it as a first-class signal trains people to tune the orchestrator. Put the courtesy rules next to the batch knobs so "go faster" has a documented ceiling.
Thresholds belong in the ADR, not in memory
Define approximate row or byte thresholds where driver-collect is forbidden. Below the line, simple paths may remain for tiny entities. Above the line, batched orchestration is mandatory. Update thresholds when you learn; do not pretend every entity is the same size forever. The point of a number is to end the argument mid-incident about whether "this one" qualifies as large.
Trade-offs
Batch orchestration adds code and knobs. Driver-collect simplicity fails at ledger scale. Lower concurrency is slower and safer; tune with metrics, not pride. Separate rebuild paths duplicate some logic; duplicated modes beat corrupted modes. Observability costs a day or two up front and saves weeks of incident archaeology.
What I would put on an ADR
- Large ledger full loads use batched writes; driver-wide InsertAll is forbidden above documented thresholds.
- Batch size and concurrency are explicit, tunable, and documented per entity.
- Progress and per-batch outcomes are logged as structured telemetry.
- Rerun filters exist for entity/key-range recovery.
- Rebuild and incremental are separate modes with separate entry points.
- Pattern reaches prod only with batch-level observability and a runbook.
Primary references: Fabric Spark job monitoring and notebook guidance on Microsoft Learn, plus source-system API throttling docs for your ledger connectors. Pair them with a runbook that names rebuild versus incremental.
Closing
OOM on bulk reload is usually not a mysterious platform mood. It is an orchestration gap.
Batch the work. Show progress. Filter reruns. Keep rebuild honest. Observe batches before you call the pattern production. Bigger VMs are optional. Clear batch boundaries are not.