Key takeaways
- Ban whole-entity driver InsertAll above documented row or byte thresholds.
- Make batch size and concurrency explicit, tunable, and documented per entity.
- Log per-batch id, rows, duration, and success or fail as structured logs.
- Support entity and key-range filters so reruns stay finite.
- Keep rebuild and incremental as separate modes with separate entry points.
- Ship the pattern to prod only with batch-level observability and a runbook.
Whole-table InsertAll on one driver is a known failure mode for large tables. The notebook looks simple. The driver dies. Someone asks for a bigger cluster. The next reload dies later, with a larger bill.
Treat bulk reload as orchestration: batched parallel page ingest, progress monitoring, and entity filters for operable reruns. Separate one-time rebuild paths from incremental watermarks. Publish the pattern to prod only after batch-level observability exists. OOM here is a design smell in notebook API usage, not proof that "Fabric is flaky."
Named model: batch → progress → filter → observe
- Batch: page or key-range writes with explicit size and concurrency; no whole-entity driver collect above threshold.
- Progress: structured per-batch logs (id, rows, duration, success or fail) so "is it stuck?" becomes a number.
- Filter: entity, partition, or key-range recovery without replaying the whole load.
- Observe: batch-level metrics and a runbook are the ship gate before prod.
- Contract: throughput for large tables comes from controlled batches and visibility, not from hoping the driver can materialize everything at once.
Figure 1. Batch, progress, filter, observe. Bigger VMs are optional. Clear batch boundaries are not.
Context and constraint
A common pattern: full reloads work for small tables and collapse on the big ones. The API and Spark were fine. The driver was asked to hold a universe. Once we batched with progress and filters, reloads became dull operations instead of weekend heroes.
Parallelism without batch boundaries still OOMs; it just fails faster. Progress monitoring turns "is it stuck?" into a number. Entity filters let you rerun the failed slice without replaying the whole load. Incremental watermarks are not a rebuild strategy; keep the paths separate. Observability is a ship gate for the pattern, not a nice-to-have dashboard later.
Figure 2 is the Learn framing for copy work inside a pipeline. Whole-table InsertAll on one driver is an orchestration smell. Batch ingest belongs in an operable pipeline.
Figure 2. Whole-table InsertAll on one driver is an orchestration smell. Batch ingest belongs in an operable pipeline. Source: Microsoft Learn: copy data activity.
Figure 3 is the Learn framing for promoting a scheduled run. Batch with progress and entity filters, then promote the pattern like any other scheduled pipeline.
Figure 3. Batch with progress and entity filters, then promote the pattern like any other scheduled pipeline. Source: Microsoft Learn: pipeline overview.
Design
1. Ban whole-entity driver collects for large tables
If the pattern requires the driver to assemble the full entity before write, it will fail at scale. Design for page or key-range batches that write incrementally. Simplicity that only works on small tables is a trap. Figure 4 is the ownership split that keeps memory thin.
Figure 4. Same API. Same Spark. Different ownership of memory. Ban whole-entity driver collects above documented thresholds.
2. Batch with explicit size and concurrency knobs
Choose batch sizes you can tune per entity. Cap concurrency so you do not DDoS the source or your own capacity. Document defaults and the dials operators may turn. Magic numbers buried in a cell become folklore under stress.
3. Emit progress per batch
Log batch id, row counts, duration, and success or fail. Operators need a trail when a long reload stops partway. Progress also feeds SLAs and capacity planning. Silent loops are not enterprise ingestion.
4. Entity filters for operable reruns
Support rerunning one partition, one entity, or one key range without truncating everything. Full restart as the only recovery path guarantees fatigue and shortcuts. Filters turn incidents into finite work.
5. Separate rebuild from incremental
Watermark-based incremental paths optimize for steady state. Rebuild paths optimize for correctness after schema or logic changes. Mixing them produces half-applied states that are hard to explain. Name the modes in the notebook and in the pipeline. Figure 5 is the split plus the ship gate.
Figure 5. Two modes. Two entry points. One ship gate: batch-level observability. Mixing modes produces half-applied states that are hard to explain.
6. Require batch-level observability before prod
Ship to production when you can answer which batch failed and why, not when a happy-path demo finished once. Metrics and structured logs are part of the ingest contract. Without them, every OOM becomes a platform complaint.
What good batch logging looks like
Each batch should leave a row or log line you could build a small table from: entity, filter, batch key, attempt, rows in, rows written, duration, error class. Aggregate for a run summary. On failure, operators replay from the failed batch key, not from zero, unless integrity requires a clean rebuild. If integrity requires clean rebuild, say so in the runbook. Ambiguous recovery is how two engineers choose opposite strategies at 2 a.m.
Source courtesy is part of reliability
Source APIs have throttling realities. Batch design must respect them: backoff, concurrency caps, and clear error classes for throttle versus hard fail. Treating throttle as a generic Spark error trains people to enlarge clusters. Treating it as a first-class signal trains people to tune the orchestrator. Put the courtesy rules next to the batch knobs so "go faster" has a documented ceiling.
Thresholds belong in the ADR, not in memory
Define approximate row or byte thresholds where driver-collect is forbidden. Below the line, simple paths may remain for tiny entities. Above the line, batched orchestration is mandatory. Update thresholds when you learn; do not pretend every entity is the same size forever. The point of a number is to end the argument mid-incident about whether "this one" qualifies as large.
Failure modes
Bigger VM as architecture. Cost up, failure delayed.
Unbounded parallelism. Source throttling and cluster chaos.
No progress. Humans cancel jobs that were still working (or babysit forever).
Rerun-all-only recovery. Engineers skip reloads; drift grows.
Incremental pretending to be rebuild. Missing history, unclear watermarks.
Prod pattern without batch metrics. Support cannot localize failure.
What to copy / what not to copy
Copy
- Large-table full loads use batched writes; driver-wide InsertAll is forbidden above documented thresholds.
- Batch size and concurrency are explicit, tunable, and documented per entity.
- Progress and per-batch outcomes are logged as structured logs.
- Rerun filters exist for entity and key-range recovery.
- Rebuild and incremental are separate modes with separate entry points.
- Pattern reaches prod only with batch-level observability and a runbook.
Do not copy blindly
Batch orchestration adds code and knobs. Driver-collect simplicity fails at large-table scale. Lower concurrency is slower and safer; tune with metrics, not pride. Separate rebuild paths duplicate some logic; duplicated modes beat corrupted modes. Observability costs a day or two up front and saves weeks of incident archaeology.
Primary references: Fabric Spark job monitoring and notebook guidance on Microsoft Learn, plus source-system API throttling docs for your source connectors. Pair them with a runbook that names rebuild versus incremental. Start from copy data activity and pipeline overview.
Batch the work. Show progress. Filter reruns. Keep rebuild honest. Observe batches before you call the pattern production. Bigger VMs are optional. Clear batch boundaries are not.