2 posts tagged with "Spark"

Materialized Lake View runs paste warning floods that mention native execution engine fallbacks. Teams panic-rewrite Spark SQL to chase every line. Meanwhile the DAG's wall-clock still hides in one slow node nobody measured.

Measure first. Pasted warnings include Delta and MLV metadata internals you cannot and need not remove. Wall-clock attribution beats fear-driven rewrites. Fix the few fallbacks that move latency. Ignore the rest deliberately. Document accepted warnings so on-call does not thrash. Pair with per-view notebooks so slow nodes are isolatable.

Figure 1. Attribute MLV time with job details and DAG reality. Do not treat every NEE warning as a rewrite mandate. Source: Microsoft Learn: Spark detail monitoring.

InsertAll of multi-million ledger rows on one driver is a known failure mode. The notebook looks simple. The driver dies. Someone asks for a bigger cluster. The next reload dies later, with a larger bill.

Treat bulk reload as orchestration: batched parallel page ingest, progress monitoring, and entity filters for operable reruns. Separate one-time rebuild paths from incremental watermarks. Publish the pattern to prod only after batch-level observability exists. OOM here is a design smell in notebook API usage, not proof that "Fabric is flaky."

Figure 1. Multi-million InsertAll on one driver is an orchestration smell. Batch ingest belongs in an operable pipeline. Source: Microsoft Learn: copy data activity.

Figure 2. Batch with progress and entity filters, then promote the pattern like any other scheduled pipeline. Source: Microsoft Learn: pipeline overview.