Apache Airflow is a good fit when you want clear scheduling, dependency control, retries, and visibility across the whole pipeline. A DAG gives you a readable contract for how a data flow should behave, which makes it easier to debug failures and easier to explain the system to other engineers.

The practical pattern is simple: keep the workload logic in Fabric, keep orchestration in Airflow, and make each task do a single thing well. That separation keeps your pipelines easier to test, easier to recover, and much easier to extend when new datasets or downstream consumers appear.

Where Airflow fits​

Use Airflow when you need more than a scheduled notebook or a manual run button. It helps when one job depends on another, when a failure should retry automatically, or when multiple pieces of the workflow need a shared execution model.

Fabric still handles the data work itself. Notebooks, pipelines, lakehouse tasks, and warehouse operations stay where they belong. Airflow coordinates them. That division keeps the orchestration layer lightweight and keeps the business logic closer to the platform that executes it.

A practical structure​

One useful pattern is:

• ingest raw data into a landing zone
• validate the structure and basic quality rules
• transform the data in Fabric notebooks or pipelines
• refresh semantic models or downstream outputs
• notify on success or failure

Each task should do one thing clearly. If a task grows too large, it becomes harder to rerun and harder to diagnose when something breaks. Smaller tasks create cleaner retries and make partial recovery much more manageable.

What to watch for​

The most common mistakes are not technical, they are architectural. Teams often try to make the orchestrator do too much, duplicate business logic across tools, or hide failures behind a single broad step.

That approach makes the system fragile. A better design is to keep the orchestration thin and explicit. When each step is visible, the entire flow is easier to reason about and easier to change later.

Why this pattern works​

Airflow is good at timing, dependency management, and operational control. Fabric is good at data processing inside Microsoft’s analytics stack. Combining them gives you a practical system where each tool does the part it is best at.

That is the real value of the pattern: not adding more tools, but reducing friction between scheduling, execution, and maintenance.

Key idea: use Airflow to coordinate the workflow, not to replace the logic already living inside Fabric.

Related reading​

• Image Classification Web App with Azure AI Services for another example of keeping platform logic and product logic separated.
• Can Databricks Genie Build My Deck and Deploy My App? for a look at AI-assisted platform workflows in practice.

Read the original LinkedIn post

A good implementation balances three concerns: getting frames into the pipeline efficiently, running inference reliably, and presenting the results in a way that is easy to understand. When the visualization is clear, the technical work underneath becomes easier to trust and easier to demo.

The most useful lesson is that real-time systems are shaped by tradeoffs. You will usually need to simplify the pipeline, reduce unnecessary processing, and measure performance in the same environment where the app will actually run.

What pose estimation really needs​

The model is only part of the product. A real-time experience also depends on frame capture, buffering, rendering, and how much delay the user can tolerate. If any one of those pieces lags, the result feels unreliable even if the model is technically accurate.

That is why the entire pipeline matters. You are not just detecting keypoints. You are building a system that has to process a continuous stream and remain responsive under changing conditions.

Practical implementation choices​

The most useful approach is to keep the pipeline lean:

• resize frames only as much as needed
• avoid redundant preprocessing
• keep inference and visualization separate
• measure end-to-end latency, not just model latency

This makes debugging simpler and helps you understand where time is actually being spent. Often the slowest part is not the model itself but the glue around it.

Why live demos fail​

Real-time demos often fail because they are built for the ideal case instead of the messy one. A smooth demo needs stable frame timing, predictable inference behavior, and a visual output that remains readable when confidence fluctuates.

The better engineering choice is usually the boring one: smaller inputs, fewer moving parts, and a UI that favors clarity over effects.

What to take away​

Pose estimation is a good reminder that AI systems are not just model exercises. They are product exercises. The model has to fit the workflow, and the workflow has to fit the user.

Key idea: real-time AI is a systems problem as much as it is a model problem.

Related reading​

• Image Classification Web App with Azure AI Services for a complementary look at production AI feature design.
• Automate Your Microsoft Fabric Workloads with Apache Airflow for orchestration patterns that keep AI systems organized.

Read the original LinkedIn post

The useful pattern is to keep the user interface simple and let the AI service do the heavy lifting. Users upload an image, the app sends it to the model, and the result comes back with a clear label and confidence score. That makes the experience understandable and easy to validate.

The bigger architectural lesson is that production-ready AI features still need the basics: input validation, request limits, logging, and a path to iterate on the model when the data changes. A good demo becomes a good product when it keeps those fundamentals in place.

Start with the user flow​

The first step is not the model, it is the journey. A clean interface should make the action obvious: choose an image, submit it, and see a result. If the flow is complicated, the AI feature feels harder than it really is.

That is why a minimal layout works well. It keeps the focus on the core interaction and avoids distracting people with unnecessary controls.

Keep the backend predictable​

Once the request reaches Azure AI Services, the application should behave predictably. The service call should be wrapped in error handling, and the app should be ready for cases where confidence is low or the response is slower than expected.

In practice, that means:

• validating file type and size before upload
• returning a clear error if inference fails
• showing confidence and labels in plain language
• logging enough detail to debug bad results later

Those basics are what turn a demo into something you can actually operate.

Why this pattern is useful​

Custom Vision is especially useful when you want to teach a model a specific domain. It gives you a fast path from labeled examples to a working classifier without having to build everything from first principles.

That makes it a good fit for internal tools, proof-of-concepts, and product experiments that need to move quickly without sacrificing structure.

What matters most​

The best AI applications are not the most complex. They are the ones that feel dependable and easy to use. Keep the interface obvious, keep the model feedback clear, and keep the system simple enough that it can be maintained.

Key idea: make the app simple, then make the AI dependable.

Related reading​

• Real-time Motion Tracking and Pose Estimation for another production-minded computer vision pattern.
• Can Databricks Genie Build My Deck and Deploy My App? for a broader look at AI tooling inside a data platform.

Read the original LinkedIn post

I have been stress-testing the new Databricks Genie Code, and it is setting a new bar for AI assistants in enterprise data platform technology.

Most of the buzz around Genie Code focuses on handling data questions or building ETL pipelines. But I wanted to push it further: how does it actually perform from a day-to-day developer and business user's perspective?

After putting it through its paces, Genie Code gives off more Claude Code and Cursor vibes than Copilot, but with enterprise data security. Because it lives natively within Databricks, it already has deep context regarding your enterprise data, governance, and platform architecture, meaning less waste of extra tokens or struggling with complex context windows.

Two use cases that stood out​

1. Making a PowerPoint heavily based on data insights. I connected Genie Code with an open-source PPTX skill from Anthropic (the same one used for Claude Code) to create a PowerPoint based on data in Databricks. Instead of just returning insights in chat, it autonomously generated a beautiful 9-slide PowerPoint deck complete with charts and graphs, and the best part is every element is editable. It is a massive time-saver that turns raw data directly into boardroom-ready presentations for monthly or quarterly meetings.

Bonus: If you customize the skill, it can decrypt your company template, then pick and replace the most suitable slides for the related data.

2. Web app development and deployment in Databricks. This is definitely possible with many IDEs like Cursor or Claude Code today using the Databricks MCP, but I still have to reference or point the agent in the right direction to deploy a web app. Genie Code, however, already has the context of your platform. So, not only did it write the web app code, but it also reasoned through platform constraints in real time. When it realized the Databricks CLI was not available on the specific serverless compute I was using, it seamlessly pivoted and successfully deployed the live dashboard to azure.databricksapps.com.

Seeing this in action makes Databricks' internal benchmarks make a lot of sense. They recently reported Genie Code hitting a 77.1% success rate on real-world tasks, compared to just 32.1% for leading coding agents using MCP servers.

Context is everything, and having an AI that actually knows your platform inside and out changes the game.

Related reading​

• Automate Your Microsoft Fabric Workloads with Apache Airflow for another orchestration-heavy workflow pattern.
• Image Classification Web App with Azure AI Services for a practical example of shipping an AI feature into a web app.

Read more​

• Genie Code: https://lnkd.in/e_rKqdsP
• Anthropic skills: https://lnkd.in/eHMcHuJp
• Extend Genie Code with agent skills: https://lnkd.in/eVEj77tU

This also leaves a question for Copilot in Microsoft Fabric: Agent mode that has context of my Fabric items and data soon?

#Databricks #Genie #DatabricksGenieCode #AI