Classification apps fail on operations, not on the label list
Custom Vision can get you from labeled images to a prediction endpoint quickly. The product work starts after the happy path: bad uploads, low-confidence answers, iteration that breaks the published endpoint, and no breadcrumbs when the label is wrong.
Keep the UI boring: upload, predict, show label and confidence. Put the complexity in data quality, validation, request limits, logging, and a controlled publish path for new iterations.
Figure 1. Labels and confidence are the user contract. The training UI is where iteration happens; the web app should not invent a second story. Source: Microsoft Learn: Custom Vision overview.
I have seen too many vision demos that look polished until the first bad upload, the first low-confidence answer, or the first model iteration that breaks the published endpoint. A good demo becomes a product when input validation, request limits, logging, and a path to retrain stay in place from day one.
The core idea
Make the app simple. Make the AI dependable. Keep model iteration a controlled change, not a surprise.
Custom Vision (or the equivalent Azure image classification path you choose) should own prediction. The web app should own upload hygiene, auth if you need it, presentation of labels and confidence, and the operational breadcrumbs that let you debug wrong answers later. If the UI tries to be clever, people stop trusting the score. If the backend skips basics, you cannot operate the cleverness you shipped.
Figure 2. Iteration lives in the project surface. Treat published endpoints as release artifacts, not as a casual overwrite. Source: Microsoft Learn: Custom Vision overview.
A model that stays explainable
1. Start with the user flow, not the model card
The journey should be obvious: choose an image, submit, see a result. Minimal layout keeps focus on the core interaction. Extra toggles, advanced panels, and decorative charts can wait until the happy path is boringly reliable. If the flow is complicated, the AI feature feels harder than it is.
2. Validate before you spend an inference call
Check file type, size, and basic image decodability on the server. Reject clearly. Do not stream arbitrary blobs to the prediction endpoint and hope. Validation protects cost, protects the service from garbage, and gives users a faster failure than a timeout wrapped as something went wrong.
3. Wrap prediction in boring, predictable backend behavior
Timeouts, retries for transient faults, and mapped error responses belong in the API layer. Return plain-language failures when inference fails. Show label and confidence in plain language when it succeeds. Low confidence is a first-class outcome: say so in the UI instead of pretending the top label is gospel.
4. Log enough to debug bad results
Store correlation ids, model name or iteration id, latency, top labels, and whether the user accepted or corrected the result if you collect that. You do not need the raw image forever in every environment, but you need a path to reproduce class of failures. Without iteration ids in logs, root cause becomes folklore after the next publish.
5. Treat Custom Vision iteration as a release
New tags, new training images, and a newly published iteration change product behavior. Promote iterations the way you promote app builds: note what changed, smoke-test a fixed image set, and keep a rollback to the previous published iteration. Domain-specific classifiers drift when the world changes. Plan for that drift instead of discovering it in a stakeholder meeting.
6. Keep secrets and keys out of the browser
Prediction keys and training keys belong on the server or in managed identity patterns, not in frontend bundles. A static page that calls Custom Vision directly is a short path to key leakage and uncontrolled quota burn. The browser uploads to your app; your app calls Azure.
Failure modes I design against
Demo happy path only. PNG of a clear object on a white background. Real users upload HEIC phone photos, screenshots, and five-megabyte scans. The app chokes or the model shrugs.
Confidence ignored. Always show the top label in bold. Users trust 0.41 the same as 0.97. That is how wrong operational decisions get a nice UI.
Keys in the client. Someone views source, copies the prediction key, and your bill explains itself next month.
No iteration discipline. Retrain in the portal, publish, and hope. Yesterday's smoke images now misclassify and nobody can say which publish did it.
Logging without model identity. You know the request failed or looked weird. You do not know which iteration served it. Support becomes archaeology.
Unbounded uploads. No size limit, no rate limit, no auth on an internal tool that somehow became external. Inference cost and abuse arrive together.
Trade-offs
Custom Vision is fast to stand up for a narrow domain. It is not a substitute for a carefully engineered vision stack when you need detection, segmentation, or strict latency on device. Choose it when labeled examples and a classification answer are enough for the product decision.
A thin web app is easy to explain and easy to replace. A thicker product shell (auth, history, human review queue) takes longer but is what internal tools often need after week two. Do not pretend the upload page is the whole system if reviewers must correct labels for retraining.
Hosting the model via Azure versus exporting for edge runtimes changes your update story and your offline story. Pick based on where images are allowed to travel, not based on which quickstart you pasted first.
What I would put on an ADR
- Server-side validation for type, size, and decodability before prediction.
- Prediction credentials only on the backend; browser talks to the app API.
- UI always shows label plus confidence; low-confidence threshold is product-defined.
- Logs include correlation id, iteration or model id, latency, and top-N labels.
- Published iterations are treated as releases with a smoke set and rollback.
- Rate limits and auth posture documented for internal versus public exposure.
- Retraining ownership and data retention rules written before the first production upload.
Primary references: Microsoft Learn on Custom Vision, image classification, and Azure AI / AI Services authentication patterns for keeping keys off the client.
Closing
Image classification apps fail in boring ways. Not because the tag list was wrong in the portal, but because we shipped a demo upload button and called validation, logging, and iteration hygiene optional.
If your architecture diagram has one box labeled Custom Vision, redraw it.
Upload. Predict. Explain. Iterate. Four jobs. One score the user can actually trust.