How to choose an AI model in Microsoft Foundry

Define the job before browsing a catalog

Write a short task contract: input types, required output, acceptable errors, latency target, throughput, context size, tool use, deployment region, data handling requirements, and cost per successful task. Include the user's fallback experience when the model is uncertain or unavailable.

Separate hard constraints from preferences. A model that lacks a required modality, deployment option, or region is not a candidate, regardless of a benchmark score.

Build a representative evaluation set

Create a small, reviewed set of real-shaped examples, including common requests, edge cases, ambiguous instructions, missing evidence, and safety-sensitive inputs. Define expected outcomes before running the comparison. Use the same prompts, retrieval context, output schema, and scoring approach for every candidate.

Score task success and critical failure rates alongside answer quality. For grounded systems, inspect whether claims are supported by retrieved evidence. For tools, test selection, arguments, authorization, and recovery rather than grading only the final prose.

• Quality: correctness, completeness, groundedness, format adherence
• Safety: policy violations, unsafe completion, unwanted disclosure
• Operations: p50/p95 latency, timeout rate, throughput, and availability
• Economics: input/output usage and total cost per accepted task
• Repeatability: model/version, deployment, region, date, and test configuration

Compare deployment fit, not just model names

Microsoft Foundry's catalog includes different model providers and deployment options; capabilities, terms, availability, and billing vary. Confirm the exact model card, license or terms, deployment type, region, quota, and support status for the model you will use. Preview or partner offerings may have different limitations from generally available Azure-sold models.

Treat this check as time-sensitive. Model lifecycle, capacity, supported features, and regional availability can change after an evaluation. Record the date and rerun tests when upgrading or migrating deployments.

Make the choice explainable

Use a decision table that records hard constraints, evaluation results, operational fit, and unresolved risks. Select the least complex and lowest-cost candidate that clears your quality and safety thresholds with adequate operational headroom. Do not select solely on a public leaderboard or one prompt demo.

For many products, routing can be useful: send routine tasks to a fast, lower-cost option and escalate only well-defined hard cases. Keep the router measurable and ensure fallback behavior does not bypass the same privacy, safety, and authorization requirements.

Re-evaluate after change

Model selection is a lifecycle decision. Re-run the regression set when the model version, prompt, retrieval pipeline, tool schema, deployment type, region, or traffic profile changes. Monitor quality and cost in production using privacy-appropriate telemetry and an approved review process.

Keep a rollback path and avoid moving a critical workload to a new model based only on availability. Preserve enough evaluation evidence to explain why the current choice remains acceptable.

Frequently asked questions