AWS and NVIDIA published a joint technical architecture and open-source AWS CDK solution on November 13, 2026 for routing coding-agent requests across models on Amazon Bedrock using LiteLLM and the NVIDIA NeMo Switchyard plugin. Co-authored by AWS specialists Aswath Ram A Srinivasan, Anshika Saxena, and Aris Tsakpinis alongside NVIDIA architects and product leaders Curt Lockhart, Ian Cairns, and Kevon Mayers, the guide and sample repository show engineering teams how to deploy a tool-aware model router on Amazon ECS on AWS Fargate and evaluate routing using cost per solved task.
- Published on November 13, 2026, the AWS and NVIDIA technical guide provides a sample AWS CDK stack (
sample-route-coding-agent-requests-Amazon-Bedrock-nemo-switchyard) that deploys LiteLLM (v1.102.0) with the experimental NVIDIA NeMo Switchyard routing plugin (switchyard-server 0.3.0) on Amazon ECS on AWS Fargate. - Instead of optimizing for per-token rates alone, the architecture evaluates coding agents—such as Claude Code and OpenAI Codex CLI—by cost per solved task: total run spend divided by the number of runs whose unit tests pass.
- Switchyard's LiteLLM integration supports two zero-judge routing strategies: Stage routing (which inspects recent tool results in the conversation history to pick between an efficient model and a capable model) and Random routing (weighted traffic splits for A/B testing).
- In the reference profile tested in US East (N. Virginia), Stage routing pairs
nvidia.nemotron-super-3-120bas the efficient default candidate withglobal.anthropic.claude-opus-5-5as the capable escalation candidate over the Amazon Bedrock Converse API.
1. Why do AWS and NVIDIA measure coding agents by cost per solved task?
Coding agents such as Claude Code and OpenAI Codex CLI execute multi-turn tasks—reading files, editing code, and running tests across dozens of calls. Because total spend depends on turn count, replayed history, and failed attempts, AWS and NVIDIA recommend evaluating model routing by cost per solved task: total run cost divided by runs whose unit tests pass.
In their November 13, 2026 AWS Machine Learning Blog post, the authors explain that price per token alone does not reflect what a full task costs because every turn replays growing history and teams pay for failed runs. By deploying a routed model group (switchyard) alongside two single-model baseline groups (claude-opus-5-5 and nemotron-super-3-120b), teams can compare how many tasks each group solves (SOLVED) and its cost per solved task (COST/SOLVED).
2. How does NVIDIA NeMo Switchyard route requests between models on Amazon Bedrock?
NVIDIA NeMo Switchyard is an open-source routing library whose LiteLLM plugin selects a model for each request without calling an external judge model. Its Stage strategy inspects recent tool activity and execution results in the conversation history to choose between an efficient model and a capable model, while Random routing splits traffic by weight.
In the sample configuration (profiles/stage/litellm.yaml and profiles/stage/switchyard.toml), both candidates share model_name: switchyard via the Amazon Bedrock Converse API, listing the capable candidate first (bedrock/converse/global.anthropic.claude-opus-5-5) and the efficient candidate second (bedrock/converse/nvidia.nemotron-super-3-120b). Stage routing uses four settings:
picker = "efficient_first": Selectsnvidia.nemotron-super-3-120bwhen history signals are inconclusive or when a new request has no tool results yet.recent_window = 3: Inspects the three most recent tool results in the conversation history for execution signals.confidence_threshold = 0.5: Sets the confidence cutoff (0 through 1) for scoring execution signals before escalating toclaude-opus-5-5.only_on_wrong_signal_escalation = true: The Switchyard default for escalation notes. Teams can also deployprofiles/random/(algorithm = "random",weights = [0.5, 0.5]) or explore LiteLLM's built-in Switchyard stage router.
| Architecture component | Configuration in the AWS CDK sample | Role in the routing workflow |
|---|---|---|
| Edge & WAF security | Amazon Route 53, ACM HTTPS certificate, IPv4 allowedCidr allowlist, and AWS WAF (2,000 requests per 5 minutes per IP). | Restricts access to approved networks, filters bad inputs without blocking source code payloads, and enforces HTTPS. |
| Application Load Balancer | 10-minute idle timeout; path allowlist for /v1/chat/completions, /v1/models, and /health/liveliness. | Forwards OpenAI-compatible chat completions to private subnets on port 4000 and returns HTTP 404 on admin UI paths. |
| ECS Fargate container | LiteLLM v1.102.0 with switchyard-server 0.3.0 wrapper (switchyard_gateway.configured.ROUTING_PLUGIN). | Validates the sk- key from AWS Secrets Manager, routes switchyard, and passes single-model baseline groups through. |
| Amazon Bedrock inference | Amazon Bedrock Converse / ConverseStream API via an Amazon Bedrock Runtime VPC interface endpoint. | Keeps inference traffic on the AWS network using the ECS task IAM role and returns the model in x-litellm-model-name. |
| Benchmark runners | bench/run.py (slugify, roman, calc, lru_ttl) and bench/multiturn.py (7-turn scripted chats). | Executes unit-tested coding tasks and multi-turn chats, reporting RUNS, SOLVED, ERRORS, COST/TASK, and COST/SOLVED. |
3. How is the LiteLLM and NeMo Switchyard gateway architected on AWS?
The AWS CDK solution runs LiteLLM and the embedded Switchyard plugin inside a single container on Amazon ECS on AWS Fargate across two Availability Zones in private subnets, fronted by an Application Load Balancer, AWS WAF, and AWS Secrets Manager, with private Amazon Bedrock Runtime VPC endpoints for model calls.
- OpenAI-compatible client request: The coding agent sends requests to
/v1/chat/completionswith"model": "switchyard", the sharedsk-gateway key in theAuthorizationheader, and the full conversation history. - WAF and ALB path allowlisting: AWS WAF enforces the Amazon IP reputation list, known bad inputs rule group, and a 2,000-requests-per-5-minutes rate limit while omitting generic XSS, SQL-injection, and body-size rules that source code triggers. The ALB forwards only
/v1/chat/completions,/v1/models, and/health/livelinessto port 4000. - Private Bedrock calls and logging: LiteLLM invokes Amazon Bedrock via a VPC interface endpoint using the ECS task's IAM role, names the answering model in the
x-litellm-model-nameheader, and logs to Amazon CloudWatch (/<stack name>/litellm, 30-day retention).
4. How do the benchmark runners test coding tasks and multi-turn conversations?
The sample repository includes two Python benchmark tools: bench/run.py, which runs coding agents against unit-tested programming tasks (slugify, roman, calc, and lru_ttl), and bench/multiturn.py, which evaluates full coding loops alongside seven-turn scripted engineering chats. Both attribute each turn's token usage through the x-litellm-model-name response header.
In bench/run.py, a coding agent lists and reads files, edits code, and runs unit tests through the gateway, marking a task solved when tests pass within the turn limit (6 turns by default, or raised to 12 turns for calc). For whole conversations, bench/multiturn.py runs an agentic loop over the coding tasks alongside four seven-turn scripted chats (such as backend and api) that alternate simple turns with design, debugging, and code-writing turns.
5. What should engineering teams consider before adopting Stage routing?
AWS and NVIDIA emphasize that the sample stack uses experimental evaluation components for non-production testing, and that teams should compare cost per solved task only after verifying solve rates, inspecting hard tasks separately, running enough repeats for statistical stability, and testing against representative tasks from their own codebase.
Alongside Amazon Bedrock AgentCore runtime V2, Gemini 4 Hydrogen in GitHub Copilot, and BigQuery Agent Analytics, teams should compare SOLVED before COST/SOLVED, inspect hard tasks where escalation replays failed attempts, apply statistical checks such as Fisher's exact test, and pair engineering infrastructure with analytics and attribution, AI SEO, and AI News tracking.
Frequently asked questions
What is the AWS and NVIDIA NeMo Switchyard solution for Amazon Bedrock?
Published on November 13, 2026 on the AWS Machine Learning Blog, the guide provides a non-production AWS CDK sample architecture that deploys LiteLLM (v1.102.0) with the experimental NVIDIA NeMo Switchyard routing plugin (switchyard-server 0.3.0) on Amazon ECS on AWS Fargate to route coding-agent requests across models on Amazon Bedrock and measure cost per solved task.
Why do AWS and NVIDIA evaluate coding agents by cost per solved task?
Coding agents such as Claude Code and OpenAI Codex CLI work in multi-turn tasks—reading files, editing code, and running unit tests across many model calls. Because total spend depends on turn count, replayed conversation history, and failed attempts, AWS and NVIDIA define cost per solved task as the total cost of runs divided by the number of runs whose tests pass.
How do Stage routing and Random routing work in the NeMo Switchyard LiteLLM plugin?
Neither strategy calls a separate judge model. Stage routing requires two candidate models—a capable model listed first (global.anthropic.claude-opus-5-5 in the sample) and an efficient model listed second (nvidia.nemotron-super-3-120b)—and inspects the three most recent tool results in the conversation history (with picker = 'efficient_first' and confidence_threshold = 0.5) to escalate when failures occur. Random routing splits traffic across models by weight for A/B comparisons.
How is the LiteLLM and Switchyard gateway secured and deployed on AWS?
The AWS CDK stack deploys a single container running LiteLLM and the Switchyard plugin on Amazon ECS on AWS Fargate in private subnets across two Availability Zones, fronted by Amazon Route 53, an ACM HTTPS certificate, an IPv4 allowedCidr allowlist, AWS WAF (limiting each IP to 2,000 requests per 5 minutes), and an Application Load Balancer forwarding only /v1/chat/completions, /v1/models, and /health/liveliness to port 4000, while calling Amazon Bedrock Converse APIs over a VPC interface endpoint.
What coding tasks and multi-turn benchmarks are included in the sample repository?
The sample repository includes bench/run.py, which tests coding agents on four unit-tested programming tasks (slugify, roman, calc, and lru_ttl) and prints RUNS, SOLVED, ERRORS, COST/TASK, and COST/SOLVED, alongside bench/multiturn.py, which runs an agentic loop over the coding tasks plus four seven-turn scripted chats while attributing each turn via the x-litellm-model-name response header.
More AI news
- Genie 4: Google's Most Interactive World Model Yet
- GPT-6.2 Astra and Sol: OpenAI's New Models for Professional and Everyday Work
- Claude Meetings: An AI Meeting Assistant That Gets the Work Started
- Gemini 4 Neon: Frontier-Level Reasoning, Built for Speed
- ChatGPT Live Dictation: Your Words Appear as You Speak
- Anthropic IPO: The Company Behind Claude Lists on Nasdaq
- Gemini 4 Carbon: Google's Strongest Model for Agentic Work, Coding and Research
- Claude Fable 5.2 and Claude Mythos 5.2: Anthropic's Most Capable Models Get Faster and Smarter
- ChatGPT Wallet: How AI Agents Can Now Pay Securely on Your Behalf
- Gemini 4 Hydrogen in GitHub Copilot: General Availability Across 11 Developer Surfaces
- Claiming Google Developer Program Cloud Credits Directly in Google AI Studio
- Amazon Bedrock AgentCore Runtime V2: Architecture Patterns for Fast Session Startup
- BigQuery Agent Analytics and Looker: Enterprise AI Agent Observability and Governance
- Microsoft Copilot Cowork, Projects and Workspaces: Unified Home, Skills and Automations
- Grok Bot 1.0: xAI's Multiplayer Team Bots for Slack, Microsoft Teams and Video Calls
- Claude Sheets, Claude Whiteboards and Claude Data: Collaborative Artifacts in Claude
- Google AI Search and Gemini 4 Hydrogen: Query Fan-Out and Visual Responses
- Businesses in ChatGPT: Business Hub, Engagement Insights and Business Agents
- Claude Plugins, Skills and Connectors: The New Plugins Page and Claude Marketplace
- ChatGPT Images 2.6: GIFs, Reference Images and What Marketers Should Know
- Google's Gemini Agent: What It Does, Coworker Agents and When You Can Use It