# Add to your Claude Code skills
git clone https://github.com/Azure-Samples/AI-Gateway
README.md
✨ AI Gateway Labs
Explore the enterprise-grade gateway for managing AI Models, Tools, and Agents
Why AI Gateway?
Building production-ready AI applications requires more than just calling model APIs. You need security, reliability, observability, and cost control—without slowing down innovation.
Each lab is a hands-on Jupyter notebook with step-by-step instructions, Bicep infrastructure templates, and APIM policies you can deploy to your Azure subscription.
🧠 AI Gateway for Models
Manage and control access to Large Language Models with enterprise-grade policies.
| Lab | Description |
|-----|-------------|
| Backend Pool Load Balancing | Distribute requests across multiple model endpoints |
| Token Rate Limiting | Control token consumption with rate limiting policies |
| Semantic Caching | Cache responses using vector similarity for faster, cheaper completions |
| Model Routing | Route requests to different backends based on model and version |
| FinOps Framework | Manage AI budgets with automated quota controls |
🔧 AI Gateway for Tools
Enable secure tool access with MCP protocol and function calling capabilities.
| Lab | Description |
|-----|-------------|
| Model Context Protocol (MCP) | Plug & play tools with OAuth credential management |
| MCP Client Authorization | Implement MCP with the client authorization flow |
| Function Calling | Use OpenAI function calling with Azure Functions backend |
| Realtime Audio + MCP | Combine realtime voice API with MCP tools |
🤖 AI Gateway for Agents
Build and control agentic applications with orchestration frameworks.
| Lab | Description |
|-----|-------------|
| AI Agent Service | Explore Foundry Agent Service with multi-service control |
| OpenAI Agents SDK | Use OpenAI Agents with Azure OpenAI and APIM-managed tools |
| Gemini MCP Agents | Integrate Google Gemini models with MCP tools |
| A2A Enabled Agents | A2A-enabled Agents with models and MCP plug & play tools |
🚀 Quick Start
Prerequisites
Python 3.12+
Python environment with the requirements.txt or run pip install -r requirements.txt in your terminal
The tools/ folder provides utilities for testing and development:
| Tool | Description |
|------|-------------|
| Tracing | Invoke AI Foundry APIs with tracing enabled |
| Streaming | Test streaming responses from AI models |
| Rate Limit Tester | Validate rate limiting configurations |
| Mock Server | OpenAI API mock for local development and testing |
| OAuth Client | Test OAuth authentication flows |
👩💻 Build Your Own Labs with AI
This repository includes Copilot Agent Skills that help you create new labs using AI-assisted development in VS Code.
Available Skills
| Skill | Description |
|-------|-------------|
| lab-creator | Scaffolds new labs with notebooks, Bicep, and policies |
| apim-bicep | Generates Azure Bicep templates for APIM resources |
| apim-terraform | Generates Terraform configurations for APIM |
| apim-policies | Creates APIM XML policies for AI gateway scenarios |
| apim-kql | Generates queries in KQL to control models, tools and agents |
| mcp-builder | Builds MCP servers for tool integration |
Example: Create a New Lab
Open this repo in VS Code with GitHub Copilot and use this prompt:
Create a new lab called "multi-model-failover" that demonstrates how to
implement automatic failover between different AI models when the primary
model is unavailable or throttled. Include:
- A backend pool with priority-based routing
- Retry policy with exponential backoff
- Circuit breaker pattern for unhealthy backends
- Built-in LLM logging to track usage across all backends
- Test the model with a LangChain agent: https://docs.langchain.com/oss/python/langchain/agents
Use gpt-4.1-mini as primary and gpt-4.1-nano as fallback, deploy to Sweden Central.
Copilot will generate the complete lab structure including:
Download the Enterprise AI Gateway e-Book for comprehensive end-to-end view of the Enterprise AI Gateway pattern, explaining why a centralized governance layer is essential for organizations adopting AI at scale and how it can be practically implemented using Azure API Management and Microsoft Foundry.
It describes the AI Gateway as a control plane that mediates all interactions between AI apps and agents and the underlying models, data, and tools, enabling consistent enforcement of security, safety, cost controls, resiliency, scalability, observability, and governance. Overall, the e-Book positions the Enterprise AI Gateway as a foundational architectural component that allows enterprises to innovate rapidly with AI while maintaining trust, compliance, visibility, and control.
🎬 Conferences & Webcasts
Learn from experts through these videos covering AI Gateway concepts and implementations.
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
