Swagger Auto-Generation on MCP Server
Would you like to generate a swagger.json directly on an MCP server on-the-fly? In many use cases, using remote MCP servers is not uncommon. In particular, if you're using Azure API Management (APIM), Azure API Center (APIC) or Copilot Studio in Power Platform, integrating with remote MCP servers is inevitable.Impariamo a conoscere MCP: Introduzione al Model Context Protocol (MCP)
Non perderti il prossimo evento “Let’s Learn – MCP” su Microsoft Reactor il 24 di Luglio, pensato per chiunque voglia conoscere meglio il nuovo standard per agenti intelligenti (il Model Context Protocol) e imparare a metterlo in pratica. La sessione è in Italiano e le demo sono in Python, ma fa parte di una serie di live-streaming disponibili in tantissime lingue.How do I get my agent to respond in a structured format like JSON?
Welcome back to Agent Support—a developer advice column for those head-scratching moments when you’re building an AI agent! Each post answers a real question from the community with simple, practical guidance to help you build smarter agents. 💬 Dear Agent Support I’m building an agent that feeds its response into another app—but sometimes the output is messy or unpredictable. Can you help? This looks like a job for JSON! While agent output often comes back as plain text, there are times when you need something more structured, especially if another system, app, or service needs to reliably parse and use that response. 🧠 What’s the Deal with JSON Output? If your agent is handing off data to another app, you need the output to be clean, consistent, and easy to parse. That’s where JSON comes in. JSON (JavaScript Object Notation) is a lightweight, widely supported format that plays nicely with almost every modern tool or platform. Whether your agent is powering a dashboard, triggering a workflow, or sending data into a UI component, JSON makes the handoff smooth. You can define exactly what shape the response should take (i.e. keys, values, types) so that other parts of your system know what to expect every single time. Without it? Things get messy fast! Unstructured text can vary wildly from one response to the next. A missing field, a misaligned format, or even just an unexpected line break can break downstream logic, crash a frontend, or silently cause bugs you won’t catch until much later. Worse, you end up writing brittle post-processing code to clean up the output just to make it usable. The bottom line: If you want your agent to work well with others, it needs to speak in a structured format. JSON isn’t just a nice-to-have, it’s the language of interoperability. 🧩 When You’d Want Structured Output Not every agent needs to speak JSON, but if your response is going anywhere beyond the chat window, structured output is your best bet. Let’s say your agent powers a dashboard. You might want to display the response in different UI components—a title, a summary, maybe a set of bullet points. That only works if the response is broken into predictable parts. Same goes for workflows. If you’re passing the output into another service, like Power Automate, an agent framework like Semantic Kernal, or even another agent, it needs to follow a format those systems can recognize. Structured output also makes logging and debugging easier. When every response follows the same format, you can spot problems faster. Missing fields, weird values, or unexpected data types stand out immediately. It also future proofs your agent. If you want to add new features later, like saving responses to a database or triggering different actions based on the content, having structured output gives you the flexibility to build on top of what’s already there. If the output is meant to do more than just be read by a human, it should be structured for a machine. 📄 How to Define a JSON Format Once you know your agent’s output needs to be structured, the next step is telling the model exactly how to structure it. That’s where defining a schema comes in. In this context, a schema is just a blueprint for the shape of your data. It outlines what fields you expect, what type of data each one should hold, and how everything should be organized. Think of it like a form template: the model just needs to fill in the blanks. Here’s a simple example of a JSON schema for a to-do app: { "task": "string", "priority": "high | medium | low", "due_date": "YYYY-MM-DD" } Once you have your format in mind, include it directly in your prompt. But if you’re using the AI Toolkit in Visual Studio Code, you don’t have to do this manually every time! 🔧 Create a JSON schema with the AI Toolkit The Agent Builder feature supports structured output natively. You can provide a JSON schema alongside your prompt, and the agent will automatically aim to match that format. This takes the guesswork out of prompting. Instead of relying on natural language instructions to shape the output, you’re giving the model a concrete set of instructions to follow. Here’s how to do it: Open the Agent Builder from the AI Toolkit panel in Visual Studio Code. Click the + New Agent button and provide a name for your agent. Select a Model for your agent. Within the System Prompt section, enter: You recommend a movie to watch. Within the User Prompt section, enter: Recommend a science-fiction movie with robots. In the Structured Output section, select json_schema as the output format. Click Prepare schema. In the wizard, select Use an example. For the example, select paper_metadata. Save the file to your desired location. You can name the file: movie_metadata. In the Agent Builder, select movie_metadata to open the file. Using the template provided, modify the schema to format the title, genre, year, and reason. Once done, save the file. { "name": "movie_metadata", "strict": true, "schema": { "type": "object", "properties": { "title": { "type": "string" }, "genre": { "type": "array", "items": { "type": "string" } }, "year": { "type": "string" }, "reason": { "type": "array", "items": { "type": "string" } } }, "required": [ "title", "genre", "year", "reason" ], "additionalProperties": false } } And just like that, you’ve set up a JSON schema for your agent’s output! 🧪 Test Before You Build You can submit a prompt with the Agent Builder to validate whether the agent adheres to the JSON schema when returning its response. When you click Run, the agent’s response will appear in the Prompt tab, ideally in JSON format. 🔁 Recap Here’s a quick rundown of what we covered: JSON lets you create reliable, machine-friendly agent responses. JSON is essential for interoperability between apps, tools, and workflows. Without JSON, you risk fragile pipelines, broken features, or confusing bugs. The AI Toolkit supports including a JSON schema with your agent’s prompt. 📺 Want to Go Deeper? Check out the AI Agents for Beginners curriculum in which we dive a bit more into agentic design patterns which includes defining structured outputs. We’ll have a video landing soon in the Build an Agent Series that takes you through a step-by-step look of creating a JSON schema for your agent!Learn Together: Building an MCP Server with AI Toolkit
AI is evolving fast—but building real, useful AI applications still requires more than just a smart model. That’s where the Model Context Protocol (MCP) comes in. MCP is an open standard that helps AI systems talk to tools, data, and services in a clean, reusable way. Think of it as the coordination layer that turns a smart model into an intelligent application. Whether you’re integrating APIs, calling scripts, or orchestrating multiple agents, MCP provides the structure that makes it all work together. And thanks to the AI Toolkit for Visual Studio Code, working with MCP is now easier than ever. The toolkit gives you an intuitive playground, built-in server tools, model integration, and everything you need to go from idea to working prototype without leaving your editor. 📅 Join the Event: Learn Together – Building an MCP Server with AI Toolkit In this live, hands-on workshop, we’ll walk through four practical modules designed to help you build, configure, and test your own MCP server from scratch. You’ll get to follow along step-by-step using the AI Toolkit extension for VS Code. Whether you're building multi-agent workflows, enabling tools in a RAG app, or just looking for a smarter way to connect your models to real-world actions, this workshop is for you. 🧪 What You’ll Learn – Inside the Labs Here’s a preview of the four modules we’ll cover during the livestream: Module 1: Getting Started with AI Toolkit In this lab, you’ll install and configure the AI Toolkit for Visual Studio Code, then dive into its key features—like the model catalog, interactive playground, and agent builder. You'll build your first functional agent, experiment with real-time model testing, and evaluate outputs using built-in metrics. Along the way, you’ll also explore batch processing and multimodal support. By the end, you’ll have a solid foundation in what the AI Toolkit can do. Module 2: Integrating MCP with Your Agents This lab introduces the Model Context Protocol (MCP) and shows you how to supercharge your AI agents with tool-using capabilities. You’ll connect to existing MCP servers, like the Playwright automation server, then integrate them into your agents using the AI Toolkit. From configuring tools to testing and deploying MCP-powered agents, you’ll see how external tools become part of your model’s workflow. This lab is your bridge between smart models and real-world actions. Module 3: Building and Debugging Custom MCP Servers Here, you’ll go beyond using prebuilt MCP servers and start creating your own. You’ll learn how to use the latest MCP Python SDK to configure and debug a custom server, including a weather tool example. The lab walks through setting up the MCP Inspector, testing tool calls, and running agents in a local dev environment. You’ll leave with a full understanding of how to build, test, and debug your own MCP endpoints. Module 4: Real-World MCP Server – GitHub Clone Agent In this final lab, you’ll build a GitHub clone MCP server that replicates real dev workflows. It features smart repo cloning, intelligent directory setup, error handling, and direct integration with VS Code. You’ll even enable GitHub Copilot’s Agent Mode to interact with your custom tools. This lab wraps everything together into a production-grade example of what’s possible with MCP. 💡 Why MCP + AI Toolkit? The magic of MCP is in its modularity. It gives developers a structured way to define tools, link models, and control interactions, all in a language-agnostic, reusable way. And when paired with the AI Toolkit, you get a guided development experience with built-in tools for testing, telemetry, evaluation, and more. Whether you're building an agent to automate tasks, assist users, or connect enterprise systems, MCP makes it possible. The AI Toolkit makes it easier! 📅 Save the Date! Don’t miss your chance to build alongside us and see MCP in action. This is more than just a demo, it’s a full walkthrough of what modern, production-aware AI development looks like. ✅ Register now to join the Learn Together livestream on the 5th August and take your first step toward building smarter AI applications. 👉 Click here to register for the event!JS AI Build-a-thon: Project Showcase
In the JS AI Build-a-thon, quest 9 was all about shipping faster, smarter, and more confidently. This quest challenged developers to skip the boilerplate and focus on what really matters, solving real-world problems with production-ready AI templates and cloud-native tools. And wow, did our builders deliver, with a massive 28 projects. From personalized chatbots to full-stack RAG applications, participants used the power of the Azure Developer CLI (azd) and robust templates to turn ideas into fully deployed AI solutions on Azure, in record time. What This Quest Was About Quest 9 focused on empowering developers to: Build AI apps faster with production-ready templates Use Azure Developer CLI (azd) to deploy with just a few commands Leverage Infrastructure-as-Code (IaC) to provision everything from databases to APIs Follow best practices out of the box — scalable, secure, and maintainable Participants explored a curated gallery of templates and learned how to adapt them to their unique use cases. No Azure experience? No problem. azd made setup, deployment, and teardown as simple as azd up. 🥁And the winner is..... With the massive votes from the community, the winning project as AI Academic advisor by Aryanjstar AI Career Navigator - Your Personal AI Career Coach by Aryanjstar [Project Submission] AI Career Navigator - Your Personal AI Career Coach · Issue #47 · Azure-Samples/JS-AI-Build-a-thon Aryan, a Troop Leader of the Geo Cyber Study Jam, created the AI Career Navigator to address common challenges faced by developers in the tech job market, such as unclear skill paths, resume uncertainty, and interview anxiety. Built using the Azure Search OpenAI Demo template, the tool offers features like resume-job description matching, skill gap analysis, and dynamic interview preparation. The project leverages a robust RAG setup and Azure integration to deliver a scalable, AI-powered solution for career planning. 🥉 Other featured projects: Deepmine-sentinel by Josephat-Onkoba [Project Submission] Deepmine-Sentinel · Issue #29 · Azure-Samples/JS-AI-Build-a-thon DeepMine Sentinel AI is an intelligent safety assistant designed to tackle the urgent risks facing workers in the mining industry, one of the most hazardous sectors globally. Built by customizing the “Get Started with Chat” Azure template, this solution offers real-time safety guidance and monitoring to prevent life-threatening incidents like cave-ins, toxic gas exposure, and equipment accidents. In regions where access to safety expertise is limited, DeepMine Sentinel bridges the gap by delivering instant, AI-powered support underground, ensuring workers can access critical information and protocols when they need it most. With a focus on accessibility, real-world impact, and life-saving potential, this project demonstrates how AI can be a powerful force for good in high-risk environments. PetPal - Your AI Pet Care Assistant by kelcho-spense [Project Submission] PetPal - Your AI Pet Care Assistant · Issue #70 · Azure-Samples/JS-AI-Build-a-thon PetPal is an AI-powered pet care assistant designed to support pet owners, especially first-timers, by offering instant, reliable answers to common pet-related concerns. From health and nutrition advice to emergency support and behavioral training, PetPal uses a serverless architecture powered by LangChain.js, Azure OpenAI, and Retrieval-Augmented Generation (RAG) to deliver accurate, context-aware responses. The app features a warm, pet-themed interface built with Lit and TypeScript, and includes thoughtful customizations like pet profile management, personalized chat history, and species-specific guidance. With backend services hosted on Azure Functions and data stored in Azure Cosmos DB, PetPal is production-ready, scalable, and focused on reducing anxiety while promoting responsible and informed pet ownership. MLSA LearnBot by Shunlexxi [Project Submission] MLSA LearnBot · Issue #67 · Azure-Samples/JS-AI-Build-a-thon Navigating the Microsoft Learn Student Ambassadors program can be overwhelming. To solve this, a student built an Intelligent Chatbot Q&A App using a Microsoft azd template, transforming a generic AI chatbot into a tailored assistant for Student Ambassadors and aspiring members. By integrating essential documentation, FAQs, and natural language support with Azure services like App Service, AI Search, and OpenAI, this tool empowers students to get instant, reliable answers and navigate their roles with ease. The front end was customized to reflect a student-friendly brand, and deployment was simplified using azd for a seamless, production-ready experience. You okay? Meet Vish AI, your mental health companion by ToshikSoni [Project Submission] You okay? Meet Vish AI, your mental health companion · Issue #38 · Azure-Samples/JS-AI-Build-a-thon Vish.AI is an empathetic GenAI companion designed to support emotional well-being, especially for individuals facing depression, loneliness, and mental burnout. Built using Azure’s AI Chat RAG template and enhanced with LangChain for conversational memory, the assistant offers a deeply personalized experience that remembers past interactions and responds with both emotional intelligence and informed support. By integrating a curated collection of resources on mental health into its RAG system, Vish.AI provides meaningful guidance and a comforting presence, available anytime, anywhere. Created to bridge the gap for those who may not feel comfortable opening up to friends or family, this project combines AI with a human touch to offer always-accessible care, demonstrating how thoughtful technology can help make life a little lighter for those quietly struggling. Want to Catch Up? If you missed the Build-a-thon or want to explore other quests (there are 9!), check them out here: 👉 GitHub - Azure-Samples/JS-AI-Build-a-thon If you want to catch up with how the challenge went and how you can get started, check out 👉JS AI Build‑a‑thon: Wrapping Up an Epic June 2025! | Microsoft Community Hub Join the Community The conversation isn’t over. The Quests are now self-paced. We’re keeping the momentum going over on Discord in the #js-ai-build-a-thon channel. Drop your questions, showcase your builds, or just come hang out with other builders. 👉 Join the community on Join the Azure AI Foundry Discord Server! Additional Resources 🔗 Microsoft for JavaScript developers 📚 Generative AI for Beginners with JavaScriptAI Toolkit for VS Code July Update
We’re back with the July update for the AI Toolkit for Visual Studio Code! This month marks a major release—version 0.16.0—bringing features we teased in the June prerelease into full availability, along with new enhancements across the board. 💳 GitHub Pay-as-you-go Model Support AI Toolkit now supports GitHub pay-as-you-go models, making it easier to continue building without friction when you exceed the free tier limits. Here’s how it works: When you hit the usage limit for GitHub’s hosted models, AI Toolkit will provide a clear prompt with a link to GitHub’s paid usage documentation. You can enable model usage billing in your GitHub Settings. Once enabled, you can continue using the same model in the Playground or Agent Builder—no changes needed in your workflow. This integration helps keep your development flow uninterrupted while giving you flexibility on how and when to scale. 🤖 Agent Builder Enhancements In last month’s June update, we introduced early access to function calling and agent version management—two highly requested features for building more reliable and dynamic AI agents. With v0.16.0, these features are now officially included in the stable release. We’ve also made several usability improvements: Resizable divider between input and output panels Cleaner, more compact evaluation results table Explore the latest features in the Agent Builder documentation. 🌐 Agent Development in VS Code: From Local to Cloud Whether you’re just getting started or ready to scale, AI Toolkit now makes it seamless to go from local experiments to production-ready deployments in the cloud. 🔍 Discover and Deploy Models with Ease You can now browse and explore a growing collection of models—including 100+ models from Azure AI Foundry—directly in the Model Catalog view. Once you’ve selected a model, you can deploy it to Azure AI Foundry with a single click, all within VS Code. ➡️ Learn more about the Model Catalog ➡️ How deployment works in Azure AI Foundry ⚙️ Iterate Locally, Scale When Ready The Playground and Agent Builder experience is designed to help you with prototyping quickly. Start with free, hosted models on GitHub to experiment with prompts and agent behavior. As your use case grows, you might hit rate limits or token caps—when that happens, the toolkit will proactively guide you to deploy the same model to Azure AI Foundry, so you can continue testing without interruption. 🚀 Get Started and Share Your Feedback If you haven’t tried the AI Toolkit for VS Code yet, now is a great time to get started. Install it directly from the Visual Studio Code Marketplace and begin building intelligent agents today. We’d love to hear from you! Whether it's a feature request, a bug report, or feedback on your experience, join the conversation and contribute directly on our GitHub repository. Let’s build the future of AI agent development together 💡💬 📄 See full changelog for v0.16.0 🧠 AI Toolkit Documentation Happy coding, and see you in August!How do I give my agent access to tools?
Welcome back to Agent Support—a developer advice column for those head-scratching moments when you’re building an AI agent! Each post answers a real question from the community with simple, practical guidance to help you build smarter agents. Today’s question comes from someone trying to move beyond chat-only agents into more capable, action-driven ones: 💬 Dear Agent Support I want my agent to do more than just respond with text. Ideally, it could look up information, call APIs, or even run code—but I’m not sure where to start. How do I give my agent access to tools? This is exactly where agents start to get interesting! Giving your agent tools is one of the most powerful ways to expand what it can do. But before we get into the “how,” let’s talk about what tools actually mean in this context—and how Model Context Protocol (MCP) helps you use them in a standardized, agent-friendly way. 🛠️ What Do We Mean by “Tools”? In agent terms, a tool is any external function or capability your agent can use to complete a task. That might be: A web search function A weather lookup API A calculator A database query A custom Python script When you give an agent tools, you’re giving it a way to take action—not just generate text. Think of tools as buttons the agent can press to interact with the outside world. ⚡ Why Give Your Agent Tools? Without tools, an agent is limited to what it “knows” from its training data and prompt. It can guess, summarize, and predict, but it can’t do. Tools change that! With the right tools, your agent can: Pull live data from external systems Perform dynamic calculations Trigger workflows in real time Make decisions based on changing conditions It’s the difference between an assistant that can answer trivia questions vs. one that can book your travel or manage your calendar. 🧩 So… How Does This Work? Enter Model Context Protocol (MCP). MCP is a simple, open protocol that helps agents use tools in a consistent way—whether those tools live in your app, your cloud project, or on a server you built yourself. Here’s what MCP does: Describes tools in a standardized format that models can understand Wraps the function call + input + output into a predictable schema Lets agents request tools as needed (with reasoning behind their choices) This makes it much easier to plug tools into your agent workflow without reinventing the wheel every time! 🔌 How to Connect an Agent to Tools Wiring tools into your agent might sound complex, but it doesn’t have to be! If you’ve already got a MCP server in mind, there’s a straightforward way within the AI Toolkit to expose it as a tool your agent can use. Here’s how to do it: Open the Agent Builder from the AI Toolkit panel in Visual Studio Code. Click the + New Agent button and provide a name for your agent. Select a Model for your agent. Within the Tools section, click + MCP Server. In the wizard that appears, click + Add Server. From there, you can select one of the MCP servers built my Microsoft, connect to an existing server that’s running, or even create your own using a template! After giving the server a Server ID, you’ll be given the option to select which tools from the server to add for your agent. Once connected, your agent can call tools dynamically based on the task at hand. 🧪 Test Before You Build Once you’ve connected your agent to an MCP server and added tools, don’t jump straight into full integration. It’s worth taking time to test whether the agent is calling the right tool for the job. You can do this directly in the Agent Builder: enter a test prompt that should trigger a tool in the User Prompt field, click Run, and observe how the model responds. This gives you a quick read on tool call accuracy. If the agent selects the wrong tool, it’s a sign that your system prompt might need tweaking before you move forward. However, if the agent calls the correct tool but the output still doesn’t look right, take a step back and check both sides of the interaction. It might be that the system prompt isn’t clearly guiding the agent on how to use or interpret the tool’s response. But it could also be an issue with the tool itself—whether that’s a bug in the logic, unexpected behavior, or a mismatch between input and expected output. Testing both the tool and the prompt in isolation can help you pinpoint where things are going wrong before you move on to full integration. 🔁 Recap Here’s a quick rundown of what we covered: Tools = external functions your agent can use to take action MCP = a protocol that helps your agent discover and use those tools reliably If the agent calls the wrong tool—or uses the right tool incorrectly—check your system prompt and test the tool logic separately to isolate the issue. 📺 Want to Go Deeper? Check out my latest video on how to connect your agent to a MCP server—it’s part of the Build an Agent Series, where I walk through the building blocks of turning an idea into a working AI agent. The MCP for Beginners curriculum covers all the essentials—MCP architecture, creating and debugging servers, and best practices for developing, testing, and deploying MCP servers and features in production environments. It also includes several hands-on exercises across .NET, Java, TypeScript, JavaScript and Python. 👉 Explore the full curriculum: aka.ms/AITKmcp And for all your general AI and AI agent questions, join us in the Azure AI Foundry Discord! You can find me hanging out there answering your questions about the AI Toolkit. I'm looking forward to chatting with you there! Whether you’re building a productivity agent, a data assistant, or a game bot—tools are how you turn your agent from smart to useful.How can I measure the quality of my agent's responses?
Welcome back to Agent Support—a developer advice column for those head-scratching moments when you’re building an AI agent! Each post answers a real question from the community with simple, practical guidance to help you build smarter agents. Today’s question comes from someone curious about measuring how well their agent responds: 💬Dear Agent Support My agent seems to be responding well—but I want to make sure I’m not just guessing. Ideally, I’d like a way to check how accurate or helpful its answers really are. How can I measure the quality of my agent’s responses? 🧠 What are Evaluations? Evaluations are how we move from “this feels good” to “this performs well.” They’re structured ways of checking how your agent is doing, based on specific goals you care about. At the simplest level, evaluations help answer: Did the agent actually answer the question? Was the output relevant and grounded in the right info? Was it easy to read or did it ramble? Did it use the tool it was supposed to? That might mean checking if the model pulled the correct file in a retrieval task. Or whether it used the right tool when multiple are registered. Or even something subjective like if the tone felt helpful or aligned with your brand. 🎯 Why Do We Do Evaluations? When you're building an agent, it’s easy to rely on instinct. You run a prompt, glance at the response, and think: “Yeah, that sounds right.” But what happens when you change a system prompt? Or upgrade the model? Or wire in a new tool? Without evaluations, there’s no way to know if things are getting better or quietly breaking. Evaluations help you: Catch regressions early: Maybe your new prompt is more detailed, but now the agent rambles. A structured evaluation can spot that before users do. Compare options: Trying out two different models? Testing a retrieval-augmented version vs. a base version? Evaluations give you a side-by-side look at which one performs better. Build trust in output quality: Whether you're handing this to a client, a customer, or just your future self, evaluations help you say, “Yes, I’ve tested this. Here’s how I know it works.” They also make debugging faster. If something’s off, a good evaluation setup helps you narrow down where it went wrong: Was the tool call incorrect? Was the retrieved content irrelevant? Did the prompt confuse the model? Ultimately, evaluations turn your agent into a system you can improve with intention, not guesswork. ⏳ When Should I Start Evaluating? Short answer: Sooner than you think! You don’t need a finished agent or a fancy framework to start evaluating. In fact, the earlier you begin, the easier it is to steer things in the right direction. Here’s a simple rule of thumb: If your agent is generating output, you can evaluate it. That could be: Manually checking if it answers the user’s question Spotting when it picks the wrong tool Comparing two prompt versions to see which sounds clearer Even informal checks can reveal big issues early before you’ve built too much around a flawed behavior. As your agent matures, you can add more structure: Create a small evaluation set with expected outputs Define categories you want to score (like fluency, groundedness, relevance) Run batch tests when you update a model Think of it like writing tests for code. You don’t wait until the end, you build them alongside your system. That way, every change gets feedback fast. The key is momentum. Start light, then layer on depth as you go. You’ll save yourself debugging pain down the line, and build an agent that gets better over time. 📊 AI Toolkit You don’t need a full evaluation pipeline or scoring rubric on day one. In fact, the best place to begin is with a simple gut check—run a few test prompts and decide whether you like the agent’s response. And if you don’t have a dataset handy, no worry! With the AI Toolkit, you can both generate datasets and keep track of your manual evaluations with the Agent Builder’s Evaluation feature. Sidebar: If you’re curious about deeper eval workflows, like using AI to assist in judging your agent output against a set of evaluators like fluency, relevance Tool Call, or even custom evaluators, we’ll cover that in a future edition of Agent Support. For now, let’s keep it simple! Here’s how to do it: Open the Agent Builder from the AI Toolkit panel in Visual Studio Code. Click the + New Agent button and provide a name for your agent. Select a Model for your agent. Within the System Prompt section, enter: You recommend a movie based on the user’s favorite genre. Within the User Prompt section, enter: What’s a good {{genre}} movie to watch? On the right side of the Agent Builder, select the Evaluation tab. Click the Generate Data icon (the first icon above the table). For the Rows of data to generate field, increase the total to 5. Click Generate. You’re now ready to start evaluating the agent responses! 🧪 Test Before You Build You can run the rows of data either individually or in bulk. I’d suggest starting with a single run to get an initial feel for how the feature works. When you click Run, the agent’s response will appear in the response column. Review the output. In the Manual Evaluation column, select either thumb up or thumb down. You can continue to run the other rows or even add your own row and pass in a value for {{city}}. Want to share the evaluation run and results with a colleague? Click the Export icon to save the run as a .JSONL file. You’ve just taken the first step toward building a more structured, reliable process for evaluating your agent’s responses! 🔁 Recap Here’s a quick rundown of what we covered: Evaluations help you measure quality and consistency in your agent’s responses. They’re useful for debugging, comparing, and iterating. Start early—even rough checks can guide better decisions. The AI Toolkit makes it easier to run and track evaluations right inside your workflow. 📺 Want to Go Deeper? Check out my previous live-stream for AgentHack: Evaluating Agents where I explore concepts and methodologies for evaluating generative AI applications. Although I focus on leveraging the Azure AI Evaluation SDK, it’s still an invaluable intro to learning more about evaluations. The Evaluate and Improve the Quality and Safety of your AI Applications lab from Microsoft Build 2025 provides a comprehensive self-guided introduction to getting started with evaluations. You’ll learn what each evaluator means, how to analyze the scores, and why observability matters—plus how to use telemetry data locally or in the cloud to assess and debug your app’s performance! 👉 Explore the lab: https://github.com/microsoft/BUILD25-LAB334/ And for all your general AI and AI agent questions, join us in the Azure AI Foundry Discord! You can find me hanging out there answering your questions about the AI Toolkit. I'm looking forward to chatting with you there! Whether you’re debugging a tool call, comparing prompt versions, or prepping for production, evaluations are how you turn responses from plausible to dependable.How do I control how my agent responds?
Welcome to Agent Support—a developer advice column for those head-scratching moments when you’re building an AI agent! Each post answers a question inspired by real conversations in the AI developer community, offering practical advice and tips. This time, we’re talking about one of the most misunderstood ingredients in agent behavior: the system prompt. Let’s dive in! 💬 Dear Agent Support I’ve written a few different prompts to guide my agent’s responses, but the output still misses the mark—sometimes it’s too vague, other times too detailed. What’s the best way to structure the instructions so the results are more consistent? Great question! It gets right to the heart of prompt engineering. When the output feels inconsistent, it’s often because the instructions aren’t doing enough to guide the model’s behavior. That’s where prompt engineering can make a difference. By refining how you frame the instructions, you can guide the model toward more reliable, purpose-driven output. 🧠 What Is Prompt Engineering (and Why It Matters for Agents) Before we can fix the prompt, let’s define the craft. Prompt engineering is the practice of designing clear, structured input instructions that guide a model toward the behavior you want. In agent systems, this usually means writing the system prompt, a behind-the-scenes instruction that sets the tone, context, and boundaries for how the agent should act. While prompt engineering feels new, it’s rooted in decades of interface design, instruction tuning, and human-computer interaction research. The big shift? With large language models (LLMs), language becomes the interface. The better your instructions, the better your outcomes. 🧩 The Anatomy of a Good System Prompt Think of your system prompt as a blueprint for how the agent should operate. It sets the stage before the conversation starts. A strong system prompt should: Define the role: Who is this agent? What’s their tone, expertise, or purpose? Clarify the goal: What task should the agent help with? What should it avoid? Establish boundaries: Are there any constraints? Should it cite sources? Stay concise? Here’s a rough template you can build from: “You are a helpful assistant that specializes in [domain]. Your job is to [task]. Keep responses [format/length/tone]. If you’re unsure, respond with ‘I don’t know’ instead of guessing.” 🛠️ Why Prompts Fail (Even When They Sound Fine) Common issues we see: Too vague (“Be helpful” isn’t helpful.) Overloaded with logic (Treating the system prompt like a config file.) Conflicting instructions (“Be friendly” + “Use legal terminology precisely.”) Even well-written prompts can underperform if they’re mismatched with the model or task. That’s why we recommend testing and refining early and often! ✏️ Skip the Struggle— let the AI Toolkit Write It! Writing a great system prompt takes practice. And even then, it’s easy to overthink it! If you’re not sure where to start (or just want to speed things up), the AI Toolkit provides a built-in way to generate a system prompt for you. All you have to do is describe what the agent needs to do, and the AI Toolkit will generate a well-defined and detailed system prompt for your agent. Here's how to do it: Open the Agent Builder from the AI Toolkit panel in Visual Studio Code. Click the + New Agent button and provide a name for your agent. Select a Model for your agent. In the Prompts section, click Generate system prompt. In the Generate a prompt window that appears, provide basic details about your task and click Generate. After the AI Toolkit generates your agent’s system prompt, it’ll appear in the System prompt field. I recommend reviewing the system prompt and modifying any parts that may need revision! Heads up: System prompts aren’t just behind-the-scenes setup, they’re submitted along with the user prompt every time you send a request. That means they count toward your total token limit, so longer prompts can impact both cost and response length. 🧪 Test Before You Build Once you’ve written (or generated) a system prompt, don’t skip straight to wiring it into your agent. It’s worth testing how the model responds with the prompt in place first. You can do that right in the Agent Builder. Just submit a test prompt in the User Prompt field, click Run, and the model will generate a response using the system prompt behind the scenes. This gives you a quick read on whether the behavior aligns with your expectations before you start building around it. 🔁 Recap Here’s a quick rundown of what we covered: Prompt engineering helps guide your agent’s behavior through language. A good system prompt sets the tone, purpose, and guardrails for the agent. Test, tweak, and simplify—especially if responses seem inconsistent or off-target. You can use the Generate system prompt feature within the AI Toolkit to quickly generate instructions for your agent. 📺 Want to Go Deeper? Check out my latest video on how to define your agent’s behavior—it’s part of the Build an Agent Series, where I walk through the building blocks of turning an idea into a working AI agent. The Prompt Engineering Fundamentals chapter from our aka.ms/AITKGenAI curriculum overs all the essentials—prompt structure, common patterns, and ways to test and improve your outputs. It also includes exercises so you can get some hands-on practice. 👉 Explore the full curriculum: aka.ms/AITKGenAI And for all your general AI and AI agent questions, join us in the Azure AI Foundry Discord! You can find me hanging out there answering your questions about the AI Toolkit. I'm looking forward to chatting with you there! And remember, great agent behavior starts with great instructions—and now you’ve got the tools to write them.I want to show my agent a picture—Can I?
Welcome to Agent Support—a developer advice column for those head-scratching moments when you’re building an AI agent! Each post answers a question inspired by real conversations in the AI developer community, offering practical advice and tips. To kick things off, we’re tackling a common challenge for anyone experimenting with multimodal agents: working with image input. Let’s dive in! Dear Agent Support, I’m building an AI agent, and I’d like to include screenshots or product photos as part of the input. But I’m not sure if that’s even possible, or if I need to use a different kind of model altogether. Can I actually upload an image and have the agent process it? Great question, and one that trips up a lot of people early on! The short answer is: yes, some models can process images—but not all of them. Let’s break that down a bit. 🧠 Understanding Image Input When we talk about image input or image attachments, we’re talking about the ability to send a non-text file (like a .png, .jpg, or screenshot) into your prompt and have the model analyze or interpret it. That could mean describing what’s in the image, extracting text from it, answering questions about a chart, or giving feedback on a design layout. 🚫 Not All Models Support Image Input That said, this isn’t something every model can do. Most base language models are trained on text data only, they’re not designed to interpret non-text inputs like images. In most tools and interfaces, the option to upload an image only appears if the selected model supports it, since platforms typically hide or disable features that aren't compatible with a model's capabilities. So, if your current chat interface doesn’t mention anything about vision or image input, it’s likely because the model itself isn’t equipped to handle it. That’s where multimodal models come in. These are models that have been trained (or extended) to understand both text and images, and sometimes other data types too. Think of them as being fluent in more than one language, except in this case, one of those “languages” is visual. 🔎 How to Find Image-Supporting Models If you’re trying to figure out which models support images, the AI Toolkit is a great place to start! The extension includes a built-in Model Catalog where you can filter models by Feature—like Image Attachment—so you can skip the guesswork. Here’s how to do it: Open the Model Catalog from the AI Toolkit panel in Visual Studio Code. Click the Feature filter near the search bar. Select Image Attachment. Browse the filtered results to see which models can accept visual input. Once you've got your filtered list, you can check out the model details or try one in the Playground to test how it handles image-based prompts. 🧪 Test Before You Build Before you plug a model into your agent and start wiring things together, it’s a good idea to test how the model handles image input on its own. This gives you a quick feel for the model’s behavior and helps you catch any limitations before you're deep into building. You can do this in the Playground, where you can upload an image and pair it with a simple prompt like: “Describe the contents of this image.” OR “Summarize what’s happening in this screenshot.” If the model supports image input, you’ll be able to attach a file and get a response based on its visual analysis. If you don’t see the option to upload an image, double-check that the model you’ve selected has image capabilities—this is usually a model issue, not a UI bug. 🔁 Recap Here’s a quick rundown of what we covered: Not all models support image input—you’ll need a multimodal model specifically built to handle visual data. Most platforms won’t let you upload an image unless the model supports it, so if you don’t see that option, it’s probably a model limitation. You can use the AI Toolkit’s Model Catalog to filter models by capability—just check the box for Image Attachment. Test the model in the Playground before integrating it into your agent to make sure it behaves the way you expect. 📺 Want to Go Deeper? Check out my latest video on how to choose the right model for your agent—it’s part of the Build an Agent Series, where I walk through the building blocks of turning an idea into a working AI agent. And if you’re looking to sharpen your model instincts, don’t miss Model Mondays—a weekly series that helps developers like you build your Model IQ, one spotlight at a time. Whether you’re just starting out or already building AI-powered apps, it’s a great way to stay current and confident in your model choices. 👉 Explore the series and catch the next episode: aka.ms/model-mondays/rsvp If you're just getting started with building agents, check out our Agents for Beginners curriculum. And for all your general AI and AI agent questions, join us in the Azure AI Foundry Discord! You can find me hanging out there answering your questions about the AI Toolkit. I'm looking forward to chatting with you there! Whatever you're building, the right model is out there—and with the right tools, you'll know exactly how to find it.
