Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Items marked (preview) in this article are currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
Monitoring your generative AI applications has never been more important, due to the complexity and rapid evolvement of the AI industry. Azure AI Foundry Observability, integrated with Azure Monitor Application Insights, enables you to continuously monitor your deployed AI applications to ensure that they're performant, safe, and produce high-quality results in production. In addition to the continuous monitoring capabilities, we also provide continuous evaluation capabilities for Agents to add further enhance the Foundry Observability dashboard with visibility into additional critical quality and safety metrics.
Note
You must use a Foundry project for this feature. A hub based project isn't supported. For more information, see Types of projects.
How to enable monitoring
To use monitoring capabilities in Azure AI Foundry, you need to connect an Application Insights resource to your Azure AI Foundry project.
- Navigate to Monitoring in the left navigation pane of the Azure AI Foundry portal.
- Select the Application analytics tab.
- Create a new Application Insights resource if you don't already have one.
- Connect the resource to your AI Foundry project.
Collecting production data for monitoring
From here, you want to begin collecting telemetry for your application that you can monitor in our built-in views. To do this, we recommend:
Instrumenting traces allows you to capture detailed telemetry data from your application. This data provides insights into the performance, latency, and behavior of your application in production.
Continuous evaluations help monitor the quality and safety of your agent in production by assessing its outputs against predefined metrics and thresholds.
Viewing monitoring results
In Azure AI Foundry portal, the Application analytics dashboard view uses signals from Azure Monitor Application Insights, querying it through Azure Workbooks and creating visualizations.
These views are designed to bring key metrics - token consumption, latency, exceptions, response quality into a single pane of glass that provides transparency to teams to track operational health and quality, understand trends, and assess continuously to improve their application.
Follow these steps to access and utilize the built-in monitoring view in your AI Foundry Project:
- Navigate to your AI Foundry Project in the Azure AI Foundry portal.
- Select Monitoring from the left navigation pane.
- Under the Application analytics tab, review the overview of your application's health.
- Use filters to specify a time range, application, and/or model to extract detailed insights.
- If you notice issues, such as declining quality metrics, go to Tracing to debug issues in your application.
- To further customize your monitoring experience and use advanced capabilities in Azure Monitor, scroll to the bottom and select View in Azure Monitor Application Insights.
Note
When you're sharing this workbook with your team members, they must have at least 'Reader' role to the connected Application Insights resource to view the displayed information.
Customize and share your dashboard
Application Insights is a powerful tool for application performance monitoring (APM) that provides insights into the health and performance of your applications.
You can open the Application analytics dashboard in Azure Monitor Application Insights workbooks gallery by selecting on View in Azure Monitor Application Insights link at the end of the page.
This dashboard is opened as an editable workbook where you can customize the workbook and save according to your needs.
Select Edit in the command bar.
Modify elements as needed per your use case. Select ... on an element to edit, add, move/resize, clone, or remove. For example, you can add a tile using KQL to track a custom attribute you're collecting and not shown in our built-in view.
Save your latest changes and create different views as needed by selecting Save.
Share with your team by selecting "Share" icon in the command bar.
Explore and analyze with Kusto Query Language (KQL)
KQL (Kusto Query Language) is a powerful query language used in Azure to explore, analyze, and visualize large volumes of telemetry and log data.
In the Application analytics dashboard view, you can Open query link by selecting on the icon in the top right for a particular tile or chart.
Once you select that, you can view and run the same KQL queries powering your monitoring view and deep dive into the related data.
Set up Azure Alerts
You can define Azure Alert rules based on the previous KQL queries to proactively detect issues with your post-production operations in the future. Select ... to view more options like New alert rule.
Selecting on the New alert rule button opens a wizard to create an alert rule on the related signal.
To learn more about setting up and managing Azure Alerts to proactively address issues, see Alerts in Azure Monitor.