Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
You can use the Microsoft Fabric eventstreams feature to monitor streaming event data, ingestion status, and ingestion performance. This article explains how to use metrics to monitor the eventstream status and check logs, errors, and data insights.
An eventstream has two types of monitoring experiences: Data insights and Runtime logs. One or both views are available, depending on the source or destination that you select.
Prerequisites
- Access to a workspace with Viewer or higher permissions where your eventstream item is located.
- An Azure event hub source or lakehouse destination added to your eventstream.
Data insights
The Data insights tab appears on the lower pane of the main editor. The tab provides metrics that you can use to monitor the status and performance of the eventstream, sources, and destinations. Different sources and destinations have different metrics.
When you select a node on the main editor canvas, the metrics for that specific node appear on the Data insights tab.
Data insights for an eventstream node
The following metrics appear for an eventstream node on the Data insights tab:
Metric | Unit | Description |
---|---|---|
IncomingMessages | Count | Number of events or messages sent to an eventstream over a specified period. |
OutgoingMessages | Count | Number of events or messages sent from an eventstream over a specified period. |
IncomingBytes | Bytes | Incoming bytes for an eventstream over a specified period. |
OutgoingBytes | Bytes | Outgoing bytes for an eventstream over a specified period. |
To view data insights for an eventstream:
On the main editor canvas, select the eventstream node.
On the lower pane, select the Data insights tab.
If data is inside the eventstream, the metrics chart appears.
On the right side of the tab, select the checkboxes next to the metrics that you want to display.
Data insights for Azure Event Hubs, Azure IoT Hub, lakehouse, eventhouse, derived stream, and Fabric Activator nodes
The following metrics are available on the Data insights tab for Azure Event Hubs, Azure IoT Hub, lakehouse, eventhouse (Event processing before ingestion mode), derived stream, and Fabric Activator nodes:
Metric | Unit | Description |
---|---|---|
Input events | Count | Number of event data items that the eventstream engine pulls from an eventstream (in a lakehouse, eventhouse, derived stream, or Fabric Activator destination) or from an event source (in an Azure Event Hubs or Azure IoT Hub source). |
Input event bytes | Bytes | Amount of event data that the eventstream engine pulls from an eventstream (in a lakehouse, eventhouse, derived stream, or Fabric Activator destination) or from an event source (in an Azure Event Hubs or Azure IoT Hub source). |
Output events | Count | Number of event data items that the eventstream engine sends to a lakehouse or eventhouse (in a lakehouse, eventhouse, derived stream, or Fabric Activator destination) or from an event source (in an Azure Event Hubs or Azure IoT Hub source). |
Backlogged input events | Count | Number of input events that are backlogged in the eventstream engine. |
Runtime errors | Count | Total number of errors related to event processing. |
Data conversion errors | Count | Number of output events that couldn't be converted to the expected output schema. |
Deserialization errors | Count | Number of input events that couldn't be deserialized inside the eventstream engine. |
Watermark delay | Second | Maximum watermark delay across all partitions of all outputs for this source or destination. It's computed as the wall clock time minus the largest watermark. |
To view the data insights for an Azure event hub, Azure IoT hub, lakehouse, eventhouse (Event processing before ingestion mode), derived stream, or Fabric activator:
On the main editor canvas, select the Azure Event Hubs, Azure IoT Hub, lakehouse, eventhouse, derived stream, or Fabric Activator node.
On the lower pane, select the Data insights tab.
If data is inside the event hub, IoT hub, lakehouse, eventhouse, derived stream, or Fabric activator, the metrics chart appears.
On the right side of the tab, select the checkboxes next to the metrics that you want to display.
Data insights for streaming connector source nodes
Streaming connector source nodes include the following sources:
- Azure SQL Database Change Data Capture (CDC)
- Azure Service Bus
- Azure Database for PostgreSQL CDC
- Azure Database for MySQL CDC
- Azure Cosmos DB CDC
- SQL Server on virtual machine (VM) database (DB) CDC
- Azure SQL Managed Instance CDC
- Google Cloud Pub/Sub
- Amazon Kinesis Data Streams
- Confluent Cloud Kafka
- Apache Kafka
- Amazon MSK Kafka
The following metrics are available on the Data insights tab for streaming connector source nodes:
Metric | Unit | Description |
---|---|---|
Source Outgoing Events | Count | Number of records sent from the transformations (if any) and written to an eventstream for the task that belongs to the named source connector in the worker (since the task was last restarted). |
Source Incoming Events | Count | Number of records produced or polled by the task that belongs to the named source connector in the worker (since the task was last restarted), before transformations are applied. |
Connector Errors Logged | Count | Number of errors that were logged for these connector tasks. |
Connector Processing Errors | Count | Number of record processing errors in these connector tasks. |
Connector Processing Failures | Count | Number of record processing failures in these connector tasks, including retry failures. |
Connector Events Skipped | Count | Number of records skipped due to errors within these connector tasks. |
To view the data insights for a streaming connector source:
Select Use external source, and then choose a streaming connector source.
Configure and publish the streaming connector source.
On the lower pane in live view, select the Data insights tab.
If data is inside the streaming connector source, the metrics chart appears.
On the right side of the tab, select the checkboxes next to the metrics that you want to display.
Runtime logs
Use the Runtime logs tab to check the detailed logs that occur in the eventstream engine. Runtime logs have three severity levels: warning, error, and information.
To view the runtime logs for an Azure event hub, Azure IoT hub, streaming connector source, lakehouse, eventhouse (Event processing before ingestion mode), or Fabric activator:
On the main editor canvas, select the node.
On the lower pane, select the Runtime logs tab.
If data is inside the Azure event hub, Azure IoT hub, streaming connector source, lakehouse, eventhouse, or Fabric activator, the logs appear.
Search the logs by using the Filter by keyword option, or filter the list by changing the severity or type.
To see the most current logs, select Refresh.