Edit

Share via

How to log events to Azure Event Hubs in Azure API Management

APPLIES TO: All API Management tiers

This article describes how to log API Management events by using Azure Event Hubs.

Azure Event Hubs is a highly scalable data ingress service that can ingest millions of events per second so that you can process and analyze the massive amounts of data produced by your connected devices and applications. Event Hubs acts as the "front door" for an event pipeline, and after data is collected into an event hub, you can transform and store it by using any real-time analytics provider or batching/storage adapters. Event Hubs decouples the production of a stream of events from the consumption of those events, so that event consumers can access the events on their own schedule.

Note

Currently, this feature isn't available in workspaces.

Prerequisites

Configure access to the event hub

To log events to the event hub, you need to configure credentials for access from API Management. API Management supports either of the two following access mechanisms:

  • A managed identity for your API Management instance (recommended)
  • An Event Hubs connection string

Note

We recommend that you use managed identity credentials when possible, for enhanced security.

Option 1: Configure an API Management managed identity

  1. Enable a system-assigned or user-assigned managed identity for API Management in your API Management instance.

    • If you enable a user-assigned managed identity, take note of the identity's Object ID.

  2. Assign the identity the Azure Event Hubs Data sender role, scoped to the Event Hubs namespace or to the event hub used for logging. To assign the role, use the Azure portal or another Azure tool.

Option 2: Configure an Event Hubs connection string

To create an Event Hubs connection string, see Get an Event Hubs connection string.

  • You can use a connection string for the Event Hubs namespace or for the specific event hub you use for logging from API Management.
  • The shared access policy for the connection string must enable at least Send permissions.

Create an API Management logger

The next step is to configure a logger in your API Management service so that it can log events to the event hub.

Create and manage API Management loggers by using the API Management REST API directly or by using other tools, like Azure PowerShell, a Bicep file, or an Azure Resource Management template.

You can configure an API Management logger to an event hub by using either system-assigned or user-assigned managed identity credentials.

Create a logger with system-assigned managed identity credentials

For prerequisites, see Configure an API Management managed identity.

Use the API Management Logger - Create or Update REST API member with the following request body.

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "Event Hub logger with system-assigned managed identity",
    "credentials": {
         "endpointAddress":"<EventHubsNamespace>.servicebus.windows.net",
         "identityClientId":"SystemAssigned",
         "name":"<EventHubName>"
    }
  }
}

Create a logger with user-assigned managed identity credentials

For prerequisites, see Configure an API Management managed identity.

Use the API Management Logger - Create or Update REST API member with the following request body.

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "Event Hub logger with user-assigned managed identity",
    "credentials": {
         "endpointAddress":"<EventHubsNamespace>.servicebus.windows.net",
         "identityClientId":"<ClientID>",
         "name":"<EventHubName>"
    }
  }
}

Option 2. Create a logger with connection string credentials

For prerequisites, see Configure an Event Hubs connection string.

Note

We recommend that you configure the logger with managed identity credentials when possible. See Configure a logger with managed identity credentials, earlier in this article.

The following example uses the New-AzApiManagementLogger cmdlet to create a logger to an event hub by configuring a connection string.

# Details specific to API Management 
$apimServiceName = "apim-hello-world"
$resourceGroupName = "myResourceGroup"

# Create logger
$context = New-AzApiManagementContext -ResourceGroupName $resourceGroupName -ServiceName $apimServiceName
New-AzApiManagementLogger -Context $context -LoggerId "ContosoLogger1" -Name "ApimEventHub" -ConnectionString "Endpoint=sb://<EventHubsNamespace>.servicebus.windows.net/;SharedAccessKeyName=<KeyName>;SharedAccessKey=<key>" -Description "Event hub logger with connection string"

Configure a log-to-eventhub policy

After your logger is configured in API Management, you can configure your log-to-eventhub policy to log the desired events. For example, use the log-to-eventhub policy in the inbound policy section to log requests, or in the outbound policy section to log responses.

  1. Go to your API Management instance.

  2. Under APIs, select APIs, and then select the API to which you want to add the policy. In this example, we're adding a policy to the Echo API in the Unlimited product.

  3. On the Design tab, select All operations.

  4. In the Inbound processing or Outbound processing pane, select the </> (Policy code editor) button. For more information, see How to set or edit policies.

  5. Position your cursor in the inbound or outbound policy section.

  6. Select Show snippets at the top of the tab. Select Advanced policies > Log to EventHub. This action inserts the log-to-eventhub policy statement template.

    <log-to-eventhub logger-id="logger-id">
    @{
        return new JObject(
            new JProperty("EventTime", DateTime.UtcNow.ToString()),
            new JProperty("ServiceName", context.Deployment.ServiceName),
            new JProperty("RequestId", context.RequestId),
            new JProperty("RequestIp", context.Request.IpAddress),
            new JProperty("OperationName", context.Operation.Name)
        ).ToString();
    }
</log-to-eventhub>
    1. Replace logger-id with the name of the logger that you created in the previous step.
    2. You can use any expression that returns a string as the value for the log-to-eventhub element. In this example, a string in JSON format containing the date and time, service name, request ID, request IP address, and operation name is logged.

  7. Select Save to save the updated policy configuration. As soon as the configuration is saved, the policy is active and events are logged to the designated event hub.

Note

The maximum supported message size that can be sent to an event hub from this API Management policy is 200 kilobytes (KB). If a message that's sent to an event hub is larger than 200 KB, it's automatically truncated, and the truncated message is transferred to the event hub. For larger messages, consider using Azure Storage with API Management as a workaround to bypass the 200-KB limit. For more information, see Send requests to Azure Storage from API Management.

Preview the log in Event Hubs by using Azure Stream Analytics

You can preview the log in Event Hubs by using Azure Stream Analytics queries.

  1. In the Azure portal, go to the event hub that the logger sends events to.
  2. Under Features, select Process data.
  3. On the Enable real time insights from events card, select Start.
  4. You should be able to preview the log on the Input preview tab. If the data shown isn't current, select Refresh to see the latest events.

Related content