« Back
read.

Azure Api Management - Log To EventHub.

A few weeks ago I was given the opportunity by the APIM team to try out the log-to-eventhub feature in a private preview. After a short proof of concept we decided to run it in production and has done successfully for a while now.

To get this up and running there are a few things that need to be configured.

Create Azure Event Hub

Service Bus namespaces are still only available in the classic portal so we need to go there.

Create the event hub by clicking New>App Services>Service Bus>Event Hub

Set the partition count to 4 and retention days to 7. (Other values can be used here).
Next we need to create shared access policies for sending to and receiving from the Event Hub. This is done under the Configure Tab for the Event Hub. Make sure you are under the Event Hub and not the Service Bus.

Last step is to copy the connectiong strings for these policies. Again, go to the dashboard for the Event Hub and click on the Key, bottom center.

Connection Strings are of the format:
Endpoint=sb://privatepreviewrules-ns.servicebus.windows.net/;SharedAccessKeyName=Send;SharedAccessKey=zL5BpULXKGheIHOacKQNKUhrgPh3Oo2Jsmx5UiNxP9U=

Create API Management Loggers

Next step is to create a Logger the API Management to send events to the Event Hub. This is done using the API Management Rest API. First we need to enable access to that api. A detailed instruction can be found here.

To create a logger. Make a HTTP PUT request to the following url:
https://{your service}.management.azure-api.net/loggers/{new logger name}?api-version=2014-02-14-preview

Make sure to use the SEND policy connection string

The body must be a json:
{ "type" : "AzureEventHub", "description" : "Sample logger description", "credentials" : { "name" : "Name of the Event Hub from the Azure portal", "connectionString" : "Endpoint=Event Hub Sender connection string" } }

Following headers must be set:

A few screenshots from Postman to help you out:
If you get a HTTP 201 response the logger has been created. You can check this by making a GET as seen below.

Modify API Management Policies

Go back to the API Management Admin Portal and go to the Policies section.

As you probably know a policy is a chain of statements applied to the request and response. At any point in this chain we can insert a log-to-eventhub statement. Each time an event is sent to the event hub.
My clients primary objective was to log bodies so we used this configuration
<log-to-eventhub logger-id="demo"> @( string.Join("##", DateTime.UtcNow, context.Deployment.ServiceName, context.User.Email, context.Deployment.Region, context.RequestId, context.Request.IpAddress, context.Operation.Name, "Request", context.Request.Body.As<string>(preserveContent: true))) </log-to-eventhub>

A few points of interest here:

  • Whatever is inside the xml tag must return a string value. Simple.
  • If you log more then one time in a policy. Logging RequestId is very useful.
  • The field we set to "Request" is for us an indication of status of the call. Request/RequestFromUser/RequestToBackend/Response/ResponseFromBackend/ResponseToUser is just a couple of values we have used.
  • Use a good delimiter or you might end up with strange errors... ;)
  • The API management portal sometimes replaces <> with > and < when a policy is saved and then opened again. If you run into problems, this is often the case. Manually put <> back in the configuration.

A full example of a policy:
<policies> <inbound> <base /> <log-to-eventhub logger-id="demo"> @( string.Join("##", DateTime.UtcNow, context.Deployment.ServiceName, context.User.Email, context.Deployment.Region, context.RequestId, context.Request.IpAddress, context.Operation.Name, "Request", context.Request.Body.As<string>(preserveContent: true))) </log-to-eventhub> </inbound> <backend> <base /> </backend> <outbound> <base /> <log-to-eventhub logger-id="demo"> @( string.Join("##", DateTime.UtcNow, context.Deployment.ServiceName, context.User.Email, context.Deployment.Region, context.RequestId, context.Request.IpAddress, context.Operation.Name, "Response", context.Response.Body.As<string>(preserveContent: true))) </log-to-eventhub> </outbound> </policies>

Read events from Event Hub

We used this guide to quickly get events read for the event hub.
Our implementation can be found at GitHub.
What we added to that code was basically converting the message to an object and then send it to an Elasticsearch instance.

The result

We can now use one of the best suites available when it comes to analyzing and searching in data, Elasticsearch + Kibana. The analytics section of API Management is good but very limited. This new feature lets us get the best of both API management and of analytics.
Our teams can now using free text search see all request and responses going through Azure Api Management.

comments powered by Disqus