Monitoring Azure Data Factory

In the previous post, we looked at the three different trigger types, as well as how to trigger pipelines on-demand. In this post, we will look at what happens after that. How does monitoring work in Azure Data Factory?

Now, if we want to look at monitoring, we probably need something to monitor first. I mean, I could show you a blank dashboard, but I kind of already did that, and that wasn’t really interesting at all 🤔 So! In the previous post, I created a schedule trigger that runs hourly, added it to my orchestration pipeline, and published it.

Let’s take a look at what has happened since then!

Monitoring Overview

First, let’s navigate to the Monitor page, which is divided into five sub-pages:

We’ll start at the top.

Dashboards

Under dashboards, you will see statistics of your pipeline, activity, and trigger runs. In my case…

…uh-oh 😅

Turns out, if you drop a table that you’re trying to load data into, your pipelines start failing. Who could ever have predicted that? 😂 Now, your goal is obviously no failures, but I wanted my pipelines to fail to show you a few things.

If you click on the pipeline chart, you will be taken to the pipeline runs page, with a pre-defined filter of either succeeded or failed.

Pipeline Runs

I clicked on failed, and everything is red and sad, aww 😔 But! You can click the filter to change the view:

If you switch from failed to succeeded…

…everything looks green and happy, yay 😃

You probably want to see everything, though, to get a full view of your environment. You can change the filter again from succeeded to all statuses, then view the results as a list…

…or as a Gantt chart:

If you are working in multiple time zones, it’s easy to see the pipeline runs in your local time zone. Select a new time zone…

…and the dates and times will adjust automatically, while the view itself remains the same:

Alright, cool! But…

What do I do about failed pipeline runs?

You can click on three buttons. One, you can view activity runs. Two, you can rerun the pipeline. Three, you can view the error message.

I usually check the error message, then see if I have to rerun the pipeline, otherwise look at the activity runs.

Activity Runs

The activity runs view should look familiar 😊 This is almost the same as the debugging output view. You can click the actions, view the details, and so on:

But there’s one difference! Remember how we couldn’t debug from? At least we can rerun from! That certainly helps with production workloads!

In this example, we have a bunch of failed pipelines. (Remember not to drop tables in production! 🤪) The trigger runs, however…

Trigger Runs

…are just fine! That’s because all the triggers started successfully. One thing to note, however, is the slight offset on some of the trigger runs:

You can click on properties to see the scheduled trigger time and the actual trigger time:

Integration Runtimes

Under integration runtimes, you can monitor… well… your integration runtimes. We haven’t created any new integration runtimes yet, so we currently only see the default AutoResolveIntegrationRuntime:

This should always be running. If this is not running, Azure is experiencing some serious issues. In that case, check the Azure Status page for more information.

Alerts and Metrics

Finally, while the dashboards and overviews are nice and handy, you may not want to open them multiple times per day to check that everything is running smoothly.

(And if you do, you can just skip this part 😅)

You can set up alerts to get notifications if something fails (or succeeds!) in your Azure Data Factory.

How do I create a new alert?

From the alerts and metrics page, click new alert rule:

Give the alert a name and description, and choose the severity. In the Azure Data Factory interface, you get no explanations of the severity levels, but these are listed in the official documentation:

• Sev 0 = Critical
• Sev 1 = Error
• Sev 2 = Warning
• Sev 3 = Informational
• Sev 4 = Verbose

Then, click + add criteria:

You can add alerts for a whole bunch of metrics. The most common ones are probably the failed metrics, but maybe you want to notify someone when a pipeline succeeds? You can do both:

Once you select a metric, you configure the alert rules for that metric:

In dimension name, choose the pipelines you want to monitor:

In failure type, choose the failure types you want to monitor:

Then, set the threshold for how many pipelines need to fail before you get an alert:

And then, the frequency for how often you want to check for failures:

Finally, configure which notifications you want to get. Click + configure notification:

Give the new action group a name and click + add notification:

Choose the type of notification you want to receive, and fill in the required information. For example, to receive an email notification, fill in your email:

Now click save, save, save, save, save until you’re done saving 😄

If you get the MissingSubscriptionRegistration error:

…follow these steps 😊

Tadaaa! Your alerts and metrics page is now slightly less empty:

Summary

In this post, we looked at how monitoring and alerting works. But what if we want to customize the monitoring views even further? Is there a way to do that in Azure Data Factory?

Well, I’m glad you asked! Because that just so happens to be the next topic I want to cover. Let’s look at adding annotations and user properties next!

About the Author

Cathrine Wilhelmsen is a Microsoft Data Platform MVP, international speaker, author, blogger, organizer, and chronic volunteer. She loves data and coding, as well as teaching and sharing knowledge - oh, and sci-fi, gaming, coffee and chocolate 🤓