Azure Monitor Alert rule not stopping

I discovered something interesting while working with Azure Monitor alert rules recently.

If you have an alert that is firing based on a condition, it will continue to fire until the condition is cleared, even if the alert rule itself is modified, disabled, or even deleted.

Here’s the example:

I have a B-Series VM, and want to alert on the “CPU Credits Remaining” metric, to pro-actively catch intervals when CPU usage is causing credit exhaustion and thus reduced compute capacity.

I created an Alert Rule to fire when the “CPU Credits Remaining” to fire when the value is 100 or less. It was configured with a frequency of 1 minute and a period of 5 minutes (because I wasn’t thinking of the implications at the time).

This worked just great for a while! And then a rogue Windows Update process got stuck consuming 60% of the CPU for a period of time, and the credit count dropped all the way to zero.

The alert began to fire as expected, once per minute, which quickly became excessive and drowned a bunch of mailboxes in alert-overload.

“Ok, lets just disable the alert” – nope, it continued to fire. I modified the rule so that the frequency and period were much greater. However, the email alerts received continued to reflect the original values:

Even deleting the rule did not stop the alerts from triggering.

Interestingly, if I modified the action group to use a different target email address, that was immediately effective. This allowed me to black-hole the emails until I had resolved the CPU utilization problem and waiting until the credits built back up.

 

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.