Alerts and Digests are not going out at all
Alerts are not being sent out immediately even though the alerts can be seen in the Viewer and Editor. The alert rule is set to send immediately, by email, everytime so it should be going out. In fact, I can see the alerts queued up to be sent out in the Metric Editor > Alerts tab, but is never sent (no timestamp of delivery). It's definitely not an issue with the Notification Schedule because we have a 3 minute schedule setup to pool alerts immediately for delivery.
What's more, digests have also not gone out lately. I can definitely manually send a Digest, and sending a Test Email from the Status Monitor page is not a problem. Also, the Status Monitor shows a zero for the Email Queue so it's not an issue with the queue.
This appears to have started around when Metric Insights was upgraded to 3.2.833. What's going on?
There appears to be no smoking gun in this case. Everything is as it should be. The issue appears to be with alerts and digests being sent out automatically by the system - it's not pooling the alerts to deliver. For example, manually sending out alerts by running notes_check.sh works. So what gives?
Turns out that in /etc/cron.d/metricinsights notes_check.sh and send_following.sh were both commented out! This was due to a mixup for the 3.2.833 build as we move to check_notification scripts for version 3.3.
Re-enabling the two scripts (delete # sign) allows for alerts and digests to go out. You can also update to 3.2.834 which corrects this issue.