No bursts sending for some period of time and there are no errors in logs

Issue:

The bursts are not getting sent, no errors in the logs. All Notification Schedules are timing out. 

RESOLUTION:

  • Identify the Notification Schedule ID for one of the problematic notification schedules that were running recently (could be found in the URL of the editor of this notification schedule)
  • Connect to the web-master pod
  • Connect to the database mysql dashboard
  • Run the next query select * from notification_schedule_run_log where notification_schedule_id = <NS_ID> order by run_finish_time desc limit 1 \G change NS_ID to the ID from the previous step and copy ns_run_id from the result
  • Run the next query select * from notification_schedule_run_log_detail where ns_run_id=<ns_run_id> where paste ns_run_id from the previous step
  • If no results on the previous query run the next one show create table notification_schedule_run_log_detail and check the value of AUTO_INCREMENT if it is reached 2147483647, need to create the backup of the existing table by running CREATE TABLE notification_schedule_run_log_detail_bkp AS SELECT * FROM notification_schedule_run_log_detail;
  • Then truncate this table by running TRUNCATE TABLE notification_schedule_run_log_detail;