Knowledge BaseMetric InsightsDistributionBurstsNo bursts sending for some period of time and there are no errors in logs

No bursts sending for some period of time and there are no errors in logs

Issue:

The bursts are not getting sent, no errors in the logs. All Notification Schedules are timing out. 

RESOLUTION:

  • Identify the Notification Schedule ID for one of the problematic notification schedules that were running recently (could be found in the URL of the editor of this notification schedule)
  • Connect to the web-master pod
  • Connect to the database mysql dashboard
  • Run the next query select * from notification_schedule_run_log where notification_schedule_id = <NS_ID> order by run_finish_time desc limit 1 \G change NS_ID to the ID from the previous step and copy ns_run_id from the result
  • Run the next query select * from notification_schedule_run_log_detail where ns_run_id=<ns_run_id> where paste ns_run_id from the previous step
  • If no results on the previous query run the next one show create table notification_schedule_run_log_detail and check the value of AUTO_INCREMENT if it is reached 2147483647, need to create the backup of the existing table by running CREATE TABLE notification_schedule_run_log_detail_bkp AS SELECT * FROM notification_schedule_run_log_detail;
  • Then truncate this table by running TRUNCATE TABLE notification_schedule_run_log_detail;