No bursts sending for some period of time and there are no errors in logs
Issue:
The bursts are not getting sent, no errors in the logs. All Notification Schedules are timing out.
RESOLUTION:
- Identify the Notification Schedule ID for one of the problematic notification schedules that were running recently (could be found in the URL of the editor of this notification schedule)
- Connect to the web-master pod
- Connect to the database
mysql dashboard
- Run the next query
select * from notification_schedule_run_log where notification_schedule_id = <NS_ID> order by run_finish_time desc limit 1 \G
change NS_ID to the ID from the previous step and copyns_run_id
from the result - Run the next query
select * from notification_schedule_run_log_detail where ns_run_id=<ns_run_id>
where paste ns_run_id from the previous step - If no results on the previous query run the next one
show create table notification_schedule_run_log_detail
and check the value of AUTO_INCREMENT if it is reached 2147483647, need to create the backup of the existing table by runningCREATE TABLE notification_schedule_run_log_detail_bkp AS SELECT * FROM notification_schedule_run_log_detail;
- Then truncate this table by running
TRUNCATE TABLE notification_schedule_run_log_detail;