Heartbeat monitoring catches the jobs that never ran

A backup job can stop running while the website continues to return successful pages. The same is true for imports, report generation, cache warming, certificate renewal tasks and scheduled maintenance scripts. An endpoint check cannot detect work that never started.

The silent-failure problem

A conventional monitor initiates the request. A heartbeat monitor reverses that relationship: the job sends a signal after it reaches a defined point, and the monitoring service alerts when the signal does not arrive inside the expected window. This is sometimes called a dead-man switch.

The signal should represent meaningful progress. A heartbeat sent at process start proves only that the scheduler launched something. For backups, the most useful completion signal normally follows a successful write and any required verification.

Define the heartbeat contract

Write down the expected schedule, tolerated delay, success point, timezone and owner. Include daylight-saving behaviour and long-running jobs. A daily task expected at 02:00 should not alert at 02:01 if normal completion takes forty minutes.

• Expected schedule and timezone are explicit.
• The grace window reflects real execution time.
• The heartbeat is sent only after the chosen success condition.
• Failures can send a distinct signal when appropriate.
• The alert identifies the job, client, last success and owner.

Implement without hiding the original failure

The heartbeat call must not turn a successful job into a failure without an intentional policy, and it must not overwrite the job's real exit status. Use bounded network timeouts. Keep the heartbeat endpoint secret because anybody who can call it may be able to create a false success signal.

For high-value jobs, consider a second verification such as checking the age, size or integrity of the output. A completion ping from a script bug can otherwise report success before the expected artifact exists.

Route missed heartbeats according to the job

A missed image-optimisation task may wait for business hours. A missed database backup may require immediate triage. Apply severity based on the protected outcome rather than giving every scheduled job the same route.

Include the last successful completion, current overdue duration, recent schedule changes and first-response instruction. Avoid repeatedly notifying the same overdue job unless escalation rules require it.

Test failure and recovery

1. Use a safe non-production job or approved test schedule.
2. Confirm a successful completion records exactly one heartbeat.
3. Suppress the heartbeat and verify the alert after the grace window.
4. Restore the signal and verify recovery is recorded once.
5. Check that the original job exit code and logs remain intact.
6. Record the test date and owner.

Never create a production failure test against a client system without explicit authorisation and a rollback plan.