Server Status

Monitor host, connectivity, services, agent, and alert channel separately. Show status, timestamp, and age of the check. An active process needs response evidence; a recent heartbeat does not attest to the health of all its tasks.

Observation points

Place the monitor outside the host and its immediate dependency to detect loss of power or network. Receive authenticated heartbeats or perform active tests with timeout. When possible, send alerts through a channel independent of the monitored process.

A local watchdog complements that monitor and can restart processes within an authorized policy. Upon a delayed signal, diagnose the process before interrupting sessions or shifts that are still running.

Signals and status

Define per host/service identity, expected interval, tolerance, last accepted heartbeat, last real test, status, and last alarm delivery. Keep tokens, private endpoints, and inventories out of examples and public alerts.

Authenticate senders, limit size and frequency, and protect against replay via sequence/nonce or authenticated protocol. Store the reception time: a client timestamp does not prove freshness. The heartbeat provides data, never commands or arbitrary URLs to execute.

Use a monotonic clock for intervals and UTC for persisted auditing. After a clock jump, restart, or gap in samples, reset candidates or return to unknown; recovery requires a valid sample.

Adjust interval and tolerance to network, load, and criticality. As a starting point, you can send every 30 seconds and alert after 3 minutes without a signal. Distinguish unknown, healthy, suspect, and down, with an initial registration window.

Alarms and recovery

• Require persistence or consecutive samples. For RAM, disk, and load use hysteresis; interpret load relative to the number of CPUs.
• Deduplicate by host, signal, and episode. Send opening and verified recovery; configure reminders separately.
• Confirm delivery before marking the alert as sent. On provider failure, preserve incident and health, and retry with progressive backoff and a limit.
• Distinguish check failure, configuration change, and maintenance from a recovery. Removing a service from the configuration does not make it healthy.
• Apply corrections only under a policy authorized for the resource. An alarm alone does not authorize restarts, updates, or privilege escalation.

Verification

Use a simulated sender or test environment to rehearse heartbeat loss, sender and monitor outage, intermittent network, delayed clock, corrupted state, alert channel failure, and recovery.

Verify external detection of absence, retry of failed alerts, unknown status without false positives, and continuity of healthy sessions. Record the result and the dependencies that remain uncovered.

Reference: https://www.freedesktop.org/software/systemd/man/latest/systemd.timer.html