Notes on taming a noisy cron schedule
Somewhere around the third year of a project, cron stops being a scheduler and
starts being a landfill. Ours had 41 entries, four of which ran every minute,
and nothing that logged its exit code anywhere useful.
The first pass was purely mechanical: pull every entry, attach an owner, and mark the ones nobody would claim. Nine were dead. Two had been failing silently since a hostname change eighteen months earlier.
Make the schedule legible
Cron's failure mode is silence. If a job does not run, you find out weeks later when a report is empty. Three changes fixed most of this:
# every job writes a heartbeat, not just an error
*/5 * * * * /usr/local/bin/collect-metrics >> /var/log/jobs/collect-metrics.log 2>&1 || logger -t cron "collect-metrics FAILED"
- Every job logs both success and failure to a directory that gets rotated.
- Long jobs acquire a lock file so overlapping runs are impossible.
- Anything with a business impact moved to a real scheduler with retries.
After consolidation we were down to 23 entries, and the fleet-wide CPU graph lost the regular sawtooth it had carried for a year. Nobody had noticed the sawtooth. Everybody noticed when it was gone.