CronCraft

CronCraft / Guide

Cron collisions: find the shared minute before it finds you

A crontab can be individually correct and collectively harmful. When several jobs share a minute, the load arrives as a spike. This guide shows how to detect it and which fix fits which job.

What a collision is

Cron has minute resolution. Every job that matches the current minute fires at the same instant, at least as far as the scheduler is concerned. Five jobs that each run hourly are harmless if their minutes differ. Five jobs that each run at minute zero are a burst of five processes, five database connections and five API calls, every hour, forever.

A collision is not automatically a bug. Two small jobs can share a minute without consequence. The problem starts when the combined work is larger than the capacity available in that minute, or when the jobs contend for the same resource: a database table, a file, a rate limited API or a single worker.

A typical collision

Consider this crontab:

*/5 * * * * health-check
0 * * * * hourly-report
15 2 * * 1-5 reindex-search
30 2 * * * rotate-logs
0 9 * * 1-5 daily-summary

The health check and the hourly report meet at minute zero of every hour: twenty four collisions a day. The reindex and the log rotation do not collide, which is good, but they both run in the maintenance window where backups may also run. The daily summary at 09:00 may meet other business hour jobs that are not in this file.

The collision checker walks the window minute by minute and reports every interval where two or more jobs fire, with the job names and the count. The weekly density map shows the shape: a solid band at the top of each hour is easy to see, and so is a quiet period that could absorb a moved job.

Fix one: stagger the schedule

The cheapest fix is to change the minute. Instead of five jobs at minute zero, use 0, 7, 14, 21 and 28. The hourly total is the same and the peak is five times smaller. For jobs that do not need an exact minute, this is almost always enough.

Choose the offset with the rest of the crontab in mind. A job at minute 7 may meet a job that runs every five minutes at minute 5 and 10, which is fine, but a job at minute 10 would meet it exactly. The collision checker makes those overlaps visible before you pick.

Fix two: random jitter inside the job

Staggering is static. If a fleet of machines all read the same crontab, they all start at the same minute even after you move it. Adding a random sleep at the start of the job spreads the work over a window:

#!/bin/sh
# spread up to 90 seconds, then do the work
sleep $(( $(od -An -N2 -tu2 /dev/urandom) % 90 ))
/usr/local/bin/actual-work.sh

Use jitter when the exact start time does not matter and the work is safe to shift. Do not use it for jobs that must align with an external clock, such as a report that must be ready before a market opens.

Fix three: a lock

Two runs of the same job overlapping is a different problem from two different jobs colliding. If a job can take longer than its interval, use a lock so the second run exits immediately:

* * * * * flock -n /var/lock/nightly.lock /usr/local/bin/nightly.sh

flock -n fails instead of waiting, which is usually right for cron. Without a lock, a slow job can pile up copies and exhaust memory or connections. Kubernetes has the same idea in concurrencyPolicy: Forbid, and most job frameworks have an equivalent.

Fix four: idempotency

A lock prevents simultaneous runs on one machine. It does not help when the same logical job runs twice because of a retry, a fall back hour or a failover. Idempotency handles that case: the job checks whether the work for this period is already done and exits if it is.

period=$(date -u +%Y-%m-%d)
if /usr/local/bin/already-done "$period"; then exit 0; fi
/usr/local/bin/work.sh "$period"
/usr/local/bin/mark-done "$period"

With a period key, the spring forward skip and the fall back duplicate both become recoverable. The job can run an extra time without harm, and a missed run can be repaired by a later check.

Fix five: move the work to a queue

Cron is a timer, not a scheduler with capacity awareness. If the work is heavy, heterogeneous or needs retries, let cron enqueue a message and let workers process it at their own rate. The cron job becomes tiny and the queue absorbs the burst. This is the standard answer for fan out jobs that would otherwise start a hundred workers at the same second.

Reading the density map

The weekly map in the collision tab counts every firing by local weekday and hour across the next seven days. A dark cell means many jobs in that hour. Look for three things:

  • A column of dark cells at the same hour every day: a daily batch that could be moved.
  • A single dark cell: a weekly job that meets a daily one, easy to stagger.
  • An empty region near a busy one: a good place to move a job without creating a new collision.

The map does not know how long each job takes or how much capacity it uses. It shows when the scheduler fires, which is the part you can control from the crontab.

A checklist before adding a job

  1. Search the existing crontab for the same minute and hour.
  2. Run the collision check with the new line added and read the new collisions.
  3. Decide whether the job needs an exact minute or can be staggered.
  4. Add a lock if the run time can exceed the interval.
  5. Add a period key if a duplicate run would be visible or expensive.
  6. Check the next runs across a DST transition if the job runs near 01:00 to 03:00.