DST was a mistake and it needs to die. It solves no problems and only creates them. And not even the type of problem that someone could profit from, just plain old complete waste of time for everyone problems.
Just stick to winter time (because that's the correct one) and adjust all working hours once and for all to be like summer time, for e.g. instead of business opening at 7AM and closing at 10PM they can open at 6AM and close a 9PM. There will be a period of adjustment, but we have a period of adjustment twice a year already, so nothing is lost.
Why match working hours to summer time? Because I want to have more sunlight by the time I'm done with work. Especially during winter. We have it completely backwards, if we are going to adjust our clocks we should adjust them in a way that gives us more sunlight during winter, not less. I don't care if it's dawning by the time I'm going to work where I will be stuck indoors for nine hours, I want sunlight when I'm free again.
I've told this story before, but it's super relevant here. When I set up all the servers for reddit, I set them all to Arizona time. Arizona does not have DST, and is only one hour off of California (where we all were) for five months, and the same as here the rest of the year.
I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.
I worked at a place that had set the servers to BST not UTC and used cron jobs extensively for reporting, which caused chaos twice a year and twice a year they had to explain to the business again why it caused chaos and ask for the time to fix it.
Company went bust before it got fixed.
Just use UTC folks unless you have a really good idea why you shouldn't.
Redundantly running cron jobs are the least of our problems. Daylight savings adjustment day causes surge in deaths and ER visits[1]. Daylight savings needs to go.
This was worked around ages ago in OpenBSD, and the workaround was already included in Debian (and by extension, Ubuntu) when I started maintaining it in 2010 (I no longer do).
Here's a link to a patch [1] from a version from when the package still kept standalone patches.
It's been a long long time and I honestly don't remember the details, but Debian's cron(8) still says [2]: Special considerations exist when the clock is changed by less than 3 hours, for example at the beginning and end of daylight savings time. If the time has moved forwards, those jobs which would have run in the time that was skipped will be run soon after the change. Conversely, if the time has moved backwards by less than 3 hours, those jobs that fall into the repeated time will not be re-run.
Edit: According to this bug report [3], this workaround first entered Debian in 1999.
The details I never really thought about until I encountered timezone issues was:
There are points in time that don't exist, and there are points in time that are ambiguous.
If we adjust the time forward by 1 hour, then then 2:30am doesn't exist. If we adjust time backward, then 2:30am is ambiguous.
2am-3am is only safe for certain countries.
America/Santiago (Chile), for example, adjusts their daylight saving on midnight. On a certain day, midnight doesn't exist because it immediately jumps to 1am. If you are building an analytics chart by day, you will encounter this issue. I wrote about several weird time zone edgecases here: https://tanin.nanakorn.com/edgecases-for-timezones/
To add more confusion, IIRC Java and Ruby handle these cases differently where Ruby raises exceptions (e.g. InvalidTime and AmbiguousTime) but Java doesn't.
I caused 3 incidents in a row at Stripe in my first month because of this. I thought I was gonna be fired. Luckily I wasn't...
I try to avoid crons at the top of the hour, partly because of this... but also because (in shared / serverless infrastructure) I assume a lot more people are setting their crons for 'on the hour' so there's more resource contention... I also aim for 'after 4am' where I can as well, or 'before midnight', to avoid this whole range.
I like cron jobs that run during the work day. If something goes wrong I would prefer to handle it at 10am rather than 10pm. Why make life unpleasant?
If it really needs to run during a low load period I line it up for 6am work time. Again so that if something goes wrong it leads into the work day rather than disrupts the on-call too badly. 6am also provides a daylight savings buffer either way so you don't have to care. Recent work has all been on 24/7 operations though.
While not always possible, in general, any mission critical scheduled job should be idempotent. This avoids so much headache, especially when there's a queue involved somewhere.
I read this a while ago and was looking forward to the mess that would happen with these default Alpine configurations (after setting non-UTC localtime)
> Alternatively, where possible set the server timezone to UTC so that no daylight savings changes will happen at all.
Is this what people actually want, though?
I'm actually working on a scheduled process right now deployed on a cloud scheduler that does not adjust for daylight saving time (it literally says "does not take into consideration daylight saving time" next to the time input).
Everybody wants the job to run four hours before the work day starts, every day of the year. I will have to change it on Sunday to ensure the job completes at the time the end users expect it. And again in March.
2 am is mid afternoon for the Pune office. If you’re running a 24/7 operation and all of your employees are in PST you’re gonna have a bad time.
But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic.
The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration.
It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.
Seems to me the real problem here is not the timezone (there's legitimate business needs to run something daily at a specific localtime...) but having multiple instances of a cron job that overlap, in which case it should wait until the previous is done or not start at all. At least prefix a job with "flock -n" if it doesn't/can't handle that.
anacron is worth a mention here for sometimes-on computers like desktops, laptops, or personal servers. It's great for ensuring something runs at the set interval, or just whenever the machine comes back online. Minimum anacron interval is one day, so this can't simply replace cron. cron might even be the most sensible thing to use to kick off anacron on your system, and it might already be in use (see if your /etc/crontab fires off anacron).
anacron won't have the OP's 2am/3am issue because it locks jobs while they run, so multiple job spawns should be no-ops. Still seems like it would be good practice to, e.g. use UTC for the host system and do whatever else might help avoid the unnecessary re-spawns.
For the most part of Europe and USA the day is either "quite" long (summer) or quite short (winter) due to the curve of the duration. Having light for long summer evenings and dark for long winter mornings is bad for many reasons, one of witch is seasonal depression, with which the Scandinavians suffer and fight. I was for the abolishment of DST until COVID cancelled it in Europe but now I am thinking it again and probably I am against.
I run a cronjob at 2:30AM every day that uses dirvish (a perl wrapper around rsync) to perform backups of several computers. In the fall, at 2AM the clock is rolled back to 1AM. The cronjob runs at 2:30AM, which is 25 hours after the previous day's run. That's not a problem.
However, in the spring, at 2AM the clock jumps to 3AM. Vixie cron seems to be smart enough to run all the cron jobs that were scheduled between 2M and 3AM at that point. However, dirvish is started but exits without running the backups! After some troubleshooting I concluded that dirvish does not make the backups because the previous day's run happened less that 24 hours ago (because that day was shortened by the change to DST) so it sees no need to run a backup yet. This problem would persist even if I moved the cronjob to a different time because it has to do with the day having only 23 hours. I solved this by adding a second entry to the crontab that runs at 3:45AM only on the 2nd Sunday in March.
The problem is not so much which timezone you pick, the problem is that cron uses 'naive' datetimes. If the timezone is explicit then you don't have duplicate hours because one is 02:00 and one is 02:00DST.
And then you can translate that time to any other timezone with ease, knowing that you are referencing a specific hour in physical time that only ever happened once.
Not a crown job, but I configured my lights to turn on progressively at my scheduled wake up time, using Home Assistant.
I figured that scheduling lights to turn on at <0 AM + wake up time> was a good approach... We got a surprise early wake up last year, so I ended up changing the formula to <4 AM - 4 hours + wake-up time>.
I avoid cron job at this time for this reason and for many other random problem because 3:00 am seem to be often choose for a maintenance like automatic update, server reboot, database backup..
UTC also solves plotting daily time plots in zones with time changes (twice a year when the plots make no sense). But there still isn't no good fix for leap year plots!
2-3am is bad due to DST but there is another reason to also avoid 3-5am or so
around 3am is the time where "in average" human attention is the worst as its where "in average" their most relevant deep sleep phases are
now I guess the average over HN readers has a good chance to be somewhat later
but either way if the job goes wrong and triggers alarms 2-5am it very bad for the the health and sleep rithm of whoever needs to fix it and the most likely time they make mistakes
it sadly is also the time where non-international interruptions are least likely to matter
if you are an international company having international teams can be an solution
and if you are a very small team adopting job times to an overlap of low usage and your admins sleep habits can be an option
but if you need to pick a generic time and don't provide for logistics or other "start work at 4/5am" companies probably 5-6am is a better time to run your batch jobs
If you want to avoid overlapping runs simply simply have the task check for the existence of a lockfile, if not create a lockfile containing the start timestamp, run the task and delete the lockfile at exit or error.
The timestamp gives you a chance to recognize stale/orphraned lock files in case of crashes based on their age.
I have a job that deliberately runs at 2am and 3am... to update the time in a bunch of really old PLCs for DST. And check that every other device on my telemetry network has correctly updated its time.
Here's my radical time idea: instead of keeping the same schedule year round, we just incrementally chop parts of the day off (we just put that time in a "bucket" and pretend it doesn't exist). Then at the summer equinox we empty the time bucket into a 6 day bender of a party. /s
This isn't a problem with actual cron and crontab. It is a problem with the systemd-timers shim "crontab" which doesn't work the same in many corner cases and often has weird bugs.
Avoid 2:00 and 3:00 am cron jobs (2013)
(endpointdev.com)319 points by pera 27 October 2025 | 357 comments
Comments
Just stick to winter time (because that's the correct one) and adjust all working hours once and for all to be like summer time, for e.g. instead of business opening at 7AM and closing at 10PM they can open at 6AM and close a 9PM. There will be a period of adjustment, but we have a period of adjustment twice a year already, so nothing is lost.
Why match working hours to summer time? Because I want to have more sunlight by the time I'm done with work. Especially during winter. We have it completely backwards, if we are going to adjust our clocks we should adjust them in a way that gives us more sunlight during winter, not less. I don't care if it's dawning by the time I'm going to work where I will be stuck indoors for nine hours, I want sunlight when I'm free again.
I didn't use UTC because subtracting eight when reading raw logs is a lot harder than subtracting one.
They use UTC now, which makes sense since they have a global workforce and log reading interfaces that can automatically change timezones on display.
Company went bust before it got fixed.
Just use UTC folks unless you have a really good idea why you shouldn't.
https://www.sciencealert.com/daylight-savings-time-change-ki...
https://mas.to/@mpirnat/115395859892135002
Here's a link to a patch [1] from a version from when the package still kept standalone patches.
It's been a long long time and I honestly don't remember the details, but Debian's cron(8) still says [2]: Special considerations exist when the clock is changed by less than 3 hours, for example at the beginning and end of daylight savings time. If the time has moved forwards, those jobs which would have run in the time that was skipped will be run soon after the change. Conversely, if the time has moved backwards by less than 3 hours, those jobs that fall into the repeated time will not be re-run.
Edit: According to this bug report [3], this workaround first entered Debian in 1999.
[1]: https://sources.debian.org/src/cron/3.0pl1-137/debian/patche...
[2]: https://manpages.debian.org/unstable/cron/cron.8.en.html
[3]: https://bugs.debian.org/8499
And tbh, don't run jobs on the hour anyway. Everything kicks off then. Set stuff to run at 01:45 or 02:15 or something instead.
(None of these suggestions are time change related)
There are points in time that don't exist, and there are points in time that are ambiguous.
If we adjust the time forward by 1 hour, then then 2:30am doesn't exist. If we adjust time backward, then 2:30am is ambiguous.
2am-3am is only safe for certain countries.
America/Santiago (Chile), for example, adjusts their daylight saving on midnight. On a certain day, midnight doesn't exist because it immediately jumps to 1am. If you are building an analytics chart by day, you will encounter this issue. I wrote about several weird time zone edgecases here: https://tanin.nanakorn.com/edgecases-for-timezones/
To add more confusion, IIRC Java and Ruby handle these cases differently where Ruby raises exceptions (e.g. InvalidTime and AmbiguousTime) but Java doesn't.
I caused 3 incidents in a row at Stripe in my first month because of this. I thought I was gonna be fired. Luckily I wasn't...
If it really needs to run during a low load period I line it up for 6am work time. Again so that if something goes wrong it leads into the work day rather than disrupts the on-call too badly. 6am also provides a daylight savings buffer either way so you don't have to care. Recent work has all been on 24/7 operations though.
Is this what people actually want, though?
I'm actually working on a scheduled process right now deployed on a cloud scheduler that does not adjust for daylight saving time (it literally says "does not take into consideration daylight saving time" next to the time input).
Everybody wants the job to run four hours before the work day starts, every day of the year. I will have to change it on Sunday to ensure the job completes at the time the end users expect it. And again in March.
Am I missing something?
But the other thing that has changed since 2013 is that most of us are using autoscaling. There were times of day when there is supplemental server capacity to run periodic processes, and it made sense to schedule them during quiet daylight hours if they existed, but now the availability of bus numbers if and when a problem happens are more pressing. And they all outweigh concerns about DST. Instead of seeking the low traffic times of day you mainly need to avoid peak traffic.
The batch processing system I renovated to use 7% if the CPU hours of the original design, used the same fixed parallelism trick that crawlers use so as not to knock over your sites. They keep k requests in flight at all times and if your server responds faster, they issue more requests per second. If the production services had excess capacity the job would run in ten minutes. If response times are slow it takes 15. You’re inverting Little’s Law and keeping the amount of servers fixed and varying the duration.
It’s a pity that spooling up more servers to handle that job took a substantial fraction of that ten minutes or I could have gotten it to five (originally it was over 30 minutes and climbing toward 35). I tried at one point to move it entirely to a Bamboo agent but we underprovisioned them. If I’d had an agent with 2x the CPUs I could have shut down a service. I was on my way out the door before I realized that upgrading the bamboo agents would have cost us less than running that service and everyone’s builds would have been faster as a consequence. And probably let someone else do the same. False economies.
this is the way
anacron won't have the OP's 2am/3am issue because it locks jobs while they run, so multiple job spawns should be no-ops. Still seems like it would be good practice to, e.g. use UTC for the host system and do whatever else might help avoid the unnecessary re-spawns.
Or learn to use lock files, which would make sure that your jobs don't run when another one is already running?
Unix shells offer the trap command, among other things, to cleanup, even in the case of errors, e.g.:
https://en.wikipedia.org/wiki/Semaphore_%28programming%29And then you can translate that time to any other timezone with ease, knowing that you are referencing a specific hour in physical time that only ever happened once.
I figured that scheduling lights to turn on at <0 AM + wake up time> was a good approach... We got a surprise early wake up last year, so I ended up changing the formula to <4 AM - 4 hours + wake-up time>.
around 3am is the time where "in average" human attention is the worst as its where "in average" their most relevant deep sleep phases are
now I guess the average over HN readers has a good chance to be somewhat later
but either way if the job goes wrong and triggers alarms 2-5am it very bad for the the health and sleep rithm of whoever needs to fix it and the most likely time they make mistakes
it sadly is also the time where non-international interruptions are least likely to matter
if you are an international company having international teams can be an solution
and if you are a very small team adopting job times to an overlap of low usage and your admins sleep habits can be an option
but if you need to pick a generic time and don't provide for logistics or other "start work at 4/5am" companies probably 5-6am is a better time to run your batch jobs
anyway just some random thoughts
just use UTC
The timestamp gives you a chance to recognize stale/orphraned lock files in case of crashes based on their age.