https://dev.arie.bovenberg.net/blog/python-datetime-pitfalls...
https://news.ycombinator.com/item?id=39417231 (147 comments)
Now at least there’s an LLM that might spot a bug every now and then so that’s nice.
I wonder what benefits this choice has that outweigh the risks of this behavior.
I learned the hard way, that dependencies kill projects.
Not saying this isn't great, thanks for creating it! It does have its use cases, of course.
I work in healthcare. If I have a choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency", I'm taking the dependency every single time.
I don't want footguns in my code, I don't want code I have to write and test myself, and I don't want to have to become an expert in a domain before I can write something that serves my purpose.
For the datetime library, specifically, I'm switching to whenever for everything, because I've been bitten by conversions and naive/aware datetime confusion too many times.
Link to Tom Scott & Computerphile from 10y ago on tz madness. https://www.youtube.com/watch?v=-5wpm-gesOY
https://whenever.readthedocs.io/en/latest/faq.html#how-can-i...
This kinda sums up the sorry state of software engineering. People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI.
Taking on a dependency does not relieve you of reading docs, but it also adds a further burden as you now need to trust the code. The stdlib is much more heavily tested and documented than any 3rd party library will be.
Sure, but the opposite applies as well. Sticking with the flawed stdlib means you are trusting that every single future developer is as careful in reading all the docs as you are - even when it's someone reviewing that random trivial-looking patch which is secretly hiding a massive footgun. A junior developer submitted a five-line patch touching datetime? Better schedule several hours for a proper analysis!
Or you can write your own wrapper code, of course. Which will almost certainly have worse documentation and testing than a popular third-party library.
External libraries aren't evil. When chosen properly, they relieve you of burdens. You shouldn't grab any random "leftpad"-like dependency like they are going out of fashion, but something as complicated as timezone handling is best left to code written by domain experts - which means using a library.
You initially said you write your own code instead of using libraries, I replied to that, and now it's that you use the stdlib instead of libraries. I won't argue against shifting goalposts.
In addition, the original post begins with, "Am I the only one to stick with the std lib". The goalposts are stable.
It's not a choice between "using a dependency" or "using something in the stdlib", where all other code remains the same, otherwise there would be no point to writing a library, as it would offer nothing over `datetime`.
That's a straw man argument. No one said "blindly". You can very well carefully consider the pros and cons of adding a dependency and arrive at the conclusion that it makes sense. Many PyPI packages are in the Debian stable repositories, you could use that as an additional barrier as well.
choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency"
I think it matters a lot how much one can trust a 3rd party library, how much it is used, how much it is maintained etc. Moreover, it also matters how central and important this is to what you are actually doing, for example if the datetimes I come across are pretty much specific and with limited scope, I would probably care about reading docs less than if I am reading data with datetimes that may be totally wild. Furthermore, there are some libraries that are just considered the standard in some fields. If I use python I call pandas to read csvs, I am obviously not gonna write my own csv parser. It will also make your code more readable for others that already know and use the same libraries if they are so standard. But that's probably a very small minority of libraries that people actually use in certain languages.
So it's you that isn't just using the built in csv parser in this project I inherited. Come back and repent.
That's why I use a Flake8 plugin to prohibit especially egregious footguns.
These things are really frustrating
As long as there is a conscious decision to build or ‘buy’, it’s fine. I think some people can be a little too careless with adding dependencies though, not realising they can have an equal if not greater maintenance burden.
The only crazier idea I can think of is implementing character encoding conversions myself.
Nobody needs a package for "left-pad", which is the most infamous example.
But there are a lot of areas where it's not a good use of your time to reinvent the wheel. There's no moral virtue in writing everything yourself. And popular packages have gone through a more bug-finding and bug-fixing than your personal code probably ever will.
Timedates are hard, and units may require even harder historical/present "political" tracking as how they are defined, and I would never want to maintain this kind of dependency: https://github.com/ryantenney/gnu-units/blob/master/units.da...
And what comes to timedate problems, I try to keep it simple if the project allows: store and operate with UTC timestamps everywhere and only temporarily convert to to local time (DSTs applied, if such) when displaying it in user-facing UI. This functionality/understanding can be locked into own 20-line microlibrary-dependency, which forces its responsible person to understand country's timezone and when e.g. DST changes and where/how/who decides DST changes and what APIs is used to get UTC time (and of course, its dependencies, e.g. NTP, and its dependencies. e.g. unperturbed ground-state hyperfine transition frequency of the caesium-133 atom, which result is then combined with the Pope Gregory XIII's Gregorian calendar, which is a type of solar calendar mixed with religious event for fixing the time, which which is then finally corrected by rewinding/forwarding "the clock" because its too bright or dark for the politicians).
I would still rather using a library for dates, a million times so.
Yes, I agree with this
> The self-written one is maintained by 1 person, the other is used by 100+ people who could jump in a collaborate on its fixing.
Libraries that have 100 people collaborating on it are very few
Most likely you'll have to vendor it and fix whatever issues you have.
Even worse when it's a dependency of a dependency and you also use it, so, let's say a dependency of yours bumps the version but this breaks your code. (Not sure this breaks only in python or in js as well, but it's possibly that it does)
I can’t for the life of me figure out why. If you update everything incrementally you bypass the upgrade version problem when you’re so far behind that so much has changed that it becomes an overwhelming burden.
I think frozen dependencies are a big anti pattern, and places where I work that regularly update their deps tended to have better software practices generally
Besides, any update risks breaking stuff. Not freezing dependencies isn't an option, because that means any commit can cause breakage in a completely unrelated part of the codebase, in a way which can be extremely confusing to debug. And you don't really want to install the very newest versions either, better wait a week or two for someone else to run into bugs and release a .1 version.
The sweet spot is somewhere in the middle: update often enough to avoid updates becoming a massive burden, stick with fixed versions between updates. I reckon it's best to just schedule some dedicated time for it every month or two.
I do this typically every couple of weeks, and it takes up almost no time at all in comparison to time spent on other work. Someone needs to review the eventual PR created, but that's also typically fairly easy. NPM makes this all very easy to do. In Python I've used tools like PDM or uv to handle dependencies similarly.
And you still have upgrades that break interfaces and such
Being judicious in selecting dependencies goes a long way too. Not always easy but certainly worth the time
Our test suite is comprehensive and will catch most breakages automatically. The key to success is robust testing, as it cuts the manual footprint significantly.
This does mean we are quite judicious with selecting dependencies.
It isn’t all that complicated when everyone is following best practices most of the time I have found[0]
It still leavings me wondering in a lot of cases
[0]: perhaps this is the real heart of the issue is best practices are systematically ignored. I’ve worked at places like that and it’s no wonder they grind through folks
You know, they import 5 libraries, each of which imports 5 more libraries, each of which imports 5 more libraries, and suddenly they're buried in 'critical' updates because there's a denial-of-service bug in the date parser used by the yaml parser used by the configuration library used by the logging library used by the application.
E.g the JS project that uses the stdlib Date API, and pulls in moment.js, and also uses date-fns.
Or the one that pulls in bits and pieces of lodash, ramda, and other functional libraries.
And maybe it uses native fetch and axios depending on the current phase of the moon.
They don’t die but time is wasted in code review trying to understand if there is any kind of deliberate approach behind the scattershot application of packages with duplicated purposes.
(picking on JS is perhaps unfair but it’s probably the most egregious example of dependency hell)
Some problems simply require using the right tools. They aren't necessarily hard, but they will be if you try to hammer a nail in with a screwdriver. The Date API, and to a certain extent Python's datetime module, are screwdrivers for a nail-shaped problem.
The rest of your example seem to have more to do with bad dependency practices than using dependencies in the first place. If you are going to include a dependency, think about it, consider whether it's worth it, document that decision, and then consistently use that dependency. Just because you've seen projects use dependencies poorly doesn't mean dependents are bad by themselves.
And even in JS, Temporal won't be available broadly for a good while yet (it will probably be rolling out in Firefox in a couple of months' time, but in Safari it's still behind a feature flag, and I don't think there's even a feature-flagged implementation for Chrome yet). In the meantime, it makes sense to use a polyfill — again a library.
By all means choose your dependencies wisely, but the point I'm trying to make is that very often a sensible use of dependencies will reduce your technical debt, and attempting to use bad approaches to complex topics just because they're built into the standard library will cause so many problems down the line.
In Temporal's case, it takes significant inspiration from several existing datetime libraries in Javascript. Those were all stepping stones (among others) that led to Temporal.
But yeah, you will have to create tests for the codepath, instead of relying on the tests the library maintainers create.
Given how many tests I see fail on 29th February at the companies I’ve been at, I don’t trust my colleagues’ tests either!
And this for every single problem: time, text, maths, network, parsing, formatting, validating, authenticating...
Avoid general terms like "Pacific Standard Time" and stick to location-specific ones like: "Vancouver/Canada". The latter is how people expect their time to work, and correctly handles whatever quirky choices jurisdictions choose to do with their time.
Searching the list here: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
I cannot find an entry for "Pacific Standard Time" nor "Vancouver/Canada", but I can see: "America/Vancouver".
Hard pass. The complexity of having to use binary packages or build things is not worth the performance benefit. The pure-Python version requires building from source and passing special flags, so it is not possible to specify it in requirements.txt.
An issue was closed as not planned: https://github.com/ariebovenberg/whenever/issues/158
1. I _love_ pure Python packages. Not everybody should be forced to use Rust. I want installing pure-Python to be as easy as possible
2. Having separate names on PyPi (with or without extras) creates confusion for libraries depending on whenever: should they depend on whenever-py or whenever-rust? If one “overwrites” the other, this adds confusion.
3. Most users expect to “pip install whenever” and start using the “fast” version
For me, points (3) and (2) weigh heavy enough to make (1) slightly more cumbersome.
But: Maybe I’ve missed something. Have a read in the above issue or add your 2 cents.
edit: formatting
2. I would expect `pip install whenever` to give me the pure-Python version, and `pip install whenever-rust` to give me the Rust extras. Both packages can be installed at the same time; the pure-Python package detects and uses the Rust implementation if present.
You can put any flags in requirements.txt, including -r[equiring] another txt etc.
Your point may apply to modern pyproject.toml tooling though, or at least that it wouldn't be simply another entry in the dependencies array.
In particular, default implementation of datetime in cpython is a C module (with a fallback to pure python one) https://github.com/python/cpython/blob/main/Modules/_datetim...
Not saying it's necessarily justified in case of this library, but if they want to compete with stdlib datetime in terms of performance, some parts will need to be compiled.
Then it would have been nice to see the benchmarks of the pure Python implementation as well. What if it's worse than arrow?
> In casual benchmarks, the pure-Python version is about 10x slower than the Rust version, making it 5x slower than the standard library but still (in general) faster than Pendulum and Arrow.
"(in general)" here since the speed compares differently per operation, while the Rust version is faster across the board. That said, there's no operation that is _significantly_ (or unnecessarily) slower than Arrow or Pendulum.
edit: I'm considering adding comparison to the pure Python version once I get the time for a more expanded "benchmarks" page in the docs
Almost all of the time UTC is enough, if I need to filter/bucket/aggregate by some range, I can reach for datetime with tz for these filter/bucket/aggregate criteria, convert them to UTC and on continues `int` comparison.
I'd imagine all of the cases handled by Whenever are mostly when datetime is a long lived object, which I don't see a need for at all.
I use it purely for allowing tz input from client, convert to UTC immediately when it arrives, or, if I really need the tz, then save it separately, which is rare (one example is calendar, where tz should be stored, although probably not even next to every UTC but at the user level, another is workforce scheduling, where 8am-4pm or 8pm-4am can mean different things for different locations -- but this is no longer datetime, it's purely time in a timezone).
And so anything server-related with calendars will be making tons of these conversions constantly. And you can't cache things long-term in UTC because the conversions of future events can change, when countries change DST etc.
So you would not store that in UTC but just in time.
But yes, I’m ignoring the standard of calendar formats , maybe they are simpler .
I read through the article listing all the weirdness of other datetime libraries and I’d say many were covering cases where you behave that timezoned datetime is long lived .
One case even pointed out datetime construction with an impossible hour.
If you have a recurring lunch, it's always at the same local time interval, but not the same UTC interval, because of DST. Calculating with it requires datetimes, not just times or UTC from the start, contrary to who I was responding to. What is unclear about that?
You’ve also stated you want to ignore the timezone and display 12 in whatever tz.
So if my interface is all events between start_utc and end_utc I will construct local datetime and can convert it to UTC and send it to frontend.
The problem with hours that don’t exist in a tz/DST needs to be dealt with separately and given Whenever raises an error and datetime does not is good. In the article that’s one of few that applies, others only exist if you have massive amounts of long lived tz datetime objects.
Yet again, no need for long lived datetime. The problem you picked is time + occurrence.
Imagine a work shift from 23-8 , DST might turn it into less or more than 9 hours. Library does not help as you have to explicitly decide what to do. To avoid the issue then you’d reinterpret user input as start time and duration. When constructing hours, you’d shortly reach out to datetime that is localized. This is again not a datetime problem as work shift is just time.
Showing data on a chart in localized time, one has to explicitly decide what to do with hours that don’t exist or are 2 hours long (visually). Having long lived tz datetime does not help.
> You’ve also stated you want to ignore the timezone and display 12 in whatever tz.
No I didn't. I literally said "You need to store things permanently with the timezone". Obviously lunch at 12 belongs to a specific timezone.
> Yet again, no need for long lived datetime.
You keep saying this. I don't know where you're getting it from. You need to store these things in databases. If my calendar goes 2 years in the future, these datetimes live for 2 years at least.
It sounds like you're arguing that datetime libraries don't need to exist. I'm genuinely baffled. That feels as nonsensical to me as saying that floating point numbers don't need to exist. Long-lived datetimes are a commonly required thing in anything regarding calendars or scheduling. If you don't believe there is a genuine use case for this, I don't know what to tell you.
I was trying initially to provide you with a basic answer to what seemed to be a genuine question, but I can't help you anymore.
Yes, therefore you won't store lunch as `datetime` you will store it as `time` and on "render" you will make sure it is 12th hour for localized datetime. Look at RFC 5545. If you move from Japan to Romania, you'll again on render (with stored 12 as `time`) display lunch at 12th hour. The only thing one stores is daily occurrence and 12th hour, with careful interpretation what is a boundary of a day and what is 12th hour. `datetime` does not help, similar to how it does not help with DST issues for workshifts (which are again `time`).
> If my calendar goes 2 years in the future, these datetimes live for 2 years at least.
Why would you materialize recurring events, if you move from Vietnam to Spain, how will all those lunches become Spain lunches? Just recompute the whole future? Sounds like a mistake of storing long-lived objects that should have been replaced with computation with short-lived ones.
> It sounds like you're arguing that datetime libraries don't need to exist.
Steelman me. 2 individuals tried to expand to include you into discussion, you seem to go in the other direction every time.
I am not implying that. I'm implying that I don't understand the performance implications, given that I'd expect to work with efficient UTC most of the time, with only a constant amount of stuff going from datetime-with-tz to UTC. I cannot imagine a case where you'd abundantly store datetime-with-tz, it sounds like a mistake.
My own impression is that majority opinion is that going from datetime-with-tz to UTC and then doing comparisons, filtering, bucketing, aggregating is somehow flawed? If I need 24 hour buckets, I can do it through UTC, if I need "daily" buckets, I can still do it through UTC -- with short-lived datetime-with-tz to UTC computation, yet UI will still have to deal with 23/25 hour days, `datetime` does not solve that problem. I understand monthly buckets, still easy to do through UTC with short-lived conversion. (go from datetime-with-tz to UTC that's used to filter/group entries in db)
https://en.wikipedia.org/wiki/Acid3
I like this new lib (Thank You) but the name unfortunately implies the opposite of what it is. "Whenever" sounds like you don't care, but you'd only be using this if you did care! Also Shakira, haha. Hmm, pedantic is taken. Timely, precise, punctual, meticulous, ahorita, pronto, etc. I like that temporal name.
Finally, none of these links mention immutability, but it should be mentioned at the top.
I am an amateur dev, though, so maybe someone who masters the language will be better off using the raw standard libraries.
I'm sure I'm in the top 1% of software devs for the most number of timestamps parsed. [1]
DST is not a problem in Python. It's parsing string timestamps. All libraries are bad, including this one, except Pandas. Pandas does great at DST too btw.
And I'm not shilling for Pandas either. I'm a Polars user who helicopters Pandas in whenever there's a timestamp that needs to be parsed.
Pandas has great defaults. Here's string timestamps I expect to be paesed by default. I'm willing to pass timezone in case of naive timestamps:
* All ISO 8601 formats and all its weird mutant children that differ by a tiny bit.
* 2025-05-01 (parsed not as date, but as timestamp)
* 2025-05-01 00:00:00 (or 00.0 or 00.000 or 0.000000 etc)
* 2025-05-01 00:00:00z (or uppercase Z or 00.0z or 00.000z or 0.000000z)
* 2025-05-01 00:00:00+02:00 (I don't need this converted to some time zone. Store offset if you must or convert to UTC. It should be comparable to other non naive timestamps).
* 2025-03-30 02:30:00+02:00 (This is a non existent timestamp wrt European DST but a legitimate timestamp in timestamp representation, therefore it should be allowed unless I specify CET or Europe/Berlin whatever)
* There's other timestamps formats that are non standard but are obvious. Allow for a Boolean parameter called accept_sensible_string_parsing and then parse the following:
\* 2025-05-01 00:00 (HH:mm format)
\* 2025-05-01 00:00+01:00 (HH:mm format)
Disclaimer: I'm on the phone and on the couch so I wasn't able to test the lib for its string parsing before posting this comment.
Javascript's big datetime redesign (Temporal) has an interesting overview of the decisions they made [1]. Whenever is currently undergoing an expansion of ISO support as well, if you'd like to chime in [2].
[1] https://tc39.es/proposal-temporal/#sec-temporal-iso8601gramm... [2] https://github.com/ariebovenberg/whenever/issues/204#issueco...
Your customers are software devs like me. When we're in control of generating timestamps, we know we must use standard ISO formatting.
However, what do I do when my customers give me access to an S3 bucket with 1 billion timestamps in an arbitrary (yet decipherable) format?
In the GitHub issue you seem to have undergone an evolution from purity to pragmatism. I support this 100%.
What I've also noticed is that you seem to try to find grounding or motivation for "where to draw the line" from what's already been done in Temporal or Python stdlib etc. This is where I'd like to challenge your intuitions and ask you instead to open the flood gates and accept any format that is theoretically sensible under ISO format.
Why? The damage has already been done. Any format you can think of, already exists out there. You just haven't realized it yet.
You know who has accepted this? Pandas devs (I assume, I don't them). The following are legitimate timestamps under Pandas (22.2.x):
* 2025-03-30T (nope, not a typo)
* 2025-03-30T01 (HH)
* 2025-03-30 01 (same as above)
* 2025-03-30 01 (two or more spaces is also acceptable)
In my opinion Pandas doesn't go far enough. Here's an example from real customer data I've seen in the past that Pandas doesn't parse.
* 2025-03-30+00:00 (this is very sensible in my opinion. Unless there's a deeper theoretical regex pattern conflicts with other parts of the ISO format)
Here's an example that isn't decipherable under a flexible ISO interpretation and shouldn't be supported.
* 2025-30-03 (theoretically you can infer that 30 is a day, and 03 is month. BUT you shouldn't accept this. Pandas used to allow such things. I believe they no longer do)
I understand writing these flexible regexes or if-else statements will hurt your benchmarks and will be painful to maintain. Maybe release them under an new call like `parse_best_effort` (or even `youre_welcome`) and document pitfalls and performance degradation. Trust me, I'd rather use a reliable generic but slow parser than spend hours writing a write a god awful regex that I will only use once (I've spent literal weeks writing regexes and fixes in the last decade).
Pandas has been around since 2012 dealing with customer data. They have seen it all and you can learn a lot from them. ISOs and RFCs when it comes to timestamps don't mean squat. If possible try to make Whenever useful rather than fast or pure. I'd rather use a slimmer faster alternative to pandas for parsing Timestamps if one is available but there aren't any at the moment.
If time permits I'll try to compile a non exhaustive list of real world timestamp formats and post in the issue.
Thank you for your work!
P.S. seeing BurntSushi in the GitHub issue gives me imposter syndrome :)
I honestly do not know the answer to that question myself. But I wouldn't necessarily look to Pandas as the shining beacon on a hill here. Not because Pandas is doing anything wrong per se, but because it's a totally different domain and use case. On the one hand, you have a general purpose library that needs to consider all of its users for all general purpose datetime use cases. On the other hand, you have a data scienc-y library designed for trying to slurp up and make sense of messy data at scale. There may be things that make sense in the latter that don't in the former.
In particular, a major gap in your reasoning, from what I can see, is that constraints beget better error reporting. I don't know how to precisely weigh error reporting versus flexible parsing, but there ought to be some deliberation there. The more flexible your format, the harder it is to give good error messages when you get invalid data.
Moreover, "flexible parsing" doesn't actually have to be in the datetime library. The task of flexible parsing is not, in and of itself, overtly challenging. It's a tedious task that can be build on top of the foundation of a good datetime library. I grant that this is a bit of a cop-out, but it's part of the calculus when designing ecosystem libraries like this.
Speaking for me personally (in the context of Jiff), something I wouldn't mind so much is adding a dedicated "flexible" parsing mode that one can opt into. But I don't think I'd want to make it the default.
I am currently enjoying DST-free life in Japan, and feel that people around the world deserve to get this much respect from their own official clocks.
And for program code, it wouldn’t really help as long as it’s still expected to be able to correctly handle dates in the past.
I’m not sure how real those costs would be beyond a transitional period, but that’s the discussions that have been going on. It’s a political risk, and nobody wants to be on the losing side of such a change.
(Schools tend to have earlier times. It's not so unusual for a school's workday to have its midpoint at about noon, I think.)
Maybe adjust the work schedule to e.g. start at 8 instead of 9?
Rather than mess with the actual clock.
I've always been in favour of keeping the clocks at non-DST all year, but now I have a new proposal: keep them at DST and just hibernate in the winter. Work an hour or two less in the winter when it's miserable.