I think UV proves that dedicated funding can make a huge impact on a project and benefit for the community. They are a doing a damn, good job.
Not that they took inspiration from PEPs, but they sought to implement those standards (for interoperability) and have been active in the discussion of new packaging-related PEPs.
Where does Astral's funding come from, anyway?
But putting that aside, a big part of uv's performance is due to things that are not the implementation language. Most of the actually necessary parts of the installation process are I/O bound, and works through C system calls even in pip. The crunchy bits are package resolution in rare cases (where lock files cache the result entirely), and pre-compiling Python to .pyc bytecode files (which is embarrassingly parallel if you don't need byte-for-byte reproducibility, and normally optional unless you're installing something with admin rights to be used by unprivileged users).
Uv simply has better internal design. You know that original performance chart "installing the Trio dependencies with a warm cache"?
It turns out that uv at the time defaulted to not pre-compiling while pip defaulted to doing it; an apples-to-apples comparison is not as drastic, but uv also does the compilation in parallel which pip hasn't been doing. I understand this functionality is coming to pip soon, but it just hasn't been a priority even though it's really not that hard (there's even higher-level support for it via the standard library `compileall` module!).
More strikingly, though, uv's cache actually caches the unpacked files from a wheel. It doesn't have to unzip anything; it just hard-links the files. Pip's cache, on the other hand, is really an HTTPS cache; it basically simulates an Internet connection locally, "downloading" a wheel by copying (the cached artifact has a few bytes of metadata prepended) and unpacking it anew. And the files are organized and named according to a hash of the original URL, so you can't even trivially reach in there and directly grab a wheel. I guess this setup is a little better for code reuse given that it was originally designed without caching and with the assumption of always downloading from PyPI. But it's worse for, like, everything else.
Or matplotlib.
Or PyTorch.
Rails. QED.
Django comes batteries included for basic apps, including an admin.
It can work...but that's not what it was designed for.
You know rust is not the only language in which one can write good code, right?
I don’t know other languages, but I’ve heard a lot of good things about zig.
Not everyone would want to get into Apple vendor-locked language (formally not, but it's like saying Chromium is opensource and not related to Alphabet/Google) and its development environment. Especially after JetBrains closed its swift oriented ide [0]. I hate xcode, sorry, after developing for too many years in it.
The language was initially very much Apple-platforms oriented (had to be), but now that pretty much all the Apple stuff works well they moved beyond that.
Finally where the language comes from does not impact whether you can write good code with it.
Led me to try uv, which fixed a couple of egregious bugs in pip. Add speed and it’s a no brainer.
I don’t think poetry has these advantages, and heard about bugs early on. Is that completely fair? Probably not. But it’s obvious astral tools have funding and a competent team.
I can do all that without having to even worry about virtual ends, or Python versions too.
Everyone seems to like uv's answer better, but I'm still a believer in composable toolchains, since I've already been using those forever. I actually was an early Poetry adopter for a few reasons. In particular, I would have been fine sticking with Setuptools for building projects if it had supported PEPS 517/518/621 promptly. 621 came later but Poetry's workaround was nicer than Setuptools' to me. And it was easier to use with the new pyproject.toml setup, and I really wanted to get away from the expectation of using setup.py even for pure-Python projects.
But that was really it. The main selling point point of Poetry was (and is) that they offered a lockfile and dependency resolution, but these weren't really things I personally needed. So there was nothing really to outweigh the downsides:
* The way Poetry does the actual installation is, as far as I can tell, not much different from what pip does. And there are a ton of problems with that model.
* The early days of Poetry were very inconsistent in terms of installation and upgrade procedures. There was at least once that it seemed that the only thing that would work was a complete manual uninstall and reinstall, and I had to do research to figure out what I had to remove for the uninstallation as there was nothing provided to automate that.
* In the end, Poetry didn't have PEP 621 support for about four years (https://github.com/python-poetry/roadmap/issues/3 ; the OP was already almost a year after PEP acceptance in https://discuss.python.org/t/_/5472/109); there was this whole thing about how you were supposed to use pyproject.toml to describe the basic metadata of your project for packaging purposes, but if you used Poetry then you used Masonry to build, and that meant using a whole separate metadata configuration. Setuptools was slow in getting PEP 621 support off the ground (and really, PEP 621 itself was slow! It's hard to justify expecting anyone to edit pyproject.toml manually without PEP 621!), but Poetry was far slower still. I had already long given up on it at that point.
So for me, Poetry was basically there to provide Masonry, and Masonry was still sub-par. I was still creating venvs manually, using `twine` to upload to PyPI etc. because that's just how I think. Writing something like `poetry shell` (or `uv run`) makes about as much sense to me as `git run-unit-tests` would.
> Some of uv's functionality cannot be expressed in the pylock.toml format; as such, uv will continue to use the uv.lock format within the project interface.
> However, uv supports pylock.toml as an export target and in the uv pip CLI.
— https://docs.astral.sh/uv/concepts/projects/layout/#the-lock...
Of course there must have been really good reasons for this decision. I hope no one will hold it against the maintainers of any of the projects. Especially because it looks like it is easy to move between the two lock files.
> The biggest limitation is that there's no support for arbitrary entrypoints to the graph, because pylock.toml includes a fixed marker for each package entry rather than recording a graph of dependencies.
Year -1: The community has a problem.
Year 0: Proposal to fix the problem.
Year 1: A small but vocal subset of the Python/Django community pops up in every thread: "It's not actually a problem." or "It's not an issue that my project would ever encounter so limited resources shouldn't be expended on it."
Year 2: People are choosing other solutions because Python/Django isn't addressing the problem.
Year 3: We'll form a committee. The committee doesn't think it's a problem.
Year 4: The community has a problem. Fine. Why doesn't the community write a Python Enhancement Proposal/Django Enhancement Proposal (PEP/DEP)?
Years 5-10: PEP/DEP ignored.
Year 11: The community has a problem. PEP/DEP implemented and released.
Year 12-22: Major packages refuse to support the change.
Year 23: Last version not supporting the change deprecated.
Year 23+1 day: Fork of last deprecated version Python not supporting change created and released.
I have 15 years of code in Python still running but spend a little more than 50% of my time in other stacks. I don't notice as many people arguing against basic features, like a REST API package in Django, in non-Python/Django communities. The precursor to a Django REST API package, content negotiation, has been a draft DEP since early 2014 (https://github.com/django/deps/blob/main/draft/content-negot...). That's going on 12 years of stalled progress for a feature that needed to be released in 2010.
With Python/Django you learn not to wait because nothing is going to change.
And yes, Python/Django are open source. And yes agin, I donate over $1,000/year to support F/OSS projects that I depend on.
I know that Guido isn’t around any more, but this is what a BDFL is useful for. To have the final say when, inevitably, the squabbling parties are unable to find a compromise.
How was the Python 2 to Python 3 migration for you? How many Python libraries are you responsible for packaging? Has that been fun for the past 15-20 years? How heavily do you rely on async?
Almost a decade for Django+major libraries to support Python 3. More than a decade for Django async support. I'm not ungrateful. And, I wish that I believed that throwing money at the problem would solve it.
Python has a large and diverse user base. That brings unique challenges that I don't see in other stacks.
It was really clear to me at that time that the resolution algorithm used by pip wasn't sound and that it worked most of the time if your systems were simple but if you added enough stuff the failure rate of your build would start creeping up.
When I started talking to people on forums I found there was little awareness that pip was broken or that it was worth doing anything about, so because of that disinterest I decided not to do anything.
Pip did fundamentally change its resolution algorithm in IIRC 2020 but yes it was bad before that. This is the hard part (I've been looking around for libraries that aren't as arcane as the `resolvelib` that was extracted from pip's internal logic) so the first announced release of PAPER will be focused on the other stuff.
I had a sense back then that pip was not great, but I had very little idea of what was wrong with it or why. My ire was more focused on Setuptools because of bootstrapping problems with setup.py and the general idea of using arbitrary code for building even just to specify metadata. So I largely ignored pip, and I regret that. What I'm making now I could have done years ago with that knowledge. (And even then I'm struggling just to make myself put in the work, since everyone will just use uv anyway.)
https://pip.pypa.io/en/stable/topics/more-dependency-resolut...
I felt the post you replied to was valuable, as it provided a general (but not grounded strictly in the facts of a specific example) outline of how these work.
I will corroborate the details. I learned Django 15 years ago, still use it, and use about 30% of its functionality. I don't think I do anything with it now I couldn't have in the earlier versions.
This is a systems problem. Successful examples wanted.
https://www.postgresql.org/community/ is a good start to get a feel of all things related to Postgres community.
PEP 751 isn’t a python feature, it’s a feature that will be implemented by 3 projects - PDM, pip and uv. Consensus isn’t optional or nice to have, it’s necessary. If any of the 3 maintainers felt their needs weren’t met they wouldn’t have implemented it.
Some projects wait too long for consensus because they prioritise not rushing into a suboptimal solution or hurting people’s feelings. Sometimes it’s ok to just go ahead and implement something. Pep 751 is not one of those projects.
It's not so much a feature as a standard. And other tools will use it as well. I'm working on one of them. PEP 751 means I don't have to design the thing myself, and I can potentially share data with the tools you mentioned.
I’m pointing out that if they didn’t have your buy-in, the work would be dead on arrival.
Unlike a python feature like let’s say, typing - there’s no way it can be pushed through without 100% consensus.
It's about dependency locking in Python packaging.
The big counter example is c++ which I feel is too productive and should slow down their decisions by a factor of 3.
Why couldn't everyone be flied to the same place and have it all figured out in a week instead of having the process drag on for years?
Python was better with a BDFL.
The best thing that could happen to Python right now would be for someone to fork it. Maybe just have Astral run away with the whole language. This lock file format should have taken a weekend, not four damn years.
The problem is that the BDFL also didn't want to think about packaging, certainly not the issues that inspired Conda.
I wouldn't mind seeing a PyPy-like fork done in Rust. Maybe take the opportunity to redesign the standard library, too.
Call them literally anything else. Freeze file, version spec, dependency pin…
There really are only two hard problems in computer science, as the saying goes. Cache invalidation and epithet manufacturing (cough).
Creating a file with the exclusive creation mode is a pretty common way to make a lock file and as far as I know that works well in posix.
pyproject.toml describes the supported dependency versions. Those dependencies are then resolved to some specific versions, and the output of that resolution is the lock file. This allows someone else to install the same dependencies in a reproducible way. It doesn't prevent someone resolving pyproject.toml to a different set of dependency versions.
If you are building a library, downstream users of your library won't use your lockfile. Lockfiles can still be useful for a library: one can use multiple lockfiles to try to validate its dependency specifications. For example you might generate a lockfile using minimum-supported-versions of all dependencies and then run your test suite against that, in addition to running the test suite against the default set of resolved dependencies.
Presumably because decades of experience has demonstrated that humans are extremely bad at maintaining compatibility between releases and dealing with fallout from badly specified package versions is probably second only to NULL in terms of engineering time wasted?
Or possibly it's just because a lot of the Python ecosystem doesn't even try and follow semver and you have no guarantee that any two versions are compatible with each other without checking the changelog and sacrificing a small chicken...
Even if they try, semver can only ever be a suggestion of the possibility of compatibility at best because people are not oracles and they misjudge the effects of changes all the time.
A little discipline and commitment to backwards compatibility and it isn’t too hard, really?
Then there’s Python - which I dearly love - that uses something that looks exactly like SemVer, but isn’t. This is often confusing for newcomers, especially because most libraries use SemVer.
Singling out two behemoths designed by committee to demonstrate SemVer's shortcomings seems misleading.
This is one of those things that becomes clearly untrue if you try to apply any amount of rigor to the definition of "just fine".
I have yet to see a single major Python project claiming to use SemVer that didn't occasionally unintentionally violate the promise made by the versioning because you cannot always easily predict whether a change will be incompatible, and, even if you could, people still regularly just make mistakes because people are not perfect. Versioning is hard. Keeping promises about not breaking things is even harder.
That may be good enough for things that don't matter, but it's not good enough for things that matter a lot.
Maybe I'm being too pedantic here, but semver for applications is always going to be broken and driven by marketing. SemVer, for my money, is only applicable realistically to libraries.
To say that you don’t have to upgrade is true, but it always comes at a price.
Whether someone else or I am the problem doesn’t matter to my customers at the end of the day, if I’m unable to ship a feature I’m at fault.
Well, yeah, it's reasonable people flame you there. What is the difference between,
- zlib-1 v 23
- zlib 1.2.3
except that automatic updates and correlation are harder for the first approach. It also will likely make typosquatting so much more fun and require namespacing at the very least (to avoid someone publishing, e. G., zlib-3 when official projects only cover zlib-{1,2}.
Literally everyone posting here is using a system built on compatible interfaces. Stable DLLs on Windows, dylib framework versions on OSX, ELF SOversioning on Linux.
It's clearly not impossible, just a question of priorities and effort, and that makes it a policy decision to do it or not. And I'll lean towards we've been shifting that policy too far in the wrong direction.
I look forward to you demonstrating your tool that can check if two python packages are compatible with each other, maybe you can solve the halting problem when you're done with that?
Also, I didn't claim any tool would give a perfect answer on compatibility. They don't for ELF libraries either, they just catch most problems, especially accidental ones. The goal is 99.9%, not 100%. Just being unable to solve a problem perfectly doesn't mean you should give up without trying.
You could call that success but I think it’s just an extra layer of cruft.
Microsoft's famously terrifyingly large amount of engineering work that goes into maintaining backwards compatibility to allow you to run Windows 3.1 software on Windows 11 is certainly impressive, but maybe also is the exception that proves the rule.
> Just being unable to solve a problem perfectly doesn't mean you should give up without trying.
Currently no one can solve that problem at all, let alone imperfectly. If you can, I'd gladly sponsor your project, since it would make my life a lot easier.