While we’re on the topic, a better typesystem is better than a worse typesystem. Thanks for coming to my TED talk.
That said, I’ve found Python’s approach lets me still build things through experimentation then fixing the types when I’m a bit more confident that I’ve built the right thing.
As a comparison, I found Go really clunky when I tried to learn it because it wouldn’t compile if the code wasn’t totally correct. It makes it hard to find the right solution through experimentation.
Python is probably an all-round better language than JavaScript. But typescript absolutely embarrasses Python’s “gradual” type system with just how good it is in comparison.
Currently, I’m using the third-party Beartype decorators to provide this feature, but I would prefer something built-in and ideally something with more readable error messages for users when type errors occur, which might be easier to provide if it was built into Python.
(I use Pyright too, but I find the runtime type checking to be more robust since it understands also the types returned by highly dynamic untyped code.)
And it does matter when 3rd party libraries are involved. Types are a wonderful gift to anyone who imports & needs to figure out your library.
1. You need to understand how to express concepts in a form the type system can understand. Each type system will have patterns it understands, and patterns it does not.
2. Go isn't a great example. It has a Java-circa-2000 type system. Modern type systems are much more expressive.
I picked Go as an example of a language that gets a lot of praise specifically for having strict compiler rules.
I've also tried Scala and found it clunky but I do suspect most of its problems came from the insane operator overloading and opaque build system rather than the type system. I understand it's gotten better since I looked 8 years ago but I've not had a reason to check it out again.
Python with types is a double edged sword. If you can enforce other collaborators and SDKs use strict typing, it's a very pleasant experience to write, coupled with Pydantic for defining POJO-like structs. For CLI tools Click + Poetry build system.
Java17+ type system is pretty good. Way better than Go in terms of dev experience. you may need a POJO generation tool (Lombok, Autovalue) and preferably validation libraries. I'd still write java over Go if efficiency and single binary is not a concern.
Once you're using types, I would recommend Typer instead. Built on-top of Click but configured via type hints:
man I wish my coworkers did this step
Personally I also find python typing useful in "smaller" programs and I use them pretty much everywhere.
For example I use it in all my automation scripts which are usually not much longer than a few pages or a few modules. Adding type hints is a very minor task, really comes straight out of brain just like the code i type.
Type hints have given me wonderful completion and documentation which has helped me to write more correct and bug free code. Often little things pop up that would only have been caught at runtime. Now I see those things as soon as I type them.
Python + Type Hints == Awesome
To me this suggests the tooling was insufficient. One of the things I value about type systems is being in a tight feedback loop with the compiler. Strong typing helps me run more experiments and to run them from within my editor. But without a good LSP and such, it would be miserable.
Perhaps but I find that type systems usually introduce friction to getting things working that, for some types of projects, can be worthwhile and, for others, not worth the overhead.
I'm not dogmatic, type systems aren't the end all. But I think a lot of the trouble people have with them comes from using the approaches they've developed in untyped systems and saying, "this is the same, I just get more warnings and errors." But the opportunity is in developing new approaches and habits that leverage the type system.
For me the "aha" moment was watching Jon Gjenset's YouTube videos, and seeing him intentionally write code with type errors in order to figure out what the type of something was. This helped me make the switch from thinking of errors as drudgery to be avoided to errors as feedback I may want to elicit.
Now I'm able to answer lots of questions within my editor that I'd previously need to use a REPL or consult the docs to answer. It's like I'm able to query my codebase, but the query language is just the language I'm already working in. The magic of strong typing is good tooling.
We shouldn’t pretend that it’s free. The fact is, sometimes running some code that I know works despite what the type system is telling me is very useful. Comparing to, say, Go, where I need to deal with everything (even if it’s incidental to the actual problem) adds friction. Sometimes I don’t want to deal with edge cases because this code will never leave my machine!
I can recall times when I've neglected a special case to write a one-time script faster, but they were always external to the type system. Eg, "this regex is pretty sketchy, but I know it will work with this particular set of logs." So I'm confused on this point, am I missing something?
All programs need to be type checked, even one-off scripts written in dynamic languages, to ensure that they function. The difference is whether the process is manual or automated. The computer is going to do it better and faster than I can, and that will free up mental bandwidth that I can apply to the problem at hand.
Go won't even compile if a variable is unused.
Makes it really hard to comment-out lines when debugging.
I agree. Typing in some code and trying to compile it is experimentation. Trying to fix the error in your code is exploration. Discarding the compiler's output as getting in the way means ignoring very, very precious advice.
There is a good reason why the ML community took Python as the favorite language overall.
On the other hand heavyweight type systems that demand preemptively writing code to cover cases that won't happen and/or aren't important, including cases that make sense but aren't needed yet, can waste time actual exploration and experimentation, compared to writing only useful code and being sometimes surprised by runtime errors when the program accidentally attempts something not implemented yet.
On the third hand "exploration and experimentation" can suffice for a proof of concept, not for production code: tests and automated checks are necessary to be confident about your program, and even rudimentary tools like Python type annotations and type checkers can be useful.
Ok, I won't.
> we have enough problems already.
I agree.
Very honestly, optional type systems tend to be the worst of all worlds. Because people who don't care, don't need to use it, you don't get safety. But people who enjoy ceremony can inflict verbosity on others. While missing the most important reason to do it.
Very much. The “problem” people try to solve with things like dependency injection isn’t even a problem in python. 99.99% of the time you can just import you dependencies everywhere they are needed.
So many times now I’ve had to unteach bulky OOP patterns for people coming from strictly typed languages to python believe are “good practices” or are “needed for reliable code” and you go from 20 different interdependent classes down to 3 functions that just directly do what they are supposed to do. And you always end up with someone being disappointed that all of these complex patterns they learned to solve problems that don’t exist in python aren’t needed, rather than someone just happy that you can proceed directly to a value adding solution and skip the crud and design pattern spam.
And then they start arguing crazy stuff like the 200 extra lines unneeded code makes the solution “more readable” or “more reliable”, when in reality it’s just a desire to use the solutions they are used to even when the problems don’t exist.
> The “problem” people try to
> solve with things like
> dependency injection isn’t
> even a problem in python.
> 99.99% of the time you can
> just import you dependencies
> everywhere they are needed.
Consider a function with `import datetime`, that gets the current time and does something with it. You want to make sure that this function still does the right thing in the extreme case of the current time being 23:59:59. How do you write this test without DI?
from datetime import datetime, timedelta
def f():
now = datetime.now()
future = now + timedelta(seconds=1)
return now.time() < future.time()
from unittest.mock import patch
with patch('__main__.datetime', spec=datetime, side_effect=datetime) as mockdt:
mockdt.now.return_value = datetime(2024, 10, 3, 23, 59, 59)
assert f()
from datetime import datetime, timedelta
def f(now = datetime.now):
timestamp = now()
future = timestamp + timedelta(seconds=1)
return timestamp.time() < future.time()
def mockdt():
return datetime(2024, 10, 3, 23, 59, 59)
assert f(mockdt)
Also, this approach scales poorly with the number of dependencies, and that’s a good thing. If `f` were to also depend on a `db_connection` in addition to `datetime.now`, then this pattern automatically alerts the API designer that it might be time to redesign `f`.
Expecting type annotations to be more generally useful is mostly projected, baseless declaration anxiety: tools and methods that are useful in other languages are expected to be relevant in Python, certainly comforting for the type-addicted programmer but not necessarily useful. Declaring types is perceived as normal, as a necessary burden and dynamic typing is perceived as missing important structural elements.
Consider the error pattern of accessing a value as if it had a different type: in C or C++ consequences are dire (possibly undetected and cascading memory corruption), likelihood is very high due to specific language features (pointers and references, raw arrays, casts, weak typing in general) and detailed type declarations are a useful mitigation because they turn run time catastrophes into actionable compile time reports, while in Python, thanks to robust duck typing without dangerous complications, consequences are mild (reasonable error messages or wrong results, before corrupting memory), likelihood is low (specific instances of unexpected and malformed data or gross API misunderstandings) and type declarations do nothing.
Why not make it easier for yourself and be able to turn that into
> The compiler will tell me when this object won't
I can prototype, script, move fast. Then I can solidify, stabilize, collaborate.
* To get top performance, of course you'll need compiler, types, static dispatch.
It’s amazing how quickly a few interfaces / typedefs repay the time you spent typing them in.
Because it would be hilarious if ChatGPT was itself written in Python.
Optional is equivalent to no type system at all, in my opinion.
def bounding_box(points: list[Point]) -> Box:
...
I also really like being able to document the imperative code like `def do_thing() -> None`. Of course, it's completely up to the programmer to follow the rule of not doing side effects in routines that return something.
But having to do it everywhere? Ugh. I don't think people realise how powerful duck typing is for doing polymorphic code. I can write something like:
def mean(things):
return sum(things)/len(things)
But of course learning when to define static types vs when not to comes down to experience. Python treats you like an adult. I feel like a lot of people who want static typing everywhere want it as training wheels for other devs they don't trust to make the right judgement calls.
> I feel like a lot of people who want static typing everywhere want it as training wheels for other devs they don't trust to make the right judgement calls.
Now, I want it because it makes my life a lot easier. It finds many classes of errors before runtime and improves auto-completion in my editor. For most programs, it's not a lot of effort. The frustrating part is interfacing with untyped libraries while using strict type-checking, but thankfully, more and more libraries are adopting typing. The type system itself is fine: most things you want to express are doable, but it's not at the same level of complexity as Typescript. It's certainly a better type system than Go's for example.
As for `collections.abc.Sequence`, using an ABC sucks because then you have to define a class and inherit from it. I don't want to do that. I just want to pass something that conforms to the duck type. Are these now also defined as structural types/Protocols?
Duck typing is indeed powerful, but it does not need to be a runtime check.
> What am I going to do, define an ABC or typing.Protocol called `ListLike` or something?
Yes, how am I going to know what types I can pass to mean instead? I would go to the extreme of saying that a lot of python code should be type annotated (if annotated at all) with protocols instead of concrete or base classes.
Of course until type checking is properly integrated in the interpreter this is all kind of pointless.
Oh, if I'm writing it for you (ie. I'm writing a library) then I'll put documentation (probably as type annotations, as I said in the first half of my comment). But a lot of code I write is just for me, or is going to be part of a completely self-contained project and it would be obvious from context what to pass. Like I said, it's a judgement call that Python expects you to be able to make.
Python dicts, when used as composite types or records, are a literal hellscape. Grepping through the code to find out where stringly-keyed fields get written takes way more time than thinking about types ever would. These should be structs.
Static typing has so many advantages:
- It lowers the software defect rate. All type errors are caught for free at compile time instead of runtime. This makes the software strong and rigid instead of brittle, and it removes an entire category of tests you would have to write and maintain.
- Static typing makes code maintainable for other people, including future you. It's self-documenting. You know precisely what things are in the immediate scope.
- Static typing makes bug-free automated refactoring with tools possible. There is no greater pleasure than mutating code via its AST.
Static typing is not hard, either. Most typed languages don't require type declarations except in structs and function declarations - that's really not a lot of effort.
I'll keep my dynamic typing, thanks!
There's a world of difference between the experience you get with untyped Python and typed Python that you check with strict mypy or Pyright. After using typed Python for a while, I don't think I could go back to a completely untyped codebase.
I'd probably call myself within the "dynamic typing crowd" and I've tried plenty of static typing. It mostly just slows down iterating on something, prevents issues that I/my projects don't really suffer from in the first place, and gets in the way more than it helps.
The statically typed languages I've tried are: C#, Crystal, Elm, Go, Haskell, Haxe, Java, Kotlin, Nim, Rust, TypeScript and probably more I'm forgetting about. Out of those, I've probably written most Rust code. I wouldn't say I despise static typing, but I'm not getting the same value from it that others seem to get.
I still come back to Clojure, ClojureScript or just straight up vanilla JavaScript, as they're much more effective at actually helping me solve the problem I have in my practical day-to-day.
I feel the same way about Lisps, but more strongly. It’s really fast and fun to sketch stuff up until you’re about 3 or 4 functions deep trying to figure out the shape of that one inner associative array and what made you think this little adventure was a good idea in the first place.
But that's exactly the situations where lisps shines! Select the form in your editor, evaluate it and your editor tells you exactly what it is, both runtime and compile-time data, pure magic :)
Another reason I've seen people not realise how good static types are is if they aren't using a proper IDE with code intelligence.
The benefits of static typing are:
1. Fewer bugs.
2. Makes code easier to understand, because you know what types things are. That gives you a lot of information (even "business domain" things) that usually aren't documented in dynamically typed code.
3. Navigating code in an IDE that understands the language is a lot faster. E.g. you can just ctrl-click something to go to its definition, auto-complete works reliably, you can find all the uses of an item reliably, etc.
4. Refactoring code becomes tractable and easy. You can rename an item and it will automatically update all the usages. Anything you miss will get caught by the type checker.
If you don't use a proper IDE you're missing out on half of that.
Even so, I don't really see how you can say it gets in the way more than it helps. Unless you're working on really small & one-off projects, the amount of time you'll save by not having to deal with type errors or spend ages deciphering code just to figure out what type something has easily offsets any time adding the types.
Ok, does Visual Studio Code and/or Visual Studio and/or the various JetBrains IDEs count as "proper IDE"? If so, those are the editors I tried, and while they're nice and all, none of those things you listed got better compared to my non-static typed languages usage. In fact, I'd argue that some of those things get worse when using statically typed languages, especially #2 and #4.
> Even so, I don't really see how you can say it gets in the way more than it helps. Unless you're working on really small & one-off projects, the amount of time you'll save by not having to deal with type errors or spend ages deciphering code just to figure out what type something has easily offsets any time adding the types.
I don't have to spend any time figuring out what type something has because that's not a typical problem I have when reading and writing code in for example Clojure. And if I do wonder about the shape of the data or whatever, I evaluate that snippet of code in my editor and it shows me what data is inside of whatever I had selected.
It's OK that we have different ways of working and our brains work differently. Static typing is not objectively better, some things just work better in one way for some people. I really love the feedback cycle of "Read code, evaluate it, change it, evaluate it, write a test, evaluate it, save file" for producing/modifying code, and others want a cycle of "Read a lot, type a bit, run type checker, run unit tests" or whatever, and that's perfectly fine.
But I have often wondered, if someone wants static typing, why not just use a statically typed language?
A very high proportion of Python devs aren’t using it for the language, but for the libraries and ecosystem. Also lots of them weren’t the ones who picked it.
Meanwhile, not at least having the kind of autocomplete and documentation that type hints provide is kinda hellish on any project of more than 200 or so lines. The time savings from spotting runtime bugs before they happen is just a bonus.
Personally, I almost never need to add a type hint outside high-level definitions and function/method signatures, so they’re not really in the way even when I’m being pretty thorough with them.
Mind you, I prefer a type system. I'd prefer to use Typescript over Javascript, etc. But I've also used a number of dynamic languages that let me work a lot faster when needed; Tcl, Ruby, and Python are examples of these.
I've also used some type systems that lift a heavier load, letting me pay more attention to the type definitions and know that it will "just work" at the end, because, mathematically, it works. Haskell falls into this category (though I rarely use it for anything other than fun).
I get it, you haven't use a dynamic system in a way that it works out for you. But that doesn't mean they're wrong... just that others have different experiences.
If this is the first time people are introduced to static typing in programming I can understand their frustrations and opposition to it. Its probably the worst type system in any modern, popular language out there, except perhaps when things like clojure(script) pretends to have types (shudder).
So no, we're not coming around. Static typing as dictated by a heavy-handed and super-strict compiler has it's place, but has gimped our industry for decades.
However, we do have to be very careful. Static, compile-time typing has kept the hipsters and junior-devs at-bay, and kept them from causing too-much havoc as we've seen in the JS world. So it's definitely an up-coming hazard for us to navigate around and make sure we don't fall prey to. Otherwise python will turn into another JS dumpster fire. Luckily, the JS developers are too-distracted and enthralled with node.js to jump ship.
That will sadly probably never happen given the momentum they’ve built so the only choice is to retroactively add static typing and begin enforcing it in individual projects.
When starting a new project, I would suggest nobody choose a dynamically typed language. Between Swift, Kotlin, C#, Go, Rust, etc… there’s no need for anything else. As long as front end web is around you might need TypeScript - and as long as ML is Python centric you might benefit from using a bit of it. But I wouldn’t make them the primary language.
Context: Recovering dynamically typed language addict of 12 years. They’re slow, error prone, and don’t scale to a large engineering team.
- Faster attribute access: your code is faster
- Slotted classes take less RAM, less L1 cache pressure, your code is faster
- Wrist friendly to with .foobar instead of ["foobar"]
- Runtime error if you misspell an attribute name
For me it’s mostly about .attribute being more in line with the rest of the language. Kwargs aside, I find overuse of dicts to clunky in Python
https://wiki.python.org/moin/UsingSlots
Whether or not the performance matters...well that's somewhat subjective since Python has a fairly high performance floor which makes performance concerns a bit of a, "Why are you doing it in Python?" question rather than a, "How do I do this faster in Python?" most of the time. That said it _is_ more memory efficient and faster on attribute lookup.
https://medium.com/@stephenjayakar/a-quick-dive-into-pythons...
Anecdotally, I have used Slotted Objects to buy performance headroom before to delay/postpone a component rewrite.
In many ways it matters more because it’s Python.
I’ve met a lot of teams throughout my career who struggle daily with a badly performing Python codebase. You can write a no-frills web service in c#, go, rust or JavaScript. And, so long as you don’t do anything stupid, it’s usually plenty fast enough from day 1 to handle your users. But in my experience, the same isn’t true of Python. I’m sure Python web services can be made to run ok, but because it’s slow by default, I bet a lot more time is spent optimising Python programs around the world than optimising JavaScript.
depends on the type in question. If you are fetching and operating on a large number of records then it can matter. But otherwise the answer is more often that it does not really matter.
T = TypeVar("T")
U = TypeVar("U")
V = TypeVar("V")
P = ParamSpec("P")
def modelargs(model: Callable[P, U]):
def _modelargs(func: Callable[[T], V]) -> Callable[P, V]:
def __modelargs(*args: P.args, **kwargs: P.kwargs) -> V:
return func(model(*args, **kwargs)) # type: ignore
return __modelargs # type: ignore
return _modelargs
class MyModel(BaseModel):
foo: str
bar: int = 4
@modelargs(MyModel)
def test_func(model: MyModel):
print(model.foo, model.bar)
return 4
test_func(foo="Hello", bar=20) # -> prints Hello 20
Pydantic is targeting other use cases. The point of TypedDicts is compile-time safety without run-time overhead. Pydantic is useful for a lot of things, but performance isn't exactly its strong suit (written as of 2.9.2, I was just revisiting it earlier this week).
Anyway, in the same spirit of function signature hacking, I've found the following useful for "inheriting" them:
_T = TypeVar("_T", bound=Callable)
def inherit_signature(_function: _T) -> Callable[..., _T]:
return lambda f: f
# Requests for example has some long signatures (via typeshed).
class CustomSession(requests.Session):
@inherit_signature(requests.Session.post)
def post(self, url: str, *args: Any, **kwargs: Any) -> requests.Response:
...
def post(
self: Session,
url: str | bytes,
data: _Data | None = None,
json: Any | None = None,
*,
params: _Params | None = ...,
headers: _HeadersUpdateMapping | None = ...,
cookies: RequestsCookieJar | _TextMapping | None = ...,
files: _Files | None = ...,
auth: _Auth | None = ...,
timeout: _Timeout | None = ...,
allow_redirects: bool = ...,
proxies: _TextMapping | None = ...,
hooks: _HooksInput | None = ...,
stream: bool | None = ...,
verify: _Verify | None = ...,
cert: _Cert | None = ...
) -> ResponseDon’t forget to use ‘ConfigDict(frozen=True)’ absolutely everywhere!
That said, there's also msgspec [1], which I've not used yet but plan to for my next project. It is supposedly quite a bit faster than Pydantic in de/serialization.
> whether or not models are faux-immutable, i.e. whether __setattr__ is allowed (default: True)
Seems everything is “faux” in Python world when it comes to typing ;)
You can store a float in an attribute annotated as a string, and default Python will not stop you, or display any kind of warning. The typing is purely for development, and does nothing once compiled.
If you want typing to actually be enforced, you need to use something like Pydantic.
If your input data is the correct type (big if #1), and if your types are sounds and consistent with each other (big if #2), then you don't need something like Pydantic, because the types guarantee that there will never be a float stored somewhere that's annotated as a string. It cannot happen, because as soon as you try to store a float in a string attribute, your type checker (Pyright, Mypy, etc) will complain. And if you're consistently running that type checker over your code, e.g. in a CI job that runs over your codebase for every push, then you can never have checked-in code where the types are incorrect.
There are the two big caveats above, but these turn out not to be such big deals. Caveat #1 is that you can't rely on external data to match the types you've described internally. This is an ideal use case for Pydantic, like you say: you check external types once, at the boundaries of your program, and then internally you can be confident that the types will always be correct.
Caveat #2 is that you don't break the type checker's limits. This generally means avoiding things like `Any`, and ensuring that your typed code always calls other typed code, and never untyped, un-annotated code. The easiest way to do this is to start by typing the leaf files in your program (the ones that only get imported, and never import anything in turn). These can't import untyped code, so if they're fully typed and the type checker passes, then you can be confident that their types are correct. Now sure, a bad caller could try and call one of these functions with invalid input, but as they say, Python is for adults, and you're free to call a function with the wrong arguments, and you're free to deal with the results.
Now, you can add types to code that only imports typed code, and because you've checked the imported code you know it's correct, so if you also check the importer and it's also correct, then you can be confident at that level too. You can keep going until you've added types to the entire codebase, and now you can be confident that the entire codebase always passes the correct types, and therefore that no runtime enforcement is necessary.
There is a third caveat, which is that this relies on having a powerful enough type system to model all the things that you were originally doing in Python's dynamic type system. This is hard, because you can do some pretty wild things in Python, and my own experience with Python type checkers has been a bit disappointing - they handle more boilerplate-heavy code fairly well, but seem to struggle on more idiomatic code that uses Python's dynamic features. However, that seems to be improving all the time, and Typescript demonstrates that it's certainly possible to model a dynamic language with static types to a very reasonable accuracy.
Type annotations do nothing in default Python, they require some other system to enforce them.
The language happily does it anyway.
int_field: int = "string"
self.int_field = "string"
There are some exceptions to this, like I said in the previous comment, but I think when getting started with typed Python, it's easier to assume that the types will be valid, because that's the case almost all of the time. Usually, overriding type assertions requires some kind of explicit cast or statement, and this is usually documented.
I read this like 10 times and I still dont understand this mess of grammar. Am I having a fucking stroke rn?
- change it to a new value
- set it to None
- leave it as-is
In the json the first two options are specified by sending an object with a "subscription" field, either set to a string or to null. The third case is expressed by omitting the field.
The OP asks how all three cases could be represented in python, and points out that one could not use subscription=None to represent both case 2 and 3 above.
set - no value
- new value
do_no_set
If the “subscription” part of the “patch” data-structure is set to “None” it is impossible to tell if this means:
1/ leave the “subscription” unchanged; or
2/ set the “subscription” to None.
—
You could solve this by requiring that a subscription is either a value or some other sentinel value meaning “not subscribed”:
from typing import Union
class T: pass
NotSubscribed = T()
Subscription = Union[int, T]
def f(s: Subscription): pass
f(44)
f(NotSubscribed)
f(None) # wrong
> Here we have a problem: for subscription, does None mean don't change or remove subscription?
"than i thought" would be compelling to read.
I get that it's attention-grabbing, but it's because it's rude.
You don't know anything about me. You don't know what I think or what I already know or what I won't believe.
I know it's not a big deal in the grand scheme of things, but it's just one of those little aggravating things that makes life just a little bit worse each time you come across them.
"X thing did Y, here's why Z"
Where X and Y can range from meaningless to national security but Z always remains an opinion (usually uninformed too).
Just search "here's why that's bad/good" on google news and there's so much "slop."
That said, it does annoy me too.
Also what annoys me is that I constantly try to play devils advocate for things like this even if I don't always agree with the conclusion of the advocate.
I worked on a python app built by a man with a stronger will to succeed than ability to code.
Data was passed as dicts. Many methods (in the god object, natch) destructured, unwrapped or mutated before passing the new dict to some other method.
TypedDict allowed me to annotate these shitty dicts. It became possible to reason about the code without spending two hours tracing code paths.
The real solution is to be better at code. But, given an app written by a dict fetishist, it's a pretty good solution.