This reminds me of the data-driven programming and state modeling they taught us at Waterloo: https://news.ycombinator.com/item?id=35758587
A few thoughts:
1) Sorry in advance; I spent too long at Google and am tainted by that. Maybe the proto/thrift/avro style of typed schemas + generated client libraries is overkill, but it made some of this automatic. You probably don’t need the binary wire format everywhere, but schema annotations (e.g., field-level PII tags, etc.) and a centralized automatic registry for these objects did help.
2) I’m mostly curious why I don’t feel this pain heavily anymore. Maybe I’ve aligned on (or am just OK with) the other side — of engineers paying the tax and dealing with it? I still code often, but heavily optimize for speed of delivery which seems to be one metric that helps career wise. Not sure yet how to square those thoughts yet.
I think this is refreshing and I like it when people push for cleanup and improvements to how we develop. I'll suggest engineers at work try and see what the feedback is.
I really loved protobuf in the past. I've used them at Square and implemented gRPC at Opendoor. Protobufs have been really helpful with the runtime compiled codegen. They are heavy though, and there are some issues.
Getting a data schema to go all the way from the backend to the front end is challenging with Protos. You can use proto over json but the clients on the fe are not well looked after. You have to fight the tooling (or add copious amounts of it) to have the same contracts flow all the way through your code.
In my experience, protos work well inside a company for backend, but they're not really made for public use (agent to agent) or runtime use.
Even with the strong contracts in protobufs, which I've personally seen improve teams using them! There's still the questions... but, what does X or Y mean!?
One of the issues with writing meaning into your proto files in terms of field or message options is that only the authors can annotate it for the authors purposes. What we need is a way for authors to write it an annotate it, but _also_ for humans and agent consumers to add their own meaning to the schemas, fields and uses. Use-cases, gotachas, relationships, service annotations.
We've been re-building slightly different versions of the same things in each company / application, and every time we do, the reason for a field, the meaning is usually buried in email or slack threads, remembered only by "those who came before". In the age of agents where they need context, that type of tribal knowledge is holding us back.
Maybe what we want is for agents to both consume annotations and write them back into an open annotation registry that overlays a schema registry?
1. For applications with a database, the database schema for core models strikes me as the canonical definition, except it's missing descriptions. 2. It feels like this is only a partial solution. Why not go all the way? It seems to me that more aspects of code needs this level of governance than just data models.