Of course it's poor taste to add the "You're welcome". But poor taste is par for the course for such startup, unfortunately.
It is definitely best to reuse the parser for the runtime when implementing an LSP but to do so properly means implementing the parser itself as a standalone library. Even better is shipping the semantic analysis as well!
Implementation drift is definitely an issue.
But great project anyways, just wanted to put my thoughts on the matter into the conversation!
A language parser should be correct. A LSP parser should be fault tolerant. My understanding is you cant have both.
Having built a lot of protobuf tooling, I'd estimate that protobuf largely falls into the former camp; most of your time working with .proto files is operating on fields which terminate using `;` at the end of the line (though multi-line is also possible).
[1] source: I've done it for SQLite SQL at https://github.com/lalitMaganti/syntaqlite/
[2] e.g. SQL naturally has this at statement and expression boundaries which covers almost all of the cases poeple care about.
Many editors use tree-sitter, but that is separate from the LSP.
That said, proto itself dissuades or forbids the kind of common things you might do with a LSP, such as renaming.
Renaming fields is a big no-no [edit: this isn't true, see corrections below], as is doing things like re-ordering fields.
A core idea of proto is that versions are strictly compatible with with previous versions. This itself has limitations and challenges for migrations, but encourages good practice about compatibility that usually gets ignored or hand-waved away in most ecosystems.
I accept however that it's often easy to offload both the re-structuring and the checking of version compatibility to an LLM and let them go at it.
There was definitely one that kept tripping up the checks and it was something people like to do.
Save files? Looser than exact version multiplayer?
Thus all the versioning overhead of protobufs is not needed for this wire protocol.
(Yes, games still use versioning everywhere else where it makes sense: save games, asset data, config etc...)
But then again, those barely count as games, I guess.
....
Tada!
If you patch clients to google services in Python to use json instead of grpc they get faster and more reliable. A lot faster. Benchmark it!
def get_json_client() -> CloudLoggingQueryClient:
"""Client for log queries (JSON transport, avoids gRPC overhead)."""
client = google.cloud.logging.Client.from_service_account_info(...)
client._use_grpc = False
return client
And look at the code quality of the implementation! It's like a team of interns wrote it while drunk. It is a complete spaghetti mess, but has tons of super convoluted micro optimizations that are slower than just doing the most obvious thing, but make the implementation confusing and indirect. It's trash code.
I always thought Thrift was waaay better than any of the alternatives, but it always had terrible documentation and I think it died mainly because of that.
If you never change the schema then you don't have to worry about it, get things working and never look back.
If you do change your schema from time to time, you need testing between the two systems. If you have good tests again a single source of truth is fully redundant, both systems are talking just fine. If you don't have tests things can and will break all the time even using protobuf.
It’s about data transmission. Being able to encode and decode in a type safe manner between different languages (and so, different platforms) is a goal that makes a lot of sense.
> If you do change your schema from time to time, you need testing between the two systems
Or you could just use a defined format that doesn’t require testing. I rarely use protobuf but I can see why people do. The guaranteed backwards compatibility is huge for people who can’t just publish a new web frontend at the drop of a hat.
It’s probably not an exaggeration to say that being able to avoid tests between different systems who have different versions of the schema is a core goal of protobuf. Why? These two different systems are probably owned by different teams, and introducing explicit tests between different versions of them increases coupling between them.
The advantages of json are not related to performance.
I'm starting to question many of protobuf's advantages (perhaps not the wire format). Add to that monorepos and other fads of the 2010s given the rise of LLMs.
I used to be a big believer in this stuff, but I'm quickly having my core assumptions change out from under me.
OTOH if LLMs are to write, but not supplant, much of software, then boundaries, delegation to deterministic layers, good compilers to bonk miscreant models on the head with error message seem essential.
At one point it would have been shocking to assert that the compiler would live in-band with the program too. and yet JS eats the world. It seems shocking today that we could have a universal prior over the world operating in the ms/us nJ/pJ range required. And yet … ?
This works up to a point, and then it doesn't. And you're left with tons of inconsistently formatted data.
My company is built on protobufs from ground up :) We use it in the database, for remote calls, on the frontend, etc. The protobuf language is not great, but it's about the right balance between too expressive and too restricting.
And the best thing is that it's compact, compared to OpenAPI.
If we do strict schemas, I'd like to see less ceremony around them. Tool calls instead of brittle build steps and protocol registries.
Perhaps we need new tools for this going forward.