A year and still no implementation for such a basic need as offloading MoE layers onto the CPU selectively. On llama.cpp I can get models like Qwen 35BA3B running partially on gpu/cpu with 40t/s on a laptop thanks to --n-cpu-moe but on this VC funded joke it would be simply unusable. I can't quite understand how you make a wrapper so much worse than the code you're ripping out.
>This funding is fuel for what’s ahead. Ollama sits front and center in the open model ecosystem
No.
https://www.slashdata.co/post/global-developer-population-tr...
Note to self, Enshittification ahead. Don't use any ollama services unless it's calling an industry standard api.
Investment firms and organizations
Benchmark — represented by Peter Fenton Theory Ventures — Tomasz Tunguz 8VC — Alex Kolicich Y Combinator Garage Capital Pace Capital 49 Palms GTMFund
Individual investors
Solomon Hykes — Docker founder Aaron Katz — ClickHouse CEO Spencer Kimball — Cockroach Labs co-founder and GIMP co-creator Quinn Slack — Amp CEO Marianna Tessel — Cisco board member Michael Montano — former Twitter head of engineering Other unnamed angel investors >like lmstudio and google/alphabet maybe!
Investors are not free
Since it pivoted, I imagine the story they are telling is the usage. But you can add to that Peter Fenton has had incredible success in commercial open source (Docker-ish, Elastic, JBoss, CockroachDB, TimeScaleDB, and if you include Benchmark there is Elastic, Confluent, etc.) so a bunch of these people are probably trying to ride the Peter Fenton/Benchmark train. My guess is that's what Theory is doing.
Investors are not free but they are very hype driven. There are < 10 investors who are truly experts in open source in my opinion in the market (and maybe < 5.) It's a really wacky cadence and I would argue there are different types of OSS businesses, which makes it even more difficult to be an expert. It's something I struggle with personally investing in OSS -- I think I understand the likely motion for an OSS startup and then > 50% of the time I am wrong.
Georgi Gerganov, the author of llama.cpp, is the real hero here.
But please don't use ollama, or their quants. Not only is the app itself slower than pure llamacpp. But their quants are often no where near the best.
I really hope people start with something like unsloth, as their software and quants are really much better all around.
I'm going to uninstall Ollama.
Ollama is/was based on that.
For those new to llama.cpp, the llama-server component running in "router mode" is fantastic; you place your common configuration in a defaults section of an ini file, and then model specific configuration in separate sections of the ini file. You can switch between models, on the fly, using the API, and the built-in web front-end for llama-server supports model switching as a drop down menu. Llama-server can just always be running, in the background (you can set an idle time for it to automatically unload weights to free vram/ram, and a maximum simultaneously running model count which will evict models on an LRU basis before loading new ones, if the count is exceeded).
llama-bench is another great tool included in llama.cpp, where you can provide ranges of options to test, and just let it run through all the variations that you requested, then take the best performing and add those settings to the ini file above.
There are a lot of optional tunables that can really improve performance with llama-server. E.g., typical average tok/sec generation speed improved over 7X for Gemma4 31B, on my laptop, with tuned batch-size / ubatch-size and enabling both spec-type draft-mtp and ngram-mod (with some custom settings on both). I.e., even though you don't need to do this for llama-server to be useful, it can be worth your while to spend a little time optimizing settings for your favorite models especially if your hardware is marginal.
It's not my favorite but we do have to be aggressive in fighting these gross attempts for vc-backed silicon valley non-contributors to exploit open source labor.
Looks like it's their second time, as Kitematic was a very thin wrapper over the Docker CLI.
I think this is probably first time ever I read one of these and was convinced immediately that I should probably look into using something else.
I wish HN would stop sharing drama-seeking posts, especially by anonymous authors, attacking the hard work of people.
Are ollama's authors perfect? No, they are just humans. But many people find their work valuable and we should support Open Source.
I can assure you that you won't like the world where nobody release Open Source projects anymore because of the constant negativity.
The llama.cpp authorized ollama to do this... Unintentionally.
>The project’s binary distributions didn’t include the required MIT license notice for the llama.cpp code they were shipping. This isn’t a matter of open-source etiquette, the MIT license has exactly one major requirement: include the copyright notice. Ollama didn’t.
It looks more like you're the anonymous author attacking the hard work of the author of this blog post.
The blog post alone has more relevance then the ad-post.
Were they so enraptured with the “docker for LLMs” line?
Then put those LLMs to work and build an open source alternative to everything that Ollama is doing.
After all, open source is a pricing weapon to race everything to $0.
I will be curious to see if their subscription ever supports the ~3T open weights models announced this week.