At the end, if you want to "fill in the blanks" llm will always "make up" stuff, based on all of its training data.
With a technology like photogrammetry you can get much better results, therefor if you have multiple angled images and dont really need to make up stuff, its better to use such
ML models, on the other hand, are in a big way, intuitive assumption machines. Through training, they learn what's likely and what's not, given both the input measurements and the state of the world. They bake in knowledge for what kind of cameras exist, what kind of measurements are being made, what results make sense in the real world.
In the past I'd say that for best results, we should combine the two approaches - have AI supply assumptions and estimates for otherwise explicitly formal, photogrammetric approach. Today, I'm no longer convinced it's the case - because relative to the fuzzy world modeling part, the actual math seems trivial and well within capabilities of ML models to do correctly. The last few years demonstrated that ML models are capable of internally modeling calculations and executing them, so I now feel it's more likely that a sufficiently trained model will just do photogrammetry calculations internally. See also: the Bitter Lesson.
Neat demo, but feels like things need to come quite a ways to make this interesting.
I'd recommend trying Instant Neural Graphics Primitives (https://github.com/NVlabs/instant-ngp) from NVIDIA. It's a couple years old, so not state-of-the-art, but it runs on just about anything and is extremely fast.
This is the heavy lifting: https://github.com/apple/ml-sharp
Previous discussion: https://news.ycombinator.com/item?id=46284658
It's pretty trivial to get running locally and generating the PLY files. Spark's a pretty good renderer for it after you've generated the gaussian splats.
I mean ok its a "demo" tho the funny thing is if you actually check the cli and requests, you clearly can see that the 3 stages the images walks through on "processing" are fake, its just doing 1 post request in the backend that runs while it traverses through the states, and at 90% it stops until (in theory) the request ends.
Better description: Pinokio is a free, open-source "AI browser" that simplifies installing, running, and managing complex, open-source AI applications and creative tools (like Stable Diffusion, ComfyUI) with one-click scripts, removing the need for coding or complex command-line setup.
But sure, click that download link, what's the worst that could happen? Get turned into a donkey and swallowed by a whale?