The system accepts a live camera feed, which is modified in real-time according to a text prompt. The result is presented on a large screen (a magic mirror) and has interactive speeds. It has about 5x the pixel throughput of StreamDiffusionV2 at a quarter of its latency, on two GPUs where it uses four.
I developed the demo all by myself (unusual for a professor) with agent swarms doing the implementation.
The development was made possible by 3 key ingredients:
- Heavy use of Karpathy's autoresearch methodology on several levels of abstraction
- Having access to a nice GPU cluster (~1000 GB200 hours used)
- Open weight models: a model 10x slower, such as Claude Fable, would not have been applicable for my main autoresearch loop, which ran for about 1 month.
The end result is a demo that shows the emerging possibility of using diffusion transformers for AR and VR systems, instead of Gaussian Splatting that so many people bet on.
There are still things I need to figure out:
- How to improve temporal consistency
- How to make edits robust for multiple humans in the frame
I learned several things, including:
- Autoresearch is unreasonably effective.
- 1000 GB200 GPU hours were wasteful. I should have planned better.
- Agent swarms on multiple heterogeneous machines are not trivial
- 87 ms is too high for deployment on AR Glasses. I need to find a way to bring it down to 20 ms
- The numerous bugs of TensorRT on Blackwell, e.g. failures for high resolution VAE encode/decode