So I've done a little bit of testing. When I loaded up the editor, I expected it to communicate with the installed binaries for my large language models, or possibly use a BYOK (bring your own key) approach. But once I understood that it's driven from the command line and you see the changes happen in the editor that gets spun up, I had no problems.
I've been experimenting a fair amount with LLMs driving procedural graphics myself, although my experiments have been mostly limited to relatively low-resolution stuff, such as EGA adventure game graphics.
So I decided to put Klad through a relatively hard test: I took a reference picture of an illustrated squirrel and asked it to recreate it, then set up the rigging for a basic walk cycle. This was using Fable 5.2 high.
Then I did the same thing, but with a more traditional approach, using standard generative models to manually create a walk cycle. This was all done relatively quickly, so bear that in mind.
Here are the results. I still think these are early days, and LLM-driven graphics are only going to improve.
And there are obvious advantages to a procedural system with rigging: Much easier to maintain consistency, palette control, and the ability to make precise alterations without having to deal with any kind of conventional in-painting nonsense.
I think the results are really interesting. I think it's very obvious the classic image-to-video approach yields much more natural results at this time and with the current models.
I think I'm going to do some testing of my own, and see where things can be improved. I do think that the bottleneck at this time, is the model you use itself.
But it's like you say, it's very early days with an LLM expressing its ability to draw/paint. In my own testing, it kind of feels like it did with coding 2 years ago: yeah sure the model is pretty Okay at building a function, but I wouldn't have let it build a whole feature back then.
Nowadays, I'm practically only reading code and thinking about how to refine my instructions for the next build.
I'm sort of betting that this evolution could go the same way for graphics (and possibly audio/midi as well). And I wanted to have a tool ready, to see how it evolves.
I envisioned it as some sort of a flipbook that would allow me to *cheaply* get a low-res idea of what the 'generate' click would get me, without having to burn movie making tokens.
This may not be exactly it, but it nudged me into a direction. Thanks!
I will give it a try and report back.
If there's anything I can add, do let me know.
The other thing is that this is also a first class drawing app with pen support. So you don't have to use an agent, and you can still profit from the editability. Or start a drawing yourself and have an agent come in later..
For example, but rough numbers here,.. the panel poster on the website took about 15 minutes of fable working it out, and the full scene with the creature following a traveler, was also about 10 minutes of ai work.
To me, it feels just about as fast as having a model code up a medium feature for an application. And that doesn't seem too strange, because I'm the end coding is kind of what it's doing with this app.
A benchmark I'd be really interested in, is how an agent would be able to clean up or improve a drawing made with pen tablet by a real artist. I've been focusing so much on the development side though that this is a test I have not been able to do, but I expect it to work really well. Every time I make an edit to a drawing myself, Claude just knows what I did and why.
If you try, definitely report back! I'm still figuring out what the best way is for an agent to use this tool
The underlying engine is not just vectors: a brush stroke is saved as data, and everything is rendered on the fly into pixels. This means that things like smudging and paint blending is possible, all the while being able to edit even the very first stroke you made.
The best mental model I have right now is: it's the editibility of vectors, with the flexibility of a digital paint brush
This is a really great tool that solves a specific pain point with image models though
I have no idea how an agent would leave a watermark in Klad. It's got no agent inside, and every mark that an agent made would have possibly been made by a human as well. I guess they would have to really specifically target Klad to be able to make a watermark happen...