Accessibility wasn’t the starting point, but the more I work on this the more it feels like a natural fit.
On nonverbal inputs, I’ve focused on gaze and gestures so far. I’ve thought about things like head nods or blink patterns for simple confirm/cancel, but not explored them deeply yet.
Right now the main challenge is keeping everything reliable without adding too much complexity.
Curious how you’d see this used in practice?
It’s a React SDK that lets you control your app with voice, gaze and hand gestures, no mouse or keyboard.
The idea is simple: instead of figuring out where to click, you can just look at something and say what you want.
Example: look at a row → “open this” | say “navigate to equipment” | say “create a ticket” → it builds it
Some interactions are instant (navigation, selection), while more complex ones use an LLM and take a few seconds.
What makes it different from most “AI agents” is that it runs inside the app, not outside of it.
It has access to: React state, routing, visible UI
So it doesn’t rely on screenshots or DOM guessing, it actually understands what’s on screen.
I originally started this thinking about environments where mouse/keyboard don’t work well (gloves, operating rooms, field work), but it’s also interesting for internal tools and dashboards.
This is v0.1 still rough in places, but the core flow works.
GitHub: https://github.com/haelo-labs/exocor npm: https://www.npmjs.com/package/exocor
Curious what you think, especially if this feels useful or just like a gimmick.