I've been building MemoryLane (https://github.com/deusXmachina-dev/memorylane) which comes at this from the other side - it records screen activity, spots repeated patterns with AI, and then tells you "hey you keep doing this, want to automate it?" Works as an MCP plugin for Claude/Cursor.
Feels like pattern detection (finding what to automate) + browser agents like yours (actually doing the automation) is the right combo. Are you thinking about the discovery side at all, or mostly focused on execution?
Structuring the task prompt into named blocks (objective, constraints, expected output format per step) before the workflow starts makes each step much more reliable. The agent has less to infer.
Built github.com/Nyrok/flompt to help with this, a visual builder that decomposes instructions into semantic blocks and compiles to Claude-optimized XML. Useful for defining the "task shape" before handing it to an agent.
So the model isn’t carrying the whole instruction chain across multiple steps, it’s just solving the current task. Similar pattern to what tools like Codex CLI or Claude Code do.
I mitigate it by giving the agent a fixed action set (no scripts, no direct API calls), and breaking tasks into focused subtasks so no single agent has broad scope. The LLM prioritises its own instructions over page content, but if someone managed to hijack it, the agent can interact with authenticated sessions. Everything's visible in real time though, and all actions are logged, so you can see exactly what it's doing and kill it.
Practically speaking, I use it similar to how people use Zapier or n8n, you set up specific workflows and make sure you're only pointing it at sites you trust. If you're sending it to random unknown websites then yeah, there's more risk.
But even then, an attacker would need to know what apps you're authenticated with and what data the agent has access to. The chances of something actually happening are pretty low, but the risk is there. No one's fully solved this yet.
Just need to see if people find this version useful