While building a browser-native privacy AI (Lokul), I hit a massive wall trying to manage memory for local LLMs. I realized that just dumping chat history into IndexedDB and blindly feeding it back into the prompt quickly blows up context windows and leads to heavy hallucinations.
I was spending 80% of my time babysitting context limits and writing custom boilerplate to prune and inject history. I got tired of it, so I ripped the memory architecture out, rewrote it, and open-sourced it.
LokulMem handles:
Smart Collection: Categorizing conversational turns vs. system states.
Strategic Context Insertion: Dynamically fetching only the most relevant context so you don't exceed token limits.
Local-first storage: Keeping everything completely in-browser for privacy.
I wrote a deeper dive into the architecture and the headache of goldfish-memory LLMs on my blog here: [Insert your Hashnode URL here]
I’d love your feedback on the codebase, the approach, or to just hear your war stories on how you're currently handling local LLM context limits.