Building memory and state layer for agents