Working memory and episodic logging are solved problems: the context window holds the current task, a checkpointer saves the conversation thread, and logs record what the agent did in prior sessions. When the agent reaches into a CRM, a data warehouse, or an HR system for facts it needs, that retrieval rarely checks whether the requesting user has permission to see what comes back. The issue is that the agent can return a record the requesting user isn't cleared to see in the source system.
What is agent memory architecture?
Quick definition: Agent memory architecture is the system that stores, retrieves, updates, and governs the information an AI agent uses across steps, conversations, and sessions, beyond what fits in the model's context window.
'Cognitive Architectures for Language Agents,' known as CoALA, splits agent memory into short-term working memory and long-term memory, then divides long-term memory into semantic, episodic, and procedural stores.
A 2026 paper, 'The Missing Knowledge Layer in Cognitive Architectures for AI Agents,' points out that CoALA groups semantic and episodic memory together under "long-term" and gives neither a defined update mechanism, ownership scope, or decay behavior. Two memory types can share a category and still need completely different permission models.
The four types of AI agent memory
Working memory holds the variables and observations an agent is actively using: the conversation so far, tool outputs from the last few steps, and anything it needs active to finish the current task.
Episodic memory stores a record of past interactions and events, timestamped, so the agent has history to draw on without replaying every prior session.
Semantic memory holds durable business facts, the kind that typically live in a system of record outside the model.
Procedural memory captures reusable workflows and learned action patterns the agent applies across tasks.
Memory type | What it stores | Typical store | Governance risk |
Working | Active variables, conversation state, recent tool outputs | In-context session state | A tool output can carry data into the context window that the requesting user shouldn't see |
Episodic | Past interactions and events | Log store, vector database | Logs can outlive a user's access to the source system they came from |
Semantic | Business facts, entities, CRM and ERP records | Vector database, relational database, live source systems | Retrieval returns records the requesting user can't see in the source system |
Procedural | Reusable workflows, learned action patterns | Prompt templates, scripts | Changes what the agent does more than what data it can see |
Episodic and semantic memory need different retention rules for a reason: an episodic experience should decay the way a person's memory of an ordinary day fades, while a semantic claim should get superseded when newer evidence shows up. Treat both the same way, and you either keep stale logs around forever or discard facts that are still true.
Semantic memory raises its own question. If it draws from Salesforce, NetSuite, or Snowflake, the safer pattern is reading it live, under the requesting user's own permissions, instead of copying it into an embedding store that has no permission model attached. An embedding doesn't know who's allowed to see the record it came from.
Procedural memory is lower risk by comparison. It's the agent's playbook, the steps it takes to close a ticket or draft a report, and it changes behavior more than data exposure.
Short-term vs. long-term memory: Context windows, retrieval, and live data
LangGraph draws a clean line between the two scopes. Short-term memory is thread-scoped and persisted by a checkpointer, tied to one conversation. Long-term memory lives in custom namespaces and can be recalled across any thread.
Stuffing more history into the context window to bridge that gap doesn't hold up at scale. In the Context-Folding approach, an agent matched or beat standard ReAct baselines while running with an active context ten times smaller, by collapsing finished subtasks into summaries instead of keeping every step in view.
Short-term memory keeps the current task coherent inside the context window. Long-term memory persists across sessions, and because it needs to be retrieved rather than carried along, retrieval is exactly where access control must apply.
The practical fix is pulling live data from the source on demand, with filtering pushed down to that source, which keeps prompts small and answers grounded in current data.
How governed access works across agent memory layers
The Open Worldwide Application Security Project's (OWASP) Top 10 for LLM Applications 2026 moved Excessive Agency up to the third spot, and its guidance is specific: enforce identity-based, least-privilege access outside the model, not inside the prompt.
How short-term agent memory is governed
Session state is isolated by thread or session ID, which LangGraph handles well. That isolation protects one user's conversation from another's. It does nothing about a tool output written into that state that happens to include data the requesting user isn't cleared to see. That output has to be checked against the requesting user's permissions before it reaches the context window.
How long-term memory retrieval enforces user-level permissions
The Model Context Protocol (MCP) authorization specification frames the server as an OAuth 2.1 resource server and the client as acting on behalf of a resource owner. In practice, that's what passthrough authentication should mean for long-term memory: the end user's identity, via OAuth or SAML, passes through to the source system where the source supports per-user authentication, and the source system enforces its own native permissions. It isn't the same as forwarding a raw token and hoping the source sorts it out.
Quick definition: Privilege drift is the gradual build-up of access an agent holds beyond what any single requesting user is entitled to, usually caused by shared service accounts or long-term memory stores that keep data without the permission context it came with.
Audit trails and preventing privilege drift
Audit logs need to record the user, the agent, the memory layer, the source system, and the request behind every retrieval, and those logs need to be ready for export and review. The NIST AI RMF places this under its GOVERN function, and the Generative AI Profile (AI 600-1) applies that to generative systems.
Implementing governed memory in LangChain, LangGraph, CrewAI, and Google ADK
Framework | Short-term mechanism | Long-term mechanism | Governance gap to close |
LangGraph | Checkpointer (thread-scoped) | Store with custom namespaces | Namespaces separate data but don't check source-system permissions |
CrewAI | Shared crew memory by default | Same store, opt-in per-agent memory | Per-user scoping is a deliberate design choice, not a default |
Google ADK | Session and State (current conversation) | MemoryService (searchable, cross-session) | External sources feeding MemoryService need governed connections |
CrewAI shares memory across every agent in a crew unless an agent is given its own. That default works fine for a single-tenant pilot. It stops working the moment two users with different permissions share a crew, which makes per-user scoping something you design in rather than something you inherit for free.
Google ADK keeps the same split in different terms. Session and State cover the current conversation. MemoryService is the searchable store that can span past sessions or pull in outside data. The outside data is where the governance question lives, since MemoryService itself has no opinion about who's allowed to see what it returns.
None of this means swapping out your orchestration layer. CData Connect AI doesn't replace LangGraph, CrewAI, Google ADK, or memory frameworks such as Mem0. It sits underneath them as an MCP-based connectivity and access-control layer, so a retrieval call from any of those frameworks runs with the requesting user's actual permissions instead of a shared service account's.
Build a governed memory layer for your agents with CData Connect AI
The orchestration framework owns working and episodic memory. CData Connect AI governs how every layer reaches business data, applying inherited user permissions and OAuth or SAML passthrough no matter which layer is asking.
For a security lead weighing sign-off, that means least-privilege access in line with OWASP's guidance on Excessive Agency, plus audit logs that record the user, agent, memory layer, and source system behind each retrieval, with metadata available for routing to your security information and event management (SIEM) system.
For the team building the agent, it means retrieval stays lean. Connect AI pushes filtering down to the source instead of pulling full tables into the model's context, so governance doesn't add context or cost.
Start a free trial of CData Connect AI.
Frequently asked questions
What is agent memory architecture, and why does it matter for enterprise AI?
Agent memory architecture is the system that stores, retrieves, updates, and governs the information an AI agent uses across steps, conversations, and sessions, beyond what fits in the model's context window. It matters because agents moving from pilot to production need that information to stay accurate and respect the requesting user's actual permissions, not just fit in the prompt.
What's the difference between short-term and long-term memory in an AI agent?
Short-term memory is thread-scoped and holds the active conversation, persisted by a checkpointer. Long-term memory lives in its own store, can be recalled across any thread, and must be retrieved rather than carried along, which is exactly where access control has to apply.
Where should different types of agent memory be stored?
Working memory stays in-context as session state. Episodic memory typically lives in a log store or vector database with its own retention rules. Semantic memory, when it reflects live business facts, is best read directly from the system of record, such as a CRM or data warehouse, rather than copied into an embedding store with no permission model attached.
How does an AI agent retrieve the right memory without overloading its context window?
Push filtering down to the source system instead of pulling full records into the prompt. Collapsing finished subtasks into summaries instead of keeping every step in view can shrink the active context substantially while matching baseline task performance.
How do you prevent an AI agent from accessing data a user isn't authorized to see?
Pass the requesting user's identity through to the source system via OAuth or SAML and let that system enforce its own native permissions, rather than routing requests through a shared service account. Log the user, agent, memory layer, source system, and request behind every retrieval so access can be audited, not just assumed.
What are the most common mistakes in agent memory architecture design?
The two that show up most often: treating episodic and semantic memory as interchangeable when they need different decay and update rules, and copying business data into an embedding store that carries no permission context from the source system it came from.
How should memory be scoped and isolated in a multi-agent system like LangGraph?
LangGraph isolates short-term state by thread or session ID and keeps long-term memory in custom namespaces. Namespaces separate data between agents, but they don't check source-system permissions on their own, so scoping by user identity still has to be built in rather than assumed.
