Agent memory or a fact layer: which one does your agent need

Memory recalls what was said. A fact layer holds what your company decided. A quick test for which one your agent needs, and how to run both.

Published:
By Neal Mann
Artificial Intelligence
Agent memory or a fact layer: which one does your agent need

If your agent needs to remember people and past conversations, it needs memory; if it needs to state your prices, policies or product details correctly, it needs a fact layer. Most agents that talk to customers need both, with one rule between them: when they disagree about the business, the fact layer wins.

Two different jobs

Memory and a fact layer both give an agent information it was not trained on, which is why they get confused. They hold different kinds of information, though, and they come from different places.

Memory recalls what was said. A memory layer watches conversations and keeps what is worth keeping: a user prefers email, a manager asks for the same report every Monday, a customer explained their setup last week. It is usually extracted by a model, without anyone reviewing it, and it is scoped to a user, a session or an account. That is exactly right for context. Nobody wants to verify every preference a user mentions.

A fact layer holds what was decided. It stores your business facts, like what each plan costs, what the refund policy says and what the product does today. Each fact is written and verified by a named person on your team, there is one current version, and every agent, app and page reads the same one. NOAN puts it this way: memory recalls what was said. NOAN holds what was decided.

Side by side

MemoryFact layer
What it holdsWho the user is, what they said, what they preferYour company's own facts: prices, policies, product details
Where it comes fromExtracted from conversations, usually by a modelWritten and verified by your team
Who stands behind itUsually no oneA named person, with a date
When something changesA memory is added, updated or marked invalid by the systemA new version supersedes the last; the old one is kept, not obeyed
Who reads itOne user's agent or sessionEvery agent, app and page

A quick test for your agent

Ask four questions about what your agent actually does.

  1. Does it talk to the same people more than once? If yes, memory will make it better: it can pick up where it left off and stop asking the same questions.
  2. Does it state things about your business to people outside it? If yes, it needs a fact layer. A price or a policy said to a customer has to be the current one.
  3. Would a wrong answer cost you money, trust or a support ticket? If yes, the answer needs someone behind it, which memory does not give you.
  4. Do several agents, apps or pages need the same answer? If yes, the answer should live in one place they all read, not in each agent's memory.

If you answered yes only to the first, memory alone may be enough. If you answered yes to the second, third or fourth, you need a fact layer, and probably memory as well.

What goes wrong when memory is the source of truth

Memory is good at its job. Problems start when it is asked to do the other job.

Nobody verified it. Memory records what was said or inferred, so a price a customer misremembered on a call can be stored as confidently as the real one. It changes from the data, not from a decision: memory systems add, update or invalidate entries based on what they see next, which is right for a user's preferences and wrong for your refund policy. And it belongs to one user or one agent, while your business facts belong to the whole company.

None of this is a flaw in memory. It is a sign that business facts need a different home.

Running both

Most production agents will want both, and they fit together without much work.

  • Memory for the conversation. Who the user is, what they asked before, what they prefer.
  • The fact layer for the business. What the product costs, what the policy says, what you offer today.
  • One rule in the system prompt. Something like: "For any price, policy or product detail, read NOAN. If memory or anything else disagrees with NOAN about the business, NOAN wins."

Both usually connect over MCP, so the same agent in Claude, ChatGPT, Cursor or your own code can read a memory server and NOAN's MCP server side by side.

Memory layers such as Mem0 and Zep are good at the memory side. Mem0 lets agents remember users, preferences and past sessions. Zep builds temporal context graphs from your data and conversations, and infers from the data when each fact in its graph became valid and when it stopped. Many teams run one of them next to NOAN. The comparisons go into detail: NOAN vs Mem0, NOAN vs Zep, and the broader agent memory vs verified facts.

Where retrieval fits

There is a third option people reach for: retrieval over your documents. RAG is good at search across large sets of text where the answer could be anywhere. It returns what was written, though, including stale and conflicting versions. If memory holds what was said and retrieval holds what was written, a fact layer holds what was decided. NOAN vs building your own RAG covers how the three fit together.

How to start

  1. List the questions your agent must answer correctly about your business.
  2. Write each answer as one fact and have its owner verify it.
  3. Connect the agent to the fact layer and add the rule above.
  4. Keep the memory layer for context, if you have one.

Next step

Connect your agent to NOAN over MCP at https://mcp.getnoan.com/mcp, or over REST at https://api.getnoan.com/v1 with the OpenAPI spec. Or paste this into your agent:

Read https://www.getnoan.com/setup.md and follow it.