← Building

Your AI Agent's Memory Is Infrastructure. Treat It That Way.

When we moved our dev directory, 55,000 agent memory chunks became stale overnight. The fix was three seconds. The lesson took longer.

·Agentic Business / memory-systems / knowledge-ops

When you rename a database column, you write a migration. When you move a critical config file, you update every reference and verify. These are table stakes for any production system.

We applied none of that discipline to our AI agent's memory — until 55,000 chunks told us to.

The Problem No One Warns You About

Most write-ups about AI memory focus on the retrieval side: which vector store to use, which embedding model, how to chunk documents, HNSW vs. flat index. These are real questions. But they're the easy part.

The hard part is provenance.

When an AI agent reads a file, processes a session, or receives a decision, it doesn't just store the content — it stores where that content came from. The source path. The origin. This matters because the agent needs to know which project a chunk belongs to, whether it's still current, and whether newer content has superseded it.

Our agent's memory system — Anamnesis — stores tens of thousands of chunks, each tagged with a source_path. It's a SQLite database. Regular schema. Indexed for both full-text search and semantic similarity.

When we reorganized the development directory structure (consolidating /Documents/Dev/ into ~/Dev/ for a cleaner filesystem), every single source_path in the database pointed to the old location. All 55,598 of them. Plus 2,915 rows in the source_meta table.

The agent couldn't reconcile its memories with current file locations. Worse: it didn't know it couldn't. It would recall a memory, see the path, and assume the path was still valid.

The Fix Is Trivial. The Insight Isn't.

The actual SQL update took three seconds:

UPDATE chunks
  SET source_path = REPLACE(source_path, '/Documents/Dev/', '/Dev/')
  WHERE source_path LIKE '%/Documents/Dev/%';

UPDATE source_meta
  SET source_path = REPLACE(source_path, '/Documents/Dev/', '/Dev/')
  WHERE source_path LIKE '%/Documents/Dev/%';

That's it. Three seconds, 55,000+ rows, done.

But the lesson is not "SQL migrations are fast." The lesson is that we were treating memory as a feature — a capability the agent has — when it's actually infrastructure that requires the same care as a production database.

We had:

  • No migration tooling for the memory store
  • No policy on what constitutes a "breaking change" to memory provenance
  • No test to catch stale paths before they cause silent recall failures
  • No documentation linking filesystem conventions to memory schema expectations

Every engineering team knows you don't rename a table without a migration. We had renamed the equivalent of the "schema" (the path prefix all files share) and done nothing.

What "Knowledge Ops" Actually Means

We're building five products simultaneously, each generating knowledge that flows back into the agent's memory. Session decisions. Code patterns. User signals. Market observations. The agent's recall quality depends on this knowledge being fresh, correctly attributed, and semantically indexed.

That requires operational discipline that most teams aren't thinking about yet:

1. Provenance stability contracts

If agents store source_path, then paths are an implicit contract. Moving files, renaming directories, or restructuring repos breaks that contract silently. You need either stable paths (enforced by convention) or migration scripts triggered automatically when paths change.

We now run orphan-audit.sh on every agent tick — a read-only scan that flags chunks whose source paths no longer exist on disk. Silent staleness becomes visible staleness.

2. Schema ownership

The memory database has a schema. Columns, indexes, tables. We treat it like any other database: no ALTER TABLE without a migration file, no ad-hoc updates in production. This seems obvious in hindsight. It wasn't when we built the first version.

3. Knowledge decay policy

Not everything an agent knows should persist forever. We set TTLs on ephemeral session state. We mark external facts (pricing, API docs, competitor status) with source URLs and staleness windows. The agent knows the difference between "decision we made that's still binding" vs. "fact we observed that may have changed."

4. Indexing as a deployment step

When we ship new documentation, agent playbooks, or decision records — that's a deployment. It requires re-indexing the affected content. We added this to the same CI step that runs tests. If the content deploys but the index doesn't update, the agent is flying blind.

The Compound Effect

Here's the thing that makes all of this worth the overhead: knowledge compounds.

When the agent recalls a decision from three months ago — "we chose Supabase over PlanetScale because of the RLS model for our multi-tenant design" — it doesn't just retrieve a fact. It skips the reasoning chain that led to that decision. It skips the alternatives considered. It skips the two failed experiments that preceded it.

A well-indexed, correctly attributed knowledge base turns a 15-minute "let me think this through" into a sub-second recall. In our system this typically runs under a second end-to-end. Across every session. Across every agent running on the system.

The economics compound too. The retrieval path — embedding the query, running cosine similarity, returning ranked results — involves one fast, cheap embedding call and then pure CPU. No large LLM inference. No per-retrieval API cost. The expensive part — chunking, generating embeddings, writing to the index — happens once, at index time, when content is created. That cost is amortized across every future query that touches it.

But all of this breaks if the index is stale. If paths are wrong. If chunks are orphaned. If the provenance chain is broken.

What We Now Do Differently

Every directory restructure ships with a UPDATE chunks SET source_path = REPLACE(...) migration, staged and verified before the filesystem change lands.

Every agent tick runs the orphan audit. Stale chunks surface as a visible signal, not a silent failure.

Every external fact stored in memory includes a fetched_at timestamp and a check_by date. After that date, the agent flags the fact as unverified before using it.

New knowledge categories get schema review before they go into the database. Not because the schema is hard to change — SQLite migrations are fast — but because schema stability is part of the provenance contract.

None of this is complicated. All of it is invisible until you skip it.

The Takeaway

AI agents are being built like features and deployed like infrastructure. The gap between those two things is where silent failures live.

Memory is the most underrated part of an agentic system. Not the model. Not the prompt. The accumulated, indexed, provenance-tracked knowledge that makes the agent's next response better than its last.

If you're building with agents, treat their memory like a production database. Write migrations when the schema changes. Track provenance. Audit for staleness. Build the ops practice now, before you have 55,000 chunks that all point to the wrong path.

Three seconds to fix. Weeks to understand why it mattered.

Golden Corpus builds AI-native products across Japan Halal Guide, MyWritingTwin, FluxDiagram, and more. We write about what we're actually building.