Back to blog

Memory Vault Has a Knowledge Graph

Most "AI knowledge graph" projects use an LLM for entity extraction. You feed in a chunk of text, ask the model "what entities are in here, what are the relationships," and pay per token forever. It works. It also costs money on every ingest, and your graph quality is now coupled to whichever model you happened to pick.

Milestone 7 shipped: Memory Vault now extracts entities and relationships from every ingested chunk and renders them as an interactive force-directed graph. No LLM. No API calls. spaCy's small English NER model plus three Postgres tables and some co-occurrence math gets you 80% of the way for 0% of the cost.

What's in it

A new Graph page on the dashboard, sitting between Browse and Ingest. Cytoscape.js force-directed layout with built-in pan and zoom, color-coded by entity type, click any node to see its mentions and related entities in a side panel.

Four entity types extracted automatically:

Plus one relationship type, related_to, generated from co-occurrence: if two entities show up in the same chunk, that's an edge. Edge weight grows with co-occurrence count across chunks, so a pair that appears together in five chunks gets a thicker line than a pair that appears once.

Storage is three new tables (migration 003): entities, entity_mentions, relationships. Per-space deduplication via a unique index on (lower(name), type, space_id) — same name + same type + same space = one entity, regardless of how many chunks it appears in. Cross-space dedup intentionally not done, because a Project called "Atlas" in your work space is probably not the same Atlas in your personal space.

Four new API endpoints under /api/graph/ — entities (list, paginated, filterable), entities/{id} (detail with mentions and related), relationships (list, filterable), visualize (nodes + edges optimized for the frontend, capped at 500 nodes by default to prevent browser freeze on large graphs).

The whole extraction pipeline runs synchronously on ingest, on the same CPU that runs the embeddings. No extra services, no GPU, no external dependencies beyond a 15 MB spaCy model bundled into the Docker image at build time.

The prologue

Before I get to M7 itself: a public credit. After I shipped M6, @rivestack dropped a tip on X about pgvector's maintenance_work_mem default. Stock PostgreSQL ships with 64 MB. For HNSW index builds on pgvector, that's painfully low — turns a two-minute index rebuild into forty minutes once your corpus grows past a few thousand chunks. He suggested 1 GB, then "ship it on by default" once we discussed it.

Bumping maintenance_work_mem to 1 GB in the bundled docker-compose.yml was the first thing I did in M7. It's a one-line change. It's also the kind of database default that passes demo and silently degrades at real-user scale — which is the entire framing Rivestack uses for his own product. The README has a Performance Tuning section now. Two more tips of the same shape (treating ef_search as a runtime knob, post-deploy cache warmup) didn't make v1.0.

Three tips from one builder, all of the same shape: postgres defaults that look fine in dev and quietly break in production. That's exactly the operational knowledge that normally takes years of running things in production to accumulate, and it's the kind of thing build-in-public makes possible — you ship publicly, someone with deeper expertise sees it, the project gets better.

The detour

The locked M7 plan called for a "static SVG force-directed layout (d3-force)." I built it. It worked. It looked amateur.

Custom d3-force gets you a working force simulation in about 130 lines of code. What it doesn't get you is good defaults. The layout was cramped, the labels overlapped, hub nodes didn't visibly dominate the way they should, and there was no zoom or pan — which means at 50+ nodes you couldn't actually read the graph, and at 500 nodes the SVG started stuttering.

I'd locked d3-force for V1.0 and noted Cytoscape as a V1.1 migration. The reasoning was: locked scope, ship faster, migrate later. The reasoning was wrong. "Ship at any quality bar" is not what locked scope means. The rest of Memory Vault — Postgres schema, async pipeline, MCP server, REST API, 114 tests — is held to a real quality bar. Shipping a graph page that looked like a student project on top of that infrastructure would be the one weak visual on a launch where screenshots matter.

So I pulled the Cytoscape migration forward into M7. Removed d3-force, added cytoscape + cytoscape-cose-bilkent for the layout extension, rewrote the canvas component. The API contract didn't change — same JSON, same shape — only the rendering layer swapped. Bundle went from 97 KB gzipped to 250 KB. Acceptable for a local-first tool where the dashboard is served from the same container as the data.

What you get for the extra 150 KB: pan, zoom, smooth canvas rendering of 500+ nodes without stutter, a mature layout algorithm (cose-bilkent) that handles disconnected components without scattering them across the canvas, label backgrounds for readability against the dark theme, and selectors-based styling that's an order of magnitude easier to extend than custom SVG. None of those were "features" the plan called for. All of them are what "professional knowledge graph viewer" actually means once you sit with the result for ten minutes.

The honest limitations

spaCy's en_core_web_sm model is English-only. Non-English content gets little to no useful extraction. Multilingual models exist; they're heavier and slower; they're not in v1.0.

spaCy's NER is also context-dependent. The same name can land as different entity types depending on the surrounding sentence. "Mihai Builds said X" tags as PERSON; "Mihai Builds is a brand of developer tools" tags as ORG. Memory Vault's per-space dedup keeps types separate by design — necessary so Apple-the-fruit doesn't merge with Apple-the-company — which means in a single space you can occasionally see the same name as both a Person and a Project node. Manual entity merging in the dashboard isn't in v1.0. LLM-based extraction would catch this kind of thing, but at a cost the current open-core tier doesn't pay.

There's no fuzzy matching. "PostgreSQL" and "Postgres" are separate entities. No alias merging in v1.0.

There's no re-extraction on edit. If you ingest a chunk, then forget it and ingest a corrected version, the new entities are added but the old ones aren't cleaned up automatically.

These are deliberate trade-offs. The whole bet of M7 is that spaCy + co-occurrence + Postgres gets you to "useful graph" without an LLM bill. The honest gaps are documented up front rather than masked.

Technical decisions

spaCy in the Docker image at build time, not first-run. The model is downloaded as a separate RUN layer in the Dockerfile. Build is slower, runtime is instant. docker compose up doesn't wait on a 15 MB download every time you start the container.

Sync extraction wrapped in asyncio.to_thread at the call site, not inside the extractor. spaCy is CPU-bound and synchronous. The extractor function signature reflects that — it's def, not async def. The ingestion pipeline calls it via asyncio.to_thread so the event loop doesn't block. This keeps the extractor honest about what it is, makes tests trivial (no async fixtures for pure CPU work), and pushes the concurrency decision to the caller.

Graph writes in their own transaction, separate from the chunk insert. The chunk lands first. Then entities, mentions, and relationships are inserted in a single second transaction. If extraction fails (spaCy crashes on weird Unicode, a model file is missing, anything), the chunk stays — graph data is best-effort. Worst case for that chunk: it's stored and searchable, just not represented in the graph yet.

Cytoscape with the cose-bilkent layout extension. Cytoscape's stock cose layout is fine for small graphs and gets weird at scale. cose-bilkent handles disconnected components much better — gravity pulls orphan clusters toward the center instead of letting each one settle in its own corner. Tuning landed on idealEdgeLength=50, nodeRepulsion=3500, gravity=0.9, gravityRangeCompound=3.0, randomize=false, plus a cy.fit() on layoutstop so the viewport zooms to actual content extent.

Node size scales aggressively, edge width scales gently. Mention count drives node radius (linear, 14 px base + 4 px per mention, capped at 80 px) so hubs visibly dominate. Relationship weight drives edge thickness (logarithmic, capped at 6 px) so high-traffic pairs are visible without overwhelming the graph.

Labels everywhere, with a dark plate background. The first version hid labels on low-mention nodes "to reduce noise." The result was a graph of unnamed dots. Cytoscape's min-zoomed-font-size already auto-hides labels when you zoom out far enough that they'd be unreadable, so explicit hiding was solving a non-problem at the cost of making the graph less informative.

What's actually running

Pull the repo, docker compose up -d, open http://localhost:8000:

114 tests passing — 78 from before plus 36 new ones for M7. 13 unit tests for the extractor against the real en_core_web_sm model (mocking spaCy would test the mock, not the code). 4 schema regression tests for migration 003. 19 integration tests for the four /api/graph/* endpoints, including the extraction-on-ingest happy path and the graceful-degrade path where spaCy fails and the chunk still lands.

What's next

Milestone 8 is v1.0 — local LLM chat via Ollama or LM Studio (RAG over your own memories, no API key, no cloud), final polish, security review, GitHub Actions CI, and the launch post. After M8 ships, Memory Vault is feature-complete for the open-core tier.

The repo: github.com/MihaiBuilds/memory-vault