[Roadmap] Brainiac product evaluation — fitness as company-wide memory, and the plan

[CheeryProgrammer]

Comprehensive critical evaluation of Brainiac's fitness as an open-source, self-hosted company-wide memory / knowledge base of everything — runs on weak hardware, scales to strong, dead-simple to self-host. Produced by a panel of specialist reviews across architecture, retrieval, data/scale, ingestion, deploy/DevEx, observability, security, and product/docs.

Verdict

Brainiac is an unusually well-engineered artifact — disciplined core/adapter split, a crisp two-layer model (vector chunks + provenance graph in one Postgres), a genuinely one-command deploy, and rationale-rich docs. It is a strong single-user / single-team memory today. It is not yet fit for the stated goal — company-wide, multi-team, everything-memory — for four structural reasons:

1. Retrieval is under-built for a real KB. Single-model dense-only search, and nomic-embed-text is used without its required search_query:/search_document: prefixes — a silent, corpus-wide recall loss that also invalidates every tuned threshold. No BM25/keyword fallback, no reranker, no fusion. ([Epic] Retrieval quality — make it a real knowledge-base engine #202)
2. The knowledge graph does not build at scale. With the default chat-driven extractor, bulk/auto-import stores chunks with zero nodes/edges; the differentiating layer only fills via manual curation. Coverage is Notion+Markdown only; Watch()/incremental-sync is dead code; no PDF/docx/html. ([Epic] Ingestion & connector breadth — memory of *everything* #205)
3. Effort is inverted. Heavy investment went into Layer-2 multi-tenant isolation (namespaces, quotas, export/import, temporal, rollups) while the retrieval core, connector breadth, and observability — the things the goal actually needs — lagged. And the isolation itself has real holes (below).
4. It's demo-grade on the ops/security/scale axes a company relies on: /metrics exposes almost nothing alertable, backups are off by default and restore is untested, there's no index behind the isolation wall, and by-id mutations bypass the wall entirely (cross-namespace takeover).

Nothing here is a rewrite — the model is lean and the seams are right. It's a matter of re-prioritizing toward the goal.

Cross-cutting themes (what the panel converged on)

• Retrieval quality first. The nomic-prefix bug (Add nomic search_query:/search_document: prefixes + recalibrate thresholds #210) + hybrid FTS/RRF (Hybrid search: Postgres FTS (tsvector) on chunk text + RRF fusion #211) are the highest-leverage changes in the whole roadmap.
• Correctness under concurrency. No unique node-identity constraint → duplicate nodes fragment the graph (Node identity unique constraint (partial unique WHERE status='current') #220); quota check in Remember is non-atomic (Transaction-wrap Remember (dedup + quota + insert) #222).
• Index the wall. discriminators->>'project' is unindexed → the isolation wall, quotas, export, and delete all seq-scan (Generated 'project' column + btree index on nodes & chunks #226); summary_embedding has no HNSW (HNSW index on nodes.summary_embedding #227).
• The graph must build itself. Stronger auto-extraction + entity resolution (Stronger default extraction: batch-to-Claude + cross-doc entity resolution #235) or the "company graph" stays empty.
• Security trust-model. By-id mutation wall gap (Enforce the isolation wall on ALL by-id mutations #265), unauthenticated MCP write path (Authenticated MCP write path (per-connection identity) #266), no audit/identity (Audit log + per-human identity #267).
• Make it operable & honest. Metrics to Prometheus (Export health/system metrics to Prometheus /metrics #255), backups on by default (Backups on by default + CI dump->restore round-trip + off-box hook #257), publish images instead of compiling on the box (Publish multi-arch release images to GHCR + release pipeline #248), fix the stale "do not deploy" doc (Reconcile/retire the stale production-readiness.md + SYSTEM status #274).

Recommended sequencing

• Wave 1 (correctness + quality floor): Add nomic search_query:/search_document: prefixes + recalibrate thresholds #210, Hybrid search: Postgres FTS (tsvector) on chunk text + RRF fusion #211, Node identity unique constraint (partial unique WHERE status='current') #220, De-duplicate recall query embedding + batch endpoint-name resolution #221, Generated 'project' column + btree index on nodes & chunks #226, HNSW index on nodes.summary_embedding #227, Index chunks(source_uri) and edges(source_uri) #228, Enforce the isolation wall on ALL by-id mutations #265, Reconcile/retire the stale production-readiness.md + SYSTEM status #274, Document the real MCP surface (12 tools) + add get_node/rollup to the capture instruction #275. Small/medium, high-leverage, mostly independent.
• Wave 2 (make it real infra & fill the graph): Wire hnsw.ef_search + iterative_scan for filtered/scoped reads #212, Real golden set + nDCG/MRR + node-recall + CI regression gate #216, Stronger default extraction: batch-to-Claude + cross-doc entity resolution #235, Wire Watch() + persisted incremental-sync state (cursors/mtimes) #236, Publish multi-arch release images to GHCR + release pipeline #248, Right-size compose memory to a real 4 GB (or restate the floor) #249, Export health/system metrics to Prometheus /metrics #255, Threshold + alert the vector-index-vs-RAM scaling signal #256, Backups on by default + CI dump->restore round-trip + off-box hook #257, Authenticated MCP write path (per-connection identity) #266, Audit log + per-human identity #267.
• Wave 3 (breadth + scale-out): connectors (Slack connector #237-240), reranker (Optional local cross-encoder reranker over merged top-k #213), partitioning (Partitioning + current-only partial indexes + superseded_at index #230), audit/rate-limit/TLS (Scope or remove /api/logs and gate /metrics under isolation #268-271), WebUI product surface (WebUI: capture + browse + node-detail (make it a product surface) #277).
• Wave 4 (polish/P2): the remaining P2s.

Epics

• [Epic] Retrieval quality — make it a real knowledge-base engine #202 — Retrieval quality (make it a real KB engine) area:retrieval
• [Epic] Core correctness & hardening #203 — Core correctness & hardening area:core
• [Epic] Scale & data model — index and partition for a large KB #204 — Scale & data model area:scale
• [Epic] Ingestion & connector breadth — memory of *everything* #205 — Ingestion & connector breadth area:ingestion
• [Epic] Deployment & DevEx — keep the one-command, weak-hardware promise honest #206 — Deployment & DevEx area:deploy
• [Epic] Observability & operability — production-grade, not demo-grade #207 — Observability & operability area:observability
• [Epic] Security, identity & compliance — fit for company-wide, multi-team use #208 — Security, identity & compliance area:security
• [Epic] Product & docs — from 'excellent engineering artifact' to 'a team adopts it' #209 — Product & docs (adoption) area:product-docs

What is superfluous / to cut or defer

• plugins.Registry[T] — dead code (Remove or wire up plugins.Registry[T] #224).
• The in-memory log ring as an ops store — duplicates docker logs, loses crash logs (Structured JSON logs to stdout + request-id correlation #258).
• Layer-2 hard isolation is heavy for the overwhelmingly single-team target — keep it opt-in and off by default; do not expand it before the retrieval/ingestion/ops gaps close.
• Coarse lifetime-cumulative latency quantiles; the _bucket series is the useful part.
• Tier split/disambiguate/rollup/as_of as advanced — they inflate the tool surface newcomers must hold (Tier the tool/verb surface (advanced) + a one-shot 'capture' macro #281).

Full per-domain analysis is captured across the epic issues above; each child issue carries the concrete file:line findings and fix direction.