meta: unified metadata layer (json + sqlite engines) replacing storage.Store[T]

[CMGS]

Canonical design: cocoon-specs/design/meta-store.md — frozen at v2.28. This body is a summary; where they differ, the doc wins. The original inline design (v1) and the addendum comments below are superseded by the doc.

Motivation (measured)

Clone storms are dominated by the metadata layer, not virtualization: at B=64 with resident N≈190, the segments containing the two VM-index writes carry p50 2347ms + 867ms of a 4069ms clone while the FC snapshot load is 14ms. Every storage.Store[T] update rewrites the whole index plus a .prev copy under an exclusive flock (~1.8KB/record, 339KB×3 IO at N=192) ⇒ storm cost O(N·B). Removing a single RMW (#132/#145) measured inside host noise — the O(N) payload per write is the problem, and Store[T]'s whole-value semantics pin that cost model. Full data: cocoonstack/sandbox#30.

Decision summary

• One record-granularity API, several engines (meta package): Scope-declared transactions with a single write namespace, per-transaction CommitMode (durable default, relaxed by explicit per-op opt-in), pure retryable closures, detached values, engine-neutral errors, Collection/Log primitives with domain repositories on top.
• json is the first-class default engine — today's files, formats, .prev recovery and flocks move inside it; existing deployments upgrade untouched, no conversion. sqlite is the opt-in scale engine (one DB, namespace = table group, WAL, modernc.org/sqlite).
• One meta.Store constructed at the command layer and injected into every backend — replacing five per-subsystem private store constructions.
• GC and every metadata-backed destructive flow move to a revalidating tombstone protocol (lease fencing, phase-directed recovery, stable VMID-keyed entity locks outside cleanup sets), replacing lock-all orchestration. Bridge lane stays out (no persistent records).
• Engine conversion is an explicit offline ops action (meta init / meta convert) with an engine-neutral manifest; config remains the sole backend authority.
• Statistical acceptance gates per engine: sqlite gets slope + absolute ceilings + storm anchors; json gets paired-ratio non-regression (its O(records) cost is contractual).

Phasing

• P0 — boundary only, json engine, semantics unchanged (byte-identical fixtures + differential trace); storage.Store[T] retires.
• P1 — protocol changes, still json-only: tombstone protocol, stable VM lock, vm rm restructure, retirement of gc.Module.Locker / Watchable.WatchPath / IndexFile/IndexLock.
• P2 — the sqlite engine and everything that exists only for it; the measured win lands here.
• P3 — optional: metering Log, networked engines, widening the relaxed set with data.

Status

Design frozen at v2.28 (v2.26-v2.28 were implementation-round amendments recorded in the doc header; v2.28 records the P2/P3 as-built decisions: generic (id, data) physical schema superseding the domain-typed sketch, flag-less meta commands with meta_backend as sole authority, referential name check, composition-root manifest guard for the json engine, and the SPI-layered Log/Seq with its opt-in metalog consumer). The doc moved to cocoon-specs — cocoon no longer carries design documents.

• P0 — complete (PR meta: unified metadata layer P0+P1 — record-granularity store, json engine, tombstone protocol #147): meta API + json engine + contract suite; all four subsystems on one command-layer meta.Store; storage.Store[T] retired; byte-fidelity goldens + differential value/error traces for every namespace.
• P1 — complete (PR meta: unified metadata layer P0+P1 — record-granularity store, json engine, tombstone protocol #147): tombstone phase protocol across vm rm / snapshot delete / CNI teardown / GC; stable VMID-keyed locks; entrypoint discipline; per-digest blob locks on every re-pin flow; Store.Events; protocol gates (fencing/ABA, subset recovery, shared-lease escalation, 256-process storm, durable-ack ordering) all green.
• P2 — complete (PR meta: sqlite engine, offline conversion, and the metering log (P2+P3) #148): sqlite engine (WAL, durable/relaxed writer pair, ctx-bounded busy retry surfacing ErrBusy, identity/meta_state fail-closed verification, prepared-statement cache, background passive checkpoint); offline conversion with the §6 manifest protocol (both directions, crash-rerunnable, quiesce probes, aside retirement); cocoon meta init|convert|backup; contract suite passes both engines unmodified. Measured on the 16-core NVMe anchor host: durable update 366µs at N=10k vs json 31.9ms (87x), get 8.8µs vs 24.5ms, concurrent creates 909-1,693/s vs json 94-129/s at resident 1,000 — the §9 absolute gates hold with 13-57x headroom.
• P3 — Log/Seq re-added (PR meta: sqlite engine, offline conversion, and the metering log (P2+P3) #148): SPI-layered primitive (identical semantics on both engines, zero engine code) with the metering.backend: meta consumer, Relaxed durability matching the file backend. Networked engines and relaxed-set widening stay data-gated per the doc.

[CMGS]

Addendum from the interface-leak audit (two more places the Store[T] lock model escaped, both to be absorbed here, plus one follow-up):

1. gc.Module[S].Locker + lock-all orchestration: gc.Orchestrator.Run takes every module's store lock up front (fail-closed on busy) purely to get a cross-module-consistent snapshot for Resolve (VMs pinning images etc.). With one DB, a single read transaction (View) over all namespaces replaces the lock-all dance: Module drops its Locker field for DB access; ops-flock TryLocks remain only for the directory-sweep side. This also removes the whole-index WriteRaw clobber semantics in collectors — mutations become targeted deletes in one Tx.
2. BackendConfig.IndexFile()/IndexLock(): storage mechanism leaked into the config interface; both retire with the json store (replaced by a single meta-DB path derived from root).
3. Follow-up enabled by this design (separate change): hypervisor.Reserver's prereserve/rollback choreography exists to close the GC-ownership window across three index writes; with row-level transactions, reserve + name claim + finalize can collapse into transactions inside Create/Clone, shrinking or removing the Reserver surface the CLI currently drives.

[CMGS]

Design v2 committed as docs/design/meta-store.md on branch design/meta-store (commit cb6cdce) — supersedes the body above. It incorporates the full external review: per-transaction CommitMode (durable default; Relaxed = placeholder insert only in phase 1), TOCTOU-safe revalidating tombstone GC with an explicit hybrid lock-all phase until all cross-referencing namespaces migrate, a two-release migration state machine (marker + migrations table + export-based rollback; 'every step revertible' withdrawn and re-scoped), Insert/Replace primitives + domain repositories instead of loose upsert, per-namespace schema exceptions (image digest/refs split, orphan-dirs table, networks vm_id index, tombstones), the modernc runtime contract (per-connection pragmas, _txlock=immediate, error taxonomy, no-network-fs, backup procedure), Watchable.WatchPath retirement via Store.Events (data_version-confirmed), an engine-portability contract (retryable pure closures, neutral errors, subscription events — a future non-file engine implements behind the same boundary), and statistical acceptance gates replacing the noise-contradicting ±10% wall gate. Review iteration continues on the design doc.

[CMGS]

Design frozen at v2.25 (225edb8, branch design/meta-store, docs/design/meta-store.md) — verdict: GO for P0.

Review record: two independent lines — a resumable Codex thread driven from this repo (12 rounds) and the maintainer's own Codex (6 rounds) — ~80 findings total, every one verified against the document and the real source before acting; over-defense findings were rejected or trimmed per the maintainer's calibration (one runtime defense removed: per-acquire lock-inode revalidation, which only guarded operator action outside the deployment contract). Strongest cross-validation: the two lines independently converged on the conversion manifest, the removal of the hybrid migration phase, vm_id-keyed network tombstones, the CNI cleanup payload shape, and the entrypoint tombstone rule.

Final structure: namespace-scoped writes with declared Scope; per-transaction durability (Durable default, RelaxedOK opt-in); json as first-class default engine (P0 moves today's implementation behind the boundary, byte-fixtures + differential trace as the oracle); sqlite as the opt-in scale engine (P2); revalidating tombstone GC with lease fencing and a stable VMID-keyed lock; offline operator-driven conversion with config as sole backend authority and a crash-safe manifest; per-phase/per-engine acceptance gates with numeric pass rules.

P0 scope (boundary only, semantics unchanged, json engine only): meta package + contract suite + legacy adapter allowlist (LockVMOps/netresize RawView; legacy GC + all four image publish sites), every subsystem onto Collections/repositories, storage.Store[T] retired. Implementation pointers recorded in the doc: single Store injection must cover cmd/images/import.go's four direct construction sites; the forced-retry suite must exercise UpdateStates/BatchMarkStarted whose closure-external metering append is the exact pattern clause 1 forbids.