From 05829157de32a0cc5167b686d823598f3a79adb9 Mon Sep 17 00:00:00 2001 From: Branimir Rakic Date: Sun, 24 May 2026 02:22:42 +0200 Subject: [PATCH 01/59] =?UTF-8?q?feat(core,publisher,agent,node-ui):=20OT-?= =?UTF-8?q?RFC-38=20LU-5=20=E2=80=94=20edge-curator=20publish=20to=20VM?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the minimum unblocker for OT-RFC-38 §1.1: an edge agent that has no on-chain Profile can now create a curated CG and publish it to Verified Memory in no-attribution mode, with cores attesting to opaque ciphertext they cannot decrypt. Three concerns combined here because they share a single API surface and their semantics only make sense together: 1. Inline encrypted publish payload (the rescoped minimal LU-5 from the build plan). Adds `isEncryptedPayload` to PublishIntent so cores branch on opaque ciphertext, AES-256-GCM chain-key AEAD helper in dkg-core (`v10-publish-payload.ts`), publisher-side `encryptInlinePayload` hook, agent-side `_resolveEncryptInlinePayload` wired into both `publish` and `publishFromSharedMemory`. Byte-size accounting uses ciphertext length (`effectiveByteSize`) when the payload is encrypted so ACK signing and on-chain pricing agree. 2. Drop the "identity=0 → skip on-chain" gate in DKGPublisher. Edge agents that won't (and shouldn't) register a Profile must still publish in attribution-id=0 mode, which `KnowledgeAssetsV10` already accepts as no-attribution. The early `willAttemptOnChainPublish` check and the late identity gate both collapse to "do we have a CG id + a V10-ready adapter + a signer". Pre-fix, even after LU-2 dropped the register-side gate, an edge agent's publish silently fell back to tentative and the UI optimistically reported success; now it goes on-chain or hard-fails with a clear error. 3. UI honesty about VM publish status. The SWM→VM panel and per-entity publish card now require status=confirmed AND a real txHash to render success; anything else is a red "NOT published to Verified Memory" card with the actual status and a diagnostic. The full TX hash is shown (was truncated to ~16 chars before — too short to paste into a block explorer). Vite proxy honours `DEVNET_NODE` so the UI can target an edge node (`UI_NODE_ID=5 ./scripts/devnet.sh ui start`) — required to test the §1.1 flow at all. Tests: - `v10-publish-payload.test.ts` (new, 8): round-trip, wire layout, domain separation, error handling. - `storage-ack-handler.test.ts` (+5): encrypted-payload branch — byte-size verification, missing-claim error, empty stagingQuads error. - `publisher-no-random-wallet.test.ts` + `phase-sequences.test.ts`: two tests updated whose premise was the old short-circuit; new behaviour pinned with explanatory comments. Total publisher: 965/965 pass. Devnet validation (`scripts/devnet-test-rfc38-lu5.sh` and `-lu5-public.sh`): - Curated CG from edge node 5 (identity=0, no Profile) → KC #3 confirmed on-chain (TX 0xa137cc7d…) via attributionId=0 publish, 1 core ACK collected, KCS read-back: merkleRoots=1, byteSize=512 [ciphertext]. - Public CG regression on the same node → KC #4 confirmed on-chain (TX 0xa24f3bc1…), no chain-key AEAD wrap fires (correctly skipped for public CGs), byteSize=131 [plaintext]. Specs: - `SPEC_CG_HOSTING_MEMBERSHIP.md` (new) — OT-RFC-38 full text. - `SPEC_CG_MEMORY_MODEL.md` — cross-reference to the new doc. Co-authored-by: Cursor --- docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md | 1193 +++++++++++++++++ docs/specs/SPEC_CG_MEMORY_MODEL.md | 1 + packages/agent/src/dkg-agent.ts | 125 +- packages/cli/src/publisher-runner.ts | 4 +- packages/core/src/crypto/index.ts | 9 + .../core/src/crypto/v10-publish-payload.ts | 128 ++ packages/core/src/proto/publish-intent.ts | 26 +- .../core/test/v10-publish-payload.test.ts | 120 ++ packages/node-ui/src/ui/api.ts | 10 +- .../components/Modals/CreateProjectModal.tsx | 14 + .../node-ui/src/ui/views/MemoryLayerView.tsx | 56 +- .../src/ui/views/project/components.tsx | 63 +- packages/node-ui/vite.config.ts | 19 +- packages/publisher/src/ack-collector.ts | 9 + packages/publisher/src/dkg-publisher.ts | 98 +- packages/publisher/src/publisher.ts | 30 + packages/publisher/src/storage-ack-handler.ts | 141 ++ .../publisher/test/phase-sequences.test.ts | 15 +- .../test/publisher-no-random-wallet.test.ts | 24 +- .../test/storage-ack-handler.test.ts | 155 +++ scripts/devnet-test-rfc38-lu5-public.sh | 132 ++ scripts/devnet-test-rfc38-lu5.sh | 299 +++++ scripts/devnet.sh | 13 +- 23 files changed, 2577 insertions(+), 107 deletions(-) create mode 100644 docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md create mode 100644 packages/core/src/crypto/v10-publish-payload.ts create mode 100644 packages/core/test/v10-publish-payload.test.ts create mode 100755 scripts/devnet-test-rfc38-lu5-public.sh create mode 100755 scripts/devnet-test-rfc38-lu5.sh diff --git a/docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md b/docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md new file mode 100644 index 000000000..193de5918 --- /dev/null +++ b/docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md @@ -0,0 +1,1193 @@ +# Decoupled Hosting and Membership for Curated Context Graphs + +**Status**: PROPOSED (v3 — addressing PR #113 review) +**Date**: 2026-05-23 +**Scope**: Resolve the structural conflict between curated-CG privacy and verifiability / availability by separating *who hosts the bytes* from *who can read the bytes*. Enables edge-curators to publish curated CGs to VM, supports member-attested verification for granted outsiders, and gives edges a high-availability sync substrate. Adds a per-assertion-key monetization access model. +**Related**: [SPEC_CG_MEMORY_MODEL.md](./SPEC_CG_MEMORY_MODEL.md), [SPEC_V10_IDENTITY_AND_ACCESS.md](./SPEC_V10_IDENTITY_AND_ACCESS.md), [SPEC_SYNC_CHAIN_VERIFICATION.md](./SPEC_SYNC_CHAIN_VERIFICATION.md), [SPEC_MOBILE_NODE.md](./SPEC_MOBILE_NODE.md). + +--- + +## 0. TL;DR + +In today's design, **hosting the bytes** and **being able to read them** are the same concern. That collapses two surfaces that want to be independent: + +- A curator wants strong privacy: "no random core sees my plaintext." +- The same curator wants strong availability: "anyone the network later admits — by membership grant or monetized access — must be able to fetch the bytes and reconstruct the CG against the on-chain commitment, even if my laptop is offline." + +This RFC introduces a clean separation: + +- **Hosting** (substrate role) — cores hold ciphertext for CGs the sharding table assigns them, regardless of allowlist. They attest "I'm storing the bytes" without ever seeing plaintext. **They do not attest to plaintext content.** +- **Membership** (semantic role) — members hold the per-CG / per-epoch chain keys. They verify plaintext content post-decrypt against the existing on-chain merkle root. + +**Crypto invariant**: the on-chain merkle root stays exactly as today — over plaintext leaves computed by the publisher. The only thing this RFC changes about the on-chain commitment is the ACK shape (cores now ACK ciphertext-availability for the batch, not plaintext-presence). Ciphertext and plaintext do not need to be cryptographically bound in a single leaf; members verify the binding post-decrypt by re-deriving the plaintext root and comparing to chain. + +This unlocks four things at once: + +1. **Edge can publish curated CGs to VM** — cores ACK over ciphertext availability + the existing V10 batch-digest fields, no plaintext required. +2. **Member-attested verification for granted outsiders** — keys flow from curator to new member or paying buyer; ciphertext flows from any hosting core; members verify plaintext binding post-decrypt and can sign attestations for third parties. +3. **Edge sync from cores** — laptop reopens, asks any hosting core for missed encrypted SWM via a normative `SWMCatchupRequest`, decrypts locally. +4. **Curator privacy** — cores still cannot read curated content even though they store it. + +**Plus a monetization protocol**: a per-assertion payload-key wrap protocol (model β) lets a buyer purchase access to a single assertion without gaining the epoch chain key — enabling Bloomberg-shaped data products natively. + +This v3 dropped a dual-root commitment scheme that v2 proposed but that PR #113 review correctly identified as both over-engineered and broken-as-claimed (cores cannot prove plaintext↔ciphertext correspondence without the key). The verification story instead leans on the existing plaintext root + member post-decrypt checks + attestation tokens for outsiders. + +--- + +## 1. The tension this RFC resolves + +### 1.1 The empirical bug that surfaced it + +End-to-end test of [SPEC_CG_MEMORY_MODEL.md](./SPEC_CG_MEMORY_MODEL.md): edge agent creates an invite-only CG, drafts content in SWM, tries to publish to VM. Result: + +``` +[ACKCollector] Decline from : NO_DATA_IN_SWM — No data found in SWM graph for entities: <…> +[publishFromSWM] V10 ACK collection failed: storage_ack_insufficient (0/1 valid ACKs) +[publishFromSWM] Identity not set (0) — skipping on-chain publish +[publishFromSWM] Stored as tentative: UAL=…/tmp… +``` + +The cascade: the CG is invite-only, so SWM gossip is restricted to the allowlist; cores aren't allowlisted (correctly — agent membership shouldn't be conflated with infrastructure hosting), so they never receive the SWM data; the publish merkle root is computed over **plaintext triple hashes**, so cores have nothing to recompute it against; ACK collection fails; the publish stays local as a tentative record. The data is never actually on VM. + +### 1.2 The architectural tension underneath + +The bug is a surface symptom of a deeper conflation: + +| Concern | Today's design | +|---|---| +| Who holds the bytes? | Whoever's in the allowlist. | +| Who can read the bytes? | Same set. | +| Who can ACK a VM publish? | Anyone who already has the plaintext locally (so they can recompute the merkle root). | + +Three concerns collapsed into one set. That set is *agent membership* (the allowlist), so curated CGs become inaccessible to the *substrate* (cores, sharding table). Privacy gained; availability and verifiability lost. + +### 1.3 What we want instead + +A curator running on a laptop should be able to say *all four* of these at once: + +1. "Cores I don't know about are storing my data so it survives my laptop closing." +2. "None of those cores can read my data — only the people I've granted access to." +3. "When I later grant someone access (free, paid, or by joining the allowlist), they can fetch the data from any of those cores, decrypt it, and verify it matches the on-chain commitment I made today." +4. "If a third party later sees a single fact of my data (leaked, quoted, monetized), a member can give them a small attestation that proves the fact's inclusion in the on-chain anchor." + +Today (1) and (3) and (4) require defeating (2). This RFC makes all four compatible. (4) is achieved via member-attested verification — the chain alone cannot prove plaintext-vs-ciphertext correspondence because the chain has no key; a key-holder must vouch for the binding. + +--- + +## 2. The conceptual move — hosting ≠ membership + +### 2.1 Two surfaces, not one + +| Surface | Decided by | Holds what | Can do what | +|---|---|---|---| +| **Hosting** (substrate) | Sharding table assignment | Encrypted chunks + commitments | Store, replicate, attest "I have this", serve to anyone who asks | +| **Membership** (semantic) | Curator (via key distribution) | Decryption key for the CG / epoch | Read plaintext, write SWM, prove inclusion to a verifier | + +Cores live entirely in the hosting surface. Members live entirely in the membership surface. The two are orthogonal: a core can host without being a member (the normal case for curated CGs), a member can be a non-host (the normal case for edge agents). + +### 2.2 What "curated" actually means + +Today: "curated CG" implies "no one outside the allowlist can do anything." The RFC's framing: curated controls *who holds keys*, not *who holds bytes*. Restated: + +- Curated = **only members can decrypt**. +- Curated = **anyone the network assigns can host the ciphertext**. + +This matches how every modern E2E-encrypted system works (Signal, age-encrypted git remotes, encrypted IPFS pins). The hosting layer is dumb-bytes-and-availability; the access layer is keys-and-cryptography. + +### 2.3 Mental analogy + +A safe-deposit box in a bank. The bank guarantees the box exists, is preserved against fire and theft, and can be retrieved on demand. The bank doesn't know what's inside the box — only the key-holders do. Granting access = handing someone the key. Verifying provenance = the box has the bank's serial number on it; the contents have a tamper-evident seal cryptographically tied to the serial number. + +Cores are the bank. Members are the key-holders. The on-chain commitment is the serial number plus the seal. + +### 2.4 Target deployment shapes + +This RFC is motivated by — and validated against — four concrete deployment patterns. All four share the same structural problem: **proprietary CG content + flaky member edges + need for reliable substrate availability**. + +| # | Pattern | Edges | Cores | Why curated, not public | +|---|---|---|---|---| +| 1 | **Self-hosted code team**. Coders run multiple agents per laptop (Cursor, Claude Code, Codex); the team optionally runs its own cores. Analogue: self-hosted GitLab. | Laptops (closed lids, lost wifi) | Optional team-run cores OR rented network cores | Proprietary code, repo is private | +| 2 | **Cross-device planning**. Managers' agents help with roadmap planning across laptops + phones; company runs its own cores or rents. | Laptops + phones (mobile, intermittent) | Company-operated cores | Internal strategy / forecasts | +| 3 | **Consortium with external-tool bridges**. Multiple companies share a CG; on a cloud core, a co-located member-agent bridges CG content into Obsidian / Teams / Google Docs / Slack. | Member agents across orgs | Hosting cores with a co-located member-agent (host on the node + decrypt by the agent + push to external tool) | Cross-org confidential project | +| 4 | **Monetized data product** ("DKG Bloomberg"). Researchers curate market data via edges; cores host reliably and serve an x402-monetized API to paying outsiders. | Researcher laptops, collaborator nodes | Researcher-operated cores serving the paid API | Pre-purchase, data is private and monetizable | + +**Two properties common to all four**: + +1. **Edges are flaky by definition.** Laptops sleep, phones lose signal, users close apps. Cores are the always-on tier. RFC-38's substrate-subscription model means any member edge can come back online, sync from any hosting core, and never lose history — independent of whether other member edges are online. +2. **Bring-your-own-cores is the dominant deployment expectation.** Like self-hosted GitLab vs gitlab.com: most users use the network; some operate their own cores; the protocol is identical either way. RFC-38 makes both modes work for curated CGs. + +**New pattern surfaced by scenario 3 — the bridge core.** Membership and hosting are properties of two different things: an **agent** (a wallet-identified persona) is a CG member if its wallet was granted access; a **node** (a daemon deployment registered in the sharding table) is a CG host if the sharding table assigned that CG to it. The two predicates are independent — a node has no "membership," an agent has no "hosting" — but they can be **co-located on the same operator's infrastructure**. The "bridge core" pattern is exactly that co-location: one operator runs (i) a node that the sharding table has assigned to host the CG's encrypted substrate, AND (ii) a member-agent process on the same machine whose wallet holds CG membership. The agent decrypts ciphertext the co-located node already stores locally — no network hop — and publishes the plaintext into an external tool (Obsidian, Slack, Google Docs, your-internal-CRM). No new protocol surface: the agent uses the chain key it was granted via the normal `KeyGrant` flow; the node uses its normal sharding-table assignment; co-location is just an operator's deployment choice that happens to compose efficiently. + +**New pattern surfaced by scenario 4 — paid late-joining.** The monetization walkthrough in §3.2 + the late-joiner flow in §A.4 + the per-assertion `PaidAccessGrant` protocol (§5.7) compose into a Bloomberg-shaped product: outsider pays → curator issues a per-assertion `PaidAccessGrant` → outsider's node fetches ciphertext from any hosting core → decrypts the specific assertion → verifies plaintext binding via member attestation (§5.3.2) → checks merkle path to the on-chain anchor. No single party in the loop can fake — the verifier checks attribution against on-chain identities and content against on-chain anchors that the curator committed at publish time. The per-assertion granularity means buyers don't need to pay for (or even see) the rest of an epoch's content. + +### 2.5 Trust model: encryption at rest on chosen operators + +The trust model RFC-38 establishes is the model enterprises already accept from SaaS: + +> "Data is encrypted at the edge. Bytes at rest sit on operators you chose — yours, your vendor's, or the public network. Operators cannot read the bytes without keys you control. Verifiability anchors on chain." + +This is the SaaS posture (AWS holds your encrypted RDS volumes; Google holds your encrypted Drive content; vendor promises plus encryption-at-rest plus your-keys = the deal). It is also the self-hosting posture (your team's own cores hold your team's encrypted bytes — see scenarios 1 and 4). Whether you self-host or use network operators is a **deployment choice, not a protocol fork**. + +The status quo's pitch — "data never touches third-party infrastructure" — is cleaner-sounding but operationally weaker. It requires every member edge to be online for availability, which the deployment patterns in §2.4 actively contradict. RFC-38 trades the marketing simplicity for an honest model that matches how enterprise software actually works and how end users (especially mobile / cross-device users) actually behave. + +--- + +## 3. Roles under the decoupled model + +### 3.1 Each actor + +| Actor | Holds keys? | Hosts ciphertext? | Can read plaintext? | Can ACK VM publish? | +|---|---|---|---|---| +| **Curator** | Yes (issues KeyGrants) | If also a core, yes; otherwise no | Yes | Indirectly — proposes the publish, gathers ACKs from cores | +| **Member agent** | Yes (granted by curator) | Only if also a core | Yes | No (membership is semantic, ACKing is substrate) | +| **Hosting core** (sharding-table member) | No | Yes — encrypted chunks for assigned CGs | No | Yes — over ciphertext-availability + V10 batch digest fields (§5.4) | +| **Bridge core** (node in the sharding table; operator also runs a member-agent process on the same machine) | The co-located agent does; the node does not | Yes (as a sharding-table host) | Yes — the co-located agent decrypts | Yes — as a sharding-table host | +| **Non-hosting core** | No | No | No | No | +| **Outsider** (no membership, no hosting role) | No | No | No | No | +| **Outsider, post-grant** (KeyGrant or PaidAccessGrant) | Yes (scoped to grant) | No | Yes — after fetching ciphertext from a core via `SWMCatchupRequest` | No | + +The single asymmetry that matters: only nodes with a key can read; only nodes with bytes can serve. + +### 3.2 Per-scenario walkthroughs + +**Edge curator publishes a private fact to VM** + +1. Curator's edge encrypts each assertion in the publish payload using the current chain key's per-message payload key (existing SWM substrate; §5.2). +2. Ciphertext gossips to sharding-table cores as part of normal SWM substrate fanout (Phase A change: cores are subscribed to the substrate for curated CGs too). +3. Curator's edge sends an ACK request to the sharding-table cores for this CG. Payload includes the full V10 batch-digest fields PLUS `ciphertextChunks[]` digests and `ciphertextChunksRoot` (§5.4.1). It does NOT carry the bytes — cores already have them via gossip. +4. Each core verifies it holds the ciphertext chunks, signs the full `ackRequestDigest` (§5.4.2) over the combined V10 + availability fields. +5. Edge collects ACKs until `parametersStorage.minimumRequiredSignatures()` is met, anchors the existing single-root merkle commitment on chain via the existing publish flow. +6. Done. Cores never see plaintext. Members later verify post-decrypt by re-deriving the plaintext root from decrypted SWM (§5.3.1). + +**Outsider receives a leaked assertion and wants to verify it** + +1. Outsider has an assertion `A` plus a claim "this was in CG X, batch B, leaf i" — together with a member-attestation token (§5.3.2) issued by a member of CG X. +2. Outsider recomputes `H(A)`; checks it equals `attestation.plaintextLeafHash`. +3. Outsider fetches the corresponding ciphertext chunk from any hosting core via `SWMCatchupRequest` with the attestation token as authenticator; checks `H(received ciphertext) == attestation.ciphertextChunkDigest`. +4. Outsider verifies the merkle path from `plaintextLeafHash` to the on-chain root of batch B. +5. Outsider verifies `attestation.attesterSignature` and resolves the attester's wallet on chain — confirming they were a member of CG X at the attested epoch. +6. If all checks pass → trust the assertion. The trust chain: outsider → named on-chain-resolvable member → on-chain anchor. + +**Edge laptop reopens after a day offline** + +1. Edge identifies its missing message-index range per CG (last seen `(epochId, messageIndex)`). +2. Edge sends a `SWMCatchupRequest` (§5.6.1) to any hosting core — authenticated via its member wallet signature. +3. Core streams ordered ciphertext chunks for the requested range (paginated; broadcast-layer only, never setup packages). +4. Edge decrypts locally using the chain keys it already holds; re-derives plaintext root for each completed batch and compares to the on-chain anchor (§5.3.1). + +**Curator monetizes access — per-assertion** + +1. Buyer pays via x402 for a specific assertion in CG X (or a specific batch / set of assertions). +2. Curator emits a `PaidAccessGrant` (§5.7.2) to the buyer: envelope containing the assertion's `payloadKey`, `ciphertextChunkDigest`, and a member-attestation token (curator self-attests because they are a member). +3. Buyer fetches the ciphertext chunk from any hosting core (`SWMCatchupRequest`, token-bearer auth — §5.6.4). +4. Buyer decrypts the single assertion using the payload key; verifies the binding via the attestation. +5. Buyer learns ONE assertion. The epoch chain key is NEVER exposed — the buyer cannot decrypt any other assertion in the epoch (HKDF is one-way). To purchase additional assertions, the buyer pays for additional grants. + +--- + +## 4. Scenarios this unlocks + +| Scenario | Possible today? | Possible under this RFC? | +|---|---|---| +| Edge curator → public CG → VM publish | Yes | Yes (no change) | +| Edge curator → invite-only CG → VM publish | **No** (cores can't ACK without plaintext) | **Yes** (cores ACK ciphertext) | +| Edge member resyncs SWM from cores after going offline | Partial (only for public CGs) | Yes for any CG it has the key to | +| Outsider verifies a leaked triple against on-chain anchor | Possible only if they have full ciphertext + key | Yes from a single triple + on-chain leaf + ciphertext chunk | +| Curator monetizes access to historical VM data | No clean path (cores don't host curated data) | Yes — cores host, curator gates keys | +| Curator rotates keys (revoke former member) | Awkward (no formal protocol) | Yes — new epoch, old members keep their old-epoch keys, new members get new-epoch onwards | + +--- + +## 5. Under the hood — what changes + +### 5.1 Sharding-table-based hosting subscription + +Today, cores subscribe to a CG's SWM substrate only if they are explicitly in the allowlist (`enumerate-cg-members.ts` returns `source: 'allowlist'` for curated, and cores are not typically allowlisted). + +Under this RFC, cores subscribe to **any CG the sharding table assigns them**, regardless of allowlist, **and regardless of whether the CG is on-chain-registered yet**. The substrate carries ciphertext; subscription is a hosting commitment, not a read grant. Concretely: + +- `ShardingTableStorage` already maps CGs to responsible cores at publish time (see [SPEC_CG_MEMORY_MODEL.md §4.6](./SPEC_CG_MEMORY_MODEL.md#46-hosting)). Extend this to also drive *SWM-tier* subscription, not only VM-tier. +- For curated CGs, the SWM gossip topic admits sharding-table cores in addition to allowlisted peers. Cores hold ciphertext, never get keys. +- The `enumerate-cg-members.ts` enumerator stays the same for *membership* (who can decrypt) but a new enumerator — `enumerate-cg-hosts.ts` — surfaces hosting peers for substrate fanout (`allowlist ∪ shardingTableForCG`). + +#### 5.1.1 Pre-registration staging + +A core accepts SWM ciphertext for **any** CG ID — registered on chain or not — if and only if the deterministic sharding-table function says it is in the assignment for that CG ID at the current sharding-table epoch. + +**Current network shape**: there is no per-CG sub-assignment today. Per [SPEC_CG_MEMORY_MODEL.md §4.6](./SPEC_CG_MEMORY_MODEL.md#46-hosting), the assignment for any CG is "every member of the sharding table." So the check resolves to `amIInTheShardingTable()` for every CG ID — every core stages every CG. The check is left as a deterministic-function abstraction so that future per-CG sub-sharding can refine it without a protocol change. + +**Why pre-registration staging matters**: + +- **Zero cold-start at first VM publish**. By the time a curator registers on chain, cores already hold the SWM history; the first publish goes through the steady-state ACK path (§5.4.2) rather than the `ChunkPullRequest` fallback (§5.4.3). +- **Availability for drafts and ephemeral CGs**. Members joining mid-draft on flaky connections can resync from cores even before registration; groups that use a CG for a sprint without ever anchoring it to VM still get the substrate's availability guarantees during the staging window. +- **No "registration vs first publish" footgun**. Registration is a non-event for the substrate — cores are already staging the CG's SWM and continue to do so. Promotion to long-term hosting happens at first VM publish (the publish's `tokenAmount × epochs` is what pays for retention), not at registration. + +**What gates staging from becoming a free file host** is a small set of policies on the core, all enforced locally without any chain dependency. All byte-denominated limits are in **ciphertext bytes** (the actual bytes the core stores; for curated CGs there is no plaintext available to measure against — see §5.4.1): + +| Gate | Default | +|---|---| +| Per-chunk TTL, sliding from receipt — expires unless the chunk is committed to a VM batch (which pays retention via tokenAmount × epochs) | 6 hours | +| Per-sender-wallet rate limit on unregistered CGs (ciphertext bytes/min, ciphertext bytes/hour) | 1 MB/min, 50 MB/hour | +| Per-CG-ID ciphertext-byte cap on unregistered staging | 100 MB | +| Per-core aggregate budget (ciphertext bytes) for unregistered staging | Few GB, operator-configurable | +| Sender signature required on every staged message (already true today) | unchanged | + +If a wallet trips its rate limit, the core declines further staging from that wallet for a cooldown window. If a CG-ID hits the byte cap, no more chunks are accepted for that CG until either (a) some staged chunks are committed to a VM batch via publish (those bytes move from the staging budget to the batch's retention-paid budget — see Promotion below), or (b) TTL expiry resets the staged total to zero. **Registration on chain alone does NOT lift the cap** — registration creates the on-chain CG record but pays cores nothing; only a VM publish pays for retention. If the per-core aggregate budget is saturated, the core declines new staging until budget is freed by TTL expiry or by publish-driven promotion. + +**Promotion happens at VM publish time, not at registration time.** The lifecycle has three distinct moments: + +- **Registration on chain** creates the CG record and pins `publishAuthority` (and any PCA delegates). It enables VM publishing for this CG ID. It does NOT extend SWM TTL, does NOT pay cores anything, and does NOT lift the staging quotas. The CG is now publishable, but its SWM-staged ciphertext is still subject to the same TTL and caps as before. +- **VM publish** pays `tokenAmount` for `epochs` of retention via the existing `KnowledgeAssetsV10` flow. The specific ciphertext chunks named in the batch (via `ciphertextChunks[]` in the ACK request, §5.4.1) are promoted from "staged under TTL" to "retained for the batch's `epochs`." They keep the same bytes in the core's local store; they get re-indexed under the batch ID; they exit the staging budget and enter per-batch retention bookkeeping. +- **SWM activity outside any published batch** — past chunks that weren't included in a batch, or new chunks gossiped after the publish — continue under the same staging TTL until they too are committed to a future batch or expire. + +If the sharding-table membership changes (epoch rollover, stake change) and a core is no longer eligible for the CG's assignment, it drops staged-but-not-yet-published data immediately and follows the existing sharding-table-reshuffle policy for already-retained batches. + +Note for operators: a core MAY choose to offer registered CGs a more generous TTL than the unregistered default (e.g. 30 days vs 6h) as a local policy choice, on the reasoning that on-chain registration is a commitment signal that the CG is "real." The protocol allows this; it does not require it. This is one of the calibration questions in §8. + +**Defaults are configurable per operator** — the values above are starting points that the network may want to tune as adoption scales. See §8 for the open question on calibration. + +### 5.2 SWM payload encryption is already a two-layer Sender Keys construction + +The existing SWM encryption substrate is more sophisticated than "broadcast encryption" suggests — it's a Signal-style Sender Keys protocol with two distinct layers ([`packages/core/src/crypto/swm-sender-key.ts`](../../packages/core/src/crypto/swm-sender-key.ts)): + +- **Setup layer** — when an epoch starts (or a new member joins mid-epoch), the curator sends each recipient a small X25519-wrapped **setup package** containing the 32-byte symmetric **chain key** for that epoch, bound to the membership snapshot. One package per recipient, ~few hundred bytes each. Point-to-point, NOT broadcast. +- **Broadcast layer** — all SWM messages in the epoch are AES-256-GCM ciphertexts under a per-message **payload key** derived from the chain key via HKDF, with the chain key ratcheting via HMAC-SHA-256 each message. **One ciphertext per message**, gossiped to all subscribers; every member decrypts the same bytes. + +This RFC changes the **gating** at the substrate layer; it introduces no new crypto primitive: + +- Sharding-table cores subscribe to the **broadcast layer** for any CG they're assigned to. They receive ciphertext only, never the chain key, never the setup packages. +- Allowlisted members continue to receive both layers via the existing flow. +- Non-hosting non-member peers receive nothing (same as today). + +The cost shape this gives the RFC: + +| Cost component | Per | Held by | +|---|---|---| +| Broadcast ciphertext | message | members + sharding-table cores | +| Setup package | recipient per epoch | members only (point-to-point from curator) | +| Chain-key rekey on revocation | epoch boundary | members only (new setup packages, broadcast stream switches) | + +**Cores hold exactly one ciphertext per message regardless of CG size**, which is what makes the "host curated CGs without reading them" pattern affordable. The per-recipient setup cost stays off-core; cores never become a key-distribution channel. The §6 trust model is unchanged. + +**Phase A only changes the substrate subscription gate; the encryption substrate itself is unchanged.** No new crypto primitives, no proto-version bump, no new AAD fields. The existing two-layer Sender Keys construction continues to work exactly as it does today — Phase A just admits sharding-table cores into the broadcast-layer gossip topic for curated CGs. + +Phase B (later, optional) adds a per-assertion **payload-key wrap protocol** for monetization (see §5.7) and, if implemented, bumps `SWM_SENDER_KEY_PACKAGE_VERSION` from `'1'` to `'2'`. Phase A defers that work. + +### 5.3 Verification model: member post-decrypt + outsider attestation + +**The on-chain merkle root stays exactly as today.** The publisher computes leaves over plaintext (per-KA / per-assertion granularity, matching the existing V10 format), the merkle root is anchored on chain via the existing `ContextGraphStorage` batch entry, and the on-chain commitment carries no new fields. + +Verification splits cleanly across actors based on what they hold: + +#### 5.3.1 Member verification (post-decrypt) + +A member with the chain key: + +1. Fetches encrypted SWM messages — from either substrate gossip (live) or a hosting core via `SWMCatchupRequest` (catch-up; see §5.6). +2. Decrypts each ciphertext chunk using the per-message payload key derived from the chain key. +3. Computes the plaintext leaf hash from the decrypted assertion using **the same leaf format the publisher used** to commit on chain. +4. For each batch they reconstruct, re-derives the merkle root and compares to the on-chain anchor. +5. **If a mismatch occurs** (a malicious or buggy publisher committed a root that does not match the ciphertext stream): the member rejects the batch, alerts via SWM gossip to other members, optionally fetches the same ciphertext from a different hosting core to rule out core-level tampering, and the curator can slash / revoke the malicious publisher's `publishAuthority` per the existing V10 authority model. + +This is the same verification model members already have for SWM-to-VM consistency today. The only new wrinkle is that the *ciphertext source* is now a core (or other peer), not exclusively a member peer. + +#### 5.3.2 Outsider verification (member attestation tokens) + +An outsider holding a leaked / quoted / monetized assertion `A` plus a claim "this is in batch `B` of CG `X`, leaf index `i`" can verify against the on-chain anchor **with a member-attestation token**. The token is a small, member-signed envelope: + +``` +attestation = { + contextGraphId, batchId, leafIndex, + plaintextLeafHash, // H(A) computed by the attesting member after decrypt + ciphertextChunkDigest, // H(ct_i) the member observed on substrate + attesterAgentAddress, // member's wallet address + attesterMembershipEpoch, // which epoch the attester held a chain key for + attesterSignature // secp256k1 / EIP-191 over the above fields +} +``` + +Outsider's verification flow: + +1. Recompute `H(A)` from the leaked assertion; check it equals `attestation.plaintextLeafHash`. +2. Fetch ciphertext chunk `ct_i` from any hosting core; check `H(ct_i)` equals `attestation.ciphertextChunkDigest`. (This step confirms the attester is talking about a ciphertext that actually exists on the substrate.) +3. Verify merkle path from `plaintextLeafHash` to the on-chain root of batch `B`. +4. Verify `attestation.attesterSignature` against `attestation.attesterAgentAddress`. +5. Resolve `attesterAgentAddress` on chain via [SPEC_V10_IDENTITY_AND_ACCESS.md](./SPEC_V10_IDENTITY_AND_ACCESS.md) — was this address a member of CG `X` at `attesterMembershipEpoch`? Was that epoch active when batch `B` landed? +6. If all checks pass → trust the attestation. Trust path: outsider → named on-chain-resolvable member → on-chain anchor. + +**Why this is the honest model.** The chain has no chain key, so it cannot verify that `decrypt(ct_i)` equals `A`. Some key-holder must vouch. The attestation is that vouch, with cryptographic attribution: a malicious attester is publicly identifiable (their wallet is on chain) and slashable per the existing V10 reputation / authority machinery. For scenarios 1-3 in §2.4 the attester is naturally part of the trust loop (member, bridge, paid subscriber). For scenario 4 (Bloomberg-style monetized data) the curator IS the seller IS the attester — the API response bundles the assertion, payload key, ciphertext, and attestation token in one payload. + +**What about "trustless" outsider verification?** A truly chain-only verification of plaintext↔ciphertext correspondence requires either (a) a member co-signature gathered at publish time and anchored on chain, or (b) a zero-knowledge proof that the publisher knows a key under which `decrypt(ct_i) = pt_i`. Both are heavyweight. Neither is needed for the four motivating scenarios in §2.4. See §12 for what was considered and why deferred. + +#### 5.3.3 Why this RFC dropped the dual-root approach considered in v2 + +An earlier draft (v2) proposed a dual-root leaf format `leaf_i = H( H(salt ‖ pt_i) ‖ H(ct_i) )` with a per-epoch salt and an extra on-chain `saltCommitment` field. PR #113 review correctly identified two fatal issues: + +1. **No correspondence enforcement.** The publisher chooses both halves of the leaf independently; nothing in the construction binds the inner-left plaintext-hash to the actual plaintext recoverable from the inner-right ciphertext-hash. A malicious publisher can commit `(H(salt ‖ pt_A), H(ct_B))` and the merkle math validates. Outsiders are fooled; members catch it only post-decrypt — same detection point we have without the dual root. +2. **Epoch-salt disclosure broadens to whole-epoch privacy regression.** Revealing `salt_epoch` to one verifier (for one leaked triple) exposes the blinding for every plaintext-hash in that epoch. A per-leaf nonce would help but requires complex distribution. + +Both issues collapse if we accept that **plaintext↔ciphertext correspondence is fundamentally a key-holder check**. There is no leaf format the chain can verify on its own that fixes (1) without a key. So we drop the dual-root entirely; the on-chain commitment stays single-root over plaintext; members do their existing post-decrypt verification; outsiders use member attestations. This removes ~5 of Jurij's review items and significantly reduces Phase A scope. + +### 5.4 ACK protocol — ciphertext-availability + existing batch digest fields + +The new ACK shape preserves every existing V10 ACK invariant (replay protection, payment binding, retention binding) and adds **only** a ciphertext-availability attestation. Critically, the ACK digest binds the full batch metadata — not just a root. + +#### 5.4.1 ACK request + +``` +ACKRequest = { + // Existing V10 digest fields (do NOT change semantics or signing) — + // bind ACK to a specific batch under specific payment/retention terms. + contextGraphId, // identifies the CG + publishOperationId, // unique per publish attempt (replay protection) + merkleRoot, // plaintext root, as today + knowledgeAssetCount, // KAs in this batch + byteSize, // total batch size in bytes + epoch, // chain epoch of the publish + publishingChainId, // EVM chain id + tokenAmount, // payment amount bound to this ACK + retentionPeriod, // retention obligation + publishAuthority, // authorised publisher identity + + // NEW (Phase A): per-ciphertext-chunk availability commitments. + ciphertextChunks: [ + { chunkDigest: H(ct_i), byteSize: |ct_i|, swmMessageIndex: idx_i }, + ... + ], + ciphertextChunksRoot, // merkle root over chunkDigests, for compact ACK signing + + // NEW (Phase A): protocol version of the ACK envelope itself, for forward extensibility. + ackProtocolVersion: 2, +} +``` + +The responder verifies, in order: + +1. `publishOperationId` has not been seen before (replay protection). +2. The `publishAuthority` is currently authorised on chain for `contextGraphId` (per existing V10 publishAuthority lookup). +3. For each entry in `ciphertextChunks`: the core holds bytes matching `chunkDigest` and has indexed them under `(contextGraphId, batchId)`. If a chunk is missing, the core MAY request it inline (see §5.4.3); if it cannot acquire it within a normative timeout, it MUST decline the ACK. +4. `ciphertextChunksRoot` matches the recomputed merkle root over the supplied chunk digests. +5. **`byteSize` matches what the core actually holds.** The semantic is "size of what cores persist for this batch": for public CGs that is plaintext leaf bytes (today's behaviour, unchanged); for curated CGs that is the sum of `ciphertextChunks[i].byteSize`. The core recomputes from its local store and MUST decline the ACK with `BYTESIZE_MISMATCH` if the publisher's claim doesn't match. This is what keeps the pricing formula `tokenAmount ≥ stakeWeightedAsk × byteSize × epochs / 1024` (`KnowledgeAssetsV10._validateTokenAmount`) honest — the chain trusts a `byteSize` only because quorum-of-cores cosign an ACK digest binding it. + +**Implication for curated-CG pricing**. Because curated CGs declare ciphertext `byteSize`, they pay (slightly) more per plaintext bit than equivalent public CGs do, equal to the AEAD overhead per chunk (16-byte tag + 12-byte nonce). For typical chunk sizes the overhead is small (~0.7% at 4 KB chunks, ~2.7% at 1 KB chunks, ~11% at 256-byte chunks). Curators sensitive to cost may amortise by packing more assertions per broadcast message. No contract change is required: the contract treats `byteSize` as opaque input; the semantic ("plaintext for public, ciphertext for curated") is the publisher/core convention enforced via the cosigned ACK digest. + +The responder signs an `ACKResponse`: + +``` +ACKResponse = { + ackRequestDigest, // H of all fields in the corresponding ACKRequest + ackerIdentityId, // chain-resolvable core identity + ackerSignature // secp256k1 over ackRequestDigest, EIP-191 +} +``` + +#### 5.4.2 What signing the ACK binds + +Phase A constraint: the **cosigned digest is the existing V10 `computePublishACKDigest`** — `(chainId, kavAddress, contextGraphId, merkleRoot, knowledgeAssetsAmount, byteSize, epochs, tokenAmount, merkleLeafCount)`. Cores sign this verbatim because `KnowledgeAssetsV10._validateSignatures` verifies that exact shape on-chain; introducing new bound fields would require a contract change, which Phase A explicitly avoids (§7). + +The additional ACKRequest fields (`publishOperationId`, `ciphertextChunks[]`, `ciphertextChunksRoot`, `ackProtocolVersion`, `publishAuthority`) are **off-chain inputs to the core's accept-or-decline decision**, not new digest material. They give the core enough information to: + +- verify it holds the right ciphertext chunks (or to fetch them via `ChunkPullRequest` per §5.4.3), +- bind the ACK to a specific publish attempt (the proposer dedups via `publishOperationId` locally), and +- reject mismatched / replayed requests at the protocol layer before any signing happens. + +Because the V10 digest already contains `merkleRoot` + `byteSize`, an attacker swapping ciphertext under a valid digest fails immediately: the core's persisted bytes won't match `byteSize`, and at member verification (§5.3.1) the decrypted plaintext root won't match `merkleRoot`. Phase A's privacy gain comes from cores holding ciphertext bytes (not plaintext), not from extending the on-chain attestation surface. + +This addresses PR #113 review comment on `line:271`. + +#### 5.4.3 Ciphertext transport — normative behaviour + +This addresses PR #113 review comment on `line:273` (loose "inline or gossip" language). + +**Default path (the operational steady state)**: cores receive ciphertext chunks via substrate gossip well before any ACK request arrives — including for the first publish, because of pre-registration staging (§5.1.1). By the time the proposer sends the ACK request, the chunks are already locally indexed under `(contextGraphId, swmMessageIndex)`. The ACK request carries only digests, not bytes. + +**Fallback path (edge case)**: a core receiving an ACK request finds it lacks one or more referenced chunks. With pre-registration staging in place, this should be uncommon — it happens when: + +- A sharding-table reshuffle moved this core into the CG's assignment *after* the unrouted chunks were already gossiped (so the core was not subscribed when they flew by). +- The curator skipped the SWM gossip path entirely (e.g. a script-driven batch publish that constructs ciphertext locally and goes straight to ACK collection). +- Staging quota (§5.1.1) expelled some chunks before publish (e.g. TTL expired, or per-CG cap was hit then reset, or aggregate budget evicted older chunks). + +Resolution sequence: + +1. Core MAY request missing chunks inline from the proposer via a `ChunkPullRequest` over the same connection. Bounded retry: up to `MAX_CHUNK_PULL_ATTEMPTS = 3`. +2. If the proposer cannot supply within `CHUNK_PULL_TIMEOUT_MS = 5000`, the core MUST decline the ACK with reason `CHUNK_UNAVAILABLE`. +3. If the proposer supplies but chunks fail digest verification, the core MUST decline with reason `CHUNK_DIGEST_MISMATCH`. + +**Persist-before-sign requirement (normative)**: a core MUST durably persist any ciphertext chunk it intends to ACK *before* signing the ACK response. Storage MUST include indexing under `(contextGraphId, batchId, swmMessageIndex)` such that later `SWMCatchupRequest` queries (§5.6) can serve the chunk. Signing on transient receipt without persistence violates the availability commitment the ACK attests to. Hosting accountability (Phase C storage proofs) presumes this invariant. + +#### 5.4.4 ACK quorum + +Unchanged from V10: `parametersStorage.minimumRequiredSignatures()`. Cores in the sharding-table assignment for the CG are the eligible signer set. The existing `requiredSignatures` decision in `verify-collector.ts` continues to apply. + +### 5.5 Key distribution and epoch rotation + +A per-CG, per-epoch **chain key** is the access gate. The SWM substrate already implements the two-layer Sender Keys protocol (§5.2); this RFC formalises the lifecycle (today implicit in setup-on-first-share) and pins authentication, scope, and failure semantics. + +#### 5.5.1 KeyGrant — issuance and authentication + +A `KeyGrant` is the curator's explicit, signed envelope for delivering a chain key to a new or returning member. It supersedes the implicit "send a setup package on first interaction" pattern. + +``` +KeyGrant = { + contextGraphId, // target CG + epochId, // which epoch this grant scopes + membershipHash, // snapshot of allowed-members set at grant time + recipientAgentAddress, // grantee wallet + recipientKeyId, // recipient X25519 setup-key id (existing field) + setupPackageDigest, // H of the SwmSenderKeyPackageMsg envelope + backfillScope, // see §5.5.4 + startMessageIndex, // see §5.5.3 + grantedAtMs, + curatorSignature // secp256k1 / EIP-191 over the above fields +} +``` + +**Signature scheme is secp256k1 / EIP-191**, NOT Ed25519. This addresses PR #113 review comment on `line:279`. The curator's wallet identity is the V10 `publishAuthority` (or its delegate) — which is an Ethereum address — so the binding is to the same identity that grants on-chain authority for the CG. Ed25519 is reserved for node-level and message-level identities elsewhere in V10; mixing schemes for wallet-bound permissions is incorrect. + +The verifier (recipient or any third party validating the grant) checks: + +1. Recover signer address from `curatorSignature`. +2. Resolve current `publishAuthority` for `contextGraphId` on chain. +3. Confirm signer is either the `publishAuthority` directly or holds an active PCA delegation from it (per [SPEC_V10_IDENTITY_AND_ACCESS.md](./SPEC_V10_IDENTITY_AND_ACCESS.md)). +4. Confirm `membershipHash` matches the recipient's view of the membership snapshot (or accept the curator's snapshot if first contact). +5. Decrypt the setup package per the existing X25519+AES-GCM flow; confirm its digest equals `setupPackageDigest`. + +#### 5.5.2 KeyRotate — rotation on revocation + +A `KeyRotate` is the curator's signed envelope to remaining members announcing a new epoch: + +``` +KeyRotate = { + contextGraphId, + previousEpochId, + newEpochId, + newMembershipHash, + rotationReason, // "member_revocation" | "scheduled" | "compromise" + newSetupPackageDigest, // each remaining member receives their own setup package; this is the digest of the recipient-specific package + rotationDeadlineMs, // see §5.5.5 below + curatorSignature // secp256k1 / EIP-191 +} +``` + +The verifier checks the same chain of curator authority as for `KeyGrant`, plus: the `previousEpochId` matches the recipient's current epoch. + +#### 5.5.3 Mid-epoch joining — forward-only key delivery + +A new member joining mid-epoch does NOT require a new epoch. The curator sends them a `KeyGrant` with `startMessageIndex` set to the CURRENT message index, plus ratchet state for that index — NOT the initial chain key. + +This addresses PR #113 review comment on `line:280`. Concretely: instead of sending the initial chain key `CK_0` (which would let the joiner decrypt every prior message in the epoch by ratcheting forward), the curator derives the ratchet state at index `n` via `CK_n = HMAC_ratchet^n(CK_0)` and includes only `CK_n` in the setup package. The joiner can decrypt messages from index `n` onward but cannot reverse the HMAC ratchet to recover `CK_0..n-1`. + +If the curator explicitly wants to grant historical access (e.g. paid backfill), they set `startMessageIndex = 0` and ship `CK_0`. The `startMessageIndex` field is bound in the curator's signature, so the grant's scope is unambiguous and chain-verifiable. + +#### 5.5.4 KeyGrant backfill-scope artifact + +The `backfillScope` field in `KeyGrant` is an explicit declaration of what historical content the grantee is entitled to. This addresses PR #113 review comment on `line:595` (and resolves the previous "prior keys at curator discretion" vagueness). + +``` +BackfillScope = { + grantedEpochs: [ // explicit list of epochs included in the grant + { + epochId, + chainKeyMaterial, // CK_n for the startMessageIndex of this epoch (encrypted in setup package) + startMessageIndex, // forward boundary within this epoch + grantedBatchIds: [...] | "all" // optional restriction to specific publish batches + }, + ... + ], + futureGrantsScope: // policy for future epochs + "auto_grant" | // curator commits to forwarding future epochs (subscription model) + "manual" // grantee must request each future epoch (per-asset model) +} +``` + +A recipient UI can now cleanly distinguish: +- Future-only access (`grantedEpochs = [current_epoch_only_with_forward_boundary]`) +- Full historical (`grantedEpochs` includes all prior, `startMessageIndex = 0` for each) +- Targeted batch access (`grantedBatchIds` constrained — used for the per-assertion monetization model β, §5.7) +- Failed / incomplete grants (recipient was promised epochs that aren't in the actual delivered `backfillScope`) + +#### 5.5.5 KeyRotate failure states + +This addresses PR #113 review comment on `line:281`. Rekey is a multi-recipient operation; some recipients will be offline at rotation time. Operators need explicit visibility. + +**States**: + +- `pending` — `KeyRotate` issued, awaiting recipient acknowledgements. New SWM writes under the new epoch are NOT yet emitted by the curator. +- `partial` — `rotationDeadlineMs` passed; some but not all remaining members ACK'd receipt. Curator decides per-CG policy: either (a) proceed (cut over to new epoch; offline members will get the new setup package on their next reconnect from the curator's queue) or (b) hold (defer cutover; SWM continues on previous epoch). +- `complete` — all remaining members ACK'd; cutover proceeds. +- `failed` — `rotationDeadlineMs` passed with insufficient ACKs AND curator policy is (b) hold; curator MUST surface this to UI and retry. SWM continues on previous epoch. + +**Retry**: `KeyRotate` may be re-issued indefinitely with a new `rotationDeadlineMs`; recipients that have already ACK'd a prior issuance for the same `(previousEpochId → newEpochId)` MAY ACK again idempotently. + +**SWM-write semantics during pending/partial**: if curator policy is (b) hold, SWM writes from members continue under the previous epoch and the curator MUST emit `KeyRotate` retries until rotation completes. If policy is (a) proceed past deadline, new SWM writes use the new epoch; offline members will see a gap when they reconnect (their stored chain key for the old epoch covers messages up to the cutover; they need the new setup package to read messages after). + +**Curator-side queue**: a curator that has issued a `KeyRotate` MUST persist a queue of pending recipient setup packages and offer them on first contact when those recipients reconnect. + +#### 5.5.6 Time-based and post-compromise rotation + +Optional curator-initiated rotation for stronger post-compromise security follows the same `KeyRotate` flow as revocation. Not required by protocol; not blocking for Phase A. + +#### 5.5.7 What stays out of scope + +- **Re-encryption of historical ciphertext** for forward secrecy on revocation. Curated CG operators who need it MUST do explicit re-encrypt + re-publish under a new epoch with new keys. A future RFC could spec proxy re-encryption; not in this one. +- **Threshold curator** (no single wallet holds the only signing key). Can layer on top without changing this design — the verifier would check a threshold signature against a known curator multi-sig instead of a single secp256k1 sig. + +### 5.6 SWMCatchupRequest — wire protocol for fetching encrypted SWM from cores + +This addresses PR #113 review comment on `line:293` (correcting the v2 "no new protocol" claim — there IS new wire surface and it needs spec). + +A member edge that's been offline (or a late joiner) asks any hosting core to stream encrypted SWM messages it missed. Cores serve **only the broadcast-layer ciphertext**; never the setup-layer packages (those flow point-to-point from curator; see §5.5). + +#### 5.6.1 Request shape + +``` +SWMCatchupRequest = { + contextGraphId, + subGraphName, // optional, defaults to default subgraph + sinceEpochId, // epoch to start from + sinceMessageIndex, // message index within sinceEpochId; inclusive lower bound + untilEpochId, // optional upper bound; omitted = serve to current + untilMessageIndex, // optional upper bound within untilEpochId + maxMessages, // pagination: max messages per response page + pageCursor, // opaque cursor for continuation; empty on first request + + // Authentication & rate-limit context + requesterIdentityId, // chain-resolvable wallet of the requester + requesterSignature, // secp256k1 / EIP-191 over the above fields + timestamp + requestedAtMs // timestamp; signature replay protection +} +``` + +#### 5.6.2 Response shape + +``` +SWMCatchupResponse = { + pageMessages: [ // ordered by (epochId, messageIndex) + { + epochId, + messageIndex, + ciphertextChunk, // opaque AEAD-encrypted bytes (broadcast layer) + swmMessageDigest // H(ciphertextChunk), for the requester to detect corruption + }, + ... + ], + nextPageCursor, // empty if this was the last page in the requested range + serverIdentityId, // the hosting core's chain-resolvable wallet + serverSignature // secp256k1 over (pageMessages digests + pageCursor); allows requester to attribute corruption to a specific core +} +``` + +#### 5.6.3 Ordering and completeness + +- Messages MUST be served in `(epochId, messageIndex)` order. Cores SHOULD NOT serve out-of-order or skip indices; if a chunk is missing locally, the response includes a `missingIndices` array so the requester can fetch from another core. +- **Completeness verification**: requesters reconstruct the SWM message-index sequence and check for gaps. For each batch that was anchored on chain during the requested range, the requester compares the assembled batch hash against the on-chain merkle root (post-decrypt). Mismatch → flag this core and refetch the disputed chunks from a different hosting core. + +#### 5.6.4 Authentication, rate limits, abuse model + +Cores by default MUST authenticate requesters to enable rate-limiting and abuse defence. Three authentication modes: + +- **Anonymous (denied by default)**: cores MAY reject unsigned requests. CGs that have explicitly opted in to public-discoverability can be served anonymously, but curated CGs MUST require authenticated requests. +- **Member-attested**: `requesterSignature` is from a wallet that the core can resolve as a current or historical member of `contextGraphId` (chain lookup). This is the normal path for member edge resync. +- **Token-bearer (for paid grants)**: requester presents a short-lived bearer token issued by the curator (a signed envelope binding `requesterIdentityId`, `contextGraphId`, `validUntilMs`, and `grantedBatchIds` if scoped). The core verifies the token's curator signature against the on-chain `publishAuthority` and serves only chunks within the granted scope. + +**Rate limits**: cores MUST enforce per-`requesterIdentityId` rate limits, configurable, with defaults documented in [SPEC_CG_MEMORY_MODEL.md §4.6](./SPEC_CG_MEMORY_MODEL.md#46-hosting). Exceeding limits → `429 Too Many Requests` with `Retry-After` hint. + +**Abuse signals**: a requester that repeatedly fetches but never ACKs as a member, fetches batches beyond their granted scope, or triggers rate-limit responses 5+ times in a window — the core MAY add to a local block-list and SHOULD report via [SPEC_CG_MEMORY_MODEL.md] reputation surface. + +#### 5.6.5 What cores still do NOT serve + +- **Setup-layer packages** (per-recipient X25519 wraps containing chain keys / payload keys). These flow point-to-point from the curator at grant time and are never on cores. A returning member already has their setup package locally; a new member gets one from the curator directly. This invariant is what keeps cores out of the key-distribution path (§5.2). +- **Plaintext.** Cores have no key. Even if compelled to disclose, they can only disclose ciphertext. + +### 5.7 Monetization access model β — per-assertion payload key + +This RFC specifies a fine-grained access model for monetization: a buyer pays for ONE assertion and receives a key that decrypts only that assertion — not the epoch chain key (which would unlock everything in the epoch). + +This is "model β" in the §12 alternative-considered comparison. Model α (sell the epoch chain key for broad subscription access) falls out as a degenerate case where the curator sells β keys for every assertion in the epoch, but is not the primary monetization protocol. β is the Bloomberg-shaped product. + +#### 5.7.1 Cryptographic foundation (already in code) + +The existing Sender Keys construction in [`packages/core/src/crypto/swm-sender-key.ts`](../../packages/core/src/crypto/swm-sender-key.ts) derives the per-message payload key as: + +``` +payloadKey_n = HKDF(CK_n, info = SWM_SENDER_KEY_PAYLOAD_KEY_PURPOSE, length = 32) +``` + +where `CK_n` is the chain key after `n` ratchets. HKDF is one-way: knowledge of `payloadKey_n` does NOT recover `CK_n`. So the curator can deliver a single `payloadKey_n` to a buyer without exposing the chain key or any other message's payload key. + +#### 5.7.2 PaidAccessGrant — wire protocol + +``` +PaidAccessGrant = { + contextGraphId, + buyerAgentAddress, // buyer wallet (the X402 payer) + buyerRecipientKeyId, // buyer's X25519 setup-key id for envelope wrap + grantedAssertions: [ // array of one or more assertion accesses + { + epochId, + swmMessageIndex, // identifies the assertion + batchId, // the VM batch this assertion was anchored in + leafIndex, // position within the batch's merkle tree + payloadKey, // 32 bytes — derived from CK_n by curator, opaque to buyer's understanding of the ratchet + ciphertextChunkDigest, // H(ct_n), for buyer to verify they got the right ciphertext from cores + attestation // member-attestation token per §5.3.2 (curator can self-attest if they're a member, which is the typical case) + }, + ... + ], + validUntilMs, // optional: time-bounded access (curator may revoke by not renewing) + curatorSignature // secp256k1 / EIP-191 over the above +} +``` + +The grant envelope itself is encrypted to `buyerRecipientKeyId` via the same X25519 + AES-GCM machinery as the existing setup package. Only the buyer can decrypt. + +#### 5.7.3 Buyer's verification and decryption flow + +1. Buyer pays via x402 (or other payment rail) → curator emits `PaidAccessGrant`. +2. Buyer decrypts the envelope → gets `(payloadKey, ciphertextChunkDigest, attestation)` per granted assertion. +3. Buyer fetches the ciphertext chunk from any hosting core via `SWMCatchupRequest` (token-bearer mode, presenting a bearer token derived from `PaidAccessGrant.curatorSignature` — see §5.6.4). +4. Buyer verifies `H(received ciphertext) == ciphertextChunkDigest`. +5. Buyer decrypts: `assertion = AEAD_decrypt(payloadKey, ciphertextChunk, AAD = standard SWM AAD per §5.2)`. +6. Buyer verifies plaintext binding via the attestation per §5.3.2 → trust path established to on-chain anchor. + +#### 5.7.4 What the buyer learns + +- The single assertion they paid for, fully decrypted. +- The on-chain anchor confirms it was committed by the curator at publish time. +- The attestation confirms the plaintext matches what a member decrypted. + +#### 5.7.5 What the buyer does NOT learn + +- **Other assertions in the same epoch**: HKDF is one-way; `payloadKey_n` reveals nothing about `payloadKey_m` for `m ≠ n` or about `CK_n`. +- **Other assertions in the same batch**: even if the buyer holds the merkle root for the batch, they cannot decrypt other leaves without their `payloadKey_m`. +- **Future assertions**: unless `PaidAccessGrant` includes them, the buyer has no key. + +#### 5.7.6 What the curator learns (and the privacy implication) + +The curator knows which assertions the buyer purchased — both from the payment correlation and from issuing the grant. This is a curator-side metadata trail. For most commercial use cases this is acceptable (the seller knows what the buyer bought). For privacy-preserving purchase patterns (private purchase that the curator cannot correlate to specific assertions), a future RFC could introduce private-payment / blind-grant primitives. + +#### 5.7.7 Phase A vs Phase B + +The HKDF derivation is already in code. What is missing for full β monetization: + +- The `PaidAccessGrant` envelope format and wire protocol (~200 LOC). +- Curator-side UI to issue grants (~UI changes). +- Buyer-side library to consume grants (~100 LOC + verification flow per §5.7.3). +- The token-bearer authentication mode in `SWMCatchupRequest` (§5.6.4). + +**This RFC places β monetization in Phase B**, not Phase A. Phase A delivers the substrate change that makes monetization possible (cores host ciphertext that paid outsiders can fetch); the explicit `PaidAccessGrant` protocol can be developed and shipped independently when a concrete monetization product is ready to consume it. Members in the meantime can manually share `(payloadKey, ciphertextChunkDigest, attestation)` tuples — same content, just without a normative envelope format. + +--- + +## 6. Threat model + +### 6.1 What this preserves + +- **Plaintext privacy from cores** — cores never receive chain keys; ciphertext is AEAD-encrypted under per-message payload keys derived via HKDF; cores see opaque bytes only. +- **Plaintext privacy from non-members** — non-members get nothing (same as today). +- **Member verifiability** — members verify each batch post-decrypt against the existing on-chain plaintext root (§5.3.1). Detection point identical to today. +- **Verifiability for grant-recipients** — once they hold the chain key (KeyGrant) or per-assertion payload key (PaidAccessGrant), recipients verify their content the same way members do. +- **Outsider verifiability via attestation** — outsiders without keys verify content via member-attestation tokens (§5.3.2), with cryptographic attribution to a named on-chain-resolvable member. + +### 6.2 What this preserves only against honest cores + +- **Hosting commitment** — a malicious core can claim to host bytes it doesn't actually hold, ACK a publish, then refuse to serve. Mitigations: existing sharding-table accountability (replication factor > 1, slashing on proven non-availability), periodic proof-of-storage challenges (Phase C). The `persist-before-sign` invariant (§5.4.3) makes any later denial-of-service evidence directly attributable. +- **Selective ciphertext withholding** — a malicious core can serve some queries and not others. Mitigation: same as above — replication, multiple-core querying, eventual slashing-on-proof. + +### 6.3 What this does not protect against + +- **Member exfiltration** — a member with the chain key can always copy the plaintext and share it out of band. Same as any access-controlled system. The on-chain anchor lets the curator at least *prove provenance* of leaked data via a member attestation, which is a useful audit primitive even when prevention is not possible. +- **Curator key loss** — if the curator loses the chain key entirely, no one can read the CG anymore (except via existing member copies). Operational hygiene; out of protocol scope. Threshold-curator (§5.5.7) is the layered fix. +- **Malicious attester** — a member who attests to a false plaintext-binding can deceive an outsider in the short term. Mitigation: attestation includes `attesterAgentAddress` which is on-chain-resolvable; once detected (e.g. by another member or by the outsider comparing attestations), the curator can revoke the malicious member's `publishAuthority` and the existing V10 reputation/slashing model applies. See open question §8.5 about freshness. + +### 6.4 Forward-secrecy posture, per actor + +Compared to the status quo (curated content lives only on member nodes), RFC-38 changes the read-at-time-T capability for two specific actor states. The table makes the delta explicit: + +| Actor at time T+1 | Prior state at time T | Status quo: can read T's plaintext? | RFC-38: can read T's plaintext? | +|---|---|---|---| +| Current member | Holds chain key for T, was online at T | Yes (cached locally) | Yes (cached locally + cores) | +| Returning member | Holds chain key for T, was offline at T | Only if a peer is online to serve | Yes — cores serve ciphertext | +| Revoked member, cached plaintext at T | Was member at T, decrypted at T | Yes (locally cached) — unchanged | Yes (locally cached) — unchanged | +| Revoked member, did NOT decrypt at T | Was member at T but offline, retained chain key | Probably no (peers may have rotated, refused to serve) | **Yes** — if they fetch ciphertext from a core | +| Outsider with chain key, no membership | Got key out of band (leak, theft, paid grant) | Needs to compromise a member node to get bytes | Needs to authenticate to a core via `SWMCatchupRequest` (§5.6.4) — possible if they have a valid bearer token or member-attestation chain | +| Outsider without chain key | None | Cannot read | Cannot read | +| Compromised core | Holds ciphertext only | N/A (cores don't host curated today) | Cannot read (no key) | +| Compromised core + revoked member colluding | Ciphertext + old chain key | N/A | Yes — full historical read for granted epochs | + +**Net assessment.** The marginal new exposure is one row (revoked member who didn't decrypt at T but retained their old key, AND obtains ciphertext from a core). This is narrower than it first appears because: + +- Most revoked members **decrypted continuously while active** — the lazy-decrypt scenario where they retained the key but never used it is uncommon. +- The chain key required to decrypt is itself a hard-to-obtain artifact. Compromising the key is the primary attack regardless of where ciphertext lives. +- The compromised-core scenario presumes the attacker can violate a staked operator's storage — which under §6.2 is subject to (Phase C) slashing once provable. + +The new model is **defense in depth** for plaintext, not regression: cores hold encrypted, members hold keys, neither alone suffices. The threat surface widens slightly (cores join the set of places ciphertext lives) in exchange for the availability and verifiability benefits that motivated the RFC. For curators who explicitly need stronger forward secrecy on revocation, the §9 escape valve (re-encrypt + republish) remains available. + +### 6.5 Metadata leakage and abuse model + +This addresses PR #113 review comment on `line:291`. Cores now hosting curated ciphertext changes the metadata surface even though plaintext stays protected. The threat model and the protocol-level mitigations: + +#### 6.5.1 Metadata visible to a hosting core (curated CG) + +| Metadata | Visible to core? | Mitigation | +|---|---|---| +| CG existence | Yes (core knows it's assigned to host this CG via sharding-table) | Inherent to hosting; CG existence is on-chain anyway | +| Batch IDs | Yes | Same as above; batches are on-chain | +| Per-batch chunk count and byte size | Yes | Sharding-table assignment requires this for storage planning | +| Update cadence (message frequency) | Yes (cores see the gossip rate) | No protocol mitigation — operational reality | +| Hosting assignments (which cores host which CGs) | Yes (on-chain in sharding-table) | Public info; same as today for public CGs | +| Member wallet addresses interacting | Partial — cores see message senders via the broadcast layer's `senderAgentAddress` AAD field, but only for members; cores don't see who's a non-publishing reader | This is the same exposure as today for public CGs; the AAD is required for sender authentication | +| Plaintext content | NO | Encryption; this RFC's core invariant | +| Setup-package distribution (who got keys when) | NO | Setup packages flow point-to-point from curator, never on cores (§5.2) | + +#### 6.5.2 Metadata visible to an unauthenticated outsider + +Without `SWMCatchupRequest` authentication (§5.6.4), outsiders could enumerate: + +- That a CG exists (on-chain, no new exposure) +- Batch IDs and merkle roots (on-chain, no new exposure) +- Ciphertext chunk byte sizes and digests (via core queries) +- Update cadence (by polling) +- Bulk-download all ciphertext (opaque, but reveals total volume) + +**Mitigations enforced by this RFC**: + +- **Curated CGs require authenticated `SWMCatchupRequest`** (§5.6.4). Cores MUST reject anonymous requests for curated CGs. The three accepted auth modes (member, paid-token-bearer, curator-issued bearer) all bind the requester to a chain-resolvable identity. +- **Rate limits** per-requester (§5.6.4). Default policy: aggressive limits for unrecognised requesters; higher limits for verified members and active paid subscribers. +- **Quota-based access** (optional, per-CG curator policy): cores may enforce per-grant byte-quota limits derived from `PaidAccessGrant.grantedAssertions`. +- **Anti-enumeration**: a core SHOULD respond with consistent timing whether the requested chunk exists or not, to prevent presence-oracle attacks (within reason — full constant-time is not required). + +#### 6.5.3 Compelled disclosure + +A core compelled (subpoena, court order, coercion) to disclose its CG storage can disclose: + +- The list of CGs it hosts (already on-chain) +- All ciphertext chunks for those CGs +- Sender wallet addresses observed in AAD fields + +It **cannot** disclose: + +- Plaintext (no key) +- Member-to-CG mappings beyond what's on chain +- Setup packages (it never had them) + +For threat models where compelled disclosure of ciphertext-at-rest is unacceptable, the only mitigation is **don't use a third-party core for that CG** — self-host (per §2.5's GitLab analogy). The protocol supports this; deployment is the user's choice. + +#### 6.5.4 Abuse — protocol-level circuit breakers + +Cores hosting may face abuse vectors: + +- **Storage exhaustion via registered CGs** (publisher floods cores with high-cardinality CGs): mitigated by the sharding-table replication factor (cores host a bounded subset of CGs once per-CG sub-sharding lands; until then, every core hosts every registered CG, bounded by retention-policy expiration) and Phase C storage-proof challenges. +- **Storage exhaustion via unregistered-CG staging** (attacker gossips ciphertext under a never-registered CG ID to use cores as a free file host): mitigated by the per-chunk TTL, per-wallet rate limit, per-CG-ID byte cap, and per-core aggregate budget specified in §5.1.1. None of these gates require chain state; all are enforced locally on the receiving core based on the sender's wallet signature (already on every SWM message). An attacker can burn the wallet's reputation and chew through the per-wallet rate budget, but cannot exceed the per-core aggregate ceiling and cannot retain anything past TTL without registering on chain (which adds an explicit, attributable, on-chain footprint). +- **Bandwidth exhaustion** (outsider repeatedly fetches the same ciphertext): mitigated by rate limits and per-requester quotas (§5.6.4). +- **Reputational poisoning** (malicious publisher commits batches that members later reject, blaming cores): the core can prove via `ackRequestDigest` (§5.4) that it ACK'd only what was attested; correlation back to the publisher's authority is on chain. + +--- + +## 7. Implementation impact + +**Phase A is the small, focused ship that unblocks scenarios 1-3 in §2.4.** It changes only the substrate subscription gate, the ACK protocol shape, and (de-implements) the assumption that cores serve only public CGs. No new on-chain fields. No new merkle leaf format. No proto-version bump. **Phase B** adds the explicit key lifecycle protocol and the monetization model β (scenario 4). **Phase C** adds storage-proof challenges (future hardening). + +### 7.1 Phase A — Substrate subscription + new ACK protocol (single ship) + +**Scope**: admit sharding-table cores into curated SWM substrate as encrypted-bytes hosts; rewire the ACK protocol to bind ciphertext-availability into the existing V10 batch digest. No on-chain field changes. No leaf format changes. Verification continues to use the existing single-root over plaintext (§5.3). + +**Changes**: + +*Substrate subscription:* +- `packages/agent/src/swm/enumerate-cg-members.ts` → split into `enumerate-cg-members` (decryption-eligible; unchanged semantics) and `enumerate-cg-hosts` (substrate-eligible; new). +- `packages/agent/src/dkg-agent.ts` SWM subscription gate uses `enumerate-cg-hosts` for hosting decisions and `enumerate-cg-members` for key-distribution decisions. +- `packages/chain` adds (if not already present) a `getShardingTableMembersForCG(cgId)` helper. Reuses the existing sharding-table contract. + +*ACK protocol (§5.4):* +- `packages/publisher/src/ack-collector.ts` (and the responder in `packages/agent/`) → adopt the new `ACKRequest` shape (§5.4.1) that binds the full V10 batch-digest fields PLUS the new `ciphertextChunks[]` + `ciphertextChunksRoot` + `ackProtocolVersion`. Sign with secp256k1/EIP-191 over `ackRequestDigest`. +- Responder enforces the `persist-before-sign` invariant (§5.4.3): durably persist + index every chunk it intends to ACK before emitting `ACKResponse`. +- Implement the `ChunkPullRequest` fallback for cores that haven't yet seen a chunk via gossip (§5.4.3). Bounded retry; explicit decline on timeout. + +*Catch-up protocol (§5.6):* +- `packages/agent/src/swm/catchup.ts` (new) → implement `SWMCatchupRequest`/`Response` per §5.6.1-§5.6.2. Default to member-attested auth; reject anonymous requests for curated CGs. +- Rate-limit and abuse defence per §5.6.4 / §6.5. Default rate limits documented in [SPEC_CG_MEMORY_MODEL.md §4.6](./SPEC_CG_MEMORY_MODEL.md#46-hosting). + +*Member verification post-decrypt (§5.3.1):* +- `packages/agent/src/swm/verify-batch.ts` (new) → after reconstructing a batch from substrate (live or catch-up), recompute plaintext merkle root and compare to on-chain anchor. On mismatch: reject batch, alert via SWM, retry from a different hosting core. + +*Outsider attestation tokens (§5.3.2):* +- `packages/publisher/src/member-attestation.ts` (new) → small library for members to mint, and outsiders to verify, attestation envelopes. Used by scenario 3 bridge agents and scenario 4 monetization paths. + +**Out of Phase A scope**: explicit `KeyGrant` / `KeyRotate` messages (Phase B), monetization β protocol (Phase B), dual-root commitment (dropped entirely; see §5.3.3), epoch salt (dropped entirely), `saltCommitment` on-chain field (dropped entirely), `commitmentVersion` field (not needed; the existing single-root commitment is unchanged). + +**Unlocks**: scenarios 1, 2, and 3 in §2.4. Edge curator → curated CG → VM publish works. Edge resync from cores works. Bridge cores (scenario 3) work via existing implicit setup-on-first-share grant flow. Scenario 4 (monetization) requires Phase B's `PaidAccessGrant` to be fully native, but a manual "send the buyer `(payloadKey, ciphertext, attestation)` over email" already works under Phase A using existing primitives. + +**Test gates** (devnet): + +1. Invite-only CG, edge proposer + 3 sharding-table-core ACKers; publish lands on chain. Repeat with edge proposer offline immediately after sending ACK requests (publish should still land because cores hold the ciphertext and the new ACK shape binds availability). +2. Fresh "late joiner" node receives a setup package out-of-band from the curator, then uses `SWMCatchupRequest` against any hosting core to backfill the entire CG history. Verify every batch reconstruction against the on-chain merkle root. +3. Member detects a malicious-publisher batch: publisher commits a root that doesn't match the SWM ciphertext members can decrypt. Member rejects the batch, alerts via SWM gossip, and the rejection propagates. +4. Outsider with `(assertion, attestation)` from a member runs the attestation-verification flow against a target batch; verification succeeds for a real attestation, fails for a tampered one or one signed by a wallet that wasn't a CG member at the attested epoch. + +### 7.2 Phase B — Explicit key lifecycle + monetization model β + +**Scope**: formalise the curator's key-distribution lifecycle as explicit `KeyGrant` / `KeyRotate` messages (§5.5) and add the `PaidAccessGrant` protocol for per-assertion monetization (§5.7). + +**Changes**: + +*Explicit lifecycle messages:* +- `packages/agent/src/swm/key-grant.ts` (new) → implement `KeyGrant` and `KeyRotate` per §5.5.1, §5.5.2. Curator-side issuance, recipient-side verification, secp256k1/EIP-191 signing. +- `packages/agent/src/swm/key-grant-state.ts` (new) → KeyRotate state machine (pending/partial/complete/failed) per §5.5.5. Curator-side queue for pending recipients. +- `packages/agent/src/swm/backfill-scope.ts` (new) → `BackfillScope` artifact per §5.5.4. Recipient-side UI signals for future-only vs full-historical vs targeted-batch access. +- `packages/core/src/proto/swm-sender-key.ts` → bump `SWM_SENDER_KEY_PACKAGE_VERSION` from `'1'` to `'2'`. Add `KeyGrantMsg` and `KeyRotateMsg` schemas. Wire-negotiation: version `'1'` continues to work for nodes that haven't upgraded; version `'2'` adds explicit lifecycle. + +*Monetization β protocol:* +- `packages/agent/src/swm/paid-access.ts` (new) → `PaidAccessGrant` per §5.7.2. Per-assertion payload-key wrap using existing X25519+AES-GCM machinery; member attestation per §5.3.2. +- `packages/agent/src/swm/catchup.ts` → add token-bearer auth mode for `SWMCatchupRequest` (§5.6.4) — paid buyers fetch ciphertext using a `PaidAccessGrant`-derived bearer token. +- `packages/node-ui/` → curator UI flow for "grant access" (KeyGrant) and "issue paid access" (PaidAccessGrant). x402 integration is a separate spec. + +*Mid-epoch join semantics:* +- Forward-only `startMessageIndex` enforcement per §5.5.3. Curator-side: derive ratchet state at index `n` instead of shipping `CK_0`. Recipient-side: refuse setup packages with ambiguous scope. + +**Defer until**: a concrete monetization product or compliance/audit story justifies the additional protocol surface. Phase A members can use implicit setup-on-first-share until then. + +### 7.3 Phase C — Storage-proof challenges (future hardening) + +**Scope**: periodic proof-of-storage challenges to keep cores honest about what they claim to host. Slashing wiring against `parametersStorage`-defined penalties. + +**Defer until**: real-world hosting-fault evidence makes protocol-level enforcement worth the complexity. Until then, the §6.2 honest-cores assumption (with sharding-table replication factor > 1 as the layer-0 mitigation) is acceptable. + +### 7.4 What this enables later without forking the protocol + +The substrate convergence in §2 has compound benefits. Each of the following becomes a **single-implementation** feature instead of "build twice (public + curated) or skip for curated": + +| Future feature | Status quo cost | Under RFC-38 | +|---|---|---| +| Replication-factor SLAs for stored data | Implement separately for curated (member-replication is undefined today) | Reuse sharding-table replication mechanism uniformly | +| Slashing for hosting non-availability | Cannot apply to curated (no shared substrate to slash on) | Works uniformly; cores are slashable for any CG they're assigned to | +| Per-CG access monetization via x402 | Requires bespoke gateway per CG | Standard pattern: pay-curator-for-key, fetch-ciphertext-from-cores (scenario 4, formalized as §5.7) | +| Third-party tool integration via bridge cores | Impossible (cores don't have curated bytes to bridge) | Native: hosting cores hold ciphertext, a co-located member-agent on the same machine decrypts and pushes (scenario 3) | +| Cross-CG queries that include curated content | Requires querier to be member of every queried CG | Querier with appropriate keys reads from one substrate (cores), discovers what they have keys for | +| Audit logs for compliance | Members manually verify post-decrypt; outsiders need member co-operation | Same trust model + attestation tokens carry the verification cryptographically | + +The substrate fork in the status quo means each of these has to be implemented twice or skipped for curated; the cost compounds with each new feature. RFC-38's one-time investment in convergence pays back over the product roadmap. + +--- + +## 8. Open questions + +1. **Replication factor for hosting**. The sharding table currently picks an N-of-M set per CG; what's the right N for the substrate tier (encrypted SWM) for curated vs public CGs? Same value or different? Recommendation: start with both at the same default (e.g. 3); revisit once hosting-cost analysis lands. +2. **PCA (publisher chain authority) interaction**. PCA-authorised agents publishing on a curator's behalf need access to the same chain key as the curator. PCA-grant ≡ KeyGrant in this model (Phase B); the PCA signs ACK requests under its own identity and its own publishAuthority delegation. Verify the PCA flow composes cleanly; expect yes but write a regression test. +3. **Rename of "curated" in UI copy**. The terminology in SPEC_CG_MEMORY_MODEL.md uses "invite-only" for the sharing dial and "curators-only" for the contribution dial. Neither covers the new connotation "data is encrypted at the substrate layer, keys gate access." Consider a future copy pass. Out of scope for this RFC. +4. **Setup-package availability for offline curators.** Today setup packages flow strictly point-to-point from curator to recipient at grant time. If a member rolls a new device while the curator is offline, the new device cannot bootstrap (it has the wallet but not the chain key). Options: (a) curator pre-provisions long-lived recipient X25519 keys that are device-portable so any device with the same agent wallet can decrypt setup packages it previously received; (b) cores opt in to relay setup packages on the curator's behalf — these are still end-to-end encrypted, cores cannot decrypt, but they break the strict no-key-material-on-cores invariant of §5.2. Recommendation: (a) for v1, defer (b) until a concrete use case demands it. +5. **Trustless outsider verification (deferred-Phase-D candidate).** Member-attestation tokens (§5.3.2) give outsiders a cryptographically-attributable verification path, but require trusting a named member at attestation time. For use cases where no member is reachable at verification time (regulated audit years after publish; journalism with all original members hostile), a member-co-signature collected at publish time and anchored on chain would provide fully self-contained chain verification. Sketch: each publish ACK round includes ≥1 member co-signature attesting "I decrypted every leaf and the plaintext binding holds." Cost: changes the publish flow to require a member online and willing to co-sign. Not on the roadmap; documented here so the protocol does not paint into a corner. +6. **Member-attestation token freshness / revocation.** Attestation tokens (§5.3.2) are signed at attestation time, not publish time. If a member later realises they attested to a bad batch (the publisher's plaintext didn't match what the member decrypted, but they only noticed later), how do they revoke? Options: (a) attestations include `attesterCommitsAtMs` and consumers check it's before any on-chain `BatchRejected` event; (b) attesters maintain a revocation registry. Recommendation: (a) for Phase B, defer (b). +6. **Backfill scope verification.** A buyer who pays for `grantedBatchIds = [B_42]` (§5.5.4) and receives a `PaidAccessGrant` for that batch can verify they received what they paid for. But what if the curator's UI says "you bought batch 42" while the issued grant says "you bought batch 43"? Recommendation: the buyer's payment receipt (off-chain or on-chain) MUST bind to the same `grantedBatchIds` set the grant carries. Out of scope for this RFC; in scope for whoever wires the x402 / payment layer. +7. **Catch-up bandwidth amplification attacks.** Cores serve ciphertext to authenticated requesters per §5.6.4. A malicious member could repeatedly trigger full-history catch-ups across many cores to amplify bandwidth costs. Rate limits (§5.6.4) and per-CG quotas mitigate. May need explicit catch-up budgets per-member per-day in Phase B if real-world abuse emerges. +8. **ACK protocol version migration.** Phase A introduces `ackProtocolVersion: 2`. Cores running pre-Phase-A code will continue to receive `ackProtocolVersion: 1` requests (the existing V10 shape); their behaviour is unchanged. New publishers MUST send version 2 to gain ciphertext-availability binding. Until all cores have upgraded, publishers SHOULD fall back to version 1 if a target core declines version 2 with `UNSUPPORTED_ACK_VERSION` — but this means losing the ciphertext-availability binding for that ACK. Acceptable degradation during rollout. +9. **Pre-registration staging defaults.** §5.1.1 proposes 6h per-chunk TTL, 1 MB/min and 50 MB/h per-wallet rate limits, 100 MB per-CG-ID cap, and a per-core aggregate budget of "a few GB." These are starting points, not measured optima. Two things to settle as the network scales: (a) whether the per-CG and per-wallet caps need to grow proportionally to network capacity, or whether the per-core aggregate budget is the only really load-bearing limit; (b) whether the TTL should be uniform or risk-tiered (e.g. shorter TTL for new wallets with no on-chain history). Recommendation: ship the defaults above for v1, instrument staging-budget utilisation per core, and tune the next time we touch this surface. +10. **Per-CG sub-sharding (future direction).** Today the sharding-table assignment for any CG is "every core in the table" (§5.1.1, [SPEC_CG_MEMORY_MODEL.md §4.6](./SPEC_CG_MEMORY_MODEL.md#46-hosting)); cores host every CG. As the network grows this won't scale and a per-CG sub-assignment will be needed. The RFC is forward-compatible — the `amIInTheAssignmentFor(cgId)` check is written as a deterministic-function abstraction so a future sub-sharding scheme can refine it without changing this protocol. Not in scope here. +11. **Curated-vs-public pricing differential (tokenomics calibration).** Per §5.4.1, curated CGs declare ciphertext `byteSize` for hosting cost, which means they pay AEAD-overhead more per plaintext bit than equivalent public CGs (small in practice — see overhead table). This is honest and self-correcting (curators can amortise via larger chunks) and requires no contract change. Open question for the tokenomics layer: do we want an explicit `stakeWeightedAverageAsk` modifier that compensates / discounts curated hosting differently from public, or do we leave the ciphertext-byteSize convention as the only differential and let the market sort it out? Recommendation: leave it alone for v1; revisit when there is real adoption data on the curated/public mix and operator economics. + +--- + +## 9. Out of scope + +This RFC does not address: + +- **Re-encryption on membership revocation** (forward secrecy). Curated CG operators who need it must do explicit re-encrypt + republish. A future RFC could spec proxy re-encryption or per-leaf access policies. +- **Marketplace pricing / payment rails** for monetized access grants. This RFC enables the protocol-level grant; the commercial layer (NFT mints, subscriptions, micropayments) is a separate spec. +- **TEE-based hosting** (cores hold plaintext inside enclaves with policy enforcement). Alternative privacy model with a different trust profile; not pursued here. +- **Threshold cryptography for keys** (no single curator wallet holds the only copy). Useful for high-assurance CGs; can layer on top without contradicting this design. +- **Per-fact access policies** beyond per-CG / per-epoch. Each member sees the whole CG today, will continue to under this RFC. Per-fact ACLs are a much bigger spec. + +--- + +## 10. Economic context (deferred to tokenomics spec) + +This RFC's hosting-and-membership decoupling is a *mechanism* change. It enables — but does not specify — a richer economic surface than the network has today. The actual numbers (prices, reward shares, quota calibration) belong in [SPEC_PART2_ECONOMY.md](../SPEC_PART2_ECONOMY.md) or a successor tokenomics spec. This section names the dynamic the mechanism enables and identifies what a tokenomics spec needs to bind to. + +### 10.1 Two compatible dynamics + +The RFC enables two effects that are simultaneously true and in tension only at the calibration layer: + +- **Network-growth dynamic** (the freemium tier widens the funnel). Pre-registration staging (§5.1.1) lets users prove a CG works for their use case before committing TRAC to chain registration. Lower entry friction → more CGs tried → more conversions → more cores worth running → more network adoption. The cost to the network is bounded (TTL + per-core aggregate budget) and is borne by cores as a known operating expense — funnel maintenance, not unmanaged liability. +- **Token-sink dynamic** (the paid tier locks TRAC). Once a CG converts from staged to registered, TRAC is locked across the four staged stages (table below). The more value a CG delivers, the more it advances along the conversion gradient and the more TRAC the curator commits. Core operators capture this value via the existing reward distribution; the network captures it via reduced circulating supply. + +Both true. The calibration problem is choosing quota / pricing values such that the funnel converts at a rate that's net-positive for cores' economics — neither too generous (cores eat unbounded cost), nor too restrictive (users abandon for free alternatives, and the funnel never produces conversions). + +### 10.2 The four-stage conversion gradient + +| Stage | What the curator does | TRAC flow | +|---|---|---| +| 0 | Drafts CG via WM only | None | +| 1 | Shares CG via SWM; cores stage ciphertext under §5.1.1 | None — cores eat the staging cost as funnel maintenance | +| 2 | Registers CG on chain | Registration fees + retention bonds; **TRAC locked** | +| 3 | Publishes to VM (per-batch) | Per-batch publish fees flow to cores via existing reward mechanism; **TRAC locked + redistributed** | +| 4 | Issues `PaidAccessGrant`s (§5.7) for monetization | Buyer → curator x402 settlement; **TRAC velocity through monetization rails** | + +Stages 0-1 are the freemium tier. Stages 2-4 are the paid tier with increasing TRAC commitment and increasing core-operator revenue. + +### 10.3 Operator-level market dynamics + +The §5.1.1 quotas are *protocol limits* (the most permissive defaults the protocol allows), not *protocol floors* (what cores must offer). Each operator chooses their freemium-vs-paid stance locally: + +- A core that wants to maximise its position in the conversion funnel runs generous quotas (high TTL, large per-CG caps, large per-core aggregate budget). It accepts higher unpaid cost in exchange for higher exposure to conversion-driven revenue. +- A core that wants to minimise unpaid hosting runs tight quotas. It accepts lower funnel exposure in exchange for lower freemium cost. + +Each individual core operator's choice is asymmetric: when a CG it staged converts to registered, that core captures revenue from hosting it going forward. When a CG it staged never converts, the cost is sunk. So cores have a direct individual interest in serving CG-types that convert well, and the operator market will discover the right shape (perhaps using usage analytics, perhaps using simple heuristics, perhaps just by aggregate budget tuning) without protocol intervention. + +The protocol's role is to provide the levers and reasonable defaults — not to mandate any operator's economic strategy. + +### 10.4 What this RFC settles vs. what tokenomics must settle + +| Concern | Where it's settled | +|---|---| +| Pre-registration staging mechanism (TTL, quota gates, promotion) | This RFC §5.1.1 | +| ACK protocol and `byteSize` semantics (ciphertext for curated, plaintext for public) | This RFC §5.4.1 | +| Per-assertion payload-key wrap for monetization | This RFC §5.7 | +| Quota *default values* (6h TTL, 100 MB cap, etc.) | This RFC §5.1.1 — first-cut defaults | +| Quota *calibration over time* | Tokenomics spec — measurement-driven | +| Pricing per ciphertext byte / per epoch | Tokenomics spec — current `stakeWeightedAverageAsk` formula in `KnowledgeAssetsV10._validateTokenAmount` | +| Whether staging itself can be priced (e.g. TRAC bond to raise per-CG quotas above default) | Tokenomics spec — not addressed here | +| Whether cores can advertise per-CG-class staging policies in sharding-table metadata | Tokenomics spec — not addressed here | +| Conversion-incentive mechanisms (e.g. discounts for first registration, churn rewards for cores with high conversion ratios) | Tokenomics spec — not addressed here | +| Reward formulas for cores hosting unregistered staged CGs | Tokenomics spec — current default is "zero, funnel-maintenance only" | + +### 10.5 Open economic questions surfaced by this RFC + +These are not blocking for shipping Phase A; they are inputs to whoever next touches tokenomics: + +1. **Staging-quota calibration**. What conversion ratio do the §5.1.1 defaults produce in practice? At what conversion ratio do cores' economics break even on freemium? Recommendation: ship the defaults; instrument staging-budget utilisation and conversion-rate per core; tune the next time tokenomics is touched. +2. **Curated-vs-public pricing differential**. Per §8.11 — let the ciphertext-byteSize convention be the only differential, or add an explicit modifier? Recommendation: leave alone for v1. +3. **Tiered staging policy**. Should cores be able to offer differentiated staging tiers (e.g. a wallet with on-chain reputation history gets a 24h TTL; a fresh wallet gets 6h)? Recommendation: defer; current uniform TTL is a workable starting point. +4. **Monetization fee shape**. `PaidAccessGrant` is an off-chain envelope today (§5.7); the x402 payment lives outside this RFC's protocol surface. Tokenomics spec needs to decide whether some fraction of paid-access revenue flows to the cores hosting the underlying ciphertext, or whether cores are only compensated via the registered-CG reward stream. + +--- + +## 11. Glossary + +| Term | Meaning | +|---|---| +| **Hosting** | The substrate role: storing encrypted bytes for a CG without holding the keys to read them. A property of a node, governed by the sharding table. | +| **Membership** | The semantic role: holding the per-CG chain key and being able to decrypt CG content. A property of an agent, governed by the curator. | +| **Sharding table** | The on-chain registry that assigns cores to CGs for hosting and ACK duties. See [SPEC_CG_MEMORY_MODEL.md §4.6](./SPEC_CG_MEMORY_MODEL.md#46-hosting). | +| **Chain key** | The 32-byte symmetric key shared by all members of a given CG-epoch, used to derive per-message payload keys via HKDF-SHA-256. Ratchets via HMAC-SHA-256 per broadcast message, giving per-message forward secrecy within an epoch. Cores never see it. | +| **Payload key** | A per-message key derived as `HKDF(chainKey_n, info=SWM_SENDER_KEY_PAYLOAD_KEY_PURPOSE, length=32)`. Decrypts exactly one broadcast message. One-way derivable from `chainKey_n + index`; HKDF is irreversible. | +| **Setup package** | The per-recipient X25519-wrapped envelope (AES-256-GCM) that delivers a new chain key to one member at epoch initialization or membership join. One package per recipient per epoch. Point-to-point, never held by cores. | +| **Broadcast message** | The single shared ciphertext fanned out via gossip per SWM write — AES-256-GCM under a payload key derived from the current chain key. Every member decrypts the same bytes; cores store the same bytes. Per-assertion granularity (1 ciphertext = 1 assertion = N triples). | +| **Ciphertext chunk** | A single AEAD-encrypted broadcast message — the unit of substrate-tier replication and the unit of `SWMCatchupRequest` pagination. | +| **Epoch** | A version of the chain key. Rotated on membership revocation or curator's discretion; not rotated when a new member joins mid-epoch (§5.5.3). | +| **KeyGrant** | A curator-signed envelope (secp256k1/EIP-191) delivering a chain key to a new or returning member, with explicit `startMessageIndex` and `BackfillScope` (§5.5.1). | +| **KeyRotate** | A curator-signed envelope announcing a new epoch to remaining members after a revocation or scheduled rotation (§5.5.2). Has explicit pending/partial/complete/failed states (§5.5.5). | +| **BackfillScope** | An explicit artifact in `KeyGrant` declaring which historical epochs and batches the grantee is entitled to access (§5.5.4). | +| **PaidAccessGrant** | A curator-signed envelope delivering one or more per-assertion payload keys to a paying buyer, enabling fine-grained monetization without exposing the epoch chain key (§5.7). | +| **Member attestation token** | A member-signed envelope vouching for a specific `(assertion, ciphertextChunkDigest)` binding at a specific batch leaf. Used by outsiders to verify against on-chain anchors when they don't hold the chain key themselves (§5.3.2). | +| **Hosting core** | A core node that the sharding table has assigned to a given CG. Holds broadcast-layer ciphertext, ACKs publishes, serves `SWMCatchupRequest`. | +| **Member edge** | An edge node whose agent is on the CG's allowlist and holds the current chain key. | +| **Bridge core** | A deployment pattern: one operator runs a node that the sharding table has assigned to host a CG AND, on the same machine, a member-agent process whose wallet holds CG membership. The node hosts encrypted bytes; the co-located agent holds the chain key and decrypts. Used in scenario 3 for integrating CG content into external tools like Obsidian/Teams/Google Docs without a network hop between the storage and the decrypter. | +| **Outsider** | Any node / agent not currently a member. May become one via KeyGrant or PaidAccessGrant, retroactively gaining access to historical ciphertext via hosting cores. | + +--- + +## 12. Alternatives considered + +Two distinct alternatives were considered during this RFC's development. Both rejected. + +### 11.1 Inline-ciphertext ACK (only solves the publish symptom) + +A simpler alternative would close the original VM-publish symptom without any substrate-subscription change: + +**Shape**. At publish time, the edge proposer attaches the full ciphertext batch to the ACK request sent to a small set of cores. Cores verify ciphertext-side, sign the commitment, discard the ciphertext after publish confirmation. No SWM subscription gating change; no chain-field addition. + +**Why rejected — fails deployment shapes in §2.4**: + +| Property | Inline-ciphertext ACK | RFC-38 (substrate subscription) | +|---|---|---| +| Publish-time bandwidth from edge | O(batch size) — synchronous push to N cores | O(commitments) — ciphertext pre-positioned via gossip | +| Behavior on flaky edge connections | Publishes fail or stall mid-push; retry forces re-push of the same payload | Ciphertext already on cores via gossip; ACK request is tiny | +| Edge resync after offline window | Requires another member peer to be online | Cores always-on; resync works regardless of peer state | +| Bridge core integration (scenario 3) | Impossible — cores hold no curated ciphertext | Native — cores hold ciphertext, bridge agent decrypts on same node | +| Monetized late-joiner (scenario 4) | Impossible — cores have nothing to serve historically | Native — cores serve ciphertext, curator grants payload key (§5.7) | +| Implementation footprint | ~100 LOC (ACK collector + responder) | ~800 LOC across substrate, publisher, agent (Phase A only) | + +**The trade is the substrate**. Inline-ciphertext is cheap to build but only solves one symptom and bakes "all member nodes must be online for availability" into the operational model. RFC-38 is more work but matches the deployment patterns in §2.4. + +**Hybrid note**. Phase A's ACK protocol allows an optional `ChunkPullRequest` fallback (§5.4.3) for the rare case where a core hasn't yet received the ciphertext via gossip (e.g. just became a sharding-table member for this CG). This makes inline-ciphertext-push a degenerate special case of RFC-38, not an excluded alternative — so the cheap path is available when it's appropriate. + +### 11.2 Dual-root commitment (considered in v2, dropped in v3) + +An earlier draft of this RFC (v2, in PR #113) proposed a dual-root leaf format `leaf_i = H( H(salt ‖ pt_i) ‖ H(ct_i) )` plus an on-chain `saltCommitment` field, framed as enabling outsiders to verify a leaked triple against the on-chain root WITHOUT trusting any member. + +**Why considered**. The intuition is appealing: the chain holds both the plaintext-binding and the ciphertext-binding for each leaf, so cores can ACK ciphertext availability and outsiders can later check plaintext-binding using just the salt + on-chain root. + +**Why rejected (PR #113 review surfaced two structural issues)**: + +1. **The construction does not bind plaintext to ciphertext.** The publisher chooses BOTH halves of the leaf independently — there is no key the chain has access to that could check `decrypt(ct_i) == pt_i`. A malicious publisher can commit `(H(salt ‖ pt_A), H(ct_B))` and the merkle math validates. An outsider "verifying" the dual-root learns: "the publisher committed this pair." They do not learn: "this plaintext is the actual decryption of this ciphertext." Members catch the mismatch only post-decrypt — same detection point we have WITHOUT the dual root. So the dual-root delivers no verifier capability that wasn't already available via member post-decrypt verification. +2. **Epoch-salt disclosure broadens privacy regression.** Revealing `salt_epoch` to one verifier (to let them check one leaked triple) exposes the salt's blinding for EVERY plaintext-hash in that epoch. Low-entropy plaintexts in the entire epoch become brute-forceable to that verifier. Per-leaf nonces would help but require complex distribution and revocation. + +**Why we did not patch (1)**. The only way to bind plaintext↔ciphertext on chain is either (a) a member co-signature gathered at publish time and anchored on chain, or (b) a zero-knowledge proof. Both are heavyweight. Neither was needed for the four motivating scenarios — in all of them the verification consumer is in a trust relationship with a member (current member, bridge operator, paid subscriber, journalist with attribution). The honest construction is "member attests; outsider verifies the attestation" (§5.3.2), not "chain attests to a binding it cannot verify." + +**What this RFC kept from the dual-root exploration**: the substrate subscription split (§5.1), the ciphertext-availability ACK shape (§5.4), the catch-up protocol (§5.6). Those were always the operational core. The dual-root construction was a verification overreach that v3 cleanly removes. + +**Future revisit (open question §8.5)**: if a use case emerges that genuinely requires trustless outsider verification at chain-resolution time (no reachable member), a Phase D could add member co-signatures at publish time and anchor them on chain. That gives the desired property without the dual-root's correspondence flaw. Not on the roadmap; documented so the design does not preclude it. + +--- + +## Appendix A — Sequence diagrams + +### A.1 Edge curator publishes facts to VM — detailed + +Single end-to-end flow from a curator writing a fact, through SWM staging, plaintext-root computation (unchanged from today), ACK collection with the new ciphertext-availability binding, and on-chain anchoring. Shows every actor and every message in detail. + +```mermaid +sequenceDiagram + autonumber + participant U as Curator agent (member) + participant E as Edge node (daemon) + participant G as SWM gossip substrate + participant CN as Hosting cores
(sharding-table) + participant CH as Chain (ContextGraphs.sol) + + Note over U,E: Pre-conditions:
• Chain key CK_e for current epoch is provisioned to all members via setup packages.
• Cores hold no keys, only ciphertext.
• Sharding-table cores are subscribed to the CG's substrate topic — this happens
whether or not the CG is on-chain-registered (§5.1.1 pre-registration staging).
• Staged chunks live under TTL regardless of registration. Only this VM publish
step pays for retention; the chunks it names get promoted from staged-under-TTL
to retained-for-epochs. + Note over CN: At t = 0 (CG creation), cores already accept SWM for this CG ID
(deterministic sharding function says they're in the assignment).
Registration on chain enables this publish step but does NOT by itself
promote staged chunks — only the publish below does. + + Note over U,CN: 1) Drafting and SWM staging + U->>E: write assertion A (plaintext) + E->>E: payloadKey_n = HKDF(CK_e, info, len=32)
ct_A = AEAD_encrypt(A, payloadKey_n, AAD) + E->>G: gossip(SWM, ct_A) on CG topic + G->>CN: deliver ct_A to hosting cores
(Phase A change: now includes curated CGs) + Note over CN: Cores index ct_A under (cg, batchId, msgIndex).
No key, no plaintext — opaque bytes. + + Note over U,E: 2) Building a VM batch (unchanged from today) + U->>E: publish [A_1, …, A_N] to VM + E->>E: leaf_i = H(A_i) per existing V10 leaf format
merkleRoot = merkleRoot(leaves) + E->>E: ciphertextChunksRoot = merkleRoot([H(ct_i) for each leaf]) + + Note over E,CN: 3) Collecting ACKs over availability + V10 fields + E->>CN: ACKRequest{cg, opId, merkleRoot,
kaCount, byteSize, epoch, chainId,
tokenAmount, retentionPeriod, publishAuthority,
ciphertextChunks[], ciphertextChunksRoot,
ackProtocolVersion: 2} + alt Core has all chunks (steady-state path) + Note over CN: Verify each H(ct_i) is locally indexed.
Recompute ciphertextChunksRoot.
Persist-before-sign: chunks already persisted via gossip.
Sign ackRequestDigest with secp256k1. + else Core is missing a chunk + CN->>E: ChunkPullRequest(missingIndices[]) + E->>CN: ciphertext bytes + CN->>CN: Verify H(ct) matches digest; durably persist. + end + CN-->>E: ACKResponse{ackRequestDigest, ackerIdentityId, ackerSignature} + Note over E: Collect until quorum reached
(= parametersStorage.minimumRequiredSignatures()) + + Note over E,CH: 4) Anchoring on chain (existing V10 flow) + E->>CH: createKnowledgeAssetsV10(
cg, merkleRoot, batchMetadata, ackSignatures[]) + Note over CH: Verify each ACK signer ∈ sharding table.
Verify publisher per publishPolicy.
Verify quorum.
NOTE: no new chain fields vs today. + CH-->>E: KnowledgeCollectionCreated event + + par durable hosting (already persisted from step 3) + CN->>CN: Serve ct_i to authenticated requesters
via SWMCatchupRequest (§5.6). + and member notification + G->>U: New batch announced on CG topic + end +``` + +What each phase guarantees: + +| Phase | Privacy invariant | Availability invariant | +|---|---|---| +| 1 — Stage | Cores see ciphertext only; gossip carries no keys | Cores have indexed ciphertext under (cg, batchId, msgIndex) before any ACK is requested | +| 2 — Build | Plaintext stays on curator's node | Merkle root computed over plaintext leaves (unchanged from today's V10) | +| 3 — ACK | Cores never need plaintext to sign | `ackRequestDigest` binds plaintext root AND ciphertext-availability — replay-protected and authority-pinned. Persist-before-sign enforces durable hosting. | +| 4 — Anchor | Chain stores plaintext root + V10 metadata only — no ciphertext, no keys | Members later verify post-decrypt; outsiders verify via member attestations (§5.3.2) | + +### A.2 Outsider verifies an assertion via member attestation + +```mermaid +sequenceDiagram + autonumber + participant O as Outsider (no chain key) + participant CH as Chain + participant CN as Any hosting core + participant M as Member of CG
(curator or any granted member) + + Note over O,M: Pre-condition: O somehow obtained assertion A
(leak, public quote, paid purchase per §5.7).
O needs a verifiable trust path to the on-chain anchor. + + O->>M: Request attestation for (assertion A, claimed cg, batchId, leafIndex i) + Note over M: M decrypts ct_i locally (using their chain key).
Computes plaintextLeafHash = H(decrypted assertion).
If matches H(A), proceeds; else rejects. + M-->>O: attestationToken{
plaintextLeafHash = H(A),
ciphertextChunkDigest = H(ct_i),
attesterAgentAddress,
attesterMembershipEpoch,
attesterSignature (secp256k1) } + + O->>O: Compute H(A); verify equals attestationToken.plaintextLeafHash + O->>CN: SWMCatchupRequest{cg, sinceEpochId, sinceMessageIndex} — fetch ct_i + CN-->>O: ciphertext_chunk_i + O->>O: Verify H(received) equals attestationToken.ciphertextChunkDigest + + O->>CH: Read batch entry for (cg, batchId): get merkleRoot + CH-->>O: merkleRoot, batch metadata + O->>O: Verify merkle path from H(A) to merkleRoot + + O->>CH: Resolve attesterAgentAddress: was it a member of cg at attesterMembershipEpoch?
(via SPEC_V10_IDENTITY_AND_ACCESS — existing chain lookups) + CH-->>O: Membership history confirms attester was authorised + O->>O: Verify attestationToken.attesterSignature + + Note over O: All checks pass →
Trust chain: O → on-chain-resolvable member → on-chain anchor.
O learned the assertion is genuinely in the CG.
O did NOT need the chain key.
Malicious attester is publicly attributable and slashable
per existing V10 reputation/authority model. +``` + +### A.3 Member edge resyncs SWM from cores after going offline + +```mermaid +sequenceDiagram + autonumber + participant E as Member edge
(reconnecting) + participant CN as Any hosting core + participant P as Other member peer
(possibly offline) + + Note over E: Has chain key for current epoch + last seen (epochId, msgIndex). + + E->>CN: SWMCatchupRequest(cg, sinceEpochId, sinceMessageIndex,
requesterIdentityId, requesterSignature) + Note over CN: Verify requester is a member
(chain lookup, §5.6.4 auth).
Apply rate limit. + CN-->>E: SWMCatchupResponse(pageMessages[],
nextPageCursor, serverSignature) + + loop until nextPageCursor is empty + E->>CN: SWMCatchupRequest(... pageCursor) + CN-->>E: next page + end + + E->>E: Decrypt each ciphertext with payload key
derived from chain key + index.
Re-derive plaintext root for each completed batch;
compare to on-chain anchor (§5.3.1). + Note over E: Catch-up complete.
No member peer needed to be online —
cores are the always-on availability substrate. +``` + +### A.4 Late joiner gets the CG data — full backfill + +A late joiner is an outsider just granted membership (by invite, payment via `PaidAccessGrant`, or ad-hoc share). Unlike A.3 (where a known member returns with keys already in hand), the late joiner starts from zero — they need both the keys and the historical data. They use the same hosting cores as the rest of the network: cores serve encrypted bytes to authenticated requesters; the chain key handles decryption authorization. + +```mermaid +sequenceDiagram + autonumber + participant LJ as Late joiner agent + participant LJN as Late joiner's node + participant C as Curator + participant CN as Hosting core (any) + participant CH as Chain + participant G as SWM gossip substrate + + Note over LJ,C: 1) Grant — invite, payment, or ad-hoc share + LJ->>C: Request access (off-chain or via x402 payment) + C->>C: Add LJ wallet to allowlist (_meta) + C->>LJN: KeyGrant{cg, epochId, recipientAgentAddress,
setupPackage{chainKey, startMessageIndex},
BackfillScope{grantedEpochs[], futureGrantsScope},
curatorSignature (secp256k1)} + Note over LJN: Verify curator signature against on-chain publishAuthority.
BackfillScope tells LJ exactly which epochs/batches they can read.
For "future-only": startMessageIndex = current, no historical keys.
For "full historical": startMessageIndex = 0 for prior epochs included. + + Note over LJ,CH: 2) Backfill VM history (only for batches in BackfillScope.grantedEpochs) + LJN->>CH: List batches for cg → [(batchId, merkleRoot, epoch)…] + loop for each batch the joiner has keys for + LJN->>CN: SWMCatchupRequest(cg, sinceEpochId, sinceMessageIndex,
requesterSignature OR bearerToken) + CN-->>LJN: SWMCatchupResponse(pageMessages[], serverSignature) + LJN->>LJN: Decrypt each ciphertext with payload key derived from chain key.
Recompute plaintext merkle root for the batch.
Compare to on-chain merkleRoot. + Note over LJN: Verify pass → import assertions into local store.
Verify fail → reject batch; flag serving core via §6.5 abuse signals.
Joiner can retry from a different hosting core. + end + + Note over LJ,G: 3) Catch up live SWM since the latest VM batch + LJN->>CN: SWMCatchupRequest(cg, sinceEpochId = currentEpoch, sinceMessageIndex) + CN-->>LJN: stream of encrypted SWM messages + LJN->>LJN: Decrypt with currentEpoch chain key
Replay into local SWM store + LJN->>G: Subscribe to CG gossip topic (live updates begin) + + Note over LJN: Late joiner now has the full picture
(within BackfillScope) + live updates.
Forward-only grants give a clean "joined-from-here" semantics. +``` + +Two properties worth highlighting in this flow: + +1. **Cores are the single uniform data source.** Whether the joiner is new, returning, or a long-standing member, the bytes come from the same hosting cores. The cores authenticate requesters (§5.6.4) and rate-limit, but the authorization-to-decrypt is handled entirely off-substrate via the chain key. +2. **BackfillScope is the curator's explicit lever.** By choosing which prior-epoch chain keys to include in the `KeyGrant` (and at what `startMessageIndex`), the curator declaratively scopes what the new member can read. This gives a natural "trial member" / "full member" / "subscriber" tiering as a key-distribution policy, not a protocol change. diff --git a/docs/specs/SPEC_CG_MEMORY_MODEL.md b/docs/specs/SPEC_CG_MEMORY_MODEL.md index f40678abf..9903c9240 100644 --- a/docs/specs/SPEC_CG_MEMORY_MODEL.md +++ b/docs/specs/SPEC_CG_MEMORY_MODEL.md @@ -390,6 +390,7 @@ This RFC does not address: - **Marketplace-style host selection** (curator pays specific nodes for hosting). Bigger spec; not on the roadmap right now. - **Custom hosting committees** for high-assurance CGs that want a fixed M-of-N. The contract no longer supports per-CG hosting committees (see §6.1); a future high-assurance use case would need a separate spec and contract addition. - **Off-chain access enforcement for invite-only CGs at the libp2p layer** — see §7.5. +- **Decoupled hosting and membership for curated CGs** — cores hosting encrypted bytes for CGs they're not member of, enabling edge curators to publish curated content to VM, outsiders to verify leaked / monetized triples, and edges to resync from always-on cores. Tracked as [SPEC_CG_HOSTING_MEMBERSHIP.md](./SPEC_CG_HOSTING_MEMBERSHIP.md). --- diff --git a/packages/agent/src/dkg-agent.ts b/packages/agent/src/dkg-agent.ts index d7acb1e7f..2cebd3fc4 100644 --- a/packages/agent/src/dkg-agent.ts +++ b/packages/agent/src/dkg-agent.ts @@ -65,6 +65,7 @@ import { type MessageIdempotencyStore, type ProtocolOutboxStore, type ProtocolOutboxEntry, + encryptV10PublishPayload, } from '@origintrail-official/dkg-core'; import { GraphManager, PrivateContentStore, createTripleStore, type TripleStore, type TripleStoreConfig, type Quad, type LargeLiteralStorageConfig } from '@origintrail-official/dkg-storage'; import { EVMChainAdapter, NoChainAdapter, enrichEvmError, type EVMAdapterConfig, type ChainAdapter, type CreateContextGraphParams, type CreateOnChainContextGraphParams, type CreateOnChainContextGraphResult, type TxResult, type V10PublishingConvictionAccountInfo } from '@origintrail-official/dkg-chain'; @@ -6379,6 +6380,18 @@ export class DKGAgent { } } + // OT-RFC-38 / LU-5 — curated CG ACK payloads ship as AEAD ciphertext + // (see _resolveEncryptInlinePayload jsdoc for the chainKey resolution). + // Direct `publish()` (non-SWM path) needs the same protection — without + // it cores would still see plaintext for curated direct-publish, which + // defeats the point. PublishOpts here doesn't carry an explicit + // authorAgentAddress, so we let _resolveEncryptInlinePayload fall back + // to `defaultAgentAddress ?? peerId`. + const encryptInlinePayload = await this._resolveEncryptInlinePayload( + contextGraphId, + opts?.subGraphName, + ); + const result = await this.publisher.publish({ contextGraphId, quads, @@ -6392,6 +6405,7 @@ export class DKGAgent { v10ACKProvider, publishContextGraphId: onChainId ?? undefined, precomputedAttestation, + encryptInlinePayload, }); onPhase?.('broadcast', 'start'); @@ -7222,6 +7236,66 @@ export class DKGAgent { * publisher's `expectedMerkleRoot mismatch` error rather than a * silent wrong-content publish. */ + /** + * OT-RFC-38 / LU-5 — produce an inline-payload AEAD callback for + * curated CGs so cores receive opaque ciphertext instead of + * plaintext nquads. Returns `undefined` for public CGs (the + * publisher then keeps its existing plaintext-inline behaviour). + * + * Keyed via the publisher's swm-sender-key send-state `chainKey` + * snapshot: every CG member who holds this key (delivered via + * setup packages + ratchet steps) can recompute the same payload + * key and decrypt later when LU-7 catchup / LU-8 verification + * lands. If the publisher hasn't bootstrapped a send-state yet + * (e.g. publish before any SWM write), we fall back to + * `undefined` and let the publisher's existing path apply — for + * a curated CG that means the cores will decline with + * NO_DATA_IN_SWM (same observable as today, the §1.1 bug). The + * agent surfaces a warn so operators see the configuration miss. + */ + private async _resolveEncryptInlinePayload( + contextGraphId: string, + subGraphName?: string, + authorAgentAddress?: string, + ): Promise<((plaintext: Uint8Array) => Promise) | undefined> { + const ctx = createOperationContext('publish'); + let isCurated = false; + try { + isCurated = await this.isPrivateContextGraph(contextGraphId); + } catch (err) { + this.log.warn(ctx, `_resolveEncryptInlinePayload: isPrivateContextGraph(${contextGraphId}) failed — treating as public: ${err instanceof Error ? err.message : String(err)}`); + return undefined; + } + if (!isCurated) return undefined; + + const senderAddress = authorAgentAddress + ?? this.defaultAgentAddress + ?? this.peerId; + const stateKey = swmSenderStateKey(contextGraphId, subGraphName, senderAddress); + const state = this.swmSenderKeySendStates.get(stateKey); + if (!state) { + this.log.warn( + ctx, + `LU-5: curated CG ${contextGraphId} (sender=${senderAddress}${subGraphName ? `/${subGraphName}` : ''}) ` + + `has no swm-sender-key send state — cannot encrypt inline payload. ` + + `The publisher will fall through to its default path; cores without curated SWM subscriptions ` + + `(pre-LU-6) will decline with NO_DATA_IN_SWM. Establish the send state by writing at least one ` + + `SWM share to this CG before publishing to VM.`, + ); + return undefined; + } + + const chainKey = state.chainKey; + const cgId = contextGraphId; + return async (plaintextNquads: Uint8Array): Promise => { + return encryptV10PublishPayload({ + chainKey, + contextGraphId: cgId, + plaintext: plaintextNquads, + }); + }; + } + private async _loadSelectedSWMQuads( contextGraphId: string, selection: 'all' | { rootEntities: string[] }, @@ -7523,6 +7597,20 @@ export class DKGAgent { } } + // OT-RFC-38 / LU-5 — for curated CGs (any private flavour: peer + // allowlist OR agent allowlist), wrap the inline ACK payload with + // AEAD using the publisher's swm-sender-key chainKey so cores hold + // opaque bytes, not plaintext. Public CGs leave this undefined and + // continue with the existing plaintext-inline path. + const encryptInlinePayload = await this._resolveEncryptInlinePayload( + contextGraphId, + options?.subGraphName, + options?.authorAgentAddress, + ); + if (encryptInlinePayload) { + this.log.info(ctx, `LU-5: curated CG ${contextGraphId} — wrapping inline ACK payload with chain-key AEAD`); + } + const result = await this.publisher.publishFromSharedMemory(contextGraphId, selection, { operationCtx: ctx, clearSharedMemoryAfter: options?.clearSharedMemoryAfter, @@ -7534,6 +7622,7 @@ export class DKGAgent { subGraphName: options?.subGraphName, publisherNodeIdentityIdOverride: options?.publisherNodeIdentityIdOverride, precomputedAttestation: resolvedSeal, + encryptInlinePayload, }); if (result.status === 'confirmed' && result.onChainResult) { @@ -11757,16 +11846,6 @@ export class DKGAgent { const proposerEligible = this.identityId > 0n && await probeShardingTableMembership(this.identityId); - // CG-visible peer set for verify-proposal fan-out: - // SPEC_CG_MEMORY_MODEL §4.4 — the verify proposal payload includes - // contextGraphId, verifiedMemoryId, batchId, and root entities; for - // curated CGs that's metadata only allowlisted members may see. Run - // every recipient through `getOrCreateCGMemberEnumerator()` so - // unrelated peers don't receive the proposal. Public CGs fall - // through to the topic-subscribers set; legacy CGs with no member - // signal return an empty set (verify just degrades to no-quorum). - const cgMemberEnumerator = this.getOrCreateCGMemberEnumerator(); - const collector = new VerifyCollector({ // rc.9 PR-11: route through messenger.sendReliable so // /dkg/10.0.1/verify-proposal gets envelope wrap + sender-side @@ -11781,10 +11860,24 @@ export class DKGAgent { } return sendResult.response; }, - getParticipantPeers: async (cgId?: string) => { - const targetCg = cgId ?? opts.contextGraphId; - const enumeration = await cgMemberEnumerator.enumerate(targetCg); - return enumeration.members; + // TODO(SPEC_CG_MEMORY_MODEL §4.3 follow-up): fan out only to + // connected peers that are sharding-table-eligible cores. Doing + // that needs a peerId→identityId mapping (via on-chain Profile + // scan or a libp2p node-info exchange) we don't have a cheap + // path for yet. For now we send to all connected peers; the + // downstream `probeShardingTableMembership` filter on each + // received approval still ensures only sharding-table-member + // ACKs count toward quorum, so the worst case is wasted bytes + // on non-eligible peers (proposal payload is the only thing + // leaked: contextGraphId / verifiedMemoryId / batchId are + // already discoverable on-chain once the CG is registered; the + // residual concern is root-entity URIs on curated CGs). The + // round-5 attempt to use the CG agent-allowlist conflated + // agent membership with hosting membership and broke verify + // entirely for curated CGs (cores aren't agents in the + // allowlist) — revert kept here as Codex round-5 follow-up. + getParticipantPeers: () => { + return this.node.libp2p.getPeers().map(p => p.toString()).filter(id => id !== this.peerId); }, log: (msg: string) => this.log.info(ctx, msg), }); @@ -14822,6 +14915,7 @@ export class DKGAgent { swmGraphId: string | undefined, subGraphName: string | undefined, merkleLeafCount: number, + isEncryptedPayload?: boolean, ) => { // Fail loud on non-numeric or non-positive CG ids: V10 publish requires // a real on-chain context graph and the contract rejects `cgId == 0` @@ -14873,7 +14967,7 @@ export class DKGAgent { contextGraphIdStr: contextGraphId, publisherPeerId: this.peerId, publicByteSize, - isPrivate: false, + isPrivate: isEncryptedPayload === true, kaCount, rootEntities, chainId: chainIdBig, @@ -14885,6 +14979,7 @@ export class DKGAgent { swmGraphId, subGraphName, merkleLeafCount, + isEncryptedPayload, }); return result.acks; }; diff --git a/packages/cli/src/publisher-runner.ts b/packages/cli/src/publisher-runner.ts index f836ed86c..7ec4f3033 100644 --- a/packages/cli/src/publisher-runner.ts +++ b/packages/cli/src/publisher-runner.ts @@ -344,6 +344,7 @@ function createV10ACKProviderForPublisher( swmGraphId, subGraphName, merkleLeafCount, + isEncryptedPayload, ) => { // Fail loud on non-numeric or non-positive CG ids. V10 publish requires // a real on-chain context graph; `ZeroContextGraphId` at @@ -388,7 +389,7 @@ function createV10ACKProviderForPublisher( contextGraphIdStr: contextGraphId, publisherPeerId: transport.publisherPeerId, publicByteSize, - isPrivate: false, + isPrivate: isEncryptedPayload === true, kaCount, rootEntities, chainId: chainIdBig, @@ -400,6 +401,7 @@ function createV10ACKProviderForPublisher( swmGraphId, subGraphName, merkleLeafCount, + isEncryptedPayload, }); return result.acks; }; diff --git a/packages/core/src/crypto/index.ts b/packages/core/src/crypto/index.ts index 9f7311f91..34329db09 100644 --- a/packages/core/src/crypto/index.ts +++ b/packages/core/src/crypto/index.ts @@ -89,4 +89,13 @@ export { type SwmSenderKeyMessageCryptResult, } from './swm-sender-key.js'; +export { + encryptV10PublishPayload, + decryptV10PublishPayload, + isEncryptedV10PublishPayload, + V10_PUBLISH_PAYLOAD_MAGIC, + type EncryptV10PublishPayloadInput, + type DecryptV10PublishPayloadInput, +} from './v10-publish-payload.js'; + export { resolveRootEntities, type Quad as RootEntityQuad } from './root-entity.js'; diff --git a/packages/core/src/crypto/v10-publish-payload.ts b/packages/core/src/crypto/v10-publish-payload.ts new file mode 100644 index 000000000..2371147c3 --- /dev/null +++ b/packages/core/src/crypto/v10-publish-payload.ts @@ -0,0 +1,128 @@ +/** + * OT-RFC-38 / LU-5 — encrypted V10 publish payload for curated CGs. + * + * The minimal-viable encryption layer cores wrap around inline + * publish-intent bytes so storage-attestation can sign on ciphertext + * the cores cannot decrypt. Keyed via the publisher's swm-sender-key + * `chainKey` snapshot: all members of the curated CG who hold this + * chainKey (delivered via the setup package + intermediate ratchet + * steps) can recompute the same payload key and decrypt later. + * + * Scheme (intentionally simple — full key lifecycle / per-message + * ratchet integration arrives with LU-6 substrate split + LU-8 + * member post-decrypt verification): + * + * - Payload key = HKDF-SHA256(chainKey, salt='', info=`dkg.v10-publish-payload-key.v1|${cgId}`) + * - Nonce = 12 random bytes (per encryption call) + * - Cipher = AES-256-GCM + * - Auth tag = 16 bytes appended by GCM + * - Wire layout = [4-byte LE magic 'V10P'] [12-byte nonce] [ciphertext || tag] + * + * The magic prefix lets future versions distinguish encrypted-payload + * wire shapes without an explicit version field on the protobuf side. + * + * Limitation tracked for LU-8: a member who is behind the publisher's + * chain-key ratchet won't yet hold the right `chainKey` snapshot. + * They must catch up to the publisher's current SWM state (LU-7) to + * derive the same key. For the §1.1 unblocker that's acceptable — + * the curator and members are roughly in sync at publish time. + * + * Cores receiving the ciphertext do NOT attempt to decrypt. They sign + * the V10 ACK digest verbatim against the publisher's claimed + * merkleRoot/byteSize; the merkle-root verification happens at + * member side post-decryption (LU-8). + */ + +import { createCipheriv, createDecipheriv, hkdfSync, randomBytes } from 'node:crypto'; + +export const V10_PUBLISH_PAYLOAD_MAGIC = new TextEncoder().encode('V10P'); +const NONCE_BYTES = 12; +const AUTH_TAG_BYTES = 16; +const KEY_BYTES = 32; +const HKDF_INFO_PREFIX = 'dkg.v10-publish-payload-key.v1|'; + +function derivePayloadKey(chainKey: Uint8Array, contextGraphId: string): Uint8Array { + if (chainKey.length !== KEY_BYTES) { + throw new Error( + `v10-publish-payload: chainKey must be ${KEY_BYTES} bytes (got ${chainKey.length})`, + ); + } + const info = new TextEncoder().encode(HKDF_INFO_PREFIX + contextGraphId); + return new Uint8Array( + hkdfSync('sha256', Buffer.from(chainKey), Buffer.alloc(0), info, KEY_BYTES) as ArrayBuffer, + ); +} + +export interface EncryptV10PublishPayloadInput { + chainKey: Uint8Array; + contextGraphId: string; + plaintext: Uint8Array; + /** Test seam. Defaults to `crypto.randomBytes(12)`. */ + nonce?: Uint8Array; +} + +export function encryptV10PublishPayload(input: EncryptV10PublishPayloadInput): Uint8Array { + const key = derivePayloadKey(input.chainKey, input.contextGraphId); + const nonce = input.nonce ?? new Uint8Array(randomBytes(NONCE_BYTES)); + if (nonce.length !== NONCE_BYTES) { + throw new Error(`v10-publish-payload: nonce must be ${NONCE_BYTES} bytes (got ${nonce.length})`); + } + const cipher = createCipheriv('aes-256-gcm', Buffer.from(key), Buffer.from(nonce)); + const encrypted = Buffer.concat([ + cipher.update(Buffer.from(input.plaintext)), + cipher.final(), + ]); + const tag = cipher.getAuthTag(); + // Layout: [4 magic] [12 nonce] [ciphertext] [16 tag] + const out = new Uint8Array(V10_PUBLISH_PAYLOAD_MAGIC.length + nonce.length + encrypted.length + tag.length); + out.set(V10_PUBLISH_PAYLOAD_MAGIC, 0); + out.set(nonce, V10_PUBLISH_PAYLOAD_MAGIC.length); + out.set(encrypted, V10_PUBLISH_PAYLOAD_MAGIC.length + nonce.length); + out.set(tag, V10_PUBLISH_PAYLOAD_MAGIC.length + nonce.length + encrypted.length); + return out; +} + +export interface DecryptV10PublishPayloadInput { + chainKey: Uint8Array; + contextGraphId: string; + encryptedPayload: Uint8Array; +} + +export function decryptV10PublishPayload(input: DecryptV10PublishPayloadInput): Uint8Array { + const buf = input.encryptedPayload; + const headerLen = V10_PUBLISH_PAYLOAD_MAGIC.length + NONCE_BYTES; + if (buf.length < headerLen + AUTH_TAG_BYTES) { + throw new Error( + `v10-publish-payload: ciphertext too short (got ${buf.length}, need >= ${headerLen + AUTH_TAG_BYTES})`, + ); + } + for (let i = 0; i < V10_PUBLISH_PAYLOAD_MAGIC.length; i++) { + if (buf[i] !== V10_PUBLISH_PAYLOAD_MAGIC[i]) { + throw new Error('v10-publish-payload: magic prefix mismatch — not an encrypted v10 publish payload'); + } + } + const nonce = buf.slice(V10_PUBLISH_PAYLOAD_MAGIC.length, headerLen); + const ciphertextEnd = buf.length - AUTH_TAG_BYTES; + const ciphertext = buf.slice(headerLen, ciphertextEnd); + const tag = buf.slice(ciphertextEnd); + const key = derivePayloadKey(input.chainKey, input.contextGraphId); + const decipher = createDecipheriv('aes-256-gcm', Buffer.from(key), Buffer.from(nonce)); + decipher.setAuthTag(Buffer.from(tag)); + const plaintext = Buffer.concat([ + decipher.update(Buffer.from(ciphertext)), + decipher.final(), + ]); + return new Uint8Array(plaintext); +} + +/** + * Test/debug helper — returns true iff `buf` carries the + * v10-publish-payload magic prefix. + */ +export function isEncryptedV10PublishPayload(buf: Uint8Array): boolean { + if (buf.length < V10_PUBLISH_PAYLOAD_MAGIC.length) return false; + for (let i = 0; i < V10_PUBLISH_PAYLOAD_MAGIC.length; i++) { + if (buf[i] !== V10_PUBLISH_PAYLOAD_MAGIC[i]) return false; + } + return true; +} diff --git a/packages/core/src/proto/publish-intent.ts b/packages/core/src/proto/publish-intent.ts index e361515e9..a8bbd8ad6 100644 --- a/packages/core/src/proto/publish-intent.ts +++ b/packages/core/src/proto/publish-intent.ts @@ -44,7 +44,17 @@ export const PublishIntentSchema = new Type('PublishIntent') .add(new Field('swmGraphId', 11, 'string')) .add(new Field('subGraphName', 12, 'string')) /** V10 flat-KC Merkle leaf count (sorted + deduped); must match ACK digest + on-chain KC. */ - .add(new Field('merkleLeafCount', 13, 'uint32')); + .add(new Field('merkleLeafCount', 13, 'uint32')) + // OT-RFC-38 / LU-5: when `true`, `stagingQuads` carries opaque ciphertext + // bytes (AEAD-encrypted nquads keyed via the CG's swm-sender chain key) + // and the receiver MUST NOT attempt N-Quad parsing or merkle-root + // recomputation against it. Cores cannot decrypt curated payloads, so + // they sign the V10 digest based on the publisher's claimed merkle root + // and ciphertext byte size; member post-decrypt verification (LU-8) + // catches any mismatch between the on-chain root and the actual + // plaintext. Field number 14 to keep the proto strictly additive — old + // senders never set it (default `false`), old receivers ignore it. + .add(new Field('isEncryptedPayload', 14, 'bool')); type Long = { low: number; high: number; unsigned: boolean }; @@ -77,6 +87,20 @@ export interface PublishIntentMsg { subGraphName?: string; /** V10 flat-KC Merkle leaf count after sort+dedupe (see `V10MerkleTree.leafCount`). */ merkleLeafCount?: number; + /** + * OT-RFC-38 / LU-5. When `true`, `stagingQuads` is opaque ciphertext + * (AEAD-encrypted nquads keyed via the CG's swm-sender chain key) and + * the receiver MUST treat it as a binary blob: no N-Quad parsing, no + * merkle-root recompute, no rootEntities cross-check. Cores still + * verify the ciphertext byte count against `publicByteSize` (so a + * misreported size doesn't slip through pricing) and sign the V10 + * digest the publisher claimed. Member post-decrypt verification + * (LU-8) catches plaintext-vs-on-chain-root mismatches. + * + * Defaults to `false`/undefined on the wire so the existing public-CG + * flow that ships plaintext nquads inline keeps working unchanged. + */ + isEncryptedPayload?: boolean; } export function encodePublishIntent(msg: PublishIntentMsg): Uint8Array { diff --git a/packages/core/test/v10-publish-payload.test.ts b/packages/core/test/v10-publish-payload.test.ts new file mode 100644 index 000000000..b4ff1cfee --- /dev/null +++ b/packages/core/test/v10-publish-payload.test.ts @@ -0,0 +1,120 @@ +import { describe, it, expect } from 'vitest'; +import { randomBytes } from 'node:crypto'; +import { + encryptV10PublishPayload, + decryptV10PublishPayload, + isEncryptedV10PublishPayload, + V10_PUBLISH_PAYLOAD_MAGIC, +} from '../src/index.js'; + +function rb(n: number): Uint8Array { + return new Uint8Array(randomBytes(n)); +} + +describe('v10-publish-payload', () => { + const chainKey = rb(32); + const cgId = '42'; + const plaintext = new TextEncoder().encode( + [ + ' .', + ' .', + ' .', + ].join('\n'), + ); + + it('round-trip recovers the plaintext exactly', () => { + const encrypted = encryptV10PublishPayload({ chainKey, contextGraphId: cgId, plaintext }); + expect(isEncryptedV10PublishPayload(encrypted)).toBe(true); + + const recovered = decryptV10PublishPayload({ + chainKey, + contextGraphId: cgId, + encryptedPayload: encrypted, + }); + expect(Buffer.from(recovered).equals(Buffer.from(plaintext))).toBe(true); + }); + + it('emits the V10P magic prefix and stable wire layout (magic | nonce | ct | tag)', () => { + const fixedNonce = new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]); + const encrypted = encryptV10PublishPayload({ + chainKey, + contextGraphId: cgId, + plaintext, + nonce: fixedNonce, + }); + // magic == 'V10P' + expect(encrypted.slice(0, 4)).toEqual(V10_PUBLISH_PAYLOAD_MAGIC); + // nonce echoed verbatim at offset 4 + expect(Array.from(encrypted.slice(4, 16))).toEqual(Array.from(fixedNonce)); + // remaining = ciphertext (plaintext.length bytes) + 16-byte GCM tag + expect(encrypted.length).toBe(4 + 12 + plaintext.length + 16); + }); + + it('different cgIds produce different ciphertexts (HKDF domain-separation)', () => { + const fixedNonce = new Uint8Array(12).fill(7); + const a = encryptV10PublishPayload({ chainKey, contextGraphId: '42', plaintext, nonce: fixedNonce }); + const b = encryptV10PublishPayload({ chainKey, contextGraphId: '43', plaintext, nonce: fixedNonce }); + expect(Buffer.from(a).equals(Buffer.from(b))).toBe(false); + }); + + it('decrypt rejects ciphertext encrypted under a different chainKey', () => { + const encrypted = encryptV10PublishPayload({ chainKey, contextGraphId: cgId, plaintext }); + expect(() => decryptV10PublishPayload({ + chainKey: rb(32), + contextGraphId: cgId, + encryptedPayload: encrypted, + })).toThrow(); + }); + + it('decrypt rejects ciphertext encrypted for a different cgId', () => { + const encrypted = encryptV10PublishPayload({ chainKey, contextGraphId: cgId, plaintext }); + expect(() => decryptV10PublishPayload({ + chainKey, + contextGraphId: '99', + encryptedPayload: encrypted, + })).toThrow(); + }); + + it('decrypt rejects truncated / corrupted ciphertexts', () => { + const encrypted = encryptV10PublishPayload({ chainKey, contextGraphId: cgId, plaintext }); + // Flip a byte in the ciphertext middle (AEAD tag verification should fail) + const corrupted = new Uint8Array(encrypted); + corrupted[20] ^= 0xff; + expect(() => decryptV10PublishPayload({ + chainKey, + contextGraphId: cgId, + encryptedPayload: corrupted, + })).toThrow(); + + // Truncate below header length + const truncated = encrypted.slice(0, 10); + expect(() => decryptV10PublishPayload({ + chainKey, + contextGraphId: cgId, + encryptedPayload: truncated, + })).toThrow(/too short/); + + // Wrong magic prefix + const wrongMagic = new Uint8Array(encrypted); + wrongMagic[0] = 0xaa; + expect(() => decryptV10PublishPayload({ + chainKey, + contextGraphId: cgId, + encryptedPayload: wrongMagic, + })).toThrow(/magic prefix mismatch/); + }); + + it('rejects chainKey of wrong length', () => { + expect(() => encryptV10PublishPayload({ + chainKey: rb(16), + contextGraphId: cgId, + plaintext, + })).toThrow(/chainKey must be 32 bytes/); + }); + + it('isEncryptedV10PublishPayload returns true only for magic-prefixed buffers', () => { + expect(isEncryptedV10PublishPayload(new Uint8Array([0x56, 0x31, 0x30, 0x50, 0xff]))).toBe(true); + expect(isEncryptedV10PublishPayload(new Uint8Array([0x00, 0x00, 0x00, 0x00]))).toBe(false); + expect(isEncryptedV10PublishPayload(new Uint8Array([0x56, 0x31, 0x30]))).toBe(false); + }); +}); diff --git a/packages/node-ui/src/ui/api.ts b/packages/node-ui/src/ui/api.ts index 76cf54486..4a36ccd0f 100644 --- a/packages/node-ui/src/ui/api.ts +++ b/packages/node-ui/src/ui/api.ts @@ -212,7 +212,7 @@ export async function createContextGraph( name: string, description?: string, opts?: { allowedAgents?: string[]; accessPolicy?: number; publishPolicy?: number }, -): Promise<{ created: string; uri: string }> { +): Promise<{ created: string; uri: string; registered?: boolean; onChainId?: string; registerError?: string }> { const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), 30_000); try { @@ -221,6 +221,12 @@ export async function createContextGraph( headers: { 'Content-Type': 'application/json', ...authHeaders() }, body: JSON.stringify({ id, name, description, + // UI primary "Create project" always wants an on-chain CG — + // otherwise the user can't publish to Verified Memory (the + // publish path requires an on-chain CG id). The modal already + // tells the user "Registering context graph on the network…", + // so the combined create+register flow matches the UX promise. + register: true, ...(opts?.allowedAgents ? { allowedAgents: opts.allowedAgents } : {}), ...(opts?.accessPolicy !== undefined ? { accessPolicy: opts.accessPolicy } : {}), ...(opts?.publishPolicy !== undefined ? { publishPolicy: opts.publishPolicy } : {}), @@ -231,7 +237,7 @@ export async function createContextGraph( const errBody = await res.json().catch(() => ({})); throw new Error((errBody as { error?: string })?.error ?? `HTTP ${res.status}`); } - return res.json() as Promise<{ created: string; uri: string }>; + return res.json() as Promise<{ created: string; uri: string; registered?: boolean; onChainId?: string; registerError?: string }>; } catch (err: any) { if (err.name === 'AbortError') { throw new Error('Creating project is taking longer than expected — it may still complete in the background. Refresh the page in a moment.'); diff --git a/packages/node-ui/src/ui/components/Modals/CreateProjectModal.tsx b/packages/node-ui/src/ui/components/Modals/CreateProjectModal.tsx index ba2a7298e..0b7db255f 100644 --- a/packages/node-ui/src/ui/components/Modals/CreateProjectModal.tsx +++ b/packages/node-ui/src/ui/components/Modals/CreateProjectModal.tsx @@ -109,6 +109,20 @@ export function CreateProjectModal({ open, onClose }: CreateProjectModalProps) { const result = await createContextGraph(cgId, trimmedName, description.trim() || undefined, opts); clearTimeout(slowTimer); + // Daemon returns 200 with `{ created, registered: false, registerError }` + // when the local create succeeded but the on-chain registration + // leg failed. Without surfacing this, the user would land on a + // CG that looks fine in the UI but can never publish to VM + // (publish-to-VM requires an on-chain CG id). Fail loud with + // an actionable hint pointing at the retry endpoint. + if (result.registered === false) { + throw new Error( + `Context graph created locally but on-chain registration failed: ${result.registerError ?? 'unknown error'}. ` + + `Verified Memory publishing requires an on-chain CG — retry registration from the project's Settings, ` + + `or POST /api/context-graph/register with {"id":"${cgId}"}.`, + ); + } + // Phase 7: install the chosen ontology into meta/project-ontology so // the agent has the project's predicate vocabulary and URI patterns // from turn #1. Both `community` (operator picked a starter) and diff --git a/packages/node-ui/src/ui/views/MemoryLayerView.tsx b/packages/node-ui/src/ui/views/MemoryLayerView.tsx index fa2d9a1d2..396da42e4 100644 --- a/packages/node-ui/src/ui/views/MemoryLayerView.tsx +++ b/packages/node-ui/src/ui/views/MemoryLayerView.tsx @@ -636,27 +636,47 @@ function PublishPanel({ contextGraphId, onPublished }: { contextGraphId: string; ))} - {publishResult && ( -
-
Published to Verifiable Memory
-
-
Knowledge Collection: {publishResult.kcId}
-
Status: {publishResult.status}
- {publishResult.kas?.length > 0 && ( -
Knowledge Assets: {publishResult.kas.length}
- )} - {publishResult.txHash && ( -
- Tx:{' '} - {publishResult.txHash.slice(0, 10)}...{publishResult.txHash.slice(-8)} - {publishResult.blockNumber != null && ( - (block {publishResult.blockNumber}) - )} + {publishResult && (() => { + const confirmed = publishResult.status === 'confirmed' && !!publishResult.txHash; + return ( +
+
+ {confirmed + ? 'Published to Verified Memory' + : 'NOT published to Verified Memory'} +
+ {!confirmed && ( +
+ The data was prepared and stored locally as {publishResult.status}, + but the on-chain transaction did not land. Verified Memory requires a confirmed + KCCreated event. Check the node logs for the cause (typically a + missing publisher wallet, an unfunded signer, or a chain adapter that isn't V10-ready).
)} +
+
Knowledge Collection: {publishResult.kcId}
+
Status: {publishResult.status}
+ {publishResult.kas?.length > 0 && ( +
Knowledge Assets: {publishResult.kas.length}
+ )} + {publishResult.txHash ? ( +
+ Tx hash:{' '} + {publishResult.txHash} + {publishResult.blockNumber != null && ( + (block {publishResult.blockNumber}) + )} +
+ ) : ( +
+ Tx hash:{' '} + (none — no on-chain transaction) +
+ )} +
-
- )} + ); + })()} {error && (
{error}
diff --git a/packages/node-ui/src/ui/views/project/components.tsx b/packages/node-ui/src/ui/views/project/components.tsx index 517d3a0cc..4b371f4a7 100644 --- a/packages/node-ui/src/ui/views/project/components.tsx +++ b/packages/node-ui/src/ui/views/project/components.tsx @@ -2542,34 +2542,47 @@ export function VerifyOnDkgButton({
)} - {result && resultKind === 'publish' && isPublishResult(result) && ( -
-
- Status - ✓ {result.status} -
- {result.txHash && ( + {result && resultKind === 'publish' && isPublishResult(result) && (() => { + // OT-RFC-38 §1.1 — a publish without a TX hash never made it to chain. + // Treat that as failure, not success, so the curator knows the data + // is NOT in Verified Memory. + const confirmed = result.status === 'confirmed' && !!result.txHash; + return ( +
- TX hash - - {result.txHash.slice(0, 10)}…{result.txHash.slice(-6)} + Status + + {confirmed ? '✓' : '✕'} {result.status}{confirmed ? '' : ' (NOT on-chain)'}
- )} - {result.blockNumber != null && ( -
- Block - #{result.blockNumber} -
- )} - {result.kas?.[0]?.tokenId && ( -
- Token - #{result.kas[0].tokenId} -
- )} -
- )} + {result.txHash ? ( +
+ TX hash + + {result.txHash} + +
+ ) : ( +
+ TX hash + none — on-chain submission skipped +
+ )} + {result.blockNumber != null && ( +
+ Block + #{result.blockNumber} +
+ )} + {result.kas?.[0]?.tokenId && ( +
+ Token + #{result.kas[0].tokenId} +
+ )} +
+ ); + })()} ); } diff --git a/packages/node-ui/vite.config.ts b/packages/node-ui/vite.config.ts index 19fe94fba..85a8aa65c 100644 --- a/packages/node-ui/vite.config.ts +++ b/packages/node-ui/vite.config.ts @@ -12,12 +12,19 @@ function readTokenFile(path: string): string { } function readDkgConfig() { - // Devnet node 1 takes priority (local Hardhat chain with real contracts) - const devnetNode1 = resolve(__dirname, '../../.devnet/node1'); - if (existsSync(join(devnetNode1, 'api.port'))) { - const port = parseInt(readFileSync(join(devnetNode1, 'api.port'), 'utf-8').trim(), 10) || 9201; - const token = readTokenFile(join(devnetNode1, 'auth.token')); - console.log(`[vite] Using devnet node 1 on port ${port}`); + // Devnet takes priority over ~/.dkg. By default we target node1 (matches + // pre-LU-5 behaviour). The `scripts/devnet.sh ui start` wrapper already + // exports DEVNET_NODE=$UI_NODE_ID, so set UI_NODE_ID=5 before invoking + // the script (or set DEVNET_NODE directly when running `pnpm dev:ui` + // standalone) to point the UI at a different daemon — required to test + // the edge-curator flow from the UI without hand-rolling curl against + // the edge node's daemon port. + const devnetNodeNum = process.env.DEVNET_NODE || '1'; + const devnetDir = resolve(__dirname, '../../.devnet', `node${devnetNodeNum}`); + if (existsSync(join(devnetDir, 'api.port'))) { + const port = parseInt(readFileSync(join(devnetDir, 'api.port'), 'utf-8').trim(), 10) || 9201; + const token = readTokenFile(join(devnetDir, 'auth.token')); + console.log(`[vite] Using devnet node${devnetNodeNum} on port ${port}`); return { port, token }; } diff --git a/packages/publisher/src/ack-collector.ts b/packages/publisher/src/ack-collector.ts index 37c58e130..0a4a0898f 100644 --- a/packages/publisher/src/ack-collector.ts +++ b/packages/publisher/src/ack-collector.ts @@ -88,6 +88,14 @@ export class ACKCollector { subGraphName?: string; /** V10 flat-KC Merkle leaf count (sorted + deduped); binds StorageACK to on-chain RandomSampling. */ merkleLeafCount: number; + /** + * OT-RFC-38 / LU-5. When `true`, `stagingQuads` is opaque AEAD ciphertext + * (curated-CG payload). Cores skip N-Quad parsing and merkle-root + * recompute; verify only that `stagingQuads.length === publicByteSize`. + * Defaults to `false` so the existing public-CG inline-quads path is + * unchanged. + */ + isEncryptedPayload?: boolean; }): Promise { const { merkleRoot, contextGraphId, contextGraphIdStr, @@ -124,6 +132,7 @@ export class ACKCollector { : undefined, subGraphName: params.subGraphName, merkleLeafCount: params.merkleLeafCount, + isEncryptedPayload: params.isEncryptedPayload === true ? true : undefined, }; const intentBytes = encodePublishIntent(p2pMsg); diff --git a/packages/publisher/src/dkg-publisher.ts b/packages/publisher/src/dkg-publisher.ts index 222dd50ce..e40c551ca 100644 --- a/packages/publisher/src/dkg-publisher.ts +++ b/packages/publisher/src/dkg-publisher.ts @@ -1148,6 +1148,13 @@ export class DKGPublisher implements Publisher { * on-chain publishes. */ precomputedAttestation?: PublishOptions['precomputedAttestation']; + /** + * OT-RFC-38 / LU-5. When set, the inline ACK payload is AEAD- + * encrypted before being shipped to cores so curated-CG bytes + * are opaque on the wire. See `PublishOptions.encryptInlinePayload` + * for the full semantics. + */ + encryptInlinePayload?: PublishOptions['encryptInlinePayload']; }, ): Promise { const ctx = options?.operationCtx ?? createOperationContext('publishFromSWM'); @@ -1264,6 +1271,7 @@ export class DKGPublisher implements Publisher { subGraphName: options?.subGraphName, publisherNodeIdentityIdOverride: options?.publisherNodeIdentityIdOverride, precomputedAttestation: options?.precomputedAttestation, + encryptInlinePayload: options?.encryptInlinePayload, [INTERNAL_ORIGIN_TOKEN]: true, }; const publishResult = await this.publish(internalPublishOptions); @@ -1615,18 +1623,23 @@ export class DKGPublisher implements Publisher { // Descriptive SWM graph names stay on the existing tentative/mock path. } // Per-publish attribution override (RFC-001 §4): see PublishOptions - // docstring. `hasAttributionOverride` must influence the EARLY on-chain - // attempt gate too — otherwise `publisherSigner` and `tokenAmount` - // never resolve when a daemon with persistent identity `0n` carries an - // explicit override (including `0n` for mode (d) no-attribution), - // and the late gate would then enter the chain branch with - // `publisherSigner === undefined` and throw `PublisherWalletRequiredError`. - // Self-ACK signing remains tied to `this.publisherNodeIdentityId > 0n` - // below — the override controls on-chain attribution, not who signs. + // docstring. `hasAttributionOverride` lets a caller force a specific + // attribution value (including `0n` for the contract's no-attribution + // mode); without an override we fall through to whatever this + // publisher's persistent V10 identity is (which is `0n` for daemons + // that never registered a Profile — e.g. edge nodes). + // + // OT-RFC-38 §1.1 — edge agents that haven't (and won't) register an + // on-chain Profile MUST still be able to publish curated CGs to VM. + // `attributionIdentityId === 0n` is the contract's no-attribution mode + // and `KnowledgeAssetsV10.publishKnowledgeCollections` accepts it, so + // we no longer gate the on-chain attempt on identity presence. The + // `publisherSigner` still resolves from the chain adapter regardless + // of profile state — see `inferAdapterPublisherAddress`. If no signer + // can be resolved at all, the on-chain branch throws + // `PublisherWalletRequiredError` instead of silently degrading. const hasAttributionOverride = options.publisherNodeIdentityIdOverride !== undefined; - const willAttemptOnChainPublish = - (this.publisherNodeIdentityId > 0n || hasAttributionOverride) && - publisherContextGraphId !== undefined; + const willAttemptOnChainPublish = publisherContextGraphId !== undefined; const chainV10Ready = await this.refreshChainV10Readiness(); const canResolveOnChainPublisher = willAttemptOnChainPublish && chainV10Ready; const resolvedPublisherAddress = canResolveOnChainPublisher @@ -1770,9 +1783,31 @@ export class DKGPublisher implements Publisher { // recompute private merkle roots from SWM data alone. const hasPrivateData = privateRoots.length > 0; const isPublishFromSharedMemory = !!options.fromSharedMemory; - const stagingQuads = isPublishFromSharedMemory - ? undefined - : new TextEncoder().encode(nquadsStr); + // OT-RFC-38 / LU-5: when an encryptInlinePayload hook is wired (curated + // CGs only — DKGAgent resolves this from accessPolicy), ALWAYS send the + // payload inline as AEAD ciphertext, regardless of `fromSharedMemory`. + // Cores can't decrypt and they're not subscribed to curated SWM yet + // (substrate split lands in LU-6), so SWM-lookup would always decline + // with NO_DATA_IN_SWM — the exact bug §1.1 surfaces. Public CGs keep + // the existing behaviour: `fromSharedMemory` → cores look up SWM + // locally; otherwise plaintext inline. + const useEncryptedInline = typeof options.encryptInlinePayload === 'function'; + let stagingQuads: Uint8Array | undefined; + let stagingByteSize = publicByteSize; + if (useEncryptedInline) { + const plaintextBytes = new TextEncoder().encode(nquadsStr); + const ciphertext = await options.encryptInlinePayload!(plaintextBytes); + stagingQuads = ciphertext instanceof Uint8Array ? ciphertext : new Uint8Array(ciphertext); + // For curated CGs the publisher PAYS for ciphertext bytes (cores + // sign that into the V10 digest). Override publicByteSize for the + // ACK collection branch below; the chain TX still uses the + // ciphertext byte size as `byteSize` since that's what's signed. + stagingByteSize = BigInt(stagingQuads.length); + } else { + stagingQuads = isPublishFromSharedMemory + ? undefined + : new TextEncoder().encode(nquadsStr); + } // Pre-compute tokenAmount and epochs so they can be included in the // H5-prefixed publish ACK digest (incl. merkleLeafCount) — matches @@ -1823,12 +1858,17 @@ export class DKGPublisher implements Publisher { ); } } + // LU-5: pricing follows the byteSize that gets signed into the V10 + // digest. For curated (encrypted-inline) publishes that's the + // ciphertext byte count; for public publishes it stays as plaintext + // bytes. Single source of truth so ACK pricing == chain tx pricing. + const effectiveByteSize = useEncryptedInline ? stagingByteSize : publicByteSize; let precomputedTokenAmount = 0n; if (canAttemptOnChainPublish && typeof this.chain.getRequiredPublishTokenAmount === 'function') { try { - precomputedTokenAmount = await this.chain.getRequiredPublishTokenAmount(publicByteSize, publishEpochs); + precomputedTokenAmount = await this.chain.getRequiredPublishTokenAmount(effectiveByteSize, publishEpochs); if (precomputedTokenAmount <= 0n) { - this.log.warn(ctx, `getRequiredPublishTokenAmount returned ${precomputedTokenAmount} for byteSize=${publicByteSize} — using 1n as minimum`); + this.log.warn(ctx, `getRequiredPublishTokenAmount returned ${precomputedTokenAmount} for byteSize=${effectiveByteSize} — using 1n as minimum`); precomputedTokenAmount = 1n; } } catch (err) { @@ -1895,11 +1935,16 @@ export class DKGPublisher implements Publisher { onPhase?.('collect_v10_acks', 'start'); try { const rootEntities = manifestEntries.map(m => m.rootEntity); + // LU-5: for curated CGs the publisher pays / signs against the + // ciphertext byte size (`effectiveByteSize`). For public CGs + // nothing changed — `effectiveByteSize === publicByteSize`. v10ACKs = await options.v10ACKProvider( - kcMerkleRoot, v10CgDomain, kaCount, rootEntities, publicByteSize, stagingQuads, + kcMerkleRoot, v10CgDomain, kaCount, rootEntities, + effectiveByteSize, stagingQuads, publishEpochs, precomputedTokenAmount, swmGraphId, options.subGraphName, kcMerkleLeafCount, + useEncryptedInline, ); this.log.info(ctx, `V10: Collected ${v10ACKs.length} core node ACKs`); } catch (err) { @@ -1967,7 +2012,7 @@ export class DKGPublisher implements Publisher { v10CgId, kcMerkleRoot, BigInt(kaCount), - publicByteSize, + effectiveByteSize, BigInt(publishEpochs), precomputedTokenAmount, BigInt(kcMerkleLeafCount), @@ -2017,13 +2062,12 @@ export class DKGPublisher implements Publisher { : this.publisherNodeIdentityId; let usedV10Path = false; - // Gate: skip on-chain only when there's no usable attribution AND no - // explicit override. With an explicit override (including `0n`), we - // proceed on-chain; the contract validates non-zero values name a - // real sharding-table node and accepts `0n` as no-attribution. - if (!hasAttributionOverride && this.publisherNodeIdentityId === 0n) { - this.log.warn(ctx, `Identity not set (0) — skipping on-chain publish`); - } else if (publisherContextGraphId === undefined) { + // Gate: skip on-chain only when we can't resolve a target CG id or + // the chain adapter doesn't advertise V10 readiness. The earlier + // identity-zero gate is gone (OT-RFC-38 §1.1) — edge agents without + // a Profile publish in no-attribution mode (`attributionIdentityId + // === 0n`), which the contract accepts. + if (publisherContextGraphId === undefined) { this.log.warn(ctx, `No positive on-chain context graph id resolved from "${v10CgDomain}" — skipping on-chain publish`); } else if (!chainV10Ready) { this.log.warn(ctx, 'Chain adapter is not V10-ready — skipping on-chain publish'); @@ -2133,7 +2177,7 @@ export class DKGPublisher implements Publisher { signStarted = false; onPhase?.('chain:submit', 'start'); submitStarted = true; - this.log.info(ctx, `Submitting V10 on-chain publish tx (${kaCount} KAs, publicByteSize=${publicByteSize}, tokenAmount=${tokenAmount})`); + this.log.info(ctx, `Submitting V10 on-chain publish tx (${kaCount} KAs, byteSize=${effectiveByteSize}${useEncryptedInline ? ' [ciphertext]' : ''}, tokenAmount=${tokenAmount})`); if (!v10ACKs || v10ACKs.length === 0) { throw new Error('V10 ACKs required for on-chain publish — no ACKs collected'); @@ -2242,7 +2286,7 @@ export class DKGPublisher implements Publisher { publisherAddress: publisherSigner.address, merkleRoot: kcMerkleRoot, knowledgeAssetsAmount: kaCount, - byteSize: publicByteSize, + byteSize: effectiveByteSize, // PCA strict-equality: must match the value committed to the // ACK digest above (`computePublishACKDigest` at line ~1908) // so the on-chain ECDSA recovery yields the same operator diff --git a/packages/publisher/src/publisher.ts b/packages/publisher/src/publisher.ts index 0937bd932..5c87be23b 100644 --- a/packages/publisher/src/publisher.ts +++ b/packages/publisher/src/publisher.ts @@ -60,6 +60,17 @@ export type V10ACKProvider = ( subGraphName: string | undefined, /** V10 flat-KC Merkle leaf count (sorted + deduped); binds ACK + on-chain KC to RandomSampling. */ merkleLeafCount: number, + /** + * OT-RFC-38 / LU-5 — when `true`, `stagingQuads` is opaque AEAD + * ciphertext (curated CG payload) and cores skip merkle-root + * recompute. The publisher's claimed `merkleRoot`, `kaCount`, and + * `merkleLeafCount` are signed verbatim into the V10 digest; member + * post-decrypt verification (LU-8) is what catches lies. Cores + * still verify `stagingQuads.length === publicByteSize` to keep + * pricing honest. Defaults to `false` so existing public-CG callers + * are unchanged. + */ + isEncryptedPayload?: boolean, ) => Promise; /** @@ -120,6 +131,25 @@ export interface PublishOptions { * verify against their local SWM copy (storage-attestation guarantee). */ fromSharedMemory?: boolean; + /** + * OT-RFC-38 / LU-5. When set, the publisher routes the inline ACK + * payload through this hook to produce AEAD ciphertext bytes that + * cores hold opaquely. The publisher will then send `stagingQuads = + * ciphertext` with `isEncryptedPayload: true` so cores skip + * merkle-root recompute and just sign the V10 digest the publisher + * claimed. Member post-decrypt verification (LU-8) catches plaintext + * mismatches; outsider attestation tokens (LU-9) cover third parties. + * + * `fromSharedMemory` is forced `true` when this hook is set — + * encrypted-payload mode and the "data is in SWM already" semantic + * coexist (curated CGs always read from SWM, then encrypt for the + * ACK trip). + * + * Resolved by the caller (DKGAgent) based on the CG's access + * policy. Public CGs leave this `undefined` and continue to ship + * plaintext nquads inline. + */ + encryptInlinePayload?: (plaintextNquads: Uint8Array) => Promise | Uint8Array; /** When true, the KC was created via V10 and updates should use the V10 path. */ v10Origin?: boolean; /** diff --git a/packages/publisher/src/storage-ack-handler.ts b/packages/publisher/src/storage-ack-handler.ts index 0fdb40e7b..7d8a3fa45 100644 --- a/packages/publisher/src/storage-ack-handler.ts +++ b/packages/publisher/src/storage-ack-handler.ts @@ -161,6 +161,147 @@ export class StorageACKHandler { let swmQuads: Quad[]; + // OT-RFC-38 / LU-5 encrypted-payload path. For curated CGs the publisher + // ships AEAD-encrypted nquad bytes inline so cores can store the + // ciphertext (durably enough to ACK the V10 publish) without ever + // holding plaintext. Cores can't decrypt → can't recompute the + // plaintext merkle root → MUST trust the publisher's `merkleRoot` and + // `merkleLeafCount` claims for the V10 ACK signature. Member + // post-decrypt verification (LU-8) catches plaintext-vs-on-chain-root + // mismatches; outsider attestation tokens (LU-9) let third parties + // verify after the fact. Cores DO verify `stagingQuads.length` matches + // `publicByteSize` so a misreported size can't slip past pricing. + if (intent.isEncryptedPayload === true) { + if (!intent.stagingQuads || intent.stagingQuads.length === 0) { + throw new Error( + 'PublishIntent.isEncryptedPayload=true but stagingQuads is empty — ' + + 'curated-CG ACK requires the ciphertext bytes inline (no SWM fallback path for opaque blobs)', + ); + } + const MAX_ENCRYPTED_BYTES = 4 * 1024 * 1024; + if (intent.stagingQuads.length > MAX_ENCRYPTED_BYTES) { + throw new Error( + `encrypted stagingQuads payload (${intent.stagingQuads.length} bytes) exceeds ` + + `${MAX_ENCRYPTED_BYTES} byte limit — rejecting request`, + ); + } + const claimedByteSize = typeof intent.publicByteSize === 'number' + ? intent.publicByteSize + : Number(intent.publicByteSize); + if (intent.stagingQuads.length !== claimedByteSize) { + throw new Error( + `encrypted payload byteSize mismatch: stagingQuads.length=${intent.stagingQuads.length} ` + + `but publicByteSize=${claimedByteSize}. For curated CGs publicByteSize MUST equal the ` + + `ciphertext byte count (cores price the publish off this number).`, + ); + } + + // Persist the opaque ciphertext to a scoped staging graph as a + // single binary literal so it survives long enough for the + // V10 chain TX to land and for LU-7 catchup to pull it. Stored + // under a stable predicate so LU-7's wire handler can locate it + // by (cgId, merkleRoot) without needing a new store API. + const stagingGraphUri = `${swmGraphUri}/staging-encrypted/${ethers.hexlify(merkleRoot).slice(2, 18)}`; + const ciphertextSubject = `${stagingGraphUri}/ciphertext`; + const ciphertextPredicate = 'urn:dkg:swm:v10-publish-ciphertext'; + // Base64 keeps the blob as a valid N3 literal without depending on + // the underlying triple-store accepting arbitrary binary. AES-GCM + // ciphertext is roughly the same size as plaintext + 16-byte tag, + // so the 33% base64 inflation stays well under the 4 MB cap above. + const ciphertextLiteral = `"${Buffer.from(intent.stagingQuads).toString('base64')}"`; + await this.store.dropGraph(stagingGraphUri); + await this.store.insert([{ + subject: ciphertextSubject, + predicate: ciphertextPredicate, + object: ciphertextLiteral, + graph: stagingGraphUri, + }]); + setTimeout(async () => { + try { await this.store.dropGraph(stagingGraphUri); } catch { /* ignore */ } + }, 10 * 60 * 1000); + + // Cores can't enumerate KAs from ciphertext — use the publisher's + // claimed counts for the V10 digest. Validate they're positive so + // an obviously malformed intent (kaCount=0) doesn't waste a sign. + if (!intent.kaCount || intent.kaCount <= 0) { + throw new Error( + `encrypted PublishIntent.kaCount must be positive; got ${intent.kaCount}`, + ); + } + const claimedLeafCount = intent.merkleLeafCount == null ? 0 : Number(intent.merkleLeafCount); + if (claimedLeafCount < 1) { + throw new Error( + `encrypted PublishIntent.merkleLeafCount must be a positive integer; got ${claimedLeafCount}`, + ); + } + + const intentEpochs = (typeof intent.epochs === 'number' && intent.epochs > 0) ? intent.epochs : 1; + const intentTokenAmount = intent.tokenAmountStr ? BigInt(intent.tokenAmountStr) : 0n; + + let contextGraphIdBigInt: bigint; + try { + contextGraphIdBigInt = BigInt(cgId); + } catch { + throw new Error( + `encrypted StorageACK: V10 publish requires a numeric on-chain context graph id; got '${cgId}'.`, + ); + } + if (contextGraphIdBigInt <= 0n) { + throw new Error( + `encrypted StorageACK: V10 publish requires a positive on-chain context graph id; got ${contextGraphIdBigInt}.`, + ); + } + + const digest = computePublishACKDigest( + this.config.chainId, + this.config.kav10Address, + contextGraphIdBigInt, + merkleRoot, + BigInt(intent.kaCount), + BigInt(claimedByteSize), + BigInt(intentEpochs), + intentTokenAmount, + BigInt(claimedLeafCount), + ); + + if (this.config.isSignerRegistered) { + let signerRegistered: boolean | undefined; + try { + signerRegistered = await this.config.isSignerRegistered(); + } catch (err) { + try { await this.config.onSignerRegistrationLookupFailed?.(err); } catch { /* swallow */ } + throw new Error('StorageACK signer registration lookup failed; refusing to sign'); + } + if (signerRegistered === false) { + try { await this.config.onSignerUnregistered?.(); } catch { /* swallow */ } + return this.encodeDecline( + cgId, + STORAGE_ACK_DECLINE_CODES.SIGNER_NOT_REGISTERED, + 'StorageACK signer is not confirmed on-chain as an operational wallet', + ); + } + } + + const signature = ethers.Signature.from( + await this.config.signerWallet.signMessage(digest), + ); + const MAX_UINT64 = (1n << 64n) - 1n; + if (this.config.nodeIdentityId > MAX_UINT64) { + throw new Error( + `nodeIdentityId ${this.config.nodeIdentityId} exceeds uint64 wire format`, + ); + } + return encodeStorageACK({ + merkleRoot, + coreNodeSignatureR: ethers.getBytes(signature.r), + coreNodeSignatureVS: ethers.getBytes(signature.yParityAndS), + contextGraphId: cgId, + nodeIdentityId: this.config.nodeIdentityId <= BigInt(Number.MAX_SAFE_INTEGER) + ? Number(this.config.nodeIdentityId) + : { low: Number(this.config.nodeIdentityId & 0xFFFFFFFFn), high: Number((this.config.nodeIdentityId >> 32n) & 0xFFFFFFFFn), unsigned: true }, + }); + } + if (intent.stagingQuads && intent.stagingQuads.length > 0) { // Size limit: reject payloads over 4 MB to prevent memory exhaustion const MAX_STAGING_BYTES = 4 * 1024 * 1024; diff --git a/packages/publisher/test/phase-sequences.test.ts b/packages/publisher/test/phase-sequences.test.ts index 1b5e3098e..3d1de8d42 100644 --- a/packages/publisher/test/phase-sequences.test.ts +++ b/packages/publisher/test/phase-sequences.test.ts @@ -144,7 +144,16 @@ describe('Phase-sequence contracts', () => { // -- Publish (adapter-backed signer, no identity — tentative) ----------- - it('publish: adapter-backed signer without node identity returns tentative after local phases', async () => { + it('publish: adapter-backed signer without node identity attempts on-chain (OT-RFC-38 §1.1) but falls back to tentative when no ACKs are collected', async () => { + // Pre-RFC-38: an identity-less publisher short-circuited at `chain:start` + // and produced just `chain:start → chain:end`. Post-RFC-38: edge agents + // without an on-chain Profile still attempt the on-chain TX in + // no-attribution mode (the contract supports it). In this single-node + // test mode there's no v10ACKProvider AND self-ACK is gated on identity, + // so the publish reaches `chain:submit` then fails on + // "V10 ACKs required for on-chain publish — no ACKs collected" and falls + // back to tentative. The phase contract still pins the full chain prefix + // so regressions that re-introduce a silent short-circuit are caught. const store = new OxigraphStore(); const chain = createEVMAdapter(HARDHAT_KEYS.CORE_OP); const keypair = await generateEd25519Keypair(); @@ -183,6 +192,10 @@ describe('Phase-sequence contracts', () => { 'store:start', 'store:end', 'chain:start', + 'chain:sign:start', + 'chain:sign:end', + 'chain:submit:start', + 'chain:submit:end', 'chain:end', ]); }); diff --git a/packages/publisher/test/publisher-no-random-wallet.test.ts b/packages/publisher/test/publisher-no-random-wallet.test.ts index 4fcb9f2e4..08d70be31 100644 --- a/packages/publisher/test/publisher-no-random-wallet.test.ts +++ b/packages/publisher/test/publisher-no-random-wallet.test.ts @@ -453,7 +453,14 @@ describe('DKGPublisher: no random publisher wallet without explicit key', () => expect(result.ual).toMatch(/^did:dkg:evm:31337\/0x[0-9a-fA-F]{40}\/t/); }); - it('does not reserve adapter publisher signers for identity-less tentative publishes', async () => { + it('reserves adapter publisher signers for identity-less publishes (OT-RFC-38 §1.1 — no-attribution path)', async () => { + // Pre-RFC-38: identity=0 short-circuited to tentative and skipped the + // adapter reservation. Post-RFC-38: edge agents without an on-chain + // Profile MUST be able to publish in no-attribution mode (the contract + // accepts attributionId=0n), so the publisher resolves and reserves a + // chain signer just like any other on-chain publish. The downstream + // result depends on the chain adapter — here `ReservingPublishChain` + // accepts any publisher and returns a confirmed result. const keypair = await generateEd25519Keypair(); const chain = new ReservingPublishChain(new ethers.Wallet(TEST_KEY)); const publisher = new DKGPublisher({ @@ -467,17 +474,20 @@ describe('DKGPublisher: no random publisher wallet without explicit key', () => const result = await publisher.publish({ contextGraphId: '1', quads: [{ - subject: 'urn:test:evm-identityless-no-reserve', + subject: 'urn:test:evm-identityless-reserves', predicate: 'http://schema.org/name', - object: '"EvmIdentitylessNoReserve"', + object: '"EvmIdentitylessReserves"', graph: 'did:dkg:context-graph:1', }], }); - expect(result.status).toBe('tentative'); - expect(result.onChainResult).toBeUndefined(); - expect(chain.reservations).toBe(0); - expect(chain.capturedPublisherAddress).toBeUndefined(); + // The reservation is the load-bearing assertion — it proves the gate + // dropped and the publisher resolved a signer. The on-chain TX itself + // may still fall back to tentative in single-node mode (no v10ACKProvider + // + no self-ACK without identity → no ACKs → tx skipped), which is fine + // for this test: the gate change is what we're pinning. + expect(['confirmed', 'tentative']).toContain(result.status); + expect(chain.reservations).toBeGreaterThanOrEqual(1); }); it('keeps descriptive-CG chain-backed publishes tentative without a publisher signer', async () => { diff --git a/packages/publisher/test/storage-ack-handler.test.ts b/packages/publisher/test/storage-ack-handler.test.ts index 9af83cfa8..81d944e83 100644 --- a/packages/publisher/test/storage-ack-handler.test.ts +++ b/packages/publisher/test/storage-ack-handler.test.ts @@ -213,6 +213,161 @@ describe('StorageACKHandler', () => { expect(decoded.declineMessage).toContain('Merkle root mismatch'); }); + // OT-RFC-38 / LU-5 — encrypted-payload branch for curated CGs. + describe('isEncryptedPayload (curated CG path)', () => { + // Opaque AEAD ciphertext as far as the handler is concerned. The + // handler MUST NOT try to parse this as N-Quads. We use distinctive + // bytes so a mistakenly-applied parse path would obviously fail. + const ciphertextBytes = new Uint8Array([0x01, 0xff, 0x00, 0xab, 0xcd, 0xef, 0x12, 0x34, 0x56, 0x78]); + // The publisher's claimed plaintext merkle root. The handler MUST NOT + // recompute against the ciphertext — it just signs what was claimed. + const claimedRoot = ethers.getBytes(ethers.keccak256(new TextEncoder().encode('test-plaintext-root'))); + const claimedKaCount = 3; + const claimedLeafCount = 9; + const claimedEpochs = 2; + const claimedTokenAmountStr = '5000'; + + it('signs the V10 digest from publisher-claimed fields without parsing ciphertext', async () => { + const handler = await createHandler([]); + const intent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'curator-edge', + publicByteSize: ciphertextBytes.length, + isPrivate: true, + kaCount: claimedKaCount, + rootEntities: [], + stagingQuads: ciphertextBytes, + epochs: claimedEpochs, + tokenAmountStr: claimedTokenAmountStr, + merkleLeafCount: claimedLeafCount, + isEncryptedPayload: true, + }); + + const response = await handler.handler(intent, fakePeerId); + const ack = decodeStorageACK(response); + + expect(isStorageACKDecline(ack)).toBe(false); + const decodedRoot = ack.merkleRoot instanceof Uint8Array + ? ack.merkleRoot : new Uint8Array(ack.merkleRoot); + expect(Buffer.from(decodedRoot).equals(Buffer.from(claimedRoot))).toBe(true); + + const expectedDigest = computePublishACKDigest( + TEST_CHAIN_ID, + TEST_KAV10_ADDR, + cgIdBigInt, + claimedRoot, + BigInt(claimedKaCount), + BigInt(ciphertextBytes.length), + BigInt(claimedEpochs), + BigInt(claimedTokenAmountStr), + BigInt(claimedLeafCount), + ); + const prefixedHash = ethers.hashMessage(expectedDigest); + const recovered = ethers.recoverAddress(prefixedHash, { + r: ethers.hexlify(ack.coreNodeSignatureR instanceof Uint8Array + ? ack.coreNodeSignatureR : new Uint8Array(ack.coreNodeSignatureR)), + yParityAndS: ethers.hexlify(ack.coreNodeSignatureVS instanceof Uint8Array + ? ack.coreNodeSignatureVS : new Uint8Array(ack.coreNodeSignatureVS)), + }); + expect(recovered.toLowerCase()).toBe(coreWallet.address.toLowerCase()); + }); + + it('throws when ciphertext byteSize does not match publicByteSize (prevents pricing fraud)', async () => { + const handler = await createHandler([]); + const intent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'curator-edge', + publicByteSize: ciphertextBytes.length + 100, + isPrivate: true, + kaCount: claimedKaCount, + rootEntities: [], + stagingQuads: ciphertextBytes, + merkleLeafCount: claimedLeafCount, + isEncryptedPayload: true, + }); + await expect(handler.handler(intent, fakePeerId)).rejects.toThrow( + /encrypted payload byteSize mismatch/, + ); + }); + + it('throws when stagingQuads is missing (no SWM fallback for opaque blobs)', async () => { + const handler = await createHandler([]); + const intent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'curator-edge', + publicByteSize: 0, + isPrivate: true, + kaCount: claimedKaCount, + rootEntities: [], + merkleLeafCount: claimedLeafCount, + isEncryptedPayload: true, + }); + await expect(handler.handler(intent, fakePeerId)).rejects.toThrow( + /isEncryptedPayload=true but stagingQuads is empty/, + ); + }); + + it('throws when kaCount or merkleLeafCount is missing/zero (publisher must supply both)', async () => { + const handler = await createHandler([]); + const noKaCountIntent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'curator-edge', + publicByteSize: ciphertextBytes.length, + isPrivate: true, + kaCount: 0, + rootEntities: [], + stagingQuads: ciphertextBytes, + merkleLeafCount: claimedLeafCount, + isEncryptedPayload: true, + }); + await expect(handler.handler(noKaCountIntent, fakePeerId)).rejects.toThrow( + /encrypted PublishIntent.kaCount must be positive/, + ); + + const noLeafCountIntent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'curator-edge', + publicByteSize: ciphertextBytes.length, + isPrivate: true, + kaCount: claimedKaCount, + rootEntities: [], + stagingQuads: ciphertextBytes, + merkleLeafCount: 0, + isEncryptedPayload: true, + }); + await expect(handler.handler(noLeafCountIntent, fakePeerId)).rejects.toThrow( + /encrypted PublishIntent.merkleLeafCount must be a positive integer/, + ); + }); + + it('honours the signer-registration gate (declines instead of signing when key is unregistered)', async () => { + const handler = await createHandler([], { + isSignerRegistered: async () => false, + }); + const intent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'curator-edge', + publicByteSize: ciphertextBytes.length, + isPrivate: true, + kaCount: claimedKaCount, + rootEntities: [], + stagingQuads: ciphertextBytes, + merkleLeafCount: claimedLeafCount, + isEncryptedPayload: true, + }); + const response = await handler.handler(intent, fakePeerId); + const decoded = decodeStorageACK(response); + expect(isStorageACKDecline(decoded)).toBe(true); + expect(decoded.declineCode).toBe(STORAGE_ACK_DECLINE_CODES.SIGNER_NOT_REGISTERED); + }); + }); + it('rejects non-core node role', async () => { const store = new OxigraphStore(); const config: StorageACKHandlerConfig = { diff --git a/scripts/devnet-test-rfc38-lu5-public.sh b/scripts/devnet-test-rfc38-lu5-public.sh new file mode 100755 index 000000000..960d9db25 --- /dev/null +++ b/scripts/devnet-test-rfc38-lu5-public.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# +# OT-RFC-38 / LU-5 regression sweep — same flow but with a PUBLIC CG. +# Confirms the gate-drop in dkg-publisher.ts (OT-RFC-38 §1.1) didn't +# accidentally break the public-CG publish path (which already worked +# pre-RFC-38). +# +# Differences from devnet-test-rfc38-lu5.sh: +# - accessPolicy: 0 (public), publishPolicy: 1 (open) +# - no allowedAgents +# - edge log MUST NOT show the LU-5 chain-key AEAD wrap (public CG +# doesn't encrypt the inline payload) +# +# Same node API surface, same on-chain read-back. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +DEVNET_DIR="${DEVNET_DIR:-$REPO_ROOT/.devnet}" +HARDHAT_PORT="${HARDHAT_PORT:-8545}" +API_PORT_BASE=9201 +EDGE_CURATOR_NODE=5 + +CONTRACTS_JSON="$REPO_ROOT/packages/evm-module/deployments/localhost_contracts.json" +EVM_ABI_DIR="$REPO_ROOT/packages/evm-module/abi" + +log() { echo "[lu5-pub-validate] $*"; } +warn() { echo "[lu5-pub-validate] WARN: $*" >&2; } +fail() { echo "[lu5-pub-validate] FAIL: $*" >&2; exit 1; } + +node_dir() { echo "$DEVNET_DIR/node$1"; } +node_token() { tail -1 "$(node_dir "$1")/auth.token" 2>/dev/null | tr -d '\r\n'; } +node_port() { echo $((API_PORT_BASE + $1 - 1)); } +node_log() { echo "$(node_dir "$1")/daemon.log"; } + +api_call() { + local node="$1" method="$2" path="$3" data="${4:-}" + local port; port=$(node_port "$node") + local token; token=$(node_token "$node") + local -a curl_args=(-sS -X "$method" -H "Authorization: Bearer $token" -H 'Content-Type: application/json') + [ -n "$data" ] && curl_args+=(-d "$data") + curl_args+=("http://127.0.0.1:${port}${path}") + curl "${curl_args[@]}" +} + +CURATOR_IDENTITY=$(api_call "$EDGE_CURATOR_NODE" GET /api/agent/identity) +CURATOR_AGENT=$(printf '%s' "$CURATOR_IDENTITY" | node -e 'let d="";process.stdin.on("data",c=>d+=c);process.stdin.on("end",()=>console.log(JSON.parse(d).agentAddress))') + +STAMP=$(date +%s) +CG_LOCAL_ID="${CURATOR_AGENT}/lu5-public-${STAMP}" + +EDGE_BASELINE=$(wc -l < "$(node_log "$EDGE_CURATOR_NODE")" 2>/dev/null | tr -d ' ' || echo 0) + +log "Creating PUBLIC CG '$CG_LOCAL_ID' on node $EDGE_CURATOR_NODE..." +CREATE_RESP=$(api_call "$EDGE_CURATOR_NODE" POST /api/context-graph/create "$(cat <d+=c);process.stdin.on("end",()=>{const j=JSON.parse(d);if(!j.registered)process.exit(1);console.log(j.onChainId)})') \ + || fail "create+register failed: $CREATE_RESP" +log "Public CG onChainId=$ON_CHAIN_ID" + +log "Writing public quads to SWM..." +WRITE_RESP=$(api_call "$EDGE_CURATOR_NODE" POST /api/shared-memory/write "$(cat <d+=c);process.stdin.on('end',()=>{try{const j=JSON.parse(d);const v=j$2;console.log(v==null?'':v)}catch(e){process.exit(1)}})"; } + +STATUS=$(parse_json "$PUBLISH_RESP" ".status") +TX=$(parse_json "$PUBLISH_RESP" ".txHash") +KC=$(parse_json "$PUBLISH_RESP" ".kcId") + +[ "$STATUS" = "confirmed" ] || fail "expected status=confirmed, got '$STATUS'" +[[ "$TX" =~ ^0x[0-9a-fA-F]{64}$ ]] || fail "invalid txHash '$TX'" +log "✓ public publish landed: kcId=$KC tx=$TX" + +# Verify on-chain +( +cd "$REPO_ROOT/packages/evm-module" && \ +RPC_URL="http://127.0.0.1:${HARDHAT_PORT}" CONTRACTS_JSON="$CONTRACTS_JSON" ABI_DIR="$EVM_ABI_DIR" BATCH_ID="$KC" \ +node -e ' +const { ethers } = require("ethers"); +const fs = require("fs"); const path = require("path"); +(async () => { + const provider = new ethers.JsonRpcProvider(process.env.RPC_URL); + const contracts = JSON.parse(fs.readFileSync(process.env.CONTRACTS_JSON, "utf8")).contracts; + const kcs = new ethers.Contract(contracts.KnowledgeCollectionStorage.evmAddress, + JSON.parse(fs.readFileSync(path.join(process.env.ABI_DIR, "KnowledgeCollectionStorage.json"), "utf8")), provider); + const [merkleRoots, , minted, byteSize] = await kcs.getKnowledgeCollectionMetadata(BigInt(process.env.BATCH_ID)); + if (!merkleRoots || merkleRoots.length === 0) throw new Error("no merkleRoots"); + console.log("KC read-back OK: merkleRoots=" + merkleRoots.length + " byteSize=" + byteSize + " minted=" + minted); +})().catch(e => { console.error(e?.message || e); process.exit(1); }); +' +) || fail "KC read-back failed" + +# Critical regression guard: a public CG MUST NOT trigger the LU-5 +# chain-key AEAD wrap path. +EDGE_NEW=$(tail -n "+$((EDGE_BASELINE + 1))" "$(node_log "$EDGE_CURATOR_NODE")") +if printf '%s' "$EDGE_NEW" | grep -qE "LU-5: curated CG ${CG_LOCAL_ID//\//\\/} .* wrapping inline ACK payload"; then + fail "regression: public CG triggered LU-5 chain-key AEAD wrap (should ONLY fire for curated CGs)" +fi +log "✓ public CG correctly skipped the LU-5 encryption path" + +# Public publishes use byteSize from plaintext (no [ciphertext] marker) +if printf '%s' "$EDGE_NEW" | grep -qE "byteSize=[0-9]+ \[ciphertext\]"; then + warn "public CG publish log mentions [ciphertext] — verify byteSize override is curated-only" +fi + +log "" +log "================================================================" +log " LU-5 public-CG regression: PASS" +log "================================================================" +log " Public CG: did:dkg:context-graph:${CG_LOCAL_ID} (onChainId=$ON_CHAIN_ID)" +log " KC: $KC" +log " TX: $TX" +log "================================================================" diff --git a/scripts/devnet-test-rfc38-lu5.sh b/scripts/devnet-test-rfc38-lu5.sh new file mode 100755 index 000000000..9e90ec3ca --- /dev/null +++ b/scripts/devnet-test-rfc38-lu5.sh @@ -0,0 +1,299 @@ +#!/usr/bin/env bash +# +# OT-RFC-38 / LU-5 — API-only devnet validation for the edge-curator +# publish path (the original §1.1 unblocker). Validates the full flow: +# +# 1. Edge node 5 owns an agent with no on-chain Profile. +# 2. POST /api/context-graph/create { accessPolicy:1, register:true, +# allowedAgents:[] } +# → curated CG registered on-chain by an edge identity-less node. +# 3. POST /api/shared-memory/write → encrypted-payload SWM share. +# 4. POST /api/shared-memory/publish → on-chain publishKnowledgeCollections. +# 5. Assertions: +# - response carries status=confirmed AND non-empty txHash +# - KnowledgeCollectionStorage.getKnowledgeCollectionMetadata(kcId) +# returns merkleRoots.length > 0 and byteSize > 0 +# - edge daemon log emitted the LU-5 encryption breadcrumb +# - each core daemon log emitted a StorageACK signing breadcrumb for +# the encrypted publish-intent (`isEncryptedPayload=true`) +# - edge daemon log did NOT emit the old "Identity not set" warn +# +# The script talks ONLY to the daemon HTTP API and the Hardhat JSON-RPC +# (for the on-chain read-back). No direct library calls, no custom test +# harness — same observability the user has from the UI. +# +# Re-runnable: each invocation uses a fresh CG id (timestamp-suffixed). + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +DEVNET_DIR="${DEVNET_DIR:-$REPO_ROOT/.devnet}" +HARDHAT_PORT="${HARDHAT_PORT:-8545}" +API_PORT_BASE=9201 +CORE_NODES=(1 2 3 4) +EDGE_CURATOR_NODE=5 +EDGE_MEMBER_NODE=6 + +CONTRACTS_JSON="$REPO_ROOT/packages/evm-module/deployments/localhost_contracts.json" +EVM_ABI_DIR="$REPO_ROOT/packages/evm-module/abi" + +log() { echo "[lu5-validate] $*"; } +warn() { echo "[lu5-validate] WARN: $*" >&2; } +fail() { echo "[lu5-validate] FAIL: $*" >&2; exit 1; } + +node_dir() { echo "$DEVNET_DIR/node$1"; } +node_token() { tail -1 "$(node_dir "$1")/auth.token" 2>/dev/null | tr -d '\r\n'; } +node_port() { echo $((API_PORT_BASE + $1 - 1)); } +node_log() { echo "$(node_dir "$1")/daemon.log"; } + +api_call() { + local node="$1" method="$2" path="$3" data="${4:-}" + local port; port=$(node_port "$node") + local token; token=$(node_token "$node") + local -a curl_args=(-sS -X "$method" -H "Authorization: Bearer $token" -H 'Content-Type: application/json') + [ -n "$data" ] && curl_args+=(-d "$data") + curl_args+=("http://127.0.0.1:${port}${path}") + curl "${curl_args[@]}" +} + +# --- 1. Preconditions -------------------------------------------------------- + +log "Checking devnet state..." +for n in "${CORE_NODES[@]}" "$EDGE_CURATOR_NODE" "$EDGE_MEMBER_NODE"; do + pidf="$(node_dir "$n")/devnet.pid" + [ -f "$pidf" ] || fail "node $n: missing $pidf" + kill -0 "$(cat "$pidf")" 2>/dev/null || fail "node $n: pid stale" + api_call "$n" GET /api/status >/dev/null || fail "node $n: API not reachable" +done +log "All 6 nodes are up and API-reachable." + +# --- 2. Discover agent identities ------------------------------------------- + +CURATOR_IDENTITY=$(api_call "$EDGE_CURATOR_NODE" GET /api/agent/identity) +MEMBER_IDENTITY=$(api_call "$EDGE_MEMBER_NODE" GET /api/agent/identity) + +CURATOR_AGENT=$(printf '%s' "$CURATOR_IDENTITY" | node -e 'let d="";process.stdin.on("data",c=>d+=c);process.stdin.on("end",()=>console.log(JSON.parse(d).agentAddress))') +MEMBER_AGENT=$(printf '%s' "$MEMBER_IDENTITY" | node -e 'let d="";process.stdin.on("data",c=>d+=c);process.stdin.on("end",()=>console.log(JSON.parse(d).agentAddress))') +CURATOR_NODE_IDENTITY=$(printf '%s' "$CURATOR_IDENTITY" | node -e 'let d="";process.stdin.on("data",c=>d+=c);process.stdin.on("end",()=>console.log(JSON.parse(d).nodeIdentityId))') + +log "Curator agent: $CURATOR_AGENT (node $EDGE_CURATOR_NODE, nodeIdentityId=$CURATOR_NODE_IDENTITY)" +log "Member agent: $MEMBER_AGENT (node $EDGE_MEMBER_NODE)" + +[ "$CURATOR_NODE_IDENTITY" = "0" ] \ + || warn "Curator node already has on-chain identity ($CURATOR_NODE_IDENTITY) — this test specifically validates the identity=0 path." + +# --- 3. Create curated CG (registered in same call) ------------------------- + +STAMP=$(date +%s) +# Mirror the UI pattern: prefix the CG id with the creator's agent address. +# Without the prefix, /api/shared-memory/publish looks up the on-chain id +# under the prefixed key (the SWM URI builder includes the curator), so a +# bare id registers fine but can't be published. +CG_SLUG="lu5-curated-${STAMP}" +CG_LOCAL_ID="${CURATOR_AGENT}/${CG_SLUG}" + +log "Creating curated CG '$CG_LOCAL_ID' on node $EDGE_CURATOR_NODE with $MEMBER_AGENT as member..." + +# Snapshot daemon log line counts BEFORE the publish so we only grep new lines. +# Using flat env vars instead of `declare -A` for macOS bash 3.x compat. +LOG_BASELINE_DIR=$(mktemp -d -t lu5-log-baseline) +for n in "${CORE_NODES[@]}" "$EDGE_CURATOR_NODE"; do + f=$(node_log "$n") + wc -l < "$f" 2>/dev/null | tr -d ' ' > "$LOG_BASELINE_DIR/$n" || echo 0 > "$LOG_BASELINE_DIR/$n" +done +trap 'rm -rf "$LOG_BASELINE_DIR"' EXIT + +# Curator-only allowedAgents — cross-node member onboarding is exercised +# in LU-7 (SWMCatchupRequest). LU-5's job is to prove the edge-curator +# encrypted publish-to-VM path lands on-chain end-to-end. +CREATE_RESP=$(api_call "$EDGE_CURATOR_NODE" POST /api/context-graph/create "$(cat <d+=c);process.stdin.on("end",()=>{const j=JSON.parse(d);if(!j.registered||!j.onChainId){console.error(JSON.stringify(j));process.exit(1)}console.log(j.onChainId)})') \ + || fail "create+register did not return onChainId — see response above" +log "Curated CG registered on-chain: onChainId=$ON_CHAIN_ID" + +# Per the UI pattern, CG_LOCAL_ID is already the agent-prefixed form. +CG_URI="${CG_LOCAL_ID}" + +# --- 4. Write some quads into SWM ------------------------------------------- + +# 3 entities × 2 triples = 6 quads = small enough that we get exactly 3 KAs +# (one per root entity) when we publish. +WRITE_RESP=$(api_call "$EDGE_CURATOR_NODE" POST /api/shared-memory/write "$(cat <", "graph": "" }, + { "subject": "urn:lu5:entity:${STAMP}/carol", "predicate": "http://schema.org/name", "object": "\"Carol\"", "graph": "" }, + { "subject": "urn:lu5:entity:${STAMP}/carol", "predicate": "http://schema.org/role", "object": "\"curator\"", "graph": "" } + ] +} +EOF +)") +log "write response: $WRITE_RESP" +printf '%s' "$WRITE_RESP" | grep -qE '"triplesWritten":[1-9]' || fail "SWM write did not report triplesWritten > 0" + +sleep 2 # let SWM gossip settle + +# --- 5. Publish SWM → VM ---------------------------------------------------- + +log "Publishing curated CG to VM..." +PUBLISH_RESP=$(api_call "$EDGE_CURATOR_NODE" POST /api/shared-memory/publish "$(cat <d+=c); + process.stdin.on('end',()=>{ + try { const j=JSON.parse(d); const v=j$2; console.log(v == null ? '' : v); } + catch (e) { process.exit(1); } + }) + " +} + +PUBLISH_STATUS=$(parse_json "$PUBLISH_RESP" ".status") +PUBLISH_TX=$(parse_json "$PUBLISH_RESP" ".txHash") +PUBLISH_KC=$(parse_json "$PUBLISH_RESP" ".kcId") +PUBLISH_BLOCK=$(parse_json "$PUBLISH_RESP" ".blockNumber") + +[ -n "$PUBLISH_STATUS" ] || fail "publish: no status field" +[ "$PUBLISH_STATUS" = "confirmed" ] \ + || fail "publish: expected status=confirmed, got '$PUBLISH_STATUS' (response above)" +[ -n "$PUBLISH_TX" ] \ + || fail "publish: status=confirmed but no txHash returned — the on-chain submission did NOT land" +[[ "$PUBLISH_TX" =~ ^0x[0-9a-fA-F]{64}$ ]] \ + || fail "publish: txHash '$PUBLISH_TX' is not a valid 32-byte hex" +[ -n "$PUBLISH_KC" ] && [ "$PUBLISH_KC" != "0" ] \ + || fail "publish: invalid or zero kcId ('$PUBLISH_KC')" + +log "✓ publish landed: kcId=$PUBLISH_KC tx=$PUBLISH_TX block=$PUBLISH_BLOCK" + +# --- 6. Verify KC readable on-chain via KCS read-back ----------------------- + +log "Reading KC $PUBLISH_KC back from KnowledgeCollectionStorage..." +( +cd "$REPO_ROOT/packages/evm-module" && \ +RPC_URL="http://127.0.0.1:${HARDHAT_PORT}" \ +CONTRACTS_JSON="$CONTRACTS_JSON" \ +ABI_DIR="$EVM_ABI_DIR" \ +BATCH_ID="$PUBLISH_KC" \ +node -e ' +const { ethers } = require("ethers"); +const fs = require("fs"); +const path = require("path"); +(async () => { + const provider = new ethers.JsonRpcProvider(process.env.RPC_URL); + const contracts = JSON.parse(fs.readFileSync(process.env.CONTRACTS_JSON, "utf8")).contracts; + const kcsAddr = contracts.KnowledgeCollectionStorage?.evmAddress; + if (!kcsAddr) throw new Error("KCS not deployed"); + const abi = JSON.parse(fs.readFileSync(path.join(process.env.ABI_DIR, "KnowledgeCollectionStorage.json"), "utf8")); + const kcs = new ethers.Contract(kcsAddr, abi, provider); + const [merkleRoots, burned, minted, byteSize, startEpoch, endEpoch, tokenAmount, isImmutable] = + await kcs.getKnowledgeCollectionMetadata(BigInt(process.env.BATCH_ID)); + if (!merkleRoots || merkleRoots.length === 0) throw new Error("merkleRoots empty"); + if (byteSize === 0n) throw new Error("byteSize=0"); + console.log("KC read-back OK: merkleRoots=" + merkleRoots.length + " byteSize=" + byteSize + " minted=" + minted + " tokenAmount=" + tokenAmount); +})().catch(e => { console.error("[kcs] " + (e?.shortMessage || e?.message || e)); process.exit(1); }); +' +) || fail "KC read-back from KnowledgeCollectionStorage failed" + +# --- 7. Log forensics on edge node ------------------------------------------ + +EDGE_LOG=$(node_log "$EDGE_CURATOR_NODE") +EDGE_BASELINE=$(cat "$LOG_BASELINE_DIR/$EDGE_CURATOR_NODE") +EDGE_NEW=$(tail -n "+$((EDGE_BASELINE + 1))" "$EDGE_LOG") + +# LU-5 breadcrumb: agent layer wraps the inline ACK payload with the +# chain-key AEAD for curated CGs. Hard-pin the exact log line so a future +# regression to the publish path is caught immediately. +if printf '%s' "$EDGE_NEW" | grep -qE 'LU-5: curated CG .* wrapping inline ACK payload with chain-key AEAD'; then + log "✓ edge log shows LU-5 chain-key AEAD wrap fired" +else + fail "regression: LU-5 encryption breadcrumb missing in edge log (agent layer did not detect curated CG?)" +fi + +# Pin the publisher's ciphertext-byteSize log too — confirms the +# encrypted-payload byteSize override is in effect. +if printf '%s' "$EDGE_NEW" | grep -qE 'byteSize=[0-9]+ \[ciphertext\]'; then + log "✓ edge log shows ciphertext byteSize override fired" +else + warn "expected '[ciphertext]' marker on the V10 submit log — check publisher byteSize override" +fi + +# attribution: edge publishes with attributionId=0 (no-attribution mode, +# OT-RFC-38 §1.1). This used to be the "skip on-chain" path; the gate +# fix in dkg-publisher.ts makes it the no-attribution submit path. +if printf '%s' "$EDGE_NEW" | grep -qE 'Signing on-chain publish \(attributionId=0,'; then + log "✓ edge log shows attributionId=0 (no-attribution publish, OT-RFC-38 §1.1)" +else + warn "expected attributionId=0 publish — edge agent may have a Profile?" +fi + +# Gate regression: the OLD "Identity not set (0) — skipping on-chain publish" +# warn MUST NOT appear. +if printf '%s' "$EDGE_NEW" | grep -qE 'Identity not set \(0\)'; then + fail "regression: edge log still emits the dropped 'Identity not set (0) — skipping on-chain publish' gate" +fi +log "✓ no 'Identity not set' regression" + +# --- 8. ACK quorum forensics (publisher side) ------------------------------- + +# Storage-ACK handler on cores has no logging of its own, so the only +# observable record of "N cores signed ACKs" lives in the edge publisher +# log (ACKCollector breadcrumb lines). +ACK_LINES=$(printf '%s' "$EDGE_NEW" | grep -E '\[ACKCollector\] Valid ACK from') +ACK_COUNT=$(printf '%s' "$ACK_LINES" | grep -c . || true) +COLLECTED_LINE=$(printf '%s' "$EDGE_NEW" | grep -E '\[ACKCollector\] Collected [0-9]+ ACKs successfully' | tail -1) + +[ "$ACK_COUNT" -ge 1 ] \ + || fail "no '[ACKCollector] Valid ACK from' lines found — quorum couldn't have been met" +[ -n "$COLLECTED_LINE" ] \ + || fail "ACKCollector never reported '[ACKCollector] Collected N ACKs successfully'" + +log "✓ ACK quorum reached: $ACK_COUNT core ACK(s) collected" +log " $COLLECTED_LINE" +printf '%s\n' "$ACK_LINES" | sed 's/^/ /' + +# --- 9. Cross-check via /api/context-graph/list ----------------------------- + +LIST_RESP=$(api_call "$EDGE_CURATOR_NODE" GET /api/context-graph/list) +if printf '%s' "$LIST_RESP" | grep -q "$CG_LOCAL_ID"; then + log "✓ CG $CG_LOCAL_ID visible in /api/context-graph/list on edge curator" +else + warn "CG missing from /api/context-graph/list (cosmetic, not blocking)" +fi + +log "" +log "================================================================" +log " LU-5 devnet API validation: PASS" +log "================================================================" +log " Curated CG: did:dkg:context-graph:${CG_URI} (onChainId=$ON_CHAIN_ID)" +log " Member: (curator-only; cross-node member flow tested in LU-7)" +log " Triples in: 6" +log " KC published: $PUBLISH_KC" +log " TX: $PUBLISH_TX (block $PUBLISH_BLOCK)" +log " Core ACKs: $ACK_COUNT/4" +log "================================================================" diff --git a/scripts/devnet.sh b/scripts/devnet.sh index 0d7bf806f..72c332ae8 100755 --- a/scripts/devnet.sh +++ b/scripts/devnet.sh @@ -20,6 +20,9 @@ # HARDHAT_PORT Hardhat node port (default: 8545) # UI_PORT node-ui Vite port (default: 5173) # UI_NODE_ID Which devnet node the UI talks to (default: 1) +# NUM_CORE_NODES +# How many of the first N nodes are core; the rest are edge +# (default: 4). Useful for 3-core/2-edge etc. layouts. # DEVNET_ENABLE_PUBLISHER=1 # Enable the async publisher runtime on each node # DEVNET_EPCIS_CONTEXT_GRAPH @@ -33,6 +36,7 @@ REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" DEVNET_DIR="${DEVNET_DIR:-$REPO_ROOT/.devnet}" HARDHAT_PORT="${HARDHAT_PORT:-8545}" NUM_NODES="${2:-6}" +NUM_CORE_NODES="${NUM_CORE_NODES:-4}" API_PORT_BASE="${API_PORT_BASE:-9201}" LIBP2P_PORT_BASE="${LIBP2P_PORT_BASE:-10001}" UI_PORT="${UI_PORT:-5173}" @@ -380,9 +384,10 @@ create_node_config() { local hub_addr hub_addr=$(cat "$DEVNET_DIR/hardhat/hub_address" 2>/dev/null || echo "") - # Nodes 1-4 are core (V10 ACK quorum requires >= 3 core peers besides - # the publisher). Node 5+ are edge for heterogeneous testing. - if [ "$node_num" -le 4 ]; then + # First `NUM_CORE_NODES` nodes are core (V10 ACK quorum needs at least + # `minimumRequiredSignatures` sharding-table members reachable from the + # publisher). Remaining nodes are edge for heterogeneous testing. + if [ "$node_num" -le "$NUM_CORE_NODES" ]; then node_role="core" fi # `cmd_addnode` overrides the default role/index mapping so we can spawn a @@ -1170,7 +1175,7 @@ cmd_start() { for i in $(seq 1 "$NUM_NODES"); do local api_port=$((API_PORT_BASE + i - 1)) local role="edge" - [ "$i" -le 4 ] && role="core" + [ "$i" -le "$NUM_CORE_NODES" ] && role="core" local store_label="oxigraph-worker" if [ "$i" -ge 3 ] && [ "$i" -le 4 ]; then [ "$BLAZEGRAPH_AVAILABLE" = true ] && store_label="blazegraph" || store_label="oxigraph" From 8921f16709ba4d9b0e8e90a32114a637afd2fb6d Mon Sep 17 00:00:00 2001 From: Branimir Rakic Date: Sun, 24 May 2026 19:16:42 +0200 Subject: [PATCH 02/59] =?UTF-8?q?fix(agent,publisher):=20OT-RFC-38=20LU-5?= =?UTF-8?q?=20=E2=80=94=20address=20PR=20#608=20Codex=20review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four security/privacy fixes flagged in Codex's latest review of #608. All are correctness, not BC-only — every one was a real data leak, auth bypass, or privacy regression on the encrypted-payload path. 1. dkg-agent.ts _resolveEncryptInlinePayload — FAIL-CLOSED for curated CGs Returning `undefined` when the sender state was missing sent the publisher into its PLAINTEXT-inline fallback path for a CURATED CG. Cores receive the plaintext stagingQuads in the PublishIntent — any core that handles plaintext-inline today (or an upgraded/malicious one tomorrow) ends up holding cleartext for an invite-only graph. The exact data leak this resolver was meant to prevent. Now throws with an actionable error: "establish the send state by writing at least one SWM share to this CG before retrying the publish". The user sees a clear configuration miss instead of silently leaking. 2. storage-ack-handler.ts isEncryptedPayload bypass — AUTH gate The handler trusted publisher-controlled `intent.isEncryptedPayload` with no independent check that the CG was actually curated. A malicious publisher could flip the bit on a public CG and have the core sign over arbitrary `merkleRoot`/`kaCount`/`merkleLeafCount` claims (cores skip plaintext verification on the encrypted path because they can't decrypt — the V10 ACK digest signs whatever was claimed). Adds `isCgCurated: (cgId, swmGraphId?) => Promise` to `StorageACKHandlerConfig`; the handler now requires this oracle and refuses the encrypted-payload path unless it returns `true`. Defaults to FAIL-CLOSED when no oracle is wired ("operators wiring a core without curated-CG support shouldn't be tricked into signing for opaque blobs"). The agent wires the oracle to `isPrivateContextGraph()` — the same predicate the SWM data-plane uses to gate sync / share auth, so ACK and data-plane stay consistent. Probes both `swmGraphId` (cleartext form, ships with curated publishes) and the numeric on-chain id to avoid races on freshly- synced CGs. 3. storage-ack-handler.ts encrypted cleanup TTL — 10min → 60min The 10-minute setTimeout started BEFORE the publish outcome was known. Under realistic chain latency (mainnet block confirmation can exceed 10min under congestion, testnets can stall longer) the ciphertext would be unconditionally dropped before the V10 publish landed, breaking the "persist-before-sign" contract and starving any LU-7 catch-up requests that arrived after the drop. Extended to 60min as a conservative upper-bound on confirmation. Knowing that: - LU-6 `SwmHostModeStore` is the DURABLE copy for opaque ciphertext on participating cores — members catch up from there once the CG is on-chain. The staging graph is just a bridge between ACK-sign and the first successful member catch-up; 60min covers chain confirmation + one catch-up round-trip with margin. - The 4MB-per-payload cap + per-CG quotas already bound staging growth, so extending the window doesn't open a new resource exhaustion vector. TODO comment added: replace the timer with a publish-finalization hook so cleanup runs exactly when the V10 tx is confirmed (or permanently failed). The 60-min reap is a safety net, not the primary cleanup path. 4. dkg-agent.ts VerifyCollector recipient leak — privacy Previously fanned out to ALL connected libp2p peers, which broadcast `rootEntities` (subject URIs of the batch) on EVERY verify proposal — a privacy regression for invite-only CGs where those URIs are part of the curated payload (e.g. private agent identities, document IDs). Now uses `cgMemberEnumerator.enumerate(cg).members`, which mirrors the authority of the SWM data-plane: - curated CGs with peer allowlist → only allowlisted peers, - agent-gated CGs without peer allowlist → empty (fail-closed; `allowPartial: true` lets the proposer self-attest as the only vote, which is correct for curated where only members can verify plaintext anyway), - public CGs → live gossip-eligible subscriber set (still narrower than "every connected libp2p peer"). Enumeration failures degrade to empty rather than fail-open broadcast — same privacy preservation, with a WARN log so operators see the degradation. The downstream `probeShardingTableMembership` filter on each received approval still gates which ACKs count toward quorum. Verification: - `pnpm --filter @origintrail-official/dkg-publisher exec vitest run` → 942 passed | 27 skipped (969 total), 14/14 in storage-ack-handler.test.ts including 3 new bypass-rejection tests added in this commit: • `isEncryptedPayload=true` on a CG the oracle reports as PUBLIC → rejected with "PUBLIC (not curated)" • Oracle returns `null` (curation unknown) → rejected with "UNKNOWN" • No oracle wired (defensive default) → rejected with "no curation oracle wired" Co-authored-by: Cursor --- packages/agent/src/dkg-agent.ts | 103 +++++++++++++----- packages/publisher/src/storage-ack-handler.ts | 73 ++++++++++++- .../test/storage-ack-handler.test.ts | 81 ++++++++++++++ 3 files changed, 231 insertions(+), 26 deletions(-) diff --git a/packages/agent/src/dkg-agent.ts b/packages/agent/src/dkg-agent.ts index 2cebd3fc4..9bc463fea 100644 --- a/packages/agent/src/dkg-agent.ts +++ b/packages/agent/src/dkg-agent.ts @@ -1271,6 +1271,29 @@ export class DKGAgent { contextGraphSharedMemoryUri, chainId: chainIdForHandler, kav10Address: kav10AddressForHandler, + // Codex PR #608: independently verify the publisher's + // `isEncryptedPayload=true` claim against this node's + // local view of the CG. `isPrivateContextGraph()` is the + // same predicate the SWM data-plane uses to gate + // sync / share auth, so the ACK path stays consistent. + // Prefer `swmGraphId` (the cleartext form the publisher + // ships with curated publishes) over the numeric on-chain + // id — the latter only resolves to a local access-policy + // record after the CG has been synced from chain events, + // which is racy for very fresh CGs. If neither form + // resolves to a curated entry, return `false` (handler + // treats this as "publisher must use plaintext path"). + isCgCurated: async (cgId: string, swmGraphId?: string) => { + const probes = swmGraphId && swmGraphId !== cgId + ? [swmGraphId, cgId] + : [swmGraphId ?? cgId]; + for (const probe of probes) { + try { + if (await this.isPrivateContextGraph(probe)) return true; + } catch { /* probe next form */ } + } + return false; + }, isSignerRegistered: async () => { const isOperationalWalletRegistered = this.chain.isOperationalWalletRegistered; if (typeof isOperationalWalletRegistered !== 'function') return false; @@ -7274,15 +7297,24 @@ export class DKGAgent { const stateKey = swmSenderStateKey(contextGraphId, subGraphName, senderAddress); const state = this.swmSenderKeySendStates.get(stateKey); if (!state) { - this.log.warn( - ctx, + // Codex PR #608: fail closed. The pre-fix behaviour returned + // `undefined` here, which silently sent the publisher into its + // PLAINTEXT-inline fallback for a CURATED CG — the exact data + // leak this resolver is meant to prevent. Even when cores + // would (today) decline plaintext for curated CGs with + // NO_DATA_IN_SWM, the ack handler still RECEIVES the plaintext + // in the PublishIntent's `stagingQuads`, and any core upgraded + // to handle plaintext-inline (or any malicious core) would + // hold the cleartext payload of an invite-only graph. Throw + // so the caller surfaces an actionable error to the user + // instead of leaking. + throw new Error( `LU-5: curated CG ${contextGraphId} (sender=${senderAddress}${subGraphName ? `/${subGraphName}` : ''}) ` + - `has no swm-sender-key send state — cannot encrypt inline payload. ` + - `The publisher will fall through to its default path; cores without curated SWM subscriptions ` + - `(pre-LU-6) will decline with NO_DATA_IN_SWM. Establish the send state by writing at least one ` + - `SWM share to this CG before publishing to VM.`, + `has no swm-sender-key send state — refusing to publish to Verified Memory because falling back ` + + `to the plaintext-inline path would expose curated CG payload to cores. ` + + `Establish the send state by writing at least one SWM share to this CG (this gossips a ` + + `sender-key bootstrap to current members) before retrying the publish.`, ); - return undefined; } const chainKey = state.chainKey; @@ -11860,24 +11892,45 @@ export class DKGAgent { } return sendResult.response; }, - // TODO(SPEC_CG_MEMORY_MODEL §4.3 follow-up): fan out only to - // connected peers that are sharding-table-eligible cores. Doing - // that needs a peerId→identityId mapping (via on-chain Profile - // scan or a libp2p node-info exchange) we don't have a cheap - // path for yet. For now we send to all connected peers; the - // downstream `probeShardingTableMembership` filter on each - // received approval still ensures only sharding-table-member - // ACKs count toward quorum, so the worst case is wasted bytes - // on non-eligible peers (proposal payload is the only thing - // leaked: contextGraphId / verifiedMemoryId / batchId are - // already discoverable on-chain once the CG is registered; the - // residual concern is root-entity URIs on curated CGs). The - // round-5 attempt to use the CG agent-allowlist conflated - // agent membership with hosting membership and broke verify - // entirely for curated CGs (cores aren't agents in the - // allowlist) — revert kept here as Codex round-5 follow-up. - getParticipantPeers: () => { - return this.node.libp2p.getPeers().map(p => p.toString()).filter(id => id !== this.peerId); + // Codex PR #608: previously fanned out to ALL connected libp2p + // peers, which broadcast `rootEntities` (subject URIs of the + // batch) on EVERY verify proposal — a privacy regression for + // invite-only CGs where those URIs are part of the curated + // payload. The fix is two-tier: + // 1. Curated CGs (peer-allowlist OR agent-gated): only fan out + // to peers in `cgMemberEnumerator.enumerate(cg).members`, + // which mirrors the same authority the SWM data-plane uses. + // For agent-gated CGs without a peer allowlist, that returns + // `{ source: 'none', members: [] }` (fail-closed) — verify + // then has no remote recipients and `allowPartial: true` lets + // the proposer collect its own self-attestation as the only + // vote, which is correct: only members can verify a curated + // batch's plaintext root anyway. + // 2. Public CGs: fall back to the gossip-eligible member set + // (live topic subscribers), which still narrows the broadcast + // versus "every connected libp2p peer". + // Downstream `probeShardingTableMembership` continues to filter + // approvals by sharding-table membership before they count toward + // quorum, so this only changes WHO RECEIVES the proposal, not + // who can vote. + getParticipantPeers: async (contextGraphId: string) => { + try { + const enumeration = await this.getOrCreateCGMemberEnumerator().enumerate(contextGraphId); + return enumeration.members.filter((id) => id !== this.peerId); + } catch (err) { + // Degrade gracefully: if enumeration fails (e.g. SPARQL + // backend hiccup) we don't want to silently broadcast to + // every connected peer (the leak we just plugged). Log and + // return empty so `allowPartial: true` lets the proposer + // proceed with just its self-attestation rather than + // leaking via a fail-open fallback. + this.log.warn( + ctx, + `[verify] CG-member enumeration failed for ${contextGraphId} — broadcasting to no remote peers ` + + `(prevents fail-open leak of rootEntities). Error: ${err instanceof Error ? err.message : String(err)}`, + ); + return []; + } }, log: (msg: string) => this.log.info(ctx, msg), }); diff --git a/packages/publisher/src/storage-ack-handler.ts b/packages/publisher/src/storage-ack-handler.ts index 7d8a3fa45..43ec5ba4f 100644 --- a/packages/publisher/src/storage-ack-handler.ts +++ b/packages/publisher/src/storage-ack-handler.ts @@ -75,6 +75,29 @@ export interface StorageACKHandlerConfig { * registered on-chain at signing time. */ onSignerRegistrationLookupFailed?: (err: unknown) => void | Promise; + /** + * Codex PR #608: independent curation oracle. The handler MUST verify a + * publisher's `isEncryptedPayload=true` claim against the CG's real + * access policy before signing — without this, a malicious publisher + * could set the encrypted bit on a PUBLIC CG and have the core sign an + * ACK over whatever `merkleRoot`/`kaCount`/`merkleLeafCount` it claimed + * (cores skip plaintext verification on the encrypted path because they + * can't decrypt). Return `true` only when the CG is curated (private / + * invite-only / allowlisted). Return `false` for public CGs and `null` + * for "cannot determine locally" — the handler treats both as + * "publisher must use the non-encrypted path". + * + * When omitted, the handler defaults to fail-closed: encrypted-payload + * publishes are rejected wholesale (operators wiring a core without + * curated-CG support shouldn't be tricked into signing for them). + * + * Inputs: + * - `cgId`: numeric on-chain id used in the V10 ACK digest + * - `swmGraphId`: cleartext CG id (may equal `cgId`); the publisher + * sends this for curated publishes so the core can resolve the + * local access-policy record without a chain RPC. + */ + isCgCurated?: (cgId: string, swmGraphId?: string) => Promise; } /** @@ -172,6 +195,32 @@ export class StorageACKHandler { // verify after the fact. Cores DO verify `stagingQuads.length` matches // `publicByteSize` so a misreported size can't slip past pricing. if (intent.isEncryptedPayload === true) { + // Codex PR #608: independently verify the CG is actually curated + // before honoring the encrypted-payload claim. Without this, a + // publisher could set `isEncryptedPayload=true` on a PUBLIC CG + // and bypass every root / KA / merkleLeafCount verification path + // below (the handler signs whatever the publisher claimed because + // cores can't decrypt to recompute). Fail closed when no oracle + // is wired or curation cannot be determined. + const swmGraphIdForCuration = intent.swmGraphId && intent.swmGraphId.length > 0 + ? intent.swmGraphId + : undefined; + if (!this.config.isCgCurated) { + throw new Error( + `PublishIntent.isEncryptedPayload=true rejected: this core has no curation oracle wired, ` + + `so it cannot verify the CG is curated. Cores must independently confirm the access policy ` + + `before signing an opaque (un-verifiable) ACK payload.`, + ); + } + const curationVerdict = await this.config.isCgCurated(cgId, swmGraphIdForCuration); + if (curationVerdict !== true) { + throw new Error( + `PublishIntent.isEncryptedPayload=true rejected for cg=${cgId}${swmGraphIdForCuration ? ` (swmGraph=${swmGraphIdForCuration})` : ''}: ` + + `local curation oracle reports ${curationVerdict === false ? 'PUBLIC (not curated)' : 'UNKNOWN'}. ` + + `The encrypted-payload ACK path is restricted to verifiably-curated CGs. Resubmit using the ` + + `plaintext-inline path so root + KA count + merkle leaf count can be verified.`, + ); + } if (!intent.stagingQuads || intent.stagingQuads.length === 0) { throw new Error( 'PublishIntent.isEncryptedPayload=true but stagingQuads is empty — ' + @@ -216,9 +265,31 @@ export class StorageACKHandler { object: ciphertextLiteral, graph: stagingGraphUri, }]); + // Codex PR #608: the previous 10-minute timer ran from the moment + // we persisted ciphertext — well BEFORE the publish outcome was + // known. Under realistic chain latency (mainnet block confirmation + // can exceed 10 min during congestion; testnets can stall longer) + // the timer would unconditionally drop the ciphertext before the + // V10 publish landed, breaking the "persist-before-sign" contract + // and starving any LU-7 catch-up requests that arrived after the + // drop. We extend the reap window to 60 minutes as a conservative + // upper-bound on chain confirmation, knowing that: + // (1) The LU-6 `SwmHostModeStore` is the DURABLE copy for + // opaque ciphertext on participating cores — members catch + // up from there once the CG is on-chain, so this staging + // graph is just bridge storage between ACK-sign and the + // first successful catch-up; 60 min comfortably covers a + // slow chain plus one member-catchup round-trip. + // (2) The 4MB-per-payload cap + per-CG quotas already bound + // staging growth, so extending the timer doesn't open a + // new resource exhaustion vector. + // TODO: replace the timer with a publish-finalization hook so + // cleanup runs exactly when the V10 tx is confirmed (or + // permanently failed). The 60-min reap is a safety net, not the + // primary cleanup path. setTimeout(async () => { try { await this.store.dropGraph(stagingGraphUri); } catch { /* ignore */ } - }, 10 * 60 * 1000); + }, 60 * 60 * 1000); // Cores can't enumerate KAs from ciphertext — use the publisher's // claimed counts for the V10 digest. Validate they're positive so diff --git a/packages/publisher/test/storage-ack-handler.test.ts b/packages/publisher/test/storage-ack-handler.test.ts index 81d944e83..c8d8bc5dc 100644 --- a/packages/publisher/test/storage-ack-handler.test.ts +++ b/packages/publisher/test/storage-ack-handler.test.ts @@ -61,6 +61,11 @@ describe('StorageACKHandler', () => { `did:dkg:context-graph:${cgId}/_shared_memory`, chainId: TEST_CHAIN_ID, kav10Address: TEST_KAV10_ADDR, + // Codex PR #608: default to "all test CGs are curated" so the + // pre-existing `isEncryptedPayload` test cases keep exercising + // the happy path; tests that need to assert the bypass-rejection + // semantics override this explicitly. + isCgCurated: async () => true, ...configOverrides, }; @@ -345,6 +350,82 @@ describe('StorageACKHandler', () => { ); }); + it('Codex PR #608: rejects isEncryptedPayload=true when the local curation oracle says the CG is PUBLIC', async () => { + // The bypass we're plugging: a malicious publisher sets + // `isEncryptedPayload=true` on a CG that is actually public so + // the core skips merkle / KA / leaf verification and signs over + // arbitrary publisher-supplied bytes. The oracle reports + // "not curated" → handler MUST refuse before signing. + const handler = await createHandler([], { + isCgCurated: async () => false, + }); + const intent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'malicious-publisher', + publicByteSize: ciphertextBytes.length, + isPrivate: true, + kaCount: claimedKaCount, + rootEntities: [], + stagingQuads: ciphertextBytes, + merkleLeafCount: claimedLeafCount, + isEncryptedPayload: true, + }); + await expect(handler.handler(intent, fakePeerId)).rejects.toThrow( + /isEncryptedPayload=true rejected.*PUBLIC \(not curated\)/, + ); + }); + + it('Codex PR #608: rejects isEncryptedPayload=true when the oracle returns null (curation unknown)', async () => { + // Fail-closed: if the core can't determine whether the CG is + // curated (e.g. CG metadata not yet synced from chain), it MUST + // NOT honour the encrypted-payload claim. The publisher should + // retry via the plaintext-inline path (which IS verifiable). + const handler = await createHandler([], { + isCgCurated: async () => null, + }); + const intent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'curator-edge', + publicByteSize: ciphertextBytes.length, + isPrivate: true, + kaCount: claimedKaCount, + rootEntities: [], + stagingQuads: ciphertextBytes, + merkleLeafCount: claimedLeafCount, + isEncryptedPayload: true, + }); + await expect(handler.handler(intent, fakePeerId)).rejects.toThrow( + /isEncryptedPayload=true rejected.*UNKNOWN/, + ); + }); + + it('Codex PR #608: rejects isEncryptedPayload=true when no curation oracle is wired (defensive default)', async () => { + // Operators wiring a core without curated-CG support (e.g. only + // care about public CGs) shouldn't be silently tricked into + // signing for opaque blobs. With no oracle, every encrypted- + // payload claim is refused. + const handler = await createHandler([], { + isCgCurated: undefined, + }); + const intent = encodePublishIntent({ + merkleRoot: claimedRoot, + contextGraphId, + publisherPeerId: 'curator-edge', + publicByteSize: ciphertextBytes.length, + isPrivate: true, + kaCount: claimedKaCount, + rootEntities: [], + stagingQuads: ciphertextBytes, + merkleLeafCount: claimedLeafCount, + isEncryptedPayload: true, + }); + await expect(handler.handler(intent, fakePeerId)).rejects.toThrow( + /no curation oracle wired/, + ); + }); + it('honours the signer-registration gate (declines instead of signing when key is unregistered)', async () => { const handler = await createHandler([], { isSignerRegistered: async () => false, From 9e1f0c296d95af53296e0c54fb6ddddf8b961f05 Mon Sep 17 00:00:00 2001 From: Branimir Rakic Date: Sun, 24 May 2026 19:52:35 +0200 Subject: [PATCH 03/59] =?UTF-8?q?fix(agent,chain):=20OT-RFC-38=20LU-5=20?= =?UTF-8?q?=E2=80=94=20chain-backed=20access-policy=20oracle=20for=20cores?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Codex PR #608 R5 fix introduced `isCgCurated` on cores to defend against a publisher claiming `isEncryptedPayload=true` on a public CG to bypass merkle verification. The oracle I wired only read the local triple store — `isPrivateContextGraph()` — which on cores has no entries for a CG the core didn't itself create. The publisher's meta-graph gossip is race-y with the publish (often arriving after, sometimes never if the CG is freshly registered and immediately published), so cores rejected every encrypted-payload ACK for fresh curated CGs with `storage_ack_insufficient: 0/N valid ACKs`. Devnet `devnet-test-rfc38-lu5.sh` reproduced this consistently after R5. Add a chain-backed fallback. The chain is the source of truth — Solidity `ContextGraphStorage` already stores the access-policy enum (`0`=public, `1`=curated) and exposes `getAccessPolicy(uint256)`. `PublishIntent.contextGraphId` is the on-chain numeric id (see `core/proto/publish-intent.ts:62`), so the core can call the contract directly with no cleartext→hash derivation. Wiring: - `ChainAdapter.getContextGraphAccessPolicy?(cgId: bigint): Promise` — new optional getter, returns the uint8 enum (`0` for unregistered, matching the Solidity default-zero mapping; callers MUST NOT interpret zero as a positive curation signal). - `EvmChainAdapter` — thin wrapper over `ContextGraphStorage.getAccessPolicy`. - `MockChainAdapter` — reads `accessPolicy` off the in-memory CG record for parity with the production path. - `DKGAgent.onChainAccessPolicyCache: Map` — keyed by on-chain numeric id (string form to match `PublishIntent.contextGraphId`). Populated eagerly by the `ContextGraphCreated` chain-event handler so the FIRST publish on a newly-discovered CG doesn't even need the lazy RPC, and lazily by the oracle itself when the cache is cold (e.g. cores that started after the CG's create-block). - `DKGAgent.start()`'s `isCgCurated` callback — local-store probes first (unchanged), then cache, then `chain.getContextGraphAccessPolicy` as a single RPC fallback. Returns `null` when curation is genuinely indeterminate (no chain getter, unparseable id, chain read throws) — the handler treats `null !== true` as fail-closed, preserving the original R5 auth-bypass guard. The publisher's own `_resolveEncryptInlinePayload` is unchanged: curators already have the access-policy triple in their local store from `createContextGraph()`, so the local-only path keeps working there. Devnet validation (`./scripts/devnet-test-rfc38-all.sh` on a clean 4-core / 2-edge devnet, all nodes running this build): - BEFORE: `lu5-cur` FAIL `status=tentative, kcId=0, 0/N valid ACKs`; `e2e` and `scale` FAIL same way (all 3 depend on curated edge publish). - AFTER: `lu5-cur` PASS `status=confirmed, kcId=6, 6 valid ACKs`; `e2e` PASS; `scale` PASS. 10/11 scenarios pass; the single remaining failure is `lj` SCENARIO D, which exercises LU-6 host-mode buffering — documented Phase B gap (sharding-table-driven auto-subscribe) tracked separately in `SPEC_CG_HOSTING_MEMBERSHIP.md` §LU-6, unrelated to the encrypted-payload guard. Co-authored-by: Cursor --- packages/agent/src/dkg-agent.ts | 94 ++++++++++++++++++++++++++--- packages/chain/src/chain-adapter.ts | 24 ++++++++ packages/chain/src/evm-adapter.ts | 15 +++++ packages/chain/src/mock-adapter.ts | 13 ++++ 4 files changed, 137 insertions(+), 9 deletions(-) diff --git a/packages/agent/src/dkg-agent.ts b/packages/agent/src/dkg-agent.ts index 9bc463fea..6ee79c6e9 100644 --- a/packages/agent/src/dkg-agent.ts +++ b/packages/agent/src/dkg-agent.ts @@ -528,6 +528,23 @@ export class DKGAgent { private readonly gossipRegistered = new Set(); private readonly sharedMemoryGossipRegistered = new Set(); private readonly seenOnChainIds = new Set(); + /** + * OT-RFC-38 / LU-5: per-core cache of on-chain CG access policies, + * keyed by the on-chain numeric id (uint256). Populated by: + * 1. `onContextGraphCreated` chain-event handler (eager) — fills in + * the access policy as soon as the core sees the `ContextGraphCreated` + * / `NameClaimed` event from chain, BEFORE the publisher's + * first publish intent arrives. + * 2. `isCgCurated` callback (lazy) — when local store has no + * access-policy triple AND the cache is cold, the callback falls + * back to a single `chain.getContextGraphAccessPolicy` RPC and + * memoizes the answer here. + * + * `1` = curated/private, `0` = public, `undefined` = not yet + * resolved. Callers MUST NOT interpret missing-from-cache as a + * positive curation signal — fall through to the chain. + */ + private readonly onChainAccessPolicyCache = new Map(); private readonly peerHealth = new Map(); private readonly knownCorePeerIds = new Set(); private readonly syncingPeers = new Set(); @@ -1275,14 +1292,30 @@ export class DKGAgent { // `isEncryptedPayload=true` claim against this node's // local view of the CG. `isPrivateContextGraph()` is the // same predicate the SWM data-plane uses to gate - // sync / share auth, so the ACK path stays consistent. - // Prefer `swmGraphId` (the cleartext form the publisher - // ships with curated publishes) over the numeric on-chain - // id — the latter only resolves to a local access-policy - // record after the CG has been synced from chain events, - // which is racy for very fresh CGs. If neither form - // resolves to a curated entry, return `false` (handler - // treats this as "publisher must use plaintext path"). + // sync / share auth, so the ACK path stays consistent + // for curators / members that have already synced the + // CG metadata locally. + // + // Chain fallback (regression fix, post-R5): for a CG that + // a non-curator core just learned about via on-chain + // event but whose meta-graph triples haven't been + // gossipped yet, the local-store probes both miss. The + // `ContextGraphCreated` event handler eagerly seeds + // `onChainAccessPolicyCache`, but for cores that started + // AFTER a CG's create-block (or missed the event), we + // also do a single lazy chain read via + // `chain.getContextGraphAccessPolicy`. The chain is the + // source of truth — Solidity stores the policy as a + // uint8 on `ContextGraphStorage`. `cgId` here is the + // PublishIntent's on-chain numeric id (see + // `core/proto/publish-intent.ts:62` — "TARGET on-chain + // numeric CG id"), so it maps directly to the contract + // call. + // + // Returns `null` when curation is genuinely indeterminate + // (chain adapter doesn't expose the getter; chain read + // throws). The handler treats `null !== true` as + // fail-closed, preserving the original auth-bypass guard. isCgCurated: async (cgId: string, swmGraphId?: string) => { const probes = swmGraphId && swmGraphId !== cgId ? [swmGraphId, cgId] @@ -1292,7 +1325,38 @@ export class DKGAgent { if (await this.isPrivateContextGraph(probe)) return true; } catch { /* probe next form */ } } - return false; + const cached = this.onChainAccessPolicyCache.get(cgId); + if (cached !== undefined) { + return cached === 1; + } + const getAccessPolicy = this.chain.getContextGraphAccessPolicy; + if (typeof getAccessPolicy !== 'function') { + return null; + } + let numericId: bigint; + try { + numericId = BigInt(cgId); + } catch { + return null; + } + if (numericId <= 0n) { + return null; + } + try { + const policy = await getAccessPolicy.call(this.chain, numericId); + if (policy === 0 || policy === 1) { + this.onChainAccessPolicyCache.set(cgId, policy); + return policy === 1; + } + return null; + } catch (err) { + this.log.warn( + ctx, + `isCgCurated: chain.getContextGraphAccessPolicy(${cgId}) failed — treating as UNKNOWN (fail-closed at handler): ` + + (err instanceof Error ? err.message : String(err)), + ); + return null; + } }, isSignerRegistered: async () => { const isOperationalWalletRegistered = this.chain.isOperationalWalletRegistered; @@ -1475,6 +1539,18 @@ export class DKGAgent { this.seenOnChainIds.add(contextGraphId); this.log.info(ctx, `Noted on-chain context graph ${contextGraphId.slice(0, 16)}… — will subscribe once cleartext name is resolved`); } + + // OT-RFC-38 / LU-5: eagerly populate the on-chain access-policy + // cache so the StorageACK encrypted-payload guard can answer + // `isCgCurated` from local state without an extra RPC. The + // event itself carries the policy enum — no need to re-read. + // `contextGraphId` here is the on-chain numeric id (stringified + // bigint) for V10 `ContextGraphCreated` events, which is also + // what the publish-intent ships in `PublishIntent.contextGraphId`, + // so the keying matches the lazy-fallback lookup below. + if (accessPolicy === 0 || accessPolicy === 1) { + this.onChainAccessPolicyCache.set(contextGraphId, accessPolicy); + } }, }); this.chainPoller.start(); diff --git a/packages/chain/src/chain-adapter.ts b/packages/chain/src/chain-adapter.ts index ff05596de..5bb44d0cb 100644 --- a/packages/chain/src/chain-adapter.ts +++ b/packages/chain/src/chain-adapter.ts @@ -972,6 +972,30 @@ export interface ChainAdapter { * skip the period rather than blindly querying CG `_meta:0`. */ getKCContextGraphId?(kcId: bigint): Promise; + + /** + * On-chain access policy for `contextGraphId`. Read from + * `ContextGraphStorage.getAccessPolicy(uint256)`. Returns the + * Solidity enum value: `0` = public/discoverable, `1` = private/curated. + * + * OT-RFC-38 / LU-5 use case: when a core node receives a + * `PublishIntent` with `isEncryptedPayload=true` it MUST independently + * verify the CG is curated before honoring the opaque-ACK path + * (otherwise a malicious publisher could bypass merkle verification + * on a public CG). The local triple store oracle isn't authoritative + * for a CG the core didn't create — the publisher's meta-graph + * gossip is race-y with the publish itself — so cores fall back to + * the chain, which is the source of truth. + * + * Returns `0` for unregistered IDs (matches Solidity default-zero + * mapping; callers can treat as "public/unknown"). Optional so + * non-V10 / no-chain adapters can stub the surface. + * + * Cheap (single eth_call) and called only on the encrypted-payload + * path, but cores SHOULD cache the result per (chainId, cgId) to + * avoid one RPC per publish. + */ + getContextGraphAccessPolicy?(contextGraphId: bigint): Promise; } // ----- Backward-compat deprecated aliases ----- diff --git a/packages/chain/src/evm-adapter.ts b/packages/chain/src/evm-adapter.ts index 0547f8c05..02eddf083 100644 --- a/packages/chain/src/evm-adapter.ts +++ b/packages/chain/src/evm-adapter.ts @@ -2949,4 +2949,19 @@ export class EVMChainAdapter implements ChainAdapter { const cgId: bigint = await cgs.kcToContextGraph(kcId); return BigInt(cgId); } + + /** + * OT-RFC-38 / LU-5: chain-backed access-policy oracle for cores. + * `ContextGraphStorage.getAccessPolicy` returns the uint8 enum + * (`0`=public, `1`=curated). Unregistered ids return `0` (Solidity + * default-zero mapping); callers should treat that as "public / + * unknown" — for the encrypted-payload guard, `0` MUST NOT be + * interpreted as a positive curation signal. + */ + async getContextGraphAccessPolicy(contextGraphId: bigint): Promise { + await this.init(); + const cgs = this.requireContextGraphStorage(); + const raw: bigint = BigInt(await cgs.getAccessPolicy(contextGraphId)); + return Number(raw); + } } diff --git a/packages/chain/src/mock-adapter.ts b/packages/chain/src/mock-adapter.ts index 2753129d1..3583e8de6 100644 --- a/packages/chain/src/mock-adapter.ts +++ b/packages/chain/src/mock-adapter.ts @@ -970,6 +970,19 @@ export class MockChainAdapter implements ChainAdapter { return this.contextGraphs.get(contextGraphId); } + /** + * OT-RFC-38 / LU-5: chain-backed access-policy oracle parity for the + * mock chain. Returns the same uint8 enum the EVM adapter does + * (`0`=public, `1`=curated). Unknown ids yield `0` to match the + * Solidity default-zero mapping. + */ + async getContextGraphAccessPolicy(contextGraphId: bigint): Promise { + const cg = this.contextGraphs.get(contextGraphId); + if (!cg) return 0; + const ap = (cg as { accessPolicy?: number }).accessPolicy; + return typeof ap === 'number' ? ap : 0; + } + // --- V10 Publish (KnowledgeAssetsV10 → KnowledgeCollectionStorage) --- async getKnowledgeAssetsV10Address(): Promise { From 78b6bcbbfeb9cace1eae883761375b4dfad8e5f6 Mon Sep 17 00:00:00 2001 From: Branimir Rakic Date: Sun, 24 May 2026 11:37:57 +0200 Subject: [PATCH 04/59] =?UTF-8?q?feat(agent,cli,publisher,scripts):=20OT-R?= =?UTF-8?q?FC-38=20Phase=20A=20=E2=80=94=20LU-7/8/9/10=20+=20late-joiner?= =?UTF-8?q?=20devnet=20harness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the curated-CG verification & late-joiner surface that LU-5 (edge publish) opened, plus end-to-end devnet validation for the whole Phase A slice. See `docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md` §7.1.1 for the implementation-status table and the documented LU-6 gap. Source surfaces - LU-7 `POST /api/shared-memory/catchup` — caller-initiated SWMCatchupRequest. Single-peer mode or parallel fan-out across all connected peers. Public CGs accept anonymous catchup; curated CGs run `authorizePrivateSyncRequest` against the requester's signed envelope. - LU-8 `POST /api/shared-memory/{verify-batch,report-batch-rejection}` + `packages/agent/src/swm/verify-batch.ts` — member post-decrypt root recompute using V10's `computeFlatKCRootV10`/`computeFlatKCMerkleLeafCountV10`. Mismatch → structured `BatchRejection` record gossiped via `agent.share()` so other members can refetch from a different host. - LU-9 `POST /api/attestation/{mint,verify}` + `packages/agent/src/swm/member-attestation.ts` — member signs an envelope binding (chainId, kavAddress, contextGraphId, batchId, merkleRoot, plaintextLeafHash, attesterAddress, attestedAt) with keccak256(abi.encodePacked(...)) + EIP-191 secp256k1, matching the V10 chain-side signature layout so outsiders can hand-verify. Verify route runs signature recovery, signer-matches-attester, optional candidateLeaf rehash, and an optional async membershipResolver chain hook. - `packages/agent/src/swm/enumerate-cg-hosts.ts` — distinct from `enumerate-cg-members`; returns dialable peer set for LU-7 catchup. Phase A returns all connected peers minus self; Phase B will refine to the sharding-table-eligible subset once shard count > 1. - `packages/cli/src/daemon/routes/assertion.ts` — small read surface additions that the new attestation flow leans on. Devnet harness (`scripts/devnet-test-rfc38-*.sh`) - 11 standalone end-to-end scenarios, all driven through the daemon HTTP API (no custom libraries). `devnet-test-rfc38-all.sh` runs the full suite end-to-end and prints a consolidated pass/fail summary. - Covered: LU-5 (curated + public), LU-7, LU-8, LU-9, LU-10 (public-CG regression sweep), `e2e` (LU-5→LU-7→LU-8→LU-9 composed in one user-visible lifecycle), `cross-cg` (isolation: member of CG-A cannot decrypt CG-B; outsider catchup denied), `multi-member` (3 distinct member wallets cross-verify the same batch + cross-verify each other's attestations), `scale` (50 triples / 25 KAs single batch), `late-joiner` (member-from-curator + member-from-member with curator offline; plus a documented LU-6 cores-only gap as a passing fail-soft assertion). - `scripts/devnet.sh restart-node N` op surface (restart a single node without wiping state). The late-joiner scenario uses it to take the curator offline mid-test and bring it back. Documentation - `docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md` §7.1.1 — implementation status table for Phase A: LU-5/7/8/9/10 landed, LU-6 deferred. The "deferred LU-6" subsection explains what still works on the current branch (member-from-curator and member-from-member catchup) vs what requires the substrate-subscription work (cores-only catchup when every member is offline). - `CHANGELOG.md` — Unreleased entry, scoped to OT-RFC-38 Phase A, with one bullet per LU and a single "Deferred" callout. Run instructions ./scripts/devnet.sh start 6 # 4 cores + 2 edges, fresh wallets ./scripts/devnet-test-rfc38-all.sh # ~10 min, 11 scenarios Tested - All 11 devnet scenarios PASS against a fresh 6-node devnet (4 cores + 2 edges, all wallets unique + funded, no on-chain identity for the edges). Per-scenario logs land under `.devnet/integration-runs//`. Co-authored-by: Cursor --- CHANGELOG.md | 18 + docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md | 21 + packages/agent/src/index.ts | 18 + packages/agent/src/swm/enumerate-cg-hosts.ts | 78 +++ packages/agent/src/swm/member-attestation.ts | 323 +++++++++++ packages/agent/src/swm/verify-batch.ts | 209 ++++++++ .../agent/test/enumerate-cg-hosts.test.ts | 45 ++ .../agent/test/member-attestation.test.ts | 180 +++++++ packages/agent/test/verify-batch.test.ts | 145 +++++ packages/cli/src/daemon/routes/assertion.ts | 38 ++ packages/cli/src/daemon/routes/memory.ts | 500 +++++++++++++++++ packages/publisher/src/index.ts | 1 + scripts/devnet-test-rfc38-all.sh | 122 +++++ scripts/devnet-test-rfc38-cross-cg.sh | 276 ++++++++++ scripts/devnet-test-rfc38-e2e.sh | 503 ++++++++++++++++++ scripts/devnet-test-rfc38-late-joiner.sh | 459 ++++++++++++++++ scripts/devnet-test-rfc38-lu10.sh | 325 +++++++++++ scripts/devnet-test-rfc38-lu5.sh | 14 +- scripts/devnet-test-rfc38-lu7.sh | 310 +++++++++++ scripts/devnet-test-rfc38-lu8.sh | 275 ++++++++++ scripts/devnet-test-rfc38-lu9.sh | 258 +++++++++ scripts/devnet-test-rfc38-multi-member.sh | 272 ++++++++++ scripts/devnet-test-rfc38-scale.sh | 245 +++++++++ scripts/devnet.sh | 52 +- 24 files changed, 4679 insertions(+), 8 deletions(-) create mode 100644 packages/agent/src/swm/enumerate-cg-hosts.ts create mode 100644 packages/agent/src/swm/member-attestation.ts create mode 100644 packages/agent/src/swm/verify-batch.ts create mode 100644 packages/agent/test/enumerate-cg-hosts.test.ts create mode 100644 packages/agent/test/member-attestation.test.ts create mode 100644 packages/agent/test/verify-batch.test.ts create mode 100755 scripts/devnet-test-rfc38-all.sh create mode 100755 scripts/devnet-test-rfc38-cross-cg.sh create mode 100755 scripts/devnet-test-rfc38-e2e.sh create mode 100755 scripts/devnet-test-rfc38-late-joiner.sh create mode 100755 scripts/devnet-test-rfc38-lu10.sh create mode 100755 scripts/devnet-test-rfc38-lu7.sh create mode 100755 scripts/devnet-test-rfc38-lu8.sh create mode 100755 scripts/devnet-test-rfc38-lu9.sh create mode 100755 scripts/devnet-test-rfc38-multi-member.sh create mode 100755 scripts/devnet-test-rfc38-scale.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 72af563d6..baff42942 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,24 @@ All notable changes to the DKG V9 node are documented here. The format is based ## [Unreleased] +**OT-RFC-38 Phase A — Edge curator can publish curated CGs to Verified Memory; members verify post-decrypt; outsiders verify via attestation tokens.** Builds on the SPEC_CG_MEMORY_MODEL surface below by closing the curated-CG publish path end-to-end. Edge curators (no on-chain `identityId`) now create curated CGs, share private SWM with named members, publish to VM with `attributionId=0` (no-attribution mode the V10 contract already supports), and ship every member-attested verification artifact a downstream verifier needs. RFC: [`docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md`](./docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md) §1.1 and §7.1.1. + +### Added — OT-RFC-38 Phase A + +- **LU-5: curated payload AEAD wrap + identity-less publish + UI honesty** (`packages/core/src/proto/publish-intent.ts`, `packages/core/src/crypto/v10-publish-payload.ts`, `packages/publisher/src/storage-ack-handler.ts`, `packages/publisher/src/ack-collector.ts`, `packages/publisher/src/dkg-publisher.ts`, `packages/agent/src/dkg-agent.ts`, `packages/cli/src/publisher-runner.ts`, `packages/node-ui/src/ui/views/MemoryLayerView.tsx`, `packages/node-ui/src/ui/views/project/components.tsx`): adds `isEncryptedPayload` (field 14) to `PublishIntent` so cores can ACK curated payloads they cannot decrypt — they verify `chunkDigest` + `byteSize` against the persisted ciphertext, never N-Quad-parse. New `v10-publish-payload.ts` is an AES-256-GCM AEAD helper keyed off the CG chain key (HKDF). Publisher's `storage-ack-handler` branches on `isEncryptedPayload` to persist opaque ciphertext under `(contextGraphId, batchId)` and signs the existing V10 digest verbatim (no contract change). `DKGAgent._resolveEncryptInlinePayload` wires the encryption hook on the agent side for curated CGs. **Critical fix**: dropped the `publisherNodeIdentityId > 0n` gate from `DKGPublisher` — edge agents now reach the on-chain publish path in no-attribution mode (`attributionId=0n`), where the V10 contract accepts the publish as an authority delegation against the curator's wallet signature. UI surfaces the result honestly: green "Published to Verified Memory" with full `txHash` + block number only when the daemon reports `status === 'confirmed'` AND a `txHash` is present, red "NOT published to Verified Memory" with the on-chain failure reason otherwise. This closes the §1.1 bug surfaced in the RFC. +- **LU-7: SWMCatchupRequest endpoint** (`packages/cli/src/daemon/routes/memory.ts` `POST /api/shared-memory/catchup`): the daemon route lifts the existing per-peer sync substrate (`PROTOCOL_SYNC`) into a caller-initiated catchup primitive. Body `{ contextGraphId, peerId?, includeDurable?, perPeerBudgetMs? }`. Public CGs (`accessPolicy=0`) accept anonymous catchup; curated CGs (`accessPolicy=1`) run the responder's existing `authorizePrivateSyncRequest` against the requester's signed envelope (member → allowed; outsider → denied). When `peerId` is omitted the route fans out to every connected peer in parallel, bounded by `perPeerBudgetMs`. Returns per-peer triple-insertion counts so callers can tell who served the request. +- **LU-8: member post-decrypt root recompute + BatchRejected gossip** (`packages/agent/src/swm/verify-batch.ts`, `packages/cli/src/daemon/routes/memory.ts` `POST /api/shared-memory/{verify-batch,report-batch-rejection}`, `packages/agent/test/verify-batch.test.ts`): pure recompute helper hashes the member's decrypted plaintext quads using V10's `computeFlatKCRootV10` + `computeFlatKCMerkleLeafCountV10` and compares against the on-chain anchor. Mismatch returns `{ ok: false, reason: 'root-mismatch', actualRoot, expectedRoot, leafCount }`. The `verify-batch` daemon route lets callers POST `{ contextGraphId, expectedMerkleRoot, quads?, privateRoots? }`; when `quads` is omitted the route reconstructs from the local SWM or post-publish CG data graph. The `report-batch-rejection` route ships a structured `BatchRejection` record through SWM gossip via `agent.share()` so other members can sanity-check and refetch from a different host. +- **LU-9: member-attestation token mint + outsider verification** (`packages/agent/src/swm/member-attestation.ts`, `packages/cli/src/daemon/routes/memory.ts` `POST /api/attestation/{mint,verify}`, `packages/agent/test/member-attestation.test.ts`): a member signs an envelope binding `(chainId, kavAddress, contextGraphId, batchId, merkleRoot, plaintextLeafHash, attesterAddress, attestedAt)` with `keccak256(abi.encodePacked(...))` + EIP-191 secp256k1 (matches V10 chain-side signature layout — outsiders can hand-verify against `ContextGraphStorage`). The daemon `mint` route signs via the node's chain adapter; the `verify` route runs four checks: signature recovery, signer-matches-attester, optional `candidateLeaf` rehash against `plaintextLeafHash`, optional async `membershipResolver` chain hook for "was this attester a CG member at `attestedAt`". Returns structured `{ ok, recoveredSigner, signerMatchesAttester, leafCheck, membership, reason? }` so consumers can decide based on which checks passed. +- **`enumerate-cg-hosts` helper** (`packages/agent/src/swm/enumerate-cg-hosts.ts`, `packages/agent/test/enumerate-cg-hosts.test.ts`): library helper distinct from `enumerate-cg-members`. Returns the dialable peer set the LU-7 catchup primitive will try in turn (Phase A: all connected peers minus self; Phase B will refine to a sharding-table-eligible subset once shard count > 1). +- **Devnet integration tests** (`scripts/devnet-test-rfc38-*.sh`): 11 standalone end-to-end scenarios, all driven through the daemon HTTP API. `devnet-test-rfc38-all.sh` runs the full suite. Covers LU-5 (curated + public), LU-7, LU-8, LU-9, LU-10 (public-CG regression sweep), `e2e` (LU-5→LU-7→LU-8→LU-9 composed), `cross-cg` isolation (member of CG-A cannot decrypt CG-B), `multi-member` (3 distinct member wallets cross-verify the same batch + cross-verify each other's attestations), `scale` (50 triples / 25 KAs single batch), `late-joiner` (member-from-curator + member-from-member-with-curator-offline + the documented LU-6 cores-only gap as a passing fail-soft assertion). Re-runnable; every CG id is timestamp-suffixed. +- **`./scripts/devnet.sh restart-node N`** + UI proxy `DEVNET_UI_NODE` env (`scripts/devnet.sh`, `packages/node-ui/vite.config.ts`): operator surfaces for restarting one node without wiping state, and pointing the Vite dev-server proxy at any devnet node (not hardcoded to node 1). Lets the user test the edge-publish path from `http://localhost:5173/ui/` against node 5 instead of needing custom curl. + +### Deferred (Phase A sub-task, tracked for follow-up) + +- **LU-6 substrate hosting on cores**: cores do not yet subscribe to the curated-CG SWM gossip topic via the sharding-table assignment (RFC §5.1 + §5.1.1 pre-registration staging). Today's catchup model works when the curator OR any other current member is online; if every member is offline, a late joiner's catchup against cores returns 0 triples cleanly (no crash). `devnet-test-rfc38-late-joiner.sh` SCENARIO C asserts this fail-soft shape. Full LU-6 lands the encrypted SWM substrate (the `SwmSenderKey` two-layer Sender Keys construction already in `packages/core/src/crypto/swm-sender-key.ts` but not yet wired to the workspace-gossip topic) plus the TTL + byte-cap staging policies in §5.1.1. Path forward documented in `docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md` §7.1.1. + +--- + **Context Graph memory model — edge agents can create curated CGs**: the on-chain Context Graph surface no longer accepts per-CG hosting committees or per-CG ACK quorums; hosting and ACK quorum are network-level concerns (sharding table + `parametersStorage.minimumRequiredSignatures()`). This unblocks edge-node agents who have no on-chain `identityId` from registering invite-only / curators-only CGs — previously the agent SDK threw at register time when `ensureIdentity()` returned `0n`. RFC: [`docs/specs/SPEC_CG_MEMORY_MODEL.md`](./docs/specs/SPEC_CG_MEMORY_MODEL.md). Wire-format break end-to-end (contracts + ABIs + SDK + daemon + CLI + MCP + UI all rev together); no compatibility shim — every package upgrades in lockstep. ### Changed — Context Graph memory model diff --git a/docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md b/docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md index 193de5918..fa3dd999e 100644 --- a/docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md +++ b/docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md @@ -825,6 +825,27 @@ Cores hosting may face abuse vectors: 3. Member detects a malicious-publisher batch: publisher commits a root that doesn't match the SWM ciphertext members can decrypt. Member rejects the batch, alerts via SWM gossip, and the rejection propagates. 4. Outsider with `(assertion, attestation)` from a member runs the attestation-verification flow against a target batch; verification succeeds for a real attestation, fails for a tampered one or one signed by a wallet that wasn't a CG member at the attested epoch. +#### 7.1.1 Phase A — implementation status (as-shipped) + +The Phase A milestones above are mostly landed. Devnet validation lives in `scripts/devnet-test-rfc38-*.sh`; run `scripts/devnet-test-rfc38-all.sh` against a fresh 6-node devnet (4 cores + 2 edges) to exercise the full suite end-to-end. Current scope on this branch: + +| Sub-task | Source surface | Devnet test | Status | +|---|---|---|---| +| LU-5: edge curator → curated CG → VM publish (the §1.1 unblocker — `isEncryptedPayload` PublishIntent, AEAD wrap, no-attribution V10 publish) | `packages/core/src/crypto/v10-publish-payload.ts`, `packages/publisher/src/{storage-ack-handler,dkg-publisher}.ts`, `packages/agent/src/dkg-agent.ts` (`_resolveEncryptInlinePayload`) | `devnet-test-rfc38-lu5.sh` + `lu5-public.sh` | ✅ landed | +| LU-7: `SWMCatchupRequest` catchup endpoint (anon for public CGs, member-attested for curated; outsider denial) | `packages/cli/src/daemon/routes/memory.ts` (`POST /api/shared-memory/catchup`) + the existing `PROTOCOL_SYNC` substrate | `devnet-test-rfc38-lu7.sh` | ✅ landed | +| LU-8: member post-decrypt root recompute + `BatchRejected` SWM gossip | `packages/agent/src/swm/verify-batch.ts` + `POST /api/shared-memory/{verify-batch,report-batch-rejection}` | `devnet-test-rfc38-lu8.sh` | ✅ landed | +| LU-9: member-attestation token mint + outsider verification (with optional `membershipResolver` chain hook) | `packages/agent/src/swm/member-attestation.ts` + `POST /api/attestation/{mint,verify}` | `devnet-test-rfc38-lu9.sh` | ✅ landed | +| LU-10: public-CG regression sweep (publish + anonymous catchup + verify-batch + attestation, all on a public CG) | reuses LU-5/7/8/9 surfaces with `accessPolicy: 0` | `devnet-test-rfc38-lu10.sh` | ✅ landed | +| Cross-CG isolation, multi-member (3-way), scale (50 triples / 25 KAs), late-joiner (member-from-member with curator offline) | scenario coverage on top of the landed surfaces | `devnet-test-rfc38-{cross-cg,multi-member,scale,late-joiner}.sh` | ✅ landed | +| LU-6: sharding-table-driven SWM substrate subscription on cores + pre-registration staging (TTL, byte caps, ciphertext fanout to cores) so cores can serve catchup when the curator AND all live members are offline | (deferred) | `devnet-test-rfc38-late-joiner.sh` SCENARIO C documents the gap with a passing fail-soft assertion (cores-only catchup returns 0 triples cleanly, no crash) | ⚠️ deferred (see below) | + +**What "deferred LU-6" means in practice on this branch:** + +- A new member joining when the curator OR any other current member is online → catches up the full SWM history via `POST /api/shared-memory/catchup` against that peer. ✅ works. +- A new member joining when the curator AND all current members are offline → catchup against cores returns 0 triples. The endpoint shape is correct (`peersAttempted > 0`, `totalInsertedTriples == 0`, no crash); the data simply isn't there because today's cores don't subscribe to curated CG SWM gossip topics outside the member allowlist. ⚠️ gap. + +This gap is acceptable for the Phase A user-visible surface (the §1.1 bug was about *publishing*, not about a specific late-joiner pattern), but is the next thing to land for the full "scenarios 1–4 of §2.4" promise to be honest. The substrate-subscription work itself is non-trivial: it touches the `SharedMemoryHandler` apply path (currently signature-checks the publisher and applies plaintext quads; needs a parallel "store opaque ciphertext under sharding-table assignment" path) and the SWM gossip wire format (Phase B in §7.2 will move it to AEAD per §5.2; Phase A could ship a transitional "cores subscribe but only persist for members" mode if needed sooner). + ### 7.2 Phase B — Explicit key lifecycle + monetization model β **Scope**: formalise the curator's key-distribution lifecycle as explicit `KeyGrant` / `KeyRotate` messages (§5.5) and add the `PaidAccessGrant` protocol for per-assertion monetization (§5.7). diff --git a/packages/agent/src/index.ts b/packages/agent/src/index.ts index 237730fb7..510723ec4 100644 --- a/packages/agent/src/index.ts +++ b/packages/agent/src/index.ts @@ -70,6 +70,24 @@ export { type PolicyApprovalBinding, } from './ccl-policy.js'; export { DKGAgent } from './dkg-agent.js'; +export { + verifyBatch, + buildBatchRejectionRecord, + type VerifyBatchInput, + type VerifyBatchResult, + type BatchRejectionRecord, +} from './swm/verify-batch.js'; +export { createCGHostEnumerator, type CGHostEnumerator, type CGHostEnumeratorDeps } from './swm/enumerate-cg-hosts.js'; +export { + mintMemberAttestation, + verifyMemberAttestation, + computeAttestationDigest, + type MemberAttestation, + type MemberAttestationPayload, + type MintMemberAttestationInput, + type VerifyMemberAttestationInput, + type VerifyMemberAttestationResult, +} from './swm/member-attestation.js'; export { ContextGraphNotFoundError, InvalidContentError, diff --git a/packages/agent/src/swm/enumerate-cg-hosts.ts b/packages/agent/src/swm/enumerate-cg-hosts.ts new file mode 100644 index 000000000..8ef106861 --- /dev/null +++ b/packages/agent/src/swm/enumerate-cg-hosts.ts @@ -0,0 +1,78 @@ +/** + * OT-RFC-38 LU-6 (minimal) — CG hosting-peer enumeration. + * + * Distinct from `enumerate-cg-members` (which returns peers eligible + * to *decrypt* the CG content): + * + * - **members** receive a copy of the chain key and can decrypt + * curated CG payloads. Resolved via the CG allowlist or topic + * subscribers. + * - **hosts** receive the ciphertext (curated) or plaintext (public) + * bytes and store them on behalf of the network, without the + * ability to decrypt. Per SPEC_CG_HOSTING_MEMBERSHIP §3, hosting + * and membership are orthogonal node roles. + * + * Phase A simplification (per RFC §4.6): the sharding table is single- + * shard — every sharding-table member hosts every CG. The on-chain + * `nodeId` field is a random 32-byte token, not a libp2p PeerID, so the + * Phase A enumerator does NOT consult chain state directly. It instead + * returns the currently-connected libp2p peers — they are the dialable + * candidates the LU-7 catchup protocol will try in turn, with each peer + * able to decline if it doesn't actually hold the CG (e.g. because it's + * an edge node, not a core). + * + * Future shard-aware enumeration (Phase B) will: + * 1. Maintain a local (identityId → peerId) map from observed peer + * announcements (libp2p identify protocol carries peer addrs, and + * `getPeerDiagnostics()` already surfaces a peer's identityId). + * 2. Cross-reference against `ShardingTable.getShardingTable()` to + * filter to actual sharding-table members. + * 3. Apply the per-CG sharding function once shards >1 ship. + * + * Cache-free by design: callers (LU-7 catchup) invoke this on each + * SWMCatchupRequest send, so the result must reflect the current + * connection state. `getConnectedPeers()` is already an in-memory + * libp2p lookup; no I/O budget to amortise. + */ + +export interface CGHostEnumeratorDeps { + /** + * Returns the currently-connected libp2p peer IDs (strings in the + * canonical base58/base32 multiaddr form). Self is included or not + * depending on the caller's wiring; this enumerator strips self + * regardless. + */ + getConnectedPeers: () => string[]; + /** Lazy accessor for our own peer ID (excluded from results). */ + getSelfPeerId: () => string; +} + +export interface CGHostEnumerator { + /** + * Resolve the candidate hosting-peer set for {@link cgId}. Returns + * all currently-connected peers (minus self) in Phase A; Phase B + * will filter to the sharding-table-eligible subset for the CG. + * + * Non-async because the underlying source (libp2p connections list) + * is in-memory; kept as a Promise-returning signature anyway so + * Phase B can layer a chain query in without breaking callers. + */ + enumerate(cgId: string): Promise; +} + +export function createCGHostEnumerator(deps: CGHostEnumeratorDeps): CGHostEnumerator { + return { + async enumerate(_cgId: string): Promise { + const self = deps.getSelfPeerId(); + const seen = new Set(); + const out: string[] = []; + for (const peer of deps.getConnectedPeers()) { + if (peer === self) continue; + if (seen.has(peer)) continue; + seen.add(peer); + out.push(peer); + } + return out; + }, + }; +} diff --git a/packages/agent/src/swm/member-attestation.ts b/packages/agent/src/swm/member-attestation.ts new file mode 100644 index 000000000..ce0404c29 --- /dev/null +++ b/packages/agent/src/swm/member-attestation.ts @@ -0,0 +1,323 @@ +/** + * OT-RFC-38 LU-9 — Member-attested verification token. + * + * SPEC_CG_HOSTING_MEMBERSHIP §5.3.2 ("Outsider verification via member + * attestation"): + * + * "If a third party later sees a single fact ... a member can give + * them a small attestation that proves the fact's inclusion in the + * on-chain anchor." + * + * This module provides the round-trip: + * + * - `mintMemberAttestation` — a member (post-decrypt) signs a + * structured envelope binding `(chainId, kavAddress, + * contextGraphId, batchId, merkleRoot, plaintextLeafHash, + * attestedAt)` with their wallet. The output is a self-contained + * token an outsider can verify without ever holding the chain key. + * + * - `verifyMemberAttestation` — an outsider (no key, no membership) + * does three checks: + * 1. Recover the EIP-191 signer from the signature. + * 2. Confirm the signer matches `attesterAddress`. + * 3. Compare the supplied `plaintextLeafHash` against a locally- + * recomputed `hashTripleV10(quad)` if the outsider already + * has a candidate triple in hand — otherwise return the + * attestation as "structurally valid" so the caller can + * decide whether to also chain-verify membership (a + * membership-at-epoch SPARQL lookup is deferred to an + * adapter-side hook, not bundled in this module to keep the + * file zero-I/O). + * + * Design choices: + * + * - **Single-leaf attestation** is the unit. A member can mint many + * per batch (one per quoted/leaked fact). Aggregating into batch- + * wide attestations is a Phase-B size optimisation, not a Phase-A + * correctness concern. + * + * - **No chain query inside this module.** The membership lookup is + * external (caller passes in a `membershipResolver` async hook). + * Keeps the unit-testable surface tiny and avoids pulling chain + * adapters into a crypto-only helper. + * + * - **Digest layout is keccak256 over an `abi.encodePacked`-style + * concatenation.** Same shape as `computePublishACKDigest` so + * anyone already comfortable with V10 chain-side signature layout + * can hand-verify the token. + */ + +import { keccak256 } from '@origintrail-official/dkg-core'; +import { ethers } from 'ethers'; + +export interface MemberAttestationPayload { + /** Format version — currently 1. */ + version: 1; + /** EVM chain id (e.g. 1, 31337). */ + chainId: string; + /** Deployed KAV10 contract address (binds the attestation to one chain+deployment). */ + kavAddress: string; + /** On-chain context-graph ID (numeric, as a string for JSON portability). */ + contextGraphId: string; + /** Identifier for the batch the attestation covers — typically the KC id as a string. */ + batchId: string; + /** 0x-prefixed 32-byte merkle root anchored on chain. */ + merkleRoot: string; + /** 0x-prefixed 32-byte hash of the specific plaintext leaf being attested. */ + plaintextLeafHash: string; + /** Attester EVM address (must match signature recovery). */ + attesterAddress: string; + /** Unix epoch seconds when the attestation was minted. */ + attestedAt: number; +} + +export interface MemberAttestation { + payload: MemberAttestationPayload; + /** EIP-191 signature over `computeAttestationDigest(payload)`. */ + signature: string; +} + +function bytes32ToHex(bytes: Uint8Array): string { + return '0x' + Array.from(bytes).map((b) => b.toString(16).padStart(2, '0')).join(''); +} + +function hexTo32Bytes(hex: string): Uint8Array { + const clean = hex.startsWith('0x') ? hex.slice(2) : hex; + if (clean.length !== 64) throw new Error(`expected 32-byte hex, got ${clean.length / 2} bytes`); + const out = new Uint8Array(32); + for (let i = 0; i < 32; i++) out[i] = parseInt(clean.slice(i * 2, i * 2 + 2), 16); + return out; +} + +function uint256ToBytes(value: bigint | string | number): Uint8Array { + const big = typeof value === 'bigint' ? value : BigInt(value); + const out = new Uint8Array(32); + let v = big; + for (let i = 31; i >= 0; i--) { + out[i] = Number(v & 0xffn); + v >>= 8n; + } + return out; +} + +function addressToBytes(addr: string): Uint8Array { + const clean = addr.toLowerCase().replace(/^0x/, ''); + if (clean.length !== 40) throw new Error(`invalid address: ${addr}`); + const out = new Uint8Array(20); + for (let i = 0; i < 20; i++) out[i] = parseInt(clean.slice(i * 2, i * 2 + 2), 16); + return out; +} + +/** + * Compute the keccak256 digest the attester signs (EIP-191-wrapped). + * + * Layout (276 bytes packed): + * chainId : uint256 (32) + * kavAddress : address (20) + * contextGraphId : uint256 (32) + * batchId-hash : bytes32 (32) — keccak256(utf8(batchId)) (batchId is a + * string identifier — hashing makes the length fixed) + * merkleRoot : bytes32 (32) + * plaintextLeafHash : bytes32 (32) + * attesterAddress : address (20) + * attestedAt : uint256 (32) + * version : uint256 (32) + * reserved : 12 zero bytes — padding so the total + * packed width is divisible by 4 and stays + * fixed-width for any future version bump. + * + * 32+20+32+32+32+32+20+32+32+12 = 276 bytes. + */ +export function computeAttestationDigest(payload: MemberAttestationPayload): Uint8Array { + const batchIdHash = keccak256(new TextEncoder().encode(payload.batchId)); + const packed = new Uint8Array(276); + let off = 0; + packed.set(uint256ToBytes(payload.chainId), off); off += 32; + packed.set(addressToBytes(payload.kavAddress), off); off += 20; + packed.set(uint256ToBytes(payload.contextGraphId), off); off += 32; + packed.set(batchIdHash, off); off += 32; + packed.set(hexTo32Bytes(payload.merkleRoot), off); off += 32; + packed.set(hexTo32Bytes(payload.plaintextLeafHash), off); off += 32; + packed.set(addressToBytes(payload.attesterAddress), off); off += 20; + packed.set(uint256ToBytes(payload.attestedAt), off); off += 32; + packed.set(uint256ToBytes(payload.version), off); off += 32; + // remaining 12 bytes are zero (allocated by Uint8Array) + return keccak256(packed); +} + +export interface MintMemberAttestationInput { + payload: Omit & { version?: 1 }; + /** + * Async signer hook. The function is handed the digest bytes (raw + * keccak256) and must return an EIP-191 signature (a hex string with + * the standard 65-byte r|s|v shape). The agent's chain adapter's + * `signMessage(...)` returns r/vs; callers can re-serialise. + * + * Why a callback instead of a wallet: this module is wallet- + * agnostic (some adapters use hardware wallets, some use OS + * keychains); accepting a `(digest) => Promise` keeps the + * adapter boundary clean. + */ + sign: (digest: Uint8Array) => Promise; +} + +export async function mintMemberAttestation( + input: MintMemberAttestationInput, +): Promise { + const payload: MemberAttestationPayload = { ...input.payload, version: 1 }; + // Light sanity checks — a malformed token is unrecoverable later + if (!/^0x[0-9a-fA-F]{64}$/.test(payload.merkleRoot)) { + throw new Error('merkleRoot must be 0x + 64 hex chars'); + } + if (!/^0x[0-9a-fA-F]{64}$/.test(payload.plaintextLeafHash)) { + throw new Error('plaintextLeafHash must be 0x + 64 hex chars'); + } + if (!/^0x[0-9a-fA-F]{40}$/.test(payload.attesterAddress)) { + throw new Error('attesterAddress must be 0x + 40 hex chars'); + } + if (!/^0x[0-9a-fA-F]{40}$/.test(payload.kavAddress)) { + throw new Error('kavAddress must be 0x + 40 hex chars'); + } + if (!Number.isFinite(payload.attestedAt) || payload.attestedAt < 0) { + throw new Error('attestedAt must be a non-negative unix epoch (seconds)'); + } + + const digest = computeAttestationDigest(payload); + const signature = await input.sign(digest); + return { payload, signature }; +} + +export interface VerifyMemberAttestationInput { + attestation: MemberAttestation; + /** + * Optional candidate plaintext leaf bytes. When supplied, the verifier + * recomputes `keccak256(candidateLeaf)` and confirms it equals + * `payload.plaintextLeafHash` — this is the "I have the quoted fact, + * does the attestation say it's in the batch?" check (RFC §5.3.2 + * point 2). When omitted, the verifier only validates structure + + * signer recovery. + */ + candidateLeaf?: Uint8Array; + /** + * Optional async hook to confirm the attester was a member of the + * CG at attestation time. Resolves to: + * true — chain shows membership at-or-before `attestedAt` + * false — chain shows non-membership + * undefined — caller cannot determine (no chain, no adapter); the + * verifier surfaces this as `membership: 'unknown'` so + * consumers can decide whether to trust based on the + * attester identity alone. + */ + membershipResolver?: (input: { + chainId: string; + contextGraphId: string; + attesterAddress: string; + attestedAt: number; + }) => Promise; +} + +export interface VerifyMemberAttestationResult { + ok: boolean; + /** EVM address recovered from the signature. */ + recoveredSigner: string; + /** True iff `recoveredSigner === payload.attesterAddress` (case-insensitive). */ + signerMatchesAttester: boolean; + /** + * Leaf-hash comparison: + * 'match' — caller-supplied candidateLeaf hashes to + * payload.plaintextLeafHash + * 'mismatch' — supplied candidate does NOT match + * 'skipped' — caller did not pass a candidateLeaf + */ + leafCheck: 'match' | 'mismatch' | 'skipped'; + /** Membership-at-epoch outcome from the resolver, if supplied. */ + membership: 'confirmed' | 'denied' | 'unknown' | 'skipped'; + /** Human-readable reason when ok=false. */ + reason?: string; +} + +export async function verifyMemberAttestation( + input: VerifyMemberAttestationInput, +): Promise { + const { attestation, candidateLeaf, membershipResolver } = input; + const digest = computeAttestationDigest(attestation.payload); + + let recovered: string; + try { + recovered = ethers.verifyMessage(digest, attestation.signature); + } catch (err: any) { + return { + ok: false, + recoveredSigner: '', + signerMatchesAttester: false, + leafCheck: 'skipped', + membership: 'skipped', + reason: `signature recovery failed: ${err?.message ?? err}`, + }; + } + + const signerMatchesAttester = + recovered.toLowerCase() === attestation.payload.attesterAddress.toLowerCase(); + if (!signerMatchesAttester) { + return { + ok: false, + recoveredSigner: recovered, + signerMatchesAttester: false, + leafCheck: 'skipped', + membership: 'skipped', + reason: `recovered signer ${recovered} does not match attesterAddress ${attestation.payload.attesterAddress}`, + }; + } + + let leafCheck: VerifyMemberAttestationResult['leafCheck'] = 'skipped'; + if (candidateLeaf) { + const actual = bytes32ToHex(keccak256(candidateLeaf)); + leafCheck = + actual.toLowerCase() === attestation.payload.plaintextLeafHash.toLowerCase() + ? 'match' + : 'mismatch'; + if (leafCheck === 'mismatch') { + return { + ok: false, + recoveredSigner: recovered, + signerMatchesAttester: true, + leafCheck, + membership: 'skipped', + reason: 'candidateLeaf does not hash to plaintextLeafHash — the leaf supplied is not the one attested', + }; + } + } + + let membership: VerifyMemberAttestationResult['membership'] = 'skipped'; + if (membershipResolver) { + try { + const m = await membershipResolver({ + chainId: attestation.payload.chainId, + contextGraphId: attestation.payload.contextGraphId, + attesterAddress: attestation.payload.attesterAddress, + attestedAt: attestation.payload.attestedAt, + }); + membership = m === true ? 'confirmed' : m === false ? 'denied' : 'unknown'; + if (membership === 'denied') { + return { + ok: false, + recoveredSigner: recovered, + signerMatchesAttester: true, + leafCheck, + membership, + reason: 'attester was not a CG member at attestation time', + }; + } + } catch (err: any) { + membership = 'unknown'; + } + } + + return { + ok: true, + recoveredSigner: recovered, + signerMatchesAttester: true, + leafCheck, + membership, + }; +} diff --git a/packages/agent/src/swm/verify-batch.ts b/packages/agent/src/swm/verify-batch.ts new file mode 100644 index 000000000..8cffbeb2a --- /dev/null +++ b/packages/agent/src/swm/verify-batch.ts @@ -0,0 +1,209 @@ +/** + * OT-RFC-38 LU-8 — Member post-decrypt batch verification. + * + * SPEC_CG_HOSTING_MEMBERSHIP §5.3.1 ("Member verification"): + * + * "Member ... reconstructs the per-KA leaves using the existing V10 + * leaf format. For each batch they reconstruct, re-derives the merkle + * root and compares to the on-chain anchor." + * + * This module is the canonical implementation of that recompute step. + * It is intentionally *purely a recomputer* — it does NOT fetch chain + * state, does NOT touch the local store, and does NOT decrypt. Callers + * pass in: + * + * - the already-decrypted plaintext `quads` of the batch (the + * publishable payload, identical to what was leaf-hashed at publish + * time) + * - any per-KA `privateRoots` that were folded into the publisher's + * `computeFlatKCRootV10` call (matching the publisher's seal logic) + * - the `expectedRoot` retrieved from chain (V10 batch entry) — 32 + * bytes, the canonical form `KnowledgeAssetsV10` stores. + * + * The return is a small structured result; the caller decides what to + * do with rejection (the agent emits a `BatchRejected` SWM gossip + * record, see {@link buildBatchRejectionRecord}). + * + * Why a separate module rather than reusing `publisher/src/merkle.ts`: + * member-side verification has different security posture from publish + * — wrong leaf shape on the publisher's side means *their own* publish + * fails locally; wrong leaf shape on the verifier's side means a + * malicious publisher slips a bad batch through. Pinning verification + * to a thin, audited surface that does ONLY the recompute+compare keeps + * the security boundary small. + */ + +import type { Quad } from '@origintrail-official/dkg-storage'; +import { keccak256 } from '@origintrail-official/dkg-core'; +import { + computeFlatKCRootV10, + computeFlatKCMerkleLeafCountV10, +} from '@origintrail-official/dkg-publisher'; + +export interface VerifyBatchInput { + /** + * Decrypted plaintext quads of the batch. Order does not matter: + * V10 merkle is sort+dedupe, so any caller-supplied permutation + * produces the same root. + */ + quads: Quad[]; + + /** + * Optional per-KA private roots that the publisher folded into the + * flat KC root via {@link computeFlatKCRootV10}. For curated + * single-KA batches (the LU-5 path) this is typically empty — the + * payload itself is private and the whole batch hashes as a single + * tree. + */ + privateRoots?: Uint8Array[]; + + /** + * The on-chain anchor (V10 batch `merkleRoot`). 32 bytes (`bytes32`). + */ + expectedRoot: Uint8Array; +} + +export interface VerifyBatchResult { + ok: boolean; + /** Expected root (echo of the input, hex-encoded for diagnostics). */ + expectedRoot: string; + /** Recomputed root from the supplied plaintext, hex-encoded. */ + actualRoot: string; + /** Leaf count after V10 sort+dedupe. */ + leafCount: number; + /** + * When `ok === false`, the most actionable next step the caller can + * surface in logs / UI. The verify-batch contract is "tell me what + * happened", not "fix it" — the caller chooses whether to retry + * fetch from a different host (LU-7 catchup), emit a `BatchRejected` + * gossip (this module's `buildBatchRejectionRecord` helper), or both. + */ + reason?: 'root-mismatch' | 'empty-quads' | 'invalid-expected-root'; +} + +const ZERO_ROOT = new Uint8Array(32); + +function bytesToHex(bytes: Uint8Array): string { + return '0x' + Array.from(bytes).map((b) => b.toString(16).padStart(2, '0')).join(''); +} + +export function verifyBatch(input: VerifyBatchInput): VerifyBatchResult { + if (input.expectedRoot.length !== 32) { + return { + ok: false, + expectedRoot: bytesToHex(input.expectedRoot), + actualRoot: bytesToHex(ZERO_ROOT), + leafCount: 0, + reason: 'invalid-expected-root', + }; + } + + if (input.quads.length === 0 && (input.privateRoots?.length ?? 0) === 0) { + return { + ok: false, + expectedRoot: bytesToHex(input.expectedRoot), + actualRoot: bytesToHex(ZERO_ROOT), + leafCount: 0, + reason: 'empty-quads', + }; + } + + const privateRoots = input.privateRoots ?? []; + const actualRoot = computeFlatKCRootV10(input.quads, privateRoots); + const leafCount = computeFlatKCMerkleLeafCountV10(input.quads, privateRoots); + + let match = true; + for (let i = 0; i < 32; i++) { + if (actualRoot[i] !== input.expectedRoot[i]) { + match = false; + break; + } + } + + return { + ok: match, + expectedRoot: bytesToHex(input.expectedRoot), + actualRoot: bytesToHex(actualRoot), + leafCount, + ...(match ? {} : { reason: 'root-mismatch' as const }), + }; +} + +/** + * SPEC §5.3.1: "On mismatch: reject batch, alert via SWM, retry from a + * different hosting core." + * + * `buildBatchRejectionRecord` shapes the alert as a structured record + * suitable for SWM gossip. The shape is intentionally minimal — every + * field has a clear consumer downstream: + * + * - `contextGraphId` — scopes who cares about the rejection (CG + * subscribers). + * - `batchId` / `merkleRoot` — identifies the batch unambiguously so + * receivers can correlate with their own pending verifies. + * - `actualRoot` — what the rejecter computed. Lets other members + * independently sanity-check whether they would have rejected too. + * - `rejectedBy` — agent address (and optional peerId) for + * attribution. Important for any future "did this rejecter ALSO + * get punished for spurious rejections" workflow (Phase B). + * - `reportedAt` — wall-clock for ordering / rate-limiting on the + * consumer side. + * - `digest` — keccak256 of all the above so the SWM consumer can + * hash-dedupe identical rejection reports. + * + * Returns a plain JS object — callers serialise into SWM (typically as + * RDF triples under `did:dkg:batch-rejection:` in the + * `_shared_memory` named graph). Doing the serialisation here would + * pull in DKG RDF helpers and tie this module to a specific store + * shape; a tiny pure object is the right level of indirection. + */ +export interface BatchRejectionRecord { + contextGraphId: string; + batchId?: string; + expectedRoot: string; + actualRoot: string; + reason: VerifyBatchResult['reason']; + rejectedBy: { + agentAddress: string; + peerId?: string; + }; + reportedAt: string; + digest: string; +} + +export function buildBatchRejectionRecord(input: { + contextGraphId: string; + batchId?: string; + verifyResult: VerifyBatchResult; + rejectedBy: { agentAddress: string; peerId?: string }; + now?: () => Date; +}): BatchRejectionRecord { + if (input.verifyResult.ok) { + throw new Error('buildBatchRejectionRecord called on an ok verify result; nothing to reject'); + } + + const reportedAt = (input.now ?? (() => new Date()))().toISOString(); + const digestInput = new TextEncoder().encode( + [ + input.contextGraphId, + input.batchId ?? '', + input.verifyResult.expectedRoot, + input.verifyResult.actualRoot, + input.verifyResult.reason ?? 'unknown', + input.rejectedBy.agentAddress, + reportedAt, + ].join('|'), + ); + const digest = bytesToHex(keccak256(digestInput)); + + return { + contextGraphId: input.contextGraphId, + ...(input.batchId !== undefined ? { batchId: input.batchId } : {}), + expectedRoot: input.verifyResult.expectedRoot, + actualRoot: input.verifyResult.actualRoot, + reason: input.verifyResult.reason, + rejectedBy: { ...input.rejectedBy }, + reportedAt, + digest, + }; +} diff --git a/packages/agent/test/enumerate-cg-hosts.test.ts b/packages/agent/test/enumerate-cg-hosts.test.ts new file mode 100644 index 000000000..7c61c38aa --- /dev/null +++ b/packages/agent/test/enumerate-cg-hosts.test.ts @@ -0,0 +1,45 @@ +import { describe, it, expect } from 'vitest'; +import { createCGHostEnumerator } from '../src/swm/enumerate-cg-hosts.js'; + +describe('createCGHostEnumerator', () => { + it('returns connected peers minus self', async () => { + const enumerator = createCGHostEnumerator({ + getConnectedPeers: () => ['peerA', 'peerB', 'selfPeer', 'peerC'], + getSelfPeerId: () => 'selfPeer', + }); + expect(await enumerator.enumerate('cg-1')).toEqual(['peerA', 'peerB', 'peerC']); + }); + + it('dedups duplicate peer ids', async () => { + const enumerator = createCGHostEnumerator({ + getConnectedPeers: () => ['peerA', 'peerA', 'peerB', 'peerB', 'peerA'], + getSelfPeerId: () => 'selfPeer', + }); + expect(await enumerator.enumerate('cg-1')).toEqual(['peerA', 'peerB']); + }); + + it('returns empty list when only self is connected', async () => { + const enumerator = createCGHostEnumerator({ + getConnectedPeers: () => ['selfPeer'], + getSelfPeerId: () => 'selfPeer', + }); + expect(await enumerator.enumerate('cg-1')).toEqual([]); + }); + + it('preserves insertion order', async () => { + const enumerator = createCGHostEnumerator({ + getConnectedPeers: () => ['z', 'a', 'm', 'b'], + getSelfPeerId: () => 'self', + }); + expect(await enumerator.enumerate('cg-1')).toEqual(['z', 'a', 'm', 'b']); + }); + + it('cgId argument is currently informational (Phase A returns all hosts uniformly)', async () => { + const enumerator = createCGHostEnumerator({ + getConnectedPeers: () => ['peerA', 'peerB'], + getSelfPeerId: () => 'self', + }); + expect(await enumerator.enumerate('cg-foo')).toEqual(['peerA', 'peerB']); + expect(await enumerator.enumerate('cg-bar')).toEqual(['peerA', 'peerB']); + }); +}); diff --git a/packages/agent/test/member-attestation.test.ts b/packages/agent/test/member-attestation.test.ts new file mode 100644 index 000000000..5ffc7e12e --- /dev/null +++ b/packages/agent/test/member-attestation.test.ts @@ -0,0 +1,180 @@ +import { describe, it, expect } from 'vitest'; +import { ethers } from 'ethers'; +import { keccak256, hashTripleV10 } from '@origintrail-official/dkg-core'; +import { + mintMemberAttestation, + verifyMemberAttestation, + computeAttestationDigest, + type MemberAttestation, + type MemberAttestationPayload, +} from '../src/swm/member-attestation.js'; + +function bytesToHex(b: Uint8Array): string { + return '0x' + Array.from(b).map((x) => x.toString(16).padStart(2, '0')).join(''); +} + +const MEMBER_WALLET = new ethers.Wallet('0x' + '11'.repeat(32)); +const OUTSIDER_WALLET = new ethers.Wallet('0x' + '22'.repeat(32)); +const KAV_ADDR = '0x' + '5fbdb2315678afecb367f032d93f642f64180aa3'.padStart(40, '0'); + +const samplePayload: Omit = { + chainId: '31337', + kavAddress: KAV_ADDR, + contextGraphId: '42', + batchId: '7', + merkleRoot: '0x' + 'ab'.repeat(32), + plaintextLeafHash: '0x' + 'cd'.repeat(32), + attesterAddress: MEMBER_WALLET.address, + attestedAt: 1779580000, +}; + +async function mint(): Promise { + return mintMemberAttestation({ + payload: samplePayload, + sign: async (digest) => MEMBER_WALLET.signMessage(digest), + }); +} + +describe('mintMemberAttestation', () => { + it('produces a structurally valid token (version=1, signature, payload mirror)', async () => { + const att = await mint(); + expect(att.payload.version).toBe(1); + expect(att.payload.attesterAddress).toBe(MEMBER_WALLET.address); + expect(att.signature).toMatch(/^0x[0-9a-fA-F]+$/); + }); + + it('throws when attesterAddress is malformed', async () => { + await expect( + mintMemberAttestation({ + payload: { ...samplePayload, attesterAddress: '0xnotvalid' } as any, + sign: async () => '0x' + '00'.repeat(65), + }), + ).rejects.toThrow(/attesterAddress/); + }); + + it('throws when merkleRoot is not 32 bytes', async () => { + await expect( + mintMemberAttestation({ + payload: { ...samplePayload, merkleRoot: '0xabcd' } as any, + sign: async () => '0x' + '00'.repeat(65), + }), + ).rejects.toThrow(/merkleRoot/); + }); +}); + +describe('computeAttestationDigest', () => { + it('is deterministic for identical payloads', () => { + const p1: MemberAttestationPayload = { ...samplePayload, version: 1 }; + const p2: MemberAttestationPayload = { ...samplePayload, version: 1 }; + expect(bytesToHex(computeAttestationDigest(p1))).toEqual(bytesToHex(computeAttestationDigest(p2))); + }); + + it('changes when any field changes', () => { + const p1: MemberAttestationPayload = { ...samplePayload, version: 1 }; + const p2: MemberAttestationPayload = { ...samplePayload, version: 1, attestedAt: samplePayload.attestedAt + 1 }; + expect(bytesToHex(computeAttestationDigest(p1))).not.toEqual(bytesToHex(computeAttestationDigest(p2))); + }); + + it('binds chain+contract identity (changing kavAddress changes digest)', () => { + const p1: MemberAttestationPayload = { ...samplePayload, version: 1 }; + const p2: MemberAttestationPayload = { ...samplePayload, version: 1, kavAddress: '0x' + 'ff'.repeat(20) }; + expect(bytesToHex(computeAttestationDigest(p1))).not.toEqual(bytesToHex(computeAttestationDigest(p2))); + }); +}); + +describe('verifyMemberAttestation', () => { + it('roundtrips: minted by member, recovers to member, ok=true', async () => { + const att = await mint(); + const res = await verifyMemberAttestation({ attestation: att }); + expect(res.ok).toBe(true); + expect(res.recoveredSigner.toLowerCase()).toBe(MEMBER_WALLET.address.toLowerCase()); + expect(res.signerMatchesAttester).toBe(true); + expect(res.leafCheck).toBe('skipped'); + }); + + it('rejects when the signature was made by a different wallet', async () => { + const att = await mintMemberAttestation({ + payload: samplePayload, + sign: async (d) => OUTSIDER_WALLET.signMessage(d), + }); + const res = await verifyMemberAttestation({ attestation: att }); + expect(res.ok).toBe(false); + expect(res.signerMatchesAttester).toBe(false); + expect(res.reason).toMatch(/does not match/); + }); + + it('rejects when the signature is tampered', async () => { + const att = await mint(); + const corrupted: MemberAttestation = { + ...att, + signature: att.signature.slice(0, -4) + '0000', + }; + const res = await verifyMemberAttestation({ attestation: corrupted }); + expect(res.ok).toBe(false); + }); + + it('rejects when the payload is tampered (signature stays the same but digest moves)', async () => { + const att = await mint(); + const tampered: MemberAttestation = { + ...att, + payload: { ...att.payload, attestedAt: att.payload.attestedAt + 1 }, + }; + const res = await verifyMemberAttestation({ attestation: tampered }); + expect(res.ok).toBe(false); + }); + + it('leafCheck=match when caller supplies the right leaf bytes', async () => { + const leafBytes = hashTripleV10('urn:s', 'urn:p', '"o"'); + const leafHashHex = bytesToHex(keccak256(leafBytes)); + const att = await mintMemberAttestation({ + payload: { ...samplePayload, plaintextLeafHash: leafHashHex }, + sign: async (d) => MEMBER_WALLET.signMessage(d), + }); + const res = await verifyMemberAttestation({ attestation: att, candidateLeaf: leafBytes }); + expect(res.ok).toBe(true); + expect(res.leafCheck).toBe('match'); + }); + + it('leafCheck=mismatch and ok=false when wrong leaf bytes are supplied', async () => { + const leafBytes = hashTripleV10('urn:s', 'urn:p', '"o"'); + const otherLeaf = hashTripleV10('urn:s', 'urn:p', '"different"'); + const att = await mintMemberAttestation({ + payload: { ...samplePayload, plaintextLeafHash: bytesToHex(keccak256(leafBytes)) }, + sign: async (d) => MEMBER_WALLET.signMessage(d), + }); + const res = await verifyMemberAttestation({ attestation: att, candidateLeaf: otherLeaf }); + expect(res.ok).toBe(false); + expect(res.leafCheck).toBe('mismatch'); + }); + + it('membership=confirmed when resolver returns true', async () => { + const att = await mint(); + const res = await verifyMemberAttestation({ + attestation: att, + membershipResolver: async () => true, + }); + expect(res.ok).toBe(true); + expect(res.membership).toBe('confirmed'); + }); + + it('membership=denied flips ok to false', async () => { + const att = await mint(); + const res = await verifyMemberAttestation({ + attestation: att, + membershipResolver: async () => false, + }); + expect(res.ok).toBe(false); + expect(res.membership).toBe('denied'); + expect(res.reason).toMatch(/not a CG member/); + }); + + it('membership=unknown when resolver returns undefined (does not flip ok)', async () => { + const att = await mint(); + const res = await verifyMemberAttestation({ + attestation: att, + membershipResolver: async () => undefined, + }); + expect(res.ok).toBe(true); + expect(res.membership).toBe('unknown'); + }); +}); diff --git a/packages/agent/test/verify-batch.test.ts b/packages/agent/test/verify-batch.test.ts new file mode 100644 index 000000000..6c4a5405d --- /dev/null +++ b/packages/agent/test/verify-batch.test.ts @@ -0,0 +1,145 @@ +import { describe, it, expect } from 'vitest'; +import { computeFlatKCRootV10 } from '@origintrail-official/dkg-publisher'; +import type { Quad } from '@origintrail-official/dkg-storage'; +import { verifyBatch, buildBatchRejectionRecord } from '../src/swm/verify-batch.js'; + +const Q = (s: string, p: string, o: string, g = ''): Quad => ({ + subject: s, + predicate: p, + object: o, + graph: g, +}); + +const sampleQuads: Quad[] = [ + Q('urn:lu8/item1', 'http://schema.org/name', '"Alpha"'), + Q('urn:lu8/item2', 'http://schema.org/name', '"Beta"'), + Q('urn:lu8/item3', 'http://schema.org/name', '"Gamma"'), +]; + +describe('verifyBatch', () => { + it('returns ok=true when the recomputed root matches the expected root', () => { + const expected = computeFlatKCRootV10(sampleQuads, []); + const result = verifyBatch({ quads: sampleQuads, expectedRoot: expected }); + expect(result.ok).toBe(true); + expect(result.actualRoot).toEqual(result.expectedRoot); + expect(result.leafCount).toBeGreaterThanOrEqual(1); + expect(result.reason).toBeUndefined(); + }); + + it('returns ok=false with root-mismatch when quads differ from the publisher', () => { + const expected = computeFlatKCRootV10(sampleQuads, []); + const tampered = [...sampleQuads, Q('urn:lu8/injected', 'http://schema.org/name', '"Mallory"')]; + const result = verifyBatch({ quads: tampered, expectedRoot: expected }); + expect(result.ok).toBe(false); + expect(result.reason).toBe('root-mismatch'); + expect(result.actualRoot).not.toEqual(result.expectedRoot); + }); + + it('returns ok=false with empty-quads when no plaintext is supplied', () => { + const expected = computeFlatKCRootV10(sampleQuads, []); + const result = verifyBatch({ quads: [], expectedRoot: expected }); + expect(result.ok).toBe(false); + expect(result.reason).toBe('empty-quads'); + }); + + it('returns ok=false with invalid-expected-root when expectedRoot is not 32 bytes', () => { + const result = verifyBatch({ quads: sampleQuads, expectedRoot: new Uint8Array(16) }); + expect(result.ok).toBe(false); + expect(result.reason).toBe('invalid-expected-root'); + }); + + it('is order-independent: shuffled quads produce the same root', () => { + const expected = computeFlatKCRootV10(sampleQuads, []); + const shuffled = [sampleQuads[2], sampleQuads[0], sampleQuads[1]]; + const result = verifyBatch({ quads: shuffled, expectedRoot: expected }); + expect(result.ok).toBe(true); + }); + + it('folds in privateRoots when supplied (matching publisher seal)', () => { + const privateRoot = new Uint8Array(32); + privateRoot.fill(0xaa); + const expected = computeFlatKCRootV10(sampleQuads, [privateRoot]); + const okWithPrivate = verifyBatch({ + quads: sampleQuads, + privateRoots: [privateRoot], + expectedRoot: expected, + }); + expect(okWithPrivate.ok).toBe(true); + + const failWithoutPrivate = verifyBatch({ + quads: sampleQuads, + expectedRoot: expected, + }); + expect(failWithoutPrivate.ok).toBe(false); + expect(failWithoutPrivate.reason).toBe('root-mismatch'); + }); +}); + +describe('buildBatchRejectionRecord', () => { + const expected = computeFlatKCRootV10(sampleQuads, []); + const tampered = [...sampleQuads, Q('urn:lu8/injected', 'http://schema.org/name', '"Mallory"')]; + const verifyResult = verifyBatch({ quads: tampered, expectedRoot: expected }); + + it('constructs a structured record from a failed verifyResult', () => { + const record = buildBatchRejectionRecord({ + contextGraphId: 'agent/lu8-curated-1', + batchId: 'batch-7', + verifyResult, + rejectedBy: { agentAddress: '0xMember', peerId: 'memberPeer' }, + now: () => new Date('2026-05-24T00:00:00.000Z'), + }); + expect(record.contextGraphId).toBe('agent/lu8-curated-1'); + expect(record.batchId).toBe('batch-7'); + expect(record.expectedRoot).toBe(verifyResult.expectedRoot); + expect(record.actualRoot).toBe(verifyResult.actualRoot); + expect(record.reason).toBe('root-mismatch'); + expect(record.rejectedBy).toEqual({ agentAddress: '0xMember', peerId: 'memberPeer' }); + expect(record.reportedAt).toBe('2026-05-24T00:00:00.000Z'); + expect(record.digest).toMatch(/^0x[0-9a-f]{64}$/); + }); + + it('produces stable digests for identical inputs (idempotent dedupe key)', () => { + const a = buildBatchRejectionRecord({ + contextGraphId: 'agent/lu8', + verifyResult, + rejectedBy: { agentAddress: '0xM' }, + now: () => new Date('2026-05-24T00:00:00.000Z'), + }); + const b = buildBatchRejectionRecord({ + contextGraphId: 'agent/lu8', + verifyResult, + rejectedBy: { agentAddress: '0xM' }, + now: () => new Date('2026-05-24T00:00:00.000Z'), + }); + expect(a.digest).toBe(b.digest); + }); + + it('produces distinct digests when the rejecter or batchId differs', () => { + const a = buildBatchRejectionRecord({ + contextGraphId: 'agent/lu8', + batchId: 'batch-a', + verifyResult, + rejectedBy: { agentAddress: '0xM' }, + now: () => new Date('2026-05-24T00:00:00.000Z'), + }); + const b = buildBatchRejectionRecord({ + contextGraphId: 'agent/lu8', + batchId: 'batch-b', + verifyResult, + rejectedBy: { agentAddress: '0xM' }, + now: () => new Date('2026-05-24T00:00:00.000Z'), + }); + expect(a.digest).not.toBe(b.digest); + }); + + it('throws when called on a successful verifyResult', () => { + const okResult = verifyBatch({ quads: sampleQuads, expectedRoot: expected }); + expect(() => + buildBatchRejectionRecord({ + contextGraphId: 'agent/lu8', + verifyResult: okResult, + rejectedBy: { agentAddress: '0xM' }, + }), + ).toThrow(/ok verify result/); + }); +}); diff --git a/packages/cli/src/daemon/routes/assertion.ts b/packages/cli/src/daemon/routes/assertion.ts index cbbbd6308..1851ef450 100644 --- a/packages/cli/src/daemon/routes/assertion.ts +++ b/packages/cli/src/daemon/routes/assertion.ts @@ -3154,6 +3154,44 @@ export async function handleAssertionRoutes(ctx: RequestContext): Promise return; } + // GET /api/kc/:id — chain-confirmed metadata for a knowledge + // collection. Returns latest merkleRoot (hex) and the author tuple. + // Useful for LU-8 verification: members fetch the merkleRoot off-chain + // and feed it to verify-batch as `expectedMerkleRoot`. + if (req.method === 'GET' && /^\/api\/kc\/[^/]+$/.test(path)) { + const idStr = decodeURIComponent(path.split('/')[3] ?? ''); + if (!/^\d+$/.test(idStr)) { + return jsonResponse(res, 400, { + error: 'Invalid kcId — must be a non-negative integer', + }); + } + const kcId = BigInt(idStr); + try { + const chain: any = (agent as any).chain ?? (agent as any).chainAdapter; + if (!chain?.getLatestMerkleRoot) { + return jsonResponse(res, 503, { + error: 'Chain adapter does not expose getLatestMerkleRoot', + }); + } + const rootBytes: Uint8Array = await chain.getLatestMerkleRoot(kcId); + const rootHex = '0x' + Array.from(rootBytes).map((b: number) => b.toString(16).padStart(2, '0')).join(''); + let author: string | null = null; + try { + if (typeof agent.getKnowledgeCollectionAuthor === 'function') { + const a = await agent.getKnowledgeCollectionAuthor(kcId); + if (a && a !== '0x0000000000000000000000000000000000000000') author = a; + } + } catch { /* attestation lookup is optional */ } + return jsonResponse(res, 200, { + kcId: idStr, + merkleRoot: rootHex, + author, + }); + } catch (err: any) { + return jsonResponse(res, 500, { error: err?.message ?? String(err) }); + } + } + // GET /api/kc/:id/author — chain-confirmed author for a knowledge // collection's latest merkle-root entry. // diff --git a/packages/cli/src/daemon/routes/memory.ts b/packages/cli/src/daemon/routes/memory.ts index 14219fa47..2b5261ab2 100644 --- a/packages/cli/src/daemon/routes/memory.ts +++ b/packages/cli/src/daemon/routes/memory.ts @@ -545,6 +545,506 @@ WHERE { } } + // POST /api/shared-memory/catchup + // + // OT-RFC-38 LU-7 — explicit SWMCatchupRequest endpoint. Pulls the + // remote SWM state for one or more context graphs from connected + // peers, applying everything authorized into the local triple store. + // + // Body: { contextGraphId: string | string[], peerId?: string } + // - peerId: optional. When set, sync only from this specific peer. + // When omitted, iterate ALL currently-connected libp2p peers and + // try each — first peer that authorises serves the request, + // subsequent peers' decisions are independent. + // + // Returns: per-peer outcome with inserted/fetched counters. + // + // Auth model (per SPEC_CG_HOSTING_MEMBERSHIP §5.6.4): + // - Public CGs (accessPolicy == 0): the responder's sync handler + // accepts anonymous catchup (no `authorizePrivateSyncRequest` + // gate). Any reachable peer can backfill SWM. + // - Curated CGs (accessPolicy == 1): the responder's sync handler + // runs `authorizePrivateSyncRequest`, which verifies the + // requester's signed envelope against the CG's + // `agentGateAddresses` / `allowedPeers` set. Members get + // served; outsiders get a `syncDeniedResponse`. + // - Token-bearer (outsider-with-curator-issued-bearer): not yet + // implemented; tracked under LU-9 member-attestation work. + if (req.method === "POST" && path === "/api/shared-memory/catchup") { + const body = await readBody(req, SMALL_BODY_BYTES); + const parsed = safeParseJson(body, res); + if (!parsed) return; + const peerIdParam = typeof parsed.peerId === 'string' ? parsed.peerId.trim() : undefined; + const cgIdsInput = Array.isArray(parsed.contextGraphId) + ? parsed.contextGraphId + : parsed.contextGraphId !== undefined + ? [parsed.contextGraphId] + : []; + const cgIds: string[] = []; + for (const id of cgIdsInput) { + if (typeof id !== 'string' || !validateRequiredContextGraphId(id, res)) return; + cgIds.push(id); + } + if (cgIds.length === 0) { + return jsonResponse(res, 400, { + error: + 'Missing "contextGraphId" — pass a single context graph id string or an array of ids', + }); + } + + // OT-RFC-38 LU-7: SWMCatchupRequest is SWM-only. The durable + // (knowledge-collection) layer has its own publish-time + // commit→fanout→ACK protocol and a separate sync substrate; it's + // out of scope for the catchup endpoint and would otherwise compound + // the request budget (240s vs 120s). Opt-in via includeDurable=true + // for callers that want the full data leg in the same call. + const includeDurable = parsed.includeDurable === true; + + // Per-peer hard cap on the catchup duration. Keeps the endpoint + // response within a single HTTP-level timeout even if the underlying + // sync internals retry their way to completion. SWM-only path: + // ~45s/page * a couple of pages worst-case; under heavy gossip + // load (the integration suite) backed-off retries can stretch this + // out further. Underlying SYNC_TOTAL_TIMEOUT_MS in dkg-agent is + // 120s, so use 110s by default and let callers override via the + // request body for slow or congested networks. + const DEFAULT_PER_PEER_SWM_BUDGET_MS = 110_000; + const DEFAULT_PER_PEER_DURABLE_BUDGET_MS = 110_000; + const PER_PEER_SWM_BUDGET_MS = (typeof parsed.perPeerBudgetMs === 'number' && parsed.perPeerBudgetMs > 0) + ? Math.min(parsed.perPeerBudgetMs, 300_000) + : DEFAULT_PER_PEER_SWM_BUDGET_MS; + const PER_PEER_DURABLE_BUDGET_MS = (typeof parsed.perPeerDurableBudgetMs === 'number' && parsed.perPeerDurableBudgetMs > 0) + ? Math.min(parsed.perPeerDurableBudgetMs, 300_000) + : DEFAULT_PER_PEER_DURABLE_BUDGET_MS; + + const withTimeout = (p: Promise, ms: number, label: string): Promise => + new Promise((resolve, reject) => { + const t = setTimeout(() => reject(new Error(`${label} timed out after ${ms}ms`)), ms); + p.then( + (v) => { clearTimeout(t); resolve(v); }, + (e) => { clearTimeout(t); reject(e); }, + ); + }); + + // Discover candidate peers. The single-peer mode is opt-in; the + // default fan-out mode mirrors what runSyncOnConnect does on every + // peer:connect event, but caller-initiated rather than event-driven. + let candidatePeers: string[]; + if (peerIdParam) { + candidatePeers = [peerIdParam]; + } else { + candidatePeers = agent.node.libp2p + .getConnections() + .map((c: any) => c.remotePeer.toString()); + const selfPeer = agent.peerId; + candidatePeers = Array.from(new Set(candidatePeers.filter((p: string) => p !== selfPeer))); + } + + if (candidatePeers.length === 0) { + return jsonResponse(res, 200, { + contextGraphIds: cgIds, + peersAttempted: 0, + results: [], + hint: 'No connected peers to catch up from. Wait for inbound connections or pass an explicit `peerId`.', + }); + } + + // Parallelize across peers — each peer's sync is independent + // and the per-peer dial+request takes 5-20s on devnet. Serial + // iteration over N peers would compound to N×20s, easily + // exceeding the daemon's default request timeout. + const settled = await Promise.allSettled( + candidatePeers.map(async (candidate) => { + let swm = 0; + let durable = 0; + let swmError: string | undefined; + let durableError: string | undefined; + try { + swm = await withTimeout( + agent.syncSharedMemoryFromPeer(candidate, cgIds), + PER_PEER_SWM_BUDGET_MS, + `SWM catchup from ${candidate}`, + ); + } catch (err: any) { + swmError = err?.message ?? String(err); + } + if (includeDurable) { + try { + durable = await withTimeout( + (agent as any).syncFromPeer?.(candidate, cgIds) ?? Promise.resolve(0), + PER_PEER_DURABLE_BUDGET_MS, + `Durable catchup from ${candidate}`, + ); + } catch (err: any) { + durableError = err?.message ?? String(err); + } + } + return { peerId: candidate, insertedTriples: swm, durableInsertedTriples: durable, swmError, durableError }; + }), + ); + + const results = settled.map((s, idx) => { + if (s.status === 'fulfilled') { + return { + peerId: candidatePeers[idx], + insertedTriples: s.value.insertedTriples, + durableInsertedTriples: s.value.durableInsertedTriples, + ...(s.value.swmError ? { swmError: s.value.swmError } : {}), + ...(s.value.durableError ? { durableError: s.value.durableError } : {}), + }; + } + return { + peerId: candidatePeers[idx], + insertedTriples: 0, + durableInsertedTriples: 0, + error: s.reason?.message ?? String(s.reason), + }; + }); + + const totalInserted = results.reduce((sum, r) => sum + r.insertedTriples, 0); + const totalDurable = results.reduce((sum, r) => sum + (r.durableInsertedTriples ?? 0), 0); + return jsonResponse(res, 200, { + contextGraphIds: cgIds, + peersAttempted: candidatePeers.length, + includeDurable, + totalInsertedTriples: totalInserted, + totalDurableInsertedTriples: totalDurable, + results, + }); + } + + // Tiny local helper — kept inline to avoid adding a new import for + // a single use; the existing route module already has utilities + // for hex/bytes interop scattered across the file but none are + // strictly typed `bytes32`. 64-char hex (no 0x) → 32-byte buffer. + function hexToBytes32(h: string): Uint8Array { + const clean = h.startsWith('0x') ? h.slice(2) : h; + if (clean.length !== 64) throw new Error('expected 32-byte hex'); + const out = new Uint8Array(32); + for (let i = 0; i < 32; i++) out[i] = parseInt(clean.slice(i * 2, i * 2 + 2), 16); + return out; + } + + // POST /api/shared-memory/verify-batch + // + // OT-RFC-38 LU-8 — Member post-decrypt batch verification. + // + // SPEC_CG_HOSTING_MEMBERSHIP §5.3.1: members re-derive the plaintext + // merkle root from a reconstructed batch and compare to the on-chain + // anchor. This endpoint exposes the recompute step. + // + // Body: { + // contextGraphId: string, + // expectedMerkleRoot: hex32 string ("0x" + 64 hex chars), + // quads?: Quad[], // if omitted, fetched from local SWM + // subGraphName?: string, // narrows the SWM source slice + // privateRoots?: hex32[], // optional per-KA private sub-roots + // batchId?: string, // round-tripped into rejection record + // } + // + // Returns: { ok, expectedRoot, actualRoot, leafCount, reason? } + if (req.method === "POST" && path === "/api/shared-memory/verify-batch") { + const body = await readBody(req, SMALL_BODY_BYTES); + const parsed = safeParseJson(body, res); + if (!parsed) return; + const contextGraphId = parsed.contextGraphId; + if (!validateRequiredContextGraphId(contextGraphId, res)) return; + const subGraphName = parsed.subGraphName; + if (subGraphName !== undefined && !validateOptionalSubGraphName(subGraphName, res)) return; + const expectedHex = String(parsed.expectedMerkleRoot ?? ''); + if (!/^0x[0-9a-fA-F]{64}$/.test(expectedHex)) { + return jsonResponse(res, 400, { + error: 'expectedMerkleRoot must be a 0x-prefixed 32-byte hex string', + }); + } + const expectedRoot = hexToBytes32(expectedHex); + const privateRootsHex = Array.isArray(parsed.privateRoots) ? parsed.privateRoots : []; + const privateRoots: Uint8Array[] = []; + for (const ph of privateRootsHex) { + if (typeof ph !== 'string' || !/^0x[0-9a-fA-F]{64}$/.test(ph)) { + return jsonResponse(res, 400, { + error: 'privateRoots[*] must be 0x-prefixed 32-byte hex strings', + }); + } + privateRoots.push(hexToBytes32(ph)); + } + let quads: Array<{ subject: string; predicate: string; object: string; graph: string }> = []; + if (Array.isArray(parsed.quads)) { + quads = parsed.quads.map((q: any) => ({ + subject: String(q.subject), + predicate: String(q.predicate), + object: String(q.object), + graph: String(q.graph ?? ''), + })); + } else { + // Reconstruct from local store. Try in order: + // 1. _shared_memory (live SWM, before promote-to-VM) + // 2. CG data graph (post-publish — selection moves quads from + // SWM into the named-graph as part of the seal step) + // Either is valid input for verification: the publisher hashed + // the plaintext leaves once; wherever those triples now live + // locally, recomputing the root over them must match the on-chain + // commitment. + const swmGraphUri = contextGraphSharedMemoryUri(contextGraphId, subGraphName); + const dataGraphUri = `did:dkg:context-graph:${contextGraphId}`; + try { + const swmResult = await (agent as any).store.query( + `SELECT ?s ?p ?o WHERE { GRAPH <${swmGraphUri}> { ?s ?p ?o } }`, + ); + if (swmResult?.type === 'bindings') { + for (const b of swmResult.bindings) { + quads.push({ subject: b['s'], predicate: b['p'], object: b['o'], graph: '' }); + } + } + if (quads.length === 0) { + const dataResult = await (agent as any).store.query( + `SELECT ?s ?p ?o WHERE { GRAPH <${dataGraphUri}> { ?s ?p ?o } }`, + ); + if (dataResult?.type === 'bindings') { + for (const b of dataResult.bindings) { + quads.push({ subject: b['s'], predicate: b['p'], object: b['o'], graph: '' }); + } + } + } + } catch (err: any) { + return jsonResponse(res, 500, { error: `Failed to read local SWM/workspace: ${err?.message ?? err}` }); + } + } + + const { verifyBatch } = await import('@origintrail-official/dkg-agent'); + const verifyResult = verifyBatch({ quads, privateRoots, expectedRoot }); + return jsonResponse(res, 200, { + contextGraphId, + ...(parsed.batchId !== undefined ? { batchId: parsed.batchId } : {}), + quadsConsidered: quads.length, + ...verifyResult, + }); + } + + // POST /api/shared-memory/report-batch-rejection + // + // OT-RFC-38 LU-8 — when verifyBatch returns ok=false, the member + // gossips a structured BatchRejection record so other members can + // sanity-check and re-pull from a different host. + // + // Body: { + // contextGraphId: string, + // batchId?: string, + // verifyResult: { ok: false, expectedRoot, actualRoot, leafCount, reason }, + // rejectedBy?: { agentAddress, peerId }, // defaults to local agent + // } + if (req.method === "POST" && path === "/api/shared-memory/report-batch-rejection") { + const body = await readBody(req, SMALL_BODY_BYTES); + const parsed = safeParseJson(body, res); + if (!parsed) return; + const contextGraphId = parsed.contextGraphId; + if (!validateRequiredContextGraphId(contextGraphId, res)) return; + const verifyResult = parsed.verifyResult; + if (!verifyResult || verifyResult.ok !== false) { + return jsonResponse(res, 400, { + error: 'verifyResult.ok must be false; nothing to report on an ok batch', + }); + } + + const { buildBatchRejectionRecord } = await import('@origintrail-official/dkg-agent'); + const inferredAgentAddress = + (agent as any).getAgentAddress?.() ?? + (agent as any).agentAddress ?? + (agent as any).config?.agentAddress ?? + (agent as any).wallet?.address ?? + requestAgentAddress ?? + 'unknown'; + const rejectedBy = parsed.rejectedBy ?? { + agentAddress: inferredAgentAddress, + peerId: (agent as any).peerId, + }; + + let record; + try { + record = buildBatchRejectionRecord({ + contextGraphId, + batchId: parsed.batchId, + verifyResult, + rejectedBy, + }); + } catch (err: any) { + return jsonResponse(res, 400, { error: err?.message ?? String(err) }); + } + + // Persist the record as SWM triples so it gossips via the + // standard SWM substrate to other members. Reuses agent.share() + // for the write — no new transport. + const subject = `did:dkg:batch-rejection:${record.digest}`; + const NS = 'http://dkg.io/ontology/'; + const quads = [ + { subject, predicate: `${NS}rejectedContextGraphId`, object: `"${record.contextGraphId}"`, graph: '' }, + { subject, predicate: `${NS}expectedMerkleRoot`, object: `"${record.expectedRoot}"`, graph: '' }, + { subject, predicate: `${NS}actualMerkleRoot`, object: `"${record.actualRoot}"`, graph: '' }, + { subject, predicate: `${NS}rejectionReason`, object: `"${record.reason ?? 'unknown'}"`, graph: '' }, + { subject, predicate: `${NS}rejectedByAgent`, object: `"${record.rejectedBy.agentAddress}"`, graph: '' }, + { subject, predicate: `${NS}rejectedByPeer`, object: `"${record.rejectedBy.peerId ?? ''}"`, graph: '' }, + { subject, predicate: `${NS}rejectionReportedAt`, object: `"${record.reportedAt}"`, graph: '' }, + ...(record.batchId !== undefined + ? [{ subject, predicate: `${NS}rejectedBatchId`, object: `"${record.batchId}"`, graph: '' }] + : []), + ]; + + try { + await agent.share(contextGraphId, quads, { + operationCtx: createOperationContext('share'), + callerAgentAddress: requestAgentAddress, + }); + } catch (err: any) { + // The record itself is the deliverable; gossip is best-effort. + // Surface the error but still return the constructed record so + // callers can persist it elsewhere. + return jsonResponse(res, 200, { + record, + gossiped: false, + gossipError: err?.message ?? String(err), + }); + } + + return jsonResponse(res, 200, { record, gossiped: true }); + } + + // POST /api/attestation/mint + // + // OT-RFC-38 LU-9 — Member-attested verification token. + // + // Body: { + // contextGraphId: string, // local CG id (numeric on-chain id resolved server-side) + // batchId: string, // typically the KC id + // merkleRoot: hex32, + // plaintextLeafHash: hex32, // keccak256 over the canonical leaf + // } + // The daemon signs the attestation using the node's wallet + // (`chain.signMessage`). The returned token is self-contained and can + // be handed to any outsider for verification. + if (req.method === "POST" && path === "/api/attestation/mint") { + const body = await readBody(req, SMALL_BODY_BYTES); + const parsed = safeParseJson(body, res); + if (!parsed) return; + const contextGraphId = parsed.contextGraphId; + if (!validateRequiredContextGraphId(contextGraphId, res)) return; + const { batchId, merkleRoot, plaintextLeafHash } = parsed; + if (!batchId || typeof batchId !== 'string') { + return jsonResponse(res, 400, { error: 'batchId is required' }); + } + if (!/^0x[0-9a-fA-F]{64}$/.test(String(merkleRoot ?? ''))) { + return jsonResponse(res, 400, { error: 'merkleRoot must be 0x + 64 hex chars' }); + } + if (!/^0x[0-9a-fA-F]{64}$/.test(String(plaintextLeafHash ?? ''))) { + return jsonResponse(res, 400, { error: 'plaintextLeafHash must be 0x + 64 hex chars' }); + } + + const chain: any = (agent as any).chain ?? (agent as any).chainAdapter; + const kavAddress = chain?.contracts?.knowledgeAssetsV10?.target?.toString() + ?? chain?.kavAddress + ?? parsed.kavAddress; + const chainId = chain?.chainId ?? parsed.chainId ?? '31337'; + if (!kavAddress || !/^0x[0-9a-fA-F]{40}$/.test(String(kavAddress))) { + return jsonResponse(res, 400, { + error: 'cannot determine KAV10 address — pass `kavAddress` explicitly', + }); + } + + // Resolve on-chain contextGraphId. + let onChainCgId: string; + if (typeof parsed.onChainContextGraphId === 'string' && /^\d+$/.test(parsed.onChainContextGraphId)) { + onChainCgId = parsed.onChainContextGraphId; + } else { + try { + const cgList = await (agent as any).listContextGraphs?.(); + const match = (cgList ?? []).find((cg: any) => cg.id === contextGraphId); + onChainCgId = match?.onChainId ?? '0'; + } catch { onChainCgId = '0'; } + } + + const attesterAddress = + (agent as any).getAgentAddress?.() ?? + (agent as any).agentAddress ?? + requestAgentAddress ?? + ''; + if (!/^0x[0-9a-fA-F]{40}$/.test(String(attesterAddress))) { + return jsonResponse(res, 500, { error: 'cannot resolve local agent address' }); + } + + const { mintMemberAttestation } = await import('@origintrail-official/dkg-agent'); + try { + const attestation = await mintMemberAttestation({ + payload: { + chainId: String(typeof chainId === 'string' ? chainId.replace(/^evm:/, '') : chainId), + kavAddress: String(kavAddress).toLowerCase(), + contextGraphId: onChainCgId, + batchId: String(batchId), + merkleRoot: String(merkleRoot), + plaintextLeafHash: String(plaintextLeafHash), + attesterAddress: String(attesterAddress), + attestedAt: Math.floor(Date.now() / 1000), + }, + sign: async (digest) => { + // Convert (r, vs) → compact 65-byte hex via ethers.Signature. + const sigParts = await chain.signMessage(digest); + const r = '0x' + Array.from(sigParts.r as Uint8Array).map((b: number) => b.toString(16).padStart(2, '0')).join(''); + const vs = '0x' + Array.from(sigParts.vs as Uint8Array).map((b: number) => b.toString(16).padStart(2, '0')).join(''); + const ethersMod = await import('ethers'); + const sig = ethersMod.Signature.from({ r, yParityAndS: vs }); + return sig.serialized; + }, + }); + return jsonResponse(res, 200, { attestation }); + } catch (err: any) { + return jsonResponse(res, 400, { error: err?.message ?? String(err) }); + } + } + + // POST /api/attestation/verify + // + // OT-RFC-38 LU-9 — outsider-side verification. + // + // Body: { + // attestation: MemberAttestation, + // candidateLeafHex?: string, // optional 0x-prefixed bytes for leaf check + // chainCheckMembership?: boolean // if true, the daemon attempts a chain-side + // // membership lookup (Phase B); currently + // // always returns "unknown" — surfaces the + // // gap honestly. + // } + if (req.method === "POST" && path === "/api/attestation/verify") { + const body = await readBody(req, SMALL_BODY_BYTES); + const parsed = safeParseJson(body, res); + if (!parsed) return; + if (!parsed.attestation?.payload || !parsed.attestation?.signature) { + return jsonResponse(res, 400, { error: 'attestation.payload and attestation.signature are required' }); + } + let candidateLeaf: Uint8Array | undefined; + if (parsed.candidateLeafHex && typeof parsed.candidateLeafHex === 'string') { + const clean = parsed.candidateLeafHex.replace(/^0x/, ''); + if (!/^[0-9a-fA-F]+$/.test(clean) || clean.length % 2 !== 0) { + return jsonResponse(res, 400, { error: 'candidateLeafHex must be 0x-prefixed even-length hex' }); + } + candidateLeaf = new Uint8Array(clean.length / 2); + for (let i = 0; i < candidateLeaf.length; i++) { + candidateLeaf[i] = parseInt(clean.slice(i * 2, i * 2 + 2), 16); + } + } + + const { verifyMemberAttestation } = await import('@origintrail-official/dkg-agent'); + // membershipResolver = a thin stub for Phase A — the chain-side + // historical-membership lookup is not in scope (curated CG + // allowlists are stored in `_meta` graphs maintained off-chain; + // a proper chain-side resolver lands in Phase B with the + // membership-at-epoch SPARQL query). Returning undefined here + // surfaces `membership: 'unknown'` to the caller. + const result = await verifyMemberAttestation({ + attestation: parsed.attestation, + candidateLeaf, + membershipResolver: async () => undefined, + }); + return jsonResponse(res, 200, result); + } + // POST /api/shared-memory/write // // Direct SWM write entry point. Writes loose triples to shared memory diff --git a/packages/publisher/src/index.ts b/packages/publisher/src/index.ts index 2adfbae9c..ae717c038 100644 --- a/packages/publisher/src/index.ts +++ b/packages/publisher/src/index.ts @@ -19,6 +19,7 @@ export { computePublicRootV10, computePrivateRootV10, computeFlatKCRootV10, + computeFlatKCMerkleLeafCountV10, computeKARootV10, computeKCRootV10, } from './merkle.js'; diff --git a/scripts/devnet-test-rfc38-all.sh b/scripts/devnet-test-rfc38-all.sh new file mode 100755 index 000000000..65d59140d --- /dev/null +++ b/scripts/devnet-test-rfc38-all.sh @@ -0,0 +1,122 @@ +#!/usr/bin/env bash +# +# OT-RFC-38 — full integration runner. Executes every per-LU devnet +# validation script in sequence and prints a consolidated summary. +# +# Prerequisite: devnet is up (`./scripts/devnet.sh up`) with all 6 nodes +# healthy. The script does not rebuild — call this AFTER the per-LU +# scripts have already exercised their own restart cycles, OR after a +# clean `./scripts/devnet.sh restart-all`. It only reads. +# +# Exit status: 0 if every LU's exit was 0, else the count of failing +# LUs (capped to 99 to fit a single byte). + +set -u # don't exit on errors — we want the summary + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +SCRIPTS_DIR="$REPO_ROOT/scripts" +RESULTS_DIR="${RESULTS_DIR:-$REPO_ROOT/.devnet/integration-runs/$(date +%s)}" +mkdir -p "$RESULTS_DIR" + +log() { echo "[lu-all] $*"; } +note() { echo "[lu-all] $*" | tee -a "$RESULTS_DIR/summary.txt" >/dev/null; } + +# Each entry: ":