Skip to content

feat: multi-instance session lease, write fencing, and fs/watch reliability#1920

Open
7Sageer wants to merge 17 commits into
mainfrom
refactor-fs-watch
Open

feat: multi-instance session lease, write fencing, and fs/watch reliability#1920
7Sageer wants to merge 17 commits into
mainfrom
refactor-fs-watch

Conversation

@7Sageer

@7Sageer 7Sageer commented Jul 19, 2026

Copy link
Copy Markdown
Collaborator

Related Issue

Follow-up to #1826. There is no single tracking issue for the remaining work; the problem and scope are described below.

Problem

Multiple Kimi Code servers can share one KIMI_CODE_HOME, but several persistence and transport paths previously assumed a single process:

  • session state, journals, configuration, the workspace registry, and MiniDB could be mutated without a shared cross-process ownership primitive;
  • a client could reach a server that listed a session while another server actually owned its live scope;
  • journal write failures, shutdown races, and filesystem-watch gaps could leave clients with stale state;
  • agent Write / Edit calls had no session-local baseline for detecting an intervening filesystem change;
  • skill changes required a restart before the catalog and Web UI converged.

The goal of this PR is to establish explicit ownership and durability boundaries for shared-home operation, while also tightening the single-instance journal and filesystem-watch paths those boundaries depend on.

What changed

Kernel-backed cross-process coordination

  • Added @moonshot-ai/kernel-file-lock, backed by fs-native-extensions, and wired its native dependency into the CLI bundle and Nix build.
  • Added the agent-core-v2 CrossProcessLockService, using kernel-held file locks plus inode identity checks rather than heartbeat / TTL takeover.
  • Moved config.toml and workspace-registry read-modify-write operations behind cross-process locks while preserving their on-disk formats.
  • Switched MiniDB writer ownership to the kernel lock primitive and made lock loss fail closed on later writes.
  • Added an mcp.json watcher so out-of-process configuration changes are observed.

Session ownership and release lifecycle

  • Added an unconditional per-session kernel lease when a v2 session scope is materialized. A second holder receives typed 40921 SESSION_HELD_BY_PEER details for creating, routable, unresponsive, and local-holder states.
  • Registered each live lease as the write authority for session-rooted persistence. File-backed document, blob, append, task, and raw-storage mutations check that authority before storage I/O.
  • Added explicit session close / archive / force-abort phases, append-log flushing, broadcaster draining, and deterministic server shutdown ordering before lease release.
  • Added shared-home instance registration and lease contact metadata so REST and WebSocket callers can identify a peer holder.

Web and WebSocket convergence

  • Added ownership details to session-list responses and handling for REST / WebSocket 40921 outcomes in Kimi Web.
  • Added peer redirect decisions with retry and loop budgets, plus replay of pending interaction state from snapshots.
  • Added the volatile session.list_changed hint and debounced Web list refresh when another instance creates, archives, or deletes a session.
  • Hardened the session event journal: queued batches are fsynced, failed writes remain explicit through JournalStorageError, cold reads do not create files, and replay failures request a snapshot resync.
  • Made global advisory events volatile and added deterministic broadcaster/session-journal disposal.

Filesystem fencing, watching, and skill reload

  • Added a per-session file ledger. Successful full Read, Write, and Edit calls record stat revisions; Write / Edit compare the current revision immediately before tool execution and block stale or read-before-write conflicts.
  • Moved v2 filesystem watching onto a shared host abstraction, added special-file filtering, .gitignore reload handling, coalescing/truncation semantics, and the v1 WebSocket bridge.
  • Added skill-root watching, catalog reloads, turn-boundary skill-list reminders, and the volatile skill_catalog.changed Web hint.

Packaging and release notes

  • Added seven patch changesets covering cross-process coordination, journal/fs-watch reliability, multi-instance Web behavior, session release, lease fencing, stale-write fencing, and skill hot reload.
  • Updated CLI native-asset checks and Nix workspace metadata for the new kernel lock package.

Verification

  • GitHub CI currently passes build, lint, typecheck, all Unix test shards, package preview, and the Nix workspace/build checks. The Windows job is currently skipped.
  • Added focused tests for kernel locks, MiniDB locking, config/workspace locked updates, session ownership, journal recovery and failure handling, shutdown/release ordering, filesystem watching, file fencing, skill hot reload, Web ownership decisions, and WebSocket lifecycle behavior.

Known limitations and open review items

  • Session write authority is currently checked before storage mutations, but it is not yet retained as an in-flight guard through the final async write/rename/append/unlink. Lease release therefore still needs a full storage-I/O drain barrier.
  • File fencing performs a pre-execution comparison but does not yet provide a cross-process compare-and-commit critical section. A peer process or external editor can still change a file between comparison and commit.
  • MiniDB's new kernel sentinel is not interoperable with a concurrently running release that still owns db.lock through the legacy PID-content protocol; rolling-upgrade compatibility still needs a dual-protocol boundary.
  • Kimi Web peer navigation still needs an explicit browser-reachable public origin and a secure credential handoff across ports/origins. The internal lease contact alone is not sufficient for remote or reverse-proxy deployments.
  • Filesystem hints are volatile and self-heal on reconnect/focus, but macOS delivery can stall under heavy multi-instance watch load. Low-level watchers are not yet multiplexed across sessions, and skill hot reload does not yet follow symlink targets.
  • The PR no longer has a multi-server experimental flag. User-facing documentation must therefore be reassessed before merge rather than skipped on the previous experimental-feature rationale.

Checklist

  • I have read the CONTRIBUTING document.
  • I have linked a related issue, or explained the problem above.
  • I have added tests that prove my feature works.
  • Ran gen-changesets skill, or this PR needs no changeset.
  • Ran gen-docs skill, or this PR needs no doc update. Documentation scope must be reassessed because the feature is no longer experimental.

@changeset-bot

changeset-bot Bot commented Jul 19, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 6a01fd2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@moonshot-ai/kimi-code Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@pkg-pr-new

pkg-pr-new Bot commented Jul 19, 2026

Copy link
Copy Markdown
pnpm dlx https://pkg.pr.new/@moonshot-ai/kimi-code@03be83f
npx https://pkg.pr.new/@moonshot-ai/kimi-code@03be83f

commit: 03be83f

@7Sageer
7Sageer force-pushed the refactor-fs-watch branch from a030953 to 3d7ffc7 Compare July 19, 2026 14:01
…bility

Single-instance reliability:
- Journal: preserve content on write failure with sticky JournalStorageError,
  per-batch fsync, last-header open, and zero-write cold reads
- Broadcaster: writeFailure gate, deterministic dispose drain, replay failure
  now triggers resync_required; graceful close ordering in start.ts
- fswatch: per-connection monotonic seq, pinned truncated semantics, and
  gitignore double-cache invalidation; __global__ session made volatile

Cross-process lock unification:
- agent-core-v2 CrossProcessLockService: parameterized lock modes
  (heartbeat-TTL implemented), token-guarded release/update, rename-isolated
  stale takeover, creation window, PID-reuse detection (conservative on macOS)
- config.toml and workspace registry locked read-modify-write; mcp.json watch
- kap-server lock protocol aligned (lock_id, process_started_at, three-layer
  stale detection); minidb release ownership fixed, legacy cleanup removed

Multi-instance session ownership (gated by multi_server flag):
- Session leases with write fencing; 40921 SESSION_HELD_BY_PEER with typed
  details (routable/creating/holder-unresponsive/held-by-local-instance) and
  unregistered-writer detection; write-authority registry hard gate in
  AppendLogStore, session metadata, and flush paths
- kimi-web redirect to the holder with pending approvals/questions replay;
  klient transparent redirect with loop guard
- session.list_changed cross-instance discovery via two-layer fs watch;
  ownership field on GET /sessions

Stale-write fencing and skill hot reload:
- Session file ledger + agent file fencing: hard gate with the flag on,
  advisory note with it off; dirty-tick signals from fs watch
- Skill root watcher hot reload: catalog reload, turn-boundary listing
  reminder, and skill_catalog.changed volatile event to web clients
@7Sageer
7Sageer force-pushed the refactor-fs-watch branch from 3d7ffc7 to 56f1a5a Compare July 19, 2026 14:28
7Sageer added 7 commits July 20, 2026 12:04
…lock

Per-session flock is now held for the entire time a server hosts a
session, making the kernel the single arbiter of ownership: process
death releases the lock, so heartbeats, TTL-based stale takeover, and
pid liveness probing are no longer needed. owner.json is reduced to a
redirect hint (instance address), written atomically after acquire.

- add @moonshot-ai/kernel-file-lock (fs-ext-extra-prebuilt; flock on
  unix, LockFileEx emulation on windows)
- crossProcessLockService shrinks to try-acquire/checkHeld/release
- minidb lockfile adopts the same flock-held semantics
- delete processProbe
Resolve conflicts with #1888 (unified transcript layer, dropped /api/v2
RPC surface):

- Accept main's deletion of the v2 ws transport (ws/eventMap,
  wsConnection, wsProtocol) and of the klient http/ws transports.
- Drop the klient-side redirect pieces that only served the removed
  http transport (sessionRedirect, SessionRedirectChannel) and the
  v2 ownership-redirect e2e; the v1 REST redirect surface is untouched.
- sessionEventBroadcaster: keep both the transcript additions and the
  ownership probe (getSubscriptionFailure); global events stay volatile
  (no __global__ journal) per the multi-instance design.
- wsConnectionV1: combine typed subscription entries
  ({agentFilter, transcriptGrades}) with SkillCatalogBridge attach.
- Restore klient's kap-server devDependency (removed on main); the
  dual-instance helpers still boot in-process servers.
- workspaceFs tests: expect the <home>/sessions dir the always-on
  session-list watch creates at boot.
@github-actions

Copy link
Copy Markdown
Contributor

❌ Nix build failed

Hash mismatch in pnpmDeps:

Hash
specified sha256-+pzJfoWJwVXIUU8oc56LVpfNjSY6MABID5g11Cm92xw=
got sha256-DG0qhbCF74KeNe0LKHuGAQufDzTdtXxvzFBYHl4xs9w=

Please update flake.nix with the got hash.

7Sageer added 9 commits July 21, 2026 13:10
…ules

Assert the ledger verdict in the fileFencing allow-path tests instead of
relying on the helper-internal expect, and narrow firstFailure to an Error
before rethrowing it from flushPersistence.
Conflicts resolved:
- toolExecutor.ts: ToolCall import moved to #/kosong/contract/message (main's move)
- start.ts: union of service imports; drop IModelCatalogService removed on main
- ws/v1/events.ts: ProviderRefresh* types now from kosong/model/discovery
- fileFencing.test.ts: ToolCall import updated likewise
Decouple the session file ledger from the fs watcher: verdicts come from
a fresh stat-tuple compare immediately before execution, with no watcher
ticks or cross-process file locks. Drop the dead dirty-tick and
ensured-roots surface from the session fs watch service so it is only a
confined, debounced change feed. Write fencing no longer keeps per-call
target bookkeeping: the did-hook baselines the ledger from the revision
the executed call attached to its own result. FileEditService verifies
stat stability across the read-transform-write window.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant