Add "Sync" command

[isc-dchui]

Description

• Closes Sync command #1117

Motivation

The primary use case is LLM-assisted development: an LLM generates or edits .cls / .inc / .mac files on disk, then the developer runs sync to reload and recompile only what changed without a full reload that reinstalls the entire module. This makes the tight edit-compile-test loop fast enough to be practical inside an agentic coding session.

Secondary use case: any developer editing IPM-managed source who wants faster incremental feedback than reload provides.

Architecture

Entry point

%IPM.Main.Sync  →  ExecutePhases(name, $lb("Sync"), 1, .params)

Sync is a first-class lifecycle phase registered in %IPM.DataType.PhaseName. It fires the standard OnBeforePhase / OnAfterPhase / <Invoke> hooks for free. With no module name, Main.Sync loops over all DeveloperMode=1 modules and collects failures without throwing.

Flags: -delete (remove deleted files from server), -test/-t (run changed test cases), -verbose/-v.

10-step pipeline (%IPM.Lifecycle.Base.%Sync)

Step	Method	What it does
1	SyncCheckModuleXml	Hash-compare module.xml; if changed, reload manifest immediately
2	—	Call GetSyncDirectory() on each SupportsSync()=1 processor; DeduplicateScanDirs; WalkAndHashDirs targeted walk; build bfsFiles subset
3	SyncBuildReverseIndex	Map relPath → processor via (a) ResolveChildren+OnItemRelativePath for individually-declared files, (b) GetSyncDirectory() prefix scan for directory-owned resources
4	FileHash.ComputeChanges	3-pass change detection; exit early with "Nothing to sync" if clean
5	SyncRoutePathSet	Partition modified/deleted paths into syncByResource
6	SyncDispatchProcessors	Call OnSync on each routed processor
7	SyncCompile	Full compile of all in-scope compilable resources; IRIS automatically recompiles any dependents (subclasses, includers)
8	SyncApplyDeletes	If -delete: delete server-side docs, then recompile
9	FileHash.CommitChanges	Commit new baseline — skipped on exception so next sync retries
10	SyncRunTests	If -test: run changed test-phase cases via owning Test processor only

Targeted-walk architecture (GetSyncDirectory)

Each processor returns the single directory it owns as a normalized relative path (no leading/trailing slash). Sync walks only those directories, never the full module root. This avoids touching build/, data/, node_modules/, etc. entirely.

DeduplicateScanDirs removes any entry that is a subdirectory of an already-kept entry (alphabetical scan, prefix check) to prevent double-walking.

Abstract.GetSyncDirectory() returns "" (sentinel: no owned directory). All callers guard syncDir '= "".

Change detection (FileHash.ComputeChanges)

Three-pass pipeline — SHA-1 hash only, no mtime or size:

• Pass 1 (BFS subset, namespace-filtered): compilable files (cls/inc/mac/int). Checks $$$comClassDefined / %RoutineMgr.Exists. Only files compiled in this namespace are considered.
• Pass 2 (manifest paths from reverse index): catches uncompiled new files and non-compilable tracked files (e.g. test dirs). No I/O since paths already resolved.
• Pass 3 (stored rows → deletions): any FileHash row whose file is missing from disk → deleted.

No baseline row → new file → modified. Baseline exists → hash-only comparison.

Filesystem walking

WalkAndHashDirs dispatches to Python os.walk() (WalkAndHashFilesPython) with automatic SQL BFS fallback (WalkAndHashFilesSQL). Walk-once: step 2 walks all declared dirs into allFiles+allHashes, shared by both SyncBuildReverseIndex and ComputeChanges.

Stage rollout

Stage	Processors	Status
1	All AbstractCompilable derivatives, Test (test-phase only)	Implemented
2	FileCopy, WebApplication, PythonWheel	Planned
3	CPF, Copy, ArtifactoryTarball, LegacyLocalizedMessages, Default.Global	Planned
—	CSPApplication, SystemSetting, ModuleExport, LocalizationExport	Permanently out of scope

Peculiar implementation details

Trailing-slash contract on GetSyncDirectory(): NormalizePath strips leading slashes but not trailing. Callers must explicitly strip the trailing slash before returning. If missed, SyncBuildReverseIndex appends "/" to form a prefix, producing "tests/unit//" which never matches any relPath, silently finding nothing.

%-prefixed classes: RelPathToDocName("src/cls/IPM/Main.cls") → "IPM.Main.CLS". Actual class is %IPM.Main. ComputeChanges Pass 1 tries $$$comClassDefined("%" _ className) as a fallback.

Test file relPaths: RelPathToDocName("tests/unit/SyncTest/Tests/Trivial.cls") → "tests.unit.SyncTest.Tests.Trivial.CLS" is non-empty but wrong. $$$comClassDefined returns false. Test files enter reverseIndex via the prefix scan path (step 3b), not the docName/namespace-check path. StampModule stamps all compilable files in scan dirs unconditionally for the same reason.

Test.OnSync owns its own delete cycle: SyncApplyDeletes skips Test resources (not AbstractCompilable). OnSync must call $system.OBJ.Delete for removed test class files directly. Otherwise the server-side %UnitTest.TestCase subclass persists with no on-disk source and no hash record, permanently invisible to future syncs.

Baseline not committed on failure: if compile fails, CommitChanges is skipped. The changed files retain their old hash rows (or no rows if new), so the next sync re-detects and retries them once the error is fixed.

Migration from no baseline: a module installed before this feature existed has zero FileHash rows. The first sync self-heals by stamping a baseline from current disk state, then exits. Only the next sync after a real edit detects changes.

Performance: what was tried and why things changed

Approach	Outcome
mtime + size as fast path, hash on mismatch	Initial design. Dropped: mtime is unreliable on Docker bind mounts. Removed entirely. SHA-1 only.
Pure ObjectScript directory walk	Original implementation. Replaced by Python os.walk() because IRIS's %Library.File_FileSet SQL walk has severe per-row overhead on large directories. Python processes the entire tree in native C and returns results in bulk.
zsearch / $zsearch	Considered for file enumeration. Rejected: too slow on large trees.
Full-root BFS (walk everything, filter after)	Used before targeted-walk. Replaced by GetSyncDirectory() architecture: avoid entering build/, node_modules/, data/ entirely rather than filtering them after the I/O.
Separate walk + hash passes	Initial Python implementation walked directories first, then hashed separately. Merged into a single Python pass (c37ac2d) to avoid double-traversal overhead.
Flat/non-canonical resource paths	Early implementation attempted to support resources with files scattered across arbitrary directories. Dropped (080e6bed): too complex, too slow. Only canonical resource paths (declared Name/Directory attributes) are supported.

Testing

All integration tests in Test.PM.Integration.Sync. Fixture: tests/integration_tests/Test/PM/Integration/_data/sync-test/ is loaded in dev mode before each test, uninstalled and deleted after.

Test	What it verifies
TestGetStoredPathsReturnsStampedPaths	StampModule records expected paths in FileHash after load
TestNoChangeIsNoOp	No-change sync exits with "Nothing to sync"
TestMigrationFromNoBaseline	Zero rows → self-heal baseline on first sync; edit before self-heal not retroactively detected; edit after baseline detected normally
TestModifiedClassRecompiles	Modified .cls file detected and recompiled
TestSuperclassEditRecompilesSubclass	Editing a superclass causes IRIS to recompile its subclasses even though the subclass file is unchanged
TestIncludeEditRecompilesConsumer	Modified .inc causes consumer class to recompile
TestUntrackedFileIgnored	File outside declared resource dirs never seen, never stamped
TestDeleteSkippedByDefault	Deleted file left on server without -delete flag
TestDeleteTestClassRemovesFromServer	Deleted test class file removed from server with -delete
TestDeleteRecompilesDependents	Deleting a superclass causes dependents to fail recompile (surfaces error)
TestModuleXmlChangedWarning	Changed module.xml reloads manifest and emits warning
TestSyncTestFlag	-test runs changed test class; without flag, loads but doesn't run
TestSyncTestFlagBatchesMultipleChangedClasses	Multiple changed test classes in one resource → single batched RunTest invocation
TestSyncTestFlagOnlyRunsOwningResource	Changed test class dispatched only through its owning resource, not all UnitTest processors
TestSyncAllDevModeModules	No module name → syncs all dev-mode modules; [sync-test] specifically reports nothing to sync
TestSyncModuleNotFound	Non-existent module name → error status
TestSyncNonDevModeModule	Non-dev-mode module → error status
TestModuleXmlAddsResourcePicksUpNewFile	module.xml change adding a new <Resource> causes that resource's files to be tracked in the same sync call
TestFailedCompileRetries	Syntax error → failed sync → baseline not committed → fixed file re-detected on retry
TestModuleXmlAndClassEditedTogether	Co-editing module.xml and a source class in one operation syncs both changes

Checklist

• This branch has the latest changes from the main branch rebased or merged.
• Changelog entry added.
• Unit (zpm test -only) and integration tests (zpm verify -only) pass.
• Style matches the style guide in the contributing guide.
• Documentation has been/will be updated
  • Source controlled docs, e.g. README.md, should be included in this PR and Wiki changes should be made after this PR is merged (add an extra issue for this if needed)
• Pull request correctly renders in the "Preview" tab.