fix(cloud-init): make Caddy runcmd POSIX-compatible

[simple-agent-manager]

Summary

• Fix the Hetzner/Ubuntu cloud-init failure set: Illegal option -o pipefail by making the scalar Caddy runcmd block POSIX-compatible (set -eu).
• Parse the rendered production YAML and execute the actual workspace/deployment command through /bin/sh with isolated logger and mkdir boundary stubs.
• Extend path-scoped cloud-init guidance so changed scalar commands are tested under their declared interpreter.
• Integrate current main normally; no force-push was used.

Validation

• Focused cloud-init suite — 178/178 tests
• Cloud-init package typecheck and production build
• Repository lint — 7/7 tasks; zero errors
• Repository typecheck — 16/16 tasks
• Repository production build — 9/9 tasks
• Repository tests — 19/19 tasks; API 438 files / 6,190 tests
• GitHub CI — all checks green on exact head f4e759a2311df8bae0607ab94b522193be586f4b

TDD evidence is retained in the active task: before the runtime fix, the two new interpreter-contract tests failed under Dash with status 2 and the exact pipefail error; after the one-line fix both role branches pass.

Staging Verification (merge-blocking)

• Exact head f4e759a2311df8bae0607ab94b522193be586f4b deployed in run 29727014256; deploy and 12/12 built-in smoke tests passed.
• Secondary staging identity had zero personal cloud-provider credentials; enabled platform Hetzner credential 01KNY6DC06C9QCYQM0389NAGNT was confirmed.
• Fresh node 01KXZAN71M1XKPSXTG79QJXVF0 was created through the Nodes UI via user→platform fallback and reached Hetzner running/healthy with a heartbeat.
• Fresh-node cloud-init reached successful final completion with zero actual error-severity entries; Illegal option -o pipefail, cc_scripts_user failure, and Failed to run module scripts-user were absent. VM-agent error entries were zero.
• Claude session ae17814d-dcec-4dec-bd7a-1a02c8161128 rendered and persisted marker PR1582_CLAUDE_20260720090229, then returned idle.
• Codex session 6cbba882-2291-457a-b5be-8d54c9848906 rendered and persisted marker PR1582_CODEX_20260720090324, then returned idle.
• Browser console, page, failed-request, and staging HTTP error count was zero across the primary journey.
• Primary test node 01KXZAN71M1XKPSXTG79QJXVF0 was deleted and verified absent.
• Final independent Staging Validator task 01KXZFJGBDVSQA0H73QJ71CDS9 passed from fresh prerequisites with no code/profile mutation or API-created chat sessions.
• Its UI-created node 01KXZGB1E2SYY1Z592EB82TBKC reached running/healthy/heartbeat on exact agent SHA f4e759a2311df8bae0607ab94b522193be586f4b; modules-final and config-scripts_user succeeded, actual cloud-init/VM-agent error records were zero, and all regression signatures were absent.
• Independent Claude session 4c337d61-1700-4f8b-b0cb-fe7743478294 / workspace 01KXZGTFKRQXP9DTCFE70DCAEN rendered and persisted STAGING-VALIDATE-CLAUDE-20260720-UNIQUE-a7b3c9; Codex session d171ed63-1902-4c56-911f-88522978dd05 / workspace 01KXZH36JCND69CVJRXNGT7Y8Y rendered and persisted STAGING-VALIDATE-CODEX-20260720-UNIQUE-d8e2f1. Both conversation turns settled into the expected waiting-for-next-input state.
• Independent node 01KXZGB1E2SYY1Z592EB82TBKC was deleted and verified absent; the secondary account returned to zero nodes.

Cloud-init also emitted recovered Hetzner metadata discovery probe messages inside normalized level=info entries. They are recorded rather than hidden; final cloud-init completion, the feature-specific failure signatures, normalized severity, node health, and agent logs are the merge gates.

Two earlier validator tasks were invalidated and stopped after violating their account/UI constraints. Their exact nodes, two unintended profiles, and one empty API-created session were validated, cleaned, and verified terminal/absent; none of their evidence is counted as independent PASS evidence.

End-to-End Verification

1. generateCloudInit() renders the production template.
2. Tests parse the generated YAML and select the actual Caddy scalar runcmd entry.
3. Both role variants execute through real /bin/sh with only external commands stubbed.
4. Exact-head staging repeated the command contract on a fresh Ubuntu/Hetzner VM through platform fallback.
5. Real project Chat UI journeys selected current cf-container Claude and Codex profiles, submitted unique prompts through the composer/Send button, required rendered replies, and verified persistence across consecutive assistant chunks.
6. The fresh node and all validator infrastructure are cleaned after evidence collection.

Post-Mortem

The Caddy scalar command introduced in PR #1308 assumed Bash even though cloud-init executes scalar runcmd entries through /bin/sh/Dash. Existing tests checked YAML structure and command ordering but never executed the rendered command. The process fix adds both executable interpreter-contract coverage and path-scoped guidance.

Specialist Review Evidence

• All required direct specialist procedures completed on the current-main integration

Reviewer	Status	Outcome
test-engineer	PASS	Rendered YAML, real /bin/sh, both roles, and exact side effects covered
constitution-validator	PASS	No configurable business value, URL, timeout, limit, or identifier added
doc-sync-validator	PASS	Path-scoped rule matches production runtime; no public/env/API drift
task-completion-validator	PASS	A/B/C/F pass; D/E N/A; lifecycle evidence is recorded above and task archive follows merge/production verification

UI Compliance

N/A: no UI changes. Core browser journeys were nevertheless validated on exact-head staging before merge.

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• external-api-change
• cross-component-change
• business-logic-change
• public-surface-change
• docs-sync-change
• security-sensitive-change
• ui-change
• infra-change

External References

N/A: no external API contract changed. Evidence comes from the exact Hetzner failure, the production template, current repository tests, and exact-head staging.

Codebase Impact Analysis

packages/cloud-init/src/template.ts, packages/cloud-init/tests/generate.test.ts, .claude/rules/06-vm-agent-patterns.md, and the active task record.

Documentation & Specs

Updated the path-scoped agent rule and active task/post-mortem. No public API, environment variable, schema, or user-visible configuration changed.

Constitution & Risk Check

Principle XI checked. /bin/sh, POSIX set -eu, and the existing Caddy paths are runtime contract invariants, not configurable business values. Fresh-node bootstrap risk is gated by exact-head staging, real platform-fallback provisioning, rendered Claude/Codex replies, and independent validation.

[codspeed-hq]

Merging this PR will not alter performance

✅ 6 untouched benchmarks

---

_{Comparing sam/fix-cloud-init-runcmd-shell (1ff249b) with main (c8a0166)}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud