Skip to content

fix(supervisor): tolerate non-empty bounding set when CAP_SETPCAP is unavailable#2075

Open
waynesun09 wants to merge 2 commits into
NVIDIA:mainfrom
waynesun09:fix-2069-cap-bounding-set-rootless-podman
Open

fix(supervisor): tolerate non-empty bounding set when CAP_SETPCAP is unavailable#2075
waynesun09 wants to merge 2 commits into
NVIDIA:mainfrom
waynesun09:fix-2069-cap-bounding-set-rootless-podman

Conversation

@waynesun09

@waynesun09 waynesun09 commented Jun 30, 2026

Copy link
Copy Markdown

Summary

When running under rootless Podman (or any container runtime that drops CAP_SETPCAP), cap_drop_bound() returns EPERM for every capability still in the bounding set. Since v0.0.73 this is fatal — the supervisor crashes on sandbox startup, breaking all rootless Podman deployments.

This PR adds a third match arm to validate_capability_bounding_set_clear() that tolerates EPERM when the bounding set is non-empty, logging a warning instead of returning an error. The sandbox still relies on seccomp and Landlock for confinement in this case.

Related Issue

Fixes #2069

Changes

  • crates/openshell-supervisor-process/src/process.rs:
    • Add EPERM + non-empty bounding set tolerance branch between the existing EPERM + empty (success) and catch-all (error) arms
    • Import warn from tracing
    • Update capability_bounding_set_clear_tolerates_nonempty_eperm test to assert is_ok() instead of is_err()
    • Simplify drop_privileges_succeeds_for_current_group test — remove conditional cfg(target_os) branching
  • .github/workflows/branch-checks.yml: Add rootless-caps CI job that runs supervisor capability tests as a non-root user without CAP_SETPCAP on ubuntu-24.04

Local Build & Test

Built and tested the fix locally before pushing:

  1. Cross-compiled supervisor as a static aarch64-unknown-linux-musl binary inside a rust:1.95-bookworm Podman container (macOS host cannot build Linux-only deps: capctl, landlock, seccompiler)
  2. Packaged into local OCI image localhost/openshell-supervisor:fix-2069 via Dockerfile.supervisor
  3. Configured gateway to use the local supervisor image (supervisor_image = "localhost/openshell-supervisor:fix-2069" in gateway.toml)
  4. Tested sandbox creation — sandbox reached Phase: Ready with the expected warning:
    CAP_SETPCAP is unavailable and the child capability bounding set is non-empty;
    the child process relies on seccomp and Landlock for confinement
    
  5. Ran a fullsend agent code review pipeline against the fixed supervisor — sandbox created (7.4s), bootstrapped, agent code copied into sandbox — all stages succeeded

Testing

  • cargo test -p openshell-supervisor-process --lib -- capability_bounding drop_privileges passes
  • Local supervisor build + sandbox creation test (rootless Podman on macOS)
  • Fullsend agent pipeline run with fixed supervisor image — sandbox lifecycle (create, bootstrap, code copy) verified
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Jun 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@waynesun09

Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

@waynesun09

Copy link
Copy Markdown
Author

recheck

@maxamillion

Copy link
Copy Markdown
Collaborator

Few points of note:

  • Security boundary behavior changed but architecture docs were not updated.
    architecture/sandbox.md still says child capability bounding-set clearing is fail-closed and that EPERM is tolerated only when the set is already empty. This PR intentionally changes that invariant. The architecture doc should be updated in the same PR to describe the degraded rootless mode and its reliance on seccomp/Landlock.

  • The degraded path relies on Landlock, but Landlock may be best-effort.
    The warning says the child relies on “seccomp and Landlock,” but elsewhere Landlock can run in best-effort mode and continue unavailable or failed. If the bounding set remains non-empty and Landlock is unavailable/best-effort, the actual confinement story is weaker than the warning implies. Consider tightening the message or adding an explicit check/comment explaining acceptable residual risk.

  • Consider emitting this as an OCSF security/config event, not only tracing::warn!.
    Per project logging guidance, degraded sandbox controls and unavailable confinement primitives are operator-visible security posture events. This warning represents a confinement degradation and may warrant a structured OCSF finding or config-state event so it appears in sandbox security telemetry.

  • Commit metadata includes Assisted-by: Claude.
    Project instructions say commits must not mention Claude or AI agents. The commit bodies in this PR include Assisted-by: Claude; those should be removed before merge.

@waynesun09 waynesun09 force-pushed the fix-2069-cap-bounding-set-rootless-podman branch from 7320552 to 1dc253d Compare June 30, 2026 22:13
@TaylorMutch

Copy link
Copy Markdown
Collaborator

/ok to test 1dc253d

@waynesun09 waynesun09 force-pushed the fix-2069-cap-bounding-set-rootless-podman branch 2 times, most recently from ad6106c to 3f95d51 Compare June 30, 2026 22:34
@elezar elezar assigned elezar and alangou and unassigned elezar Jul 1, 2026
@elezar

elezar commented Jul 1, 2026

Copy link
Copy Markdown
Member

@alangou could this have been introduced in #2001? Do you mind having a look?

@NVIDIA NVIDIA deleted a comment from copy-pr-bot Bot Jul 1, 2026
@alangou

alangou commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test 3f95d51

@alangou

alangou commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

The new OCSF degraded-mode alert may not fire.

The parent-side probe returns early if CAP_SETPCAP is present in the effective set, but the commit message says Podman can grant CAP_SETPCAP while AppArmor still makes bounding::clear() fail with EPERM. Then we skip the parent DetectionFinding, and only hit the warn! inside pre_exec, which is exactly the context this patch says cannot reliably emit structured logs.

Could we make the readiness probe test the actual bounding-set clear behavior ?

@waynesun09 waynesun09 force-pushed the fix-2069-cap-bounding-set-rootless-podman branch 2 times, most recently from adf21ac to b3a0e2a Compare July 1, 2026 12:39
@alangou

alangou commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test b3a0e2a

@alangou

alangou commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@waynesun09 look to me. Just fix the format issue and we're good to merge

@waynesun09

Copy link
Copy Markdown
Author

@alangou cool, I'm on it now.

@waynesun09 waynesun09 force-pushed the fix-2069-cap-bounding-set-rootless-podman branch from b3a0e2a to 9ab2fcb Compare July 1, 2026 14:28
@waynesun09

Copy link
Copy Markdown
Author

@alangou it's updated, please check, thanks

@alangou

alangou commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test 9ab2fcb

…unavailable

When running inside rootless Podman on Ubuntu 24.04 with AppArmor's
apparmor_restrict_unprivileged_userns=1, prctl(PR_CAPBSET_DROP)
returns EPERM even though CAP_SETPCAP may be nominally granted.
The capability bounding set remains non-empty, causing the supervisor
to abort sandbox creation.

Add a new match arm in validate_capability_bounding_set_clear() that
tolerates EPERM when the bounding set is non-empty: log a warning and
continue, relying on seccomp to block dangerous syscalls. The existing
privileged-environment behavior (fail-closed on non-empty success) is
unchanged.

Emit a parent-side OCSF DetectionFinding alert so the degraded mode
is visible to operators and SIEM. The readiness probe performs a
non-destructive bounding::drop() on an already-absent capability to
detect AppArmor restrictions even when CAP_SETPCAP is nominally
present in the effective set.

Closes NVIDIA#2069

Signed-off-by: Wayne Sun <gsun@redhat.com>
Add a rootless-caps job to branch-checks.yml that runs the supervisor
capability bounding set and drop_privileges tests as an unprivileged
user on ubuntu-24.04 where AppArmor restricts PR_CAPBSET_DROP.

Update architecture/sandbox.md to describe the degraded rootless mode
where seccomp provides confinement when the bounding set cannot be
cleared.

Signed-off-by: Wayne Sun <gsun@redhat.com>
@waynesun09 waynesun09 force-pushed the fix-2069-cap-bounding-set-rootless-podman branch from 9ab2fcb to 4d5f5ad Compare July 1, 2026 15:28
@waynesun09

Copy link
Copy Markdown
Author

@alangou the new ci clippy failure on backticked AppArmor is fixed, sorry I missed that in the local test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

supervisor v0.0.73 crashes in rootless Podman: drop_capability_bounding_set() EPERM with non-empty bounding set

5 participants