Skip to content

Set Partial Clone flags on sparse-checkout#4044

Draft
lizrabuya wants to merge 5 commits into
mainfrom
fix/set-clone-flags-on-sparsecheckout
Draft

Set Partial Clone flags on sparse-checkout#4044
lizrabuya wants to merge 5 commits into
mainfrom
fix/set-clone-flags-on-sparsecheckout

Conversation

@lizrabuya

@lizrabuya lizrabuya commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Description

When BUILDKITE_GIT_SPARSE_CHECKOUT_PATHS is set, the agent now injects --filter=blob:none into git clone so blobs outside the sparse set aren't downloaded up front. The filter is only injected when the user hasn't already supplied their own --filter (git takes the last --filter on the command line, so stacking ours after would silently override theirs) and when the local git actually supports sparse-checkout set --cone (>= 2.27). On older git, the agent falls back to a plain full checkout — no partial clone, no --cone attempt.

The sparse-checkout decision is resolved once at the top of defaultCheckoutPhase via planSparseCheckout, which runs and validates the git --version at most once per checkout and only when sparse paths are configured.

Changes

  • add a new sparseCheckoutPlan struct and planSparseCheckout function to perform git version check only once before clone starts
  • version floor bumped from 2.26 → 2.27 (single-call sparse-checkout set --cone landed in 2.27; 2.26 only accepted --cone on init).
  • inject --filter=blob:none into gitCloneFlags only when sparsePlan.supported and the user hasn't set their own --filter; pass the plan through to setupSparseCheckout.
  • Added unit and integration tests

Testing

  • Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
  • Code is formatted (with go tool gofumpt -extra -w .)

Disclosures / Credits

Claude code drafted all the tests and helped diagnose build failures. I applied and reviewed all changes.

@omehegan omehegan added the bug label Jul 2, 2026
@lizrabuya lizrabuya added feature New user-facing feature! and removed bug labels Jul 3, 2026
@lizrabuya lizrabuya marked this pull request as ready for review July 3, 2026 05:35
@lizrabuya lizrabuya requested review from a team as code owners July 3, 2026 05:35

@buildsworth-bk-app buildsworth-bk-app Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one checkout-path issue that looks like it prevents the new auto-filter behavior from meeting the sparse-checkout intent when users haven't also supplied clone flags such as --sparse.

Want to dig deeper? The full session log is attached to this Buildkite build. Download the session file and open a new pi session with it:

Download the buildsworth logs from build 2623, then answer my questions about the findings.

Comment thread internal/job/checkout.go
if hasPartialCloneFilter(gitCloneFlags) {
e.shell.Commentf("Sparse checkout is configured and BUILDKITE_GIT_CLONE_FLAGS already contains a --filter (preserving user-supplied filter).")
} else {
gitCloneFlags = append(gitCloneFlags, "--filter=blob:none")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: This auto-add also runs when the clone flags don't include --sparse (or another flag that suppresses the initial checkout). In that case git clone --filter=blob:none still checks out the full worktree before we call sparse-checkout set --cone, so Git hydrates blobs outside the requested sparse paths up front and this doesn't deliver the intended partial-clone saving. The new integration coverage includes --sparse, which is the case where the filter works; could we either add --sparse along with the filter when supported, or only add the filter when the clone flags already prevent the full initial checkout?

@lizrabuya lizrabuya marked this pull request as draft July 3, 2026 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New user-facing feature!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants