eve-k: use stable :latest tag for eve-external-boot-image in VMIRSes#6100
Draft
andrewd-zededa wants to merge 1 commit into
Draft
eve-k: use stable :latest tag for eve-external-boot-image in VMIRSes#6100andrewd-zededa wants to merge 1 commit into
andrewd-zededa wants to merge 1 commit into
Conversation
github-actions
Bot
requested review from
OhmSpectator,
eriknordmark,
naiming-zededa,
rene,
rucoder,
shjala and
zedi-pramodh
June 29, 2026 21:47
| @@ -763,8 +766,7 @@ func Run(ps *pubsub.PubSub, loggerArg *logrus.Logger, logArg *base.LogObject, ar | |||
| zedkubeCtx.reconcileSRIOVDevicePlugin(&aa) | |||
| } | |||
| } | |||
| zedkubeCtx.applyLonghornNodeDrainPolicy() | |||
|
|
|||
| zedkubeCtx.bootImgMigrator.step(&zedkubeCtx) | |||
Contributor
There was a problem hiding this comment.
can we check if this is not done yet then go into this?
| patchCtx, patchCancel := context.WithTimeout(context.Background(), kubeAPITimeout) | ||
| _, merr := virtClient.ReplicaSet(kubeapi.EVEKubeNameSpace).Update( | ||
| patchCtx, vmirs, metav1.UpdateOptions{}) | ||
| patchCancel() |
Contributor
There was a problem hiding this comment.
to see if we can use patch like:
_, merr := virtClient.ReplicaSet(kubeapi.EVEKubeNameSpace).Patch(
patchCtx, vmirs.Name, k8stypes.MergePatchType, patch, metav1.PatchOptions{})
which should give a limited change than the update one.
Contributor
Author
There was a problem hiding this comment.
Yeah this will avoid resource version issues between the list and update, will incorporate the change.
andrewd-zededa
force-pushed
the
eve-k-external-boot-image-tag
branch
from
July 1, 2026 18:10
25ddcc8 to
96b156d
Compare
andrewd-zededa
force-pushed
the
eve-k-external-boot-image-tag
branch
from
July 6, 2026 23:10
96b156d to
c615ef2
Compare
andrewd-zededa
force-pushed
the
eve-k-external-boot-image-tag
branch
from
July 6, 2026 23:13
c615ef2 to
28ba07c
Compare
Contributor
Author
|
Fixed timing race with descheduler, and issue with bumping the latest->latest tag forward. Need to test locally before moving to ready to review. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #6100 +/- ##
==========================================
+ Coverage 22.33% 22.78% +0.45%
==========================================
Files 491 502 +11
Lines 91901 93704 +1803
==========================================
+ Hits 20529 21354 +825
- Misses 69699 70495 +796
- Partials 1673 1855 +182 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Container-as-VM VMIRSes were embedding a versioned eve-external-boot-image tag (e.g. :1.2.3) that gets deleted by cluster-init.sh cleanup on every baseOS upgrade, permanently breaking virt-launcher pod scheduling on upgraded nodes (ErrImageNeverPull, ImagePullPolicy: PullNever). Fix by decoupling image lifecycle from VMIRS references: - cluster-init.sh: tag the imported image as both :VERSION and :latest, and exclude :latest from cleanup_old_external_boot_images so it survives across upgrades. The :latest re-tag uses `ctr image tag --force`: the image store is persistent (/persist/vault/containerd), so :latest already exists from the prior version and a plain tag would fail with AlreadyExists — which, combined with the inspecti early-return guard, would leave :latest stuck on the old image permanently. - kubevirt.go CreateReplicaVMIConfig: use :latest instead of reading /run/eve-release, so new VMIRSes never embed a version-specific tag. - zedkube bootImgMigrator (new): one-shot state machine (WaitReady → Migrate → Done) that migrates existing VMIRSes still referencing a versioned tag. WaitReady polls independently for both eve-external-boot-image:latest in k3s containerd and the KubeVirt CR Available condition, and requires nodeName to be known before advancing (an empty nodeName would skip every VMIRS and latch to Done). Scoped to VMIRSes whose kubernetes.io/hostname node affinity matches the local node, so each node only patches its own apps. Once complete the event loop skips it (isDone) to avoid rebuilding a kubevirt client every tick. This races the descheduler in a rolling upgrade: an app can be rescheduled onto the upgraded node before its VMIRS is patched. step() collapses WaitReady → Migrate → Done into a single tick to shrink that window, and the case self-heals — checkStuckPendingVMI force-deletes the stuck Pending VMI and the ReplicaSet recreates it from the patched template. Also documents the migration in docs/zedkube.md and normalizes a pre-existing markdown table separator in the same file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Andrew Durbin <andrewd@zededa.com>
andrewd-zededa
force-pushed
the
eve-k-external-boot-image-tag
branch
from
July 23, 2026 18:25
28ba07c to
7eff58a
Compare
Contributor
Author
|
rebased on latest master, moving to test next |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Container-as-VM VMIRSes were embedding a versioned eve-external-boot-image
tag (e.g. :1.2.3) that gets deleted by cluster-init.sh cleanup on every
baseOS upgrade, permanently breaking virt-launcher pod scheduling on
upgraded nodes (ErrImageNeverPull, ImagePullPolicy: PullNever).
Fix by decoupling image lifecycle from VMIRS references:
cluster-init.sh: tag the imported image as both :VERSION and :latest,
and exclude :latest from cleanup_old_external_boot_images so it survives
across upgrades. The :latest re-tag uses
ctr image tag --force: theimage store is persistent (/persist/vault/containerd), so :latest
already exists from the prior version and a plain tag would fail with
AlreadyExists — which, combined with the inspecti early-return guard,
would leave :latest stuck on the old image permanently.
kubevirt.go CreateReplicaVMIConfig: use :latest instead of reading
/run/eve-release, so new VMIRSes never embed a version-specific tag.
zedkube bootImgMigrator (new): one-shot state machine
(WaitReady → Migrate → Done) that migrates existing VMIRSes still
referencing a versioned tag. WaitReady polls independently for both
eve-external-boot-image:latest in k3s containerd and the KubeVirt CR
Available condition, and requires nodeName to be known before advancing
(an empty nodeName would skip every VMIRS and latch to Done). Scoped to
VMIRSes whose kubernetes.io/hostname node affinity matches the local
node, so each node only patches its own apps. Once complete the event
loop skips it (isDone) to avoid rebuilding a kubevirt client every tick.
This races the descheduler in a rolling upgrade: an app can be
rescheduled onto the upgraded node before its VMIRS is patched. step()
collapses WaitReady → Migrate → Done into a single tick to shrink that
window, and the case self-heals — checkStuckPendingVMI force-deletes the
stuck Pending VMI and the ReplicaSet recreates it from the patched
template.
Also documents the migration in docs/zedkube.md and normalizes a
pre-existing markdown table separator in the same file.
PR dependencies
None
How to test and validate this PR
TODO
Changelog notes
TODO
PR Backports
TODO
Checklist
And the last but not least:
check them.
Please, check the boxes above after submitting the PR in interactive mode.