[Fix] s3-bucket-init job IMDS region crash#303
Merged
tal-haim merged 5 commits intoJul 12, 2026
Conversation
…crash The create-bucket container never set AWS_DEFAULT_REGION/AWS_REGION, so botocore fell back to EC2 IMDS region discovery inside the pod. That call fails in most Kubernetes environments (especially Azure) and hits a long-standing botocore bug that prints a raw AWSRequest object repr instead of a proper error, crashing the job even on a fresh install with no pre-existing bucket. Co-authored-by: Cursor <cursoragent@cursor.com>
…nit job The AWS_REGION environment variable was removed from the s3-bucket-init job as it was redundant. The job now only sets AWS_DEFAULT_REGION, preventing unnecessary configuration and potential confusion. Co-authored-by: Cursor <cursoragent@cursor.com>
royischoss
reviewed
Jul 9, 2026
royischoss
left a comment
royischoss
left a comment
Collaborator
There was a problem hiding this comment.
LGTM small improvement suggestion
royischoss
changed the title
| name: mlrun-ce | ||
| version: 0.12.0-rc.3 | ||
| version: 0.12.0-rc.4 | ||
| appVersion: 1.12.0-rc16 |
Collaborator
There was a problem hiding this comment.
Please update to the latest RC - https://github.com/mlrun/mlrun/releases and also in the values.yaml
Contributor
Author
There was a problem hiding this comment.
Can you verify now?
…gion variable This commit updates the MLRun app version in Chart.yaml and values.yaml to 1.12.0-rc18. Additionally, the redundant S3 region variable has been removed from the templates, simplifying the configuration for the s3-bucket-init job.
The action's install script creates its cache dir before verifying the downloaded binary's signature via cosign/Rekor, so a transient upstream verification failure leaves a stale empty cache dir behind. Retry once after clearing it so a flaky Rekor lookup doesn't fail the whole job. Co-authored-by: Cursor <cursoragent@cursor.com>
…ommand Improves troubleshooting when both aws s3 mb and ls fail by echoing a clear failure reason in job logs (PR review feedback). Co-authored-by: Cursor <cursoragent@cursor.com>
tal-haim
force-pushed
the
fix/CEML-721-s3-bucket-init-region
branch
from
July 12, 2026 08:19
1f07012 to
f1fe14d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Description
Fixes CEML-721: the
mlrun-ce-s3-bucket-initpost-install hook job fails on fresh installs because thecreate-bucketcontainer runsaws s3without setting a region. Botocore then attempts EC2 IMDS region discovery inside the pod, which fails in most Kubernetes environments (especially Azure) and hits a long-standing botocore bug (aws/aws-cli#8988) that prints a raw<botocore.awsrequest.AWSRequest object at 0x...>repr instead of a proper error. Withhelm install --wait, the failed hook causes the entire release to report FAILED.🛠️ Changes Made
seaweedfs.s3.region: us-east-1tovalues.yamlas the configurable region for the bucket-init job's AWS CLI callsmlrun-ce.s3.regionhelper in_helpers.tplAWS_DEFAULT_REGIONandAWS_EC2_METADATA_DISABLED=trueon thecreate-bucketcontainer ins3-bucket-init-job.yaml2>&1) on bothaws s3commands so future failures produce readable log output0.12.0-rc.3→0.12.0-rc.4✅ Checklist
charts/mlrun-ce/Chart.yaml.🧪 Testing
helm template mlrun charts/mlrun-ce -f charts/mlrun-ce/values.yaml --show-only templates/seaweedfs/s3-bucket-init-job.yaml— confirmsAWS_DEFAULT_REGION=us-east-1andAWS_EC2_METADATA_DISABLED=truerender correctlyhelm lint charts/mlrun-ce— passes (0 failed)🔗 References
🚨 Breaking Changes?
🔍️ Additional Notes
The existing
aws s3 mb || aws s3 lsidempotency fallback is unchanged — it already handles the "bucket already exists" case from CEML-698. This fix addresses a separate failure mode (IMDS region discovery) that reproduces on genuinely fresh installs with no pre-existing bucket.No changes needed to the three install-mode values files — they only toggle
seaweedfs.enabledand do not overrideseaweedfs.s3.