[fs] Support AWS S3 credentials provider mode#3540
Open
litiliu wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #3493.
This PR adds a server-side AWS S3 credentials provider mode for the S3 filesystem. When
fs.s3a.aws.credentials.provideris explicitly configured in Fluss configuration, Fluss treats it as the authoritative server-side credential source instead of injecting the client delegated-token provider.This is intended for deployments that use standard AWS SDK/Hadoop S3A providers such as
com.amazonaws.auth.profile.ProfileCredentialsProvider, so rotated long-term credentials can be picked up by the provider without restarting Fluss servers.The full motivation and design discussion are in #3493. This PR description keeps the reviewer-facing summary and the final implemented behavior.
Brief change log
s3.aws.credentials.providers3a.aws.credentials.providerfs.s3a.aws.credentials.providerS3DelegationTokenProvidercan distinguish an explicit Fluss provider from Hadoop default resources.DynamicTemporaryAWSCredentialsProvider;fs.s3a.assumed.role.arn;Credential mode resolution:
Tests
mvn -pl fluss-filesystems/fluss-fs-s3 test -Dtest=S3FileSystemPluginTest,S3DelegationTokenProviderTestAdded/updated coverage for:
DynamicTemporaryAWSCredentialsProvider;DynamicTemporaryAWSCredentialsProviderfor server mode is rejected;API and Format
No public API or storage format changes.
The PR adds an internal Hadoop configuration marker under
fluss.fs.s3.aws.credentials.provider.explicitly.configured. It is not a user-facing option; it only carries whether the provider was explicitly configured through Fluss config.Documentation
No separate documentation update in this PR. The user-facing behavior and operational motivation are described in #3493.
Generative AI disclosure
AGENTS.mdguidance.