Skip to content

Conversation

@TayPark
Copy link

@TayPark TayPark commented Jan 22, 2026

Summary

This PR introduces a new RefreshingFileCredentialsProvider that periodically re-reads AWS credentials from file, solving the issue where ProfileFileCredentialsProvider caches credentials indefinitely.

Closes #18591

Problem

Vector's current file-based AWS authentication uses ProfileFileCredentialsProvider which reads the credentials file once at startup and caches them indefinitely. This causes ExpiredToken errors in long-running processes where credentials are rotated externally:

  • IRSA (IAM Roles for Service Accounts) in Kubernetes
  • Credential rotation tools (aws-vault, saml2aws, etc.)
  • External credential providers
ERROR vector::sinks::util::retries: Non-retriable error; dropping the request.
error=ServiceError { source: ErrorMetadata { 
  code: Some("ExpiredToken"), 
  message: Some("The provided token has expired.")
}}

Root Cause Analysis

  1. ProfileFileCredentialsProvider caches indefinitely: The AWS SDK's ProfileFileCredentialsProvider reads credentials once and doesn't re-read the file even when credentials expire.

  2. IdentityCache limitations: While IdentityCache::lazy() supports automatic refresh, it relies on the underlying provider to return new credentials - which ProfileFileCredentialsProvider doesn't do.

  3. SIGHUP doesn't help: Config reload via SIGHUP only recreates sinks if their config changed, not for credential refresh.

Solution

Introduce RefreshingFileCredentialsProvider that:

Feature Description
Periodic refresh Re-reads credentials file at configurable intervals (default: 5 min)
Proactive refresh Refreshes credentials 5 minutes before expiration
Graceful fallback Uses cached credentials if file read fails temporarily
Thread-safe Uses RwLock for concurrent access

New Configuration Option

[sinks.my_s3_sink]
type = "aws_s3"

[sinks.my_s3_sink.auth]
credentials_file = "/path/to/credentials"
profile = "default"
refresh_interval_secs = 300  # Re-read file every 5 minutes (default)

Changes

File Change
src/aws/refreshing_file_credentials.rs New - RefreshingFileCredentialsProvider implementation
src/aws/mod.rs Add module export
src/aws/auth.rs Add refresh_interval_secs to File auth, use new provider

Testing

  • Unit tests for ProviderState refresh logic
  • Unit tests for minimum refresh interval clamping
  • Parsing tests for new refresh_interval_secs config option

Backwards Compatibility

  • Default behavior unchanged (5-minute refresh interval)
  • Existing configs work without modification
  • New refresh_interval_secs option is optional

🤖 Generated with Claude Code

This PR introduces a new `RefreshingFileCredentialsProvider` that
periodically re-reads AWS credentials from file, solving the issue
where `ProfileFileCredentialsProvider` caches credentials indefinitely.

## Problem

Vector's current file-based AWS authentication uses
`ProfileFileCredentialsProvider` which reads the credentials file once
at startup and caches them indefinitely. This causes `ExpiredToken`
errors in long-running processes where credentials are rotated
externally (e.g., IRSA in Kubernetes, credential rotation tools).

Related: vectordotdev#18591

## Solution

Introduce `RefreshingFileCredentialsProvider` that:
- Re-reads credentials file at configurable intervals (default: 5 min)
- Proactively refreshes credentials 5 minutes before expiration
- Falls back to cached credentials if file read fails
- Adds `refresh_interval_secs` config option for File authentication

## Configuration

```toml
[sinks.my_s3_sink]
type = "aws_s3"

[sinks.my_s3_sink.auth]
credentials_file = "/path/to/credentials"
profile = "default"
refresh_interval_secs = 300  # Re-read file every 5 minutes
```

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@TayPark TayPark requested a review from a team as a code owner January 22, 2026 03:01
@github-actions
Copy link

github-actions bot commented Jan 22, 2026

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@TayPark TayPark changed the title fix(aws): add RefreshingFileCredentialsProvider for credential rotation fix(aws/auth): add RefreshingFileCredentialsProvider for credential rotation Jan 22, 2026
@TayPark TayPark changed the title fix(aws/auth): add RefreshingFileCredentialsProvider for credential rotation fix(auth): add RefreshingFileCredentialsProvider for credential rotation Jan 22, 2026
TayPark and others added 2 commits January 22, 2026 12:21
Closes vectordotdev#18591

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@TayPark
Copy link
Author

TayPark commented Jan 22, 2026

recheck

@TayPark
Copy link
Author

TayPark commented Jan 22, 2026

I have read the CLA Document and I hereby sign the CLA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vector Does Not Properly Reload Updated AWS Credentials

1 participant