DOT NOT MERGE -- RDS/Aurora logical refresh component #262
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a new component that automates DBLab full refresh using temporary RDS/Aurora database clones created from snapshots. This enables a hassle-free data sync workflow that doesn't impact production. Features: - Create temporary RDS/Aurora clones from latest automated snapshots - Wait for clone availability with proper timeout handling - Trigger DBLab full refresh via API - Poll refresh status until completion - Clean up temporary clones automatically Deployment options: - AWS Lambda with SAM template and EventBridge scheduling - Standalone CLI binary for cron/manual execution - Docker container for containerized environments Includes comprehensive documentation with IAM policy examples and example configuration files.
Add a fully standalone version of the RDS/Aurora refresh tool that can be built and deployed immediately without requiring a DBLab Engine release. This version has no dependencies on DBLab internal packages. The standalone component includes: - Self-contained Go module with minimal dependencies - Makefile for easy building (CLI, Lambda, Docker) - SAM template for AWS Lambda deployment - Dockerfile for container builds - Comprehensive documentation Can be used immediately by: 1. go build -o rds-refresh . 2. ./rds-refresh -config config.yaml
- Add UpdateSourceConfig method to update DBLab source connection before triggering refresh, ensuring DBLab knows the clone endpoint - Add source database credentials (dbName, username, password) to config for DBLab connection configuration - Remove Lambda-specific code, focusing on container/CLI deployment which better suits the long-running nature of refresh operations - Remove duplicate engine-integrated code (engine/internal/rdsrefresh and engine/cmd/rds-refresh) since standalone component is preferred - Update documentation with container deployment options (Docker, ECS Task, Kubernetes CronJob, CLI with cron) - Update workflow to include config update step before refresh trigger
DBLab expects a flat JSON structure matching ConfigProjection fields,
not a nested structure. The API automatically reloads the config after
update (calls reloadFn internally), so no SIGHUP is needed.
Changed from nested:
{"retrieval": {"dbSource": {"host": "..."}}}
To flat projection format:
{"host": "...", "port": 5432, "dbname": "...", ...}
Contributor
Author
|
@cursor review |
NikolayS
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR is being reviewed by Cursor Bugbot
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
| } | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Bug: Race condition: premature completion on stale status
The WaitForRefreshComplete function can return success prematurely if the DBLab Engine's status is StatusFinished from a previous refresh operation. Since the engine's /full-refresh API triggers the refresh in a goroutine, the status may still show StatusFinished when this function first polls. The function then returns success immediately, causing the tool to delete the clone before the actual refresh has started or completed. The function needs to first wait for a non-StatusFinished status (indicating the new refresh has started) before waiting for it to become StatusFinished again.
The function could return success prematurely if DBLab's status still showed StatusFinished from a previous refresh. Since /full-refresh triggers the operation in a goroutine, polling immediately after could see stale status. Now tracks whether refresh has actually started before accepting StatusFinished as success. This prevents premature clone deletion.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Introduce a standalone
rds-refreshtool that clones from RDS/Aurora snapshots, updates DBLab config, triggers full refresh, and cleans up the clone.rds-refresh/admin/config(host/port/db/user/pass) → trigger/full-refresh→ poll until complete → delete clone.go.mod/go.sum,Dockerfile,Makefile,README.md, andconfig.example.yamlfor CLI/Docker/ECS/K8s usage.config.go,dblab.go(DBLab API client),rds.go(AWS RDS ops),refresher.go(orchestration),main.go(CLI).engine/configs/rds-refresh.example.yaml.engine/go.modwith indirect AWS SDK/Lambda deps.Written by Cursor Bugbot for commit cb8223a. Configure here.