Skip to content

Add grace period to project stuck creating alert#547

Open
scotwells wants to merge 2 commits intomainfrom
fix/project-stuck-alert-grace-period
Open

Add grace period to project stuck creating alert#547
scotwells wants to merge 2 commits intomainfrom
fix/project-stuck-alert-grace-period

Conversation

@scotwells
Copy link
Copy Markdown
Contributor

@scotwells scotwells commented Mar 28, 2026

Summary

Adds a 2-minute grace period to the ProjectStuckCreatingSLOViolation alert rule. Previously, the alert fired immediately (for: 0s), which meant any brief gap in monitoring data would trigger a false page.

Context

On March 28, a short monitoring database disruption caused this alert to fire incorrectly. The 2-minute grace period gives the system time to recover from brief outages before paging on-call.

Tracking issue: https://github.com/datum-cloud/infra/issues/2059

scotwells and others added 2 commits March 28, 2026 15:51
Add a 2-minute grace period before the ProjectStuckCreatingSLOViolation
alert fires. This prevents false pages when monitoring infrastructure
has brief disruptions that cause incomplete query results.
🤖 Automatically added newlines to 1 file(s)

Co-Authored-By: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown
Contributor

🤖 I automatically added missing newlines at the end of 1 file(s) in this PR.

All files should now end with a newline character as per coding standards.

@joggrbot

This comment has been minimized.

@cla-assistant
Copy link
Copy Markdown

cla-assistant bot commented Mar 28, 2026

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ scotwells
❌ github-actions[bot]
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@cla-assistant
Copy link
Copy Markdown

cla-assistant bot commented Mar 28, 2026

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ scotwells
❌ github-actions[bot]
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant