[Draft] use current metrics for adaptive circuit breaker reporting#913
Draft
adriangudas wants to merge 129 commits intomainfrom
Draft
[Draft] use current metrics for adaptive circuit breaker reporting#913adriangudas wants to merge 129 commits intomainfrom
adriangudas wants to merge 129 commits intomainfrom
Conversation
Update variable names Fill sliding window with 1 hr worth data Add comment Update experiemnt resource to be deterministic Change deterministic default value to false Cleanup Remove unused variable Make initial seed error rate more customizable Add seed_error_rate as a property
* Prefilling added * Change initial duration to 900 s
* testing different circuit breaking scenarios * adding concurrency * adds more puts to get further information during phases * Fixing concurrency, unprotected ping, extras * update classic sustained test * cleaning up outdated tests, and testing without ping rate * modify ki instead of dividing by window size --------- Co-authored-by: Abdulrahman Alhamali <abdulrahman.alhamali@shopify.com>
…reads Different Semians between different threads
* Set up automated test for PID controller Reset Update experiment graphs Delete existing image Done Update experiment graphs Update experiment graphs Refactor experiments folder and CI Update scope of commit Update experiment graphs Update experiment graphs Update Gemfile Update experiment graphs Move windup file into test folder Update experiment graphs CI should only commit main graphs Update experiment graphs New bot Fail fast Test Delete comment Update experiment graphs test Revert test Add the commit Test Test * Rework CI check Done Final
* allow max threads, and introduce slow query experiment * fix my experiment, and run all other experiments
* add tables, and fix bug in helper * remove useless check * replace with csv tables for better display
) * Replace p2 estimator with SES * Fix PID tests to use new smoother * Remove p2 * update images * Correct comment * update images * Clean up comments, address Abed's concerns * Update comment * Update images * run all experiments * Unit tests for success criteria * Add incident detection and adaptive convergence * Update success criteria version 2 * Address Abed's comments * Update experiments Gemfile to prevent error * Update experiments * Address Abed comments again * Remove init * Experiments --------- Co-authored-by: Abdulrahman Alhamali <abdulrahman.alhamali@shopify.com>
Remove Throughput and Duration Graphs
Test Near target error rate
* switch algorithm to a sliding window instead of discrete * clean up AI slop PR * provide observations per minute to the smoother from the pid controller * fix tests * remove unused smoother in setup * fix alpha value * fix tests and run experiments * fix max size for sliding window and rerun experiments
Remove unnecessary comments and fix smoother initalization
* add elastic defensiveness * remove kd * update docs and run experiments * fix tests --------- Co-authored-by: Fernando Aguasvivas <fernando.aguasvivas@shopify.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use current metrics instead of last window for the error state transition message. This way, we get an up-to-date understanding of the error rate at the moment of transition.
This is also missing
last_error_messageanderror_last_atto be parity with legacy.On a 100% error rate spike, now prints: