Skip to content

Extract early stopping replay utilities to OSS #4744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
shrutipatel31 wants to merge 1 commit into from
Closed

Extract early stopping replay utilities to OSS #4744

shrutipatel31 wants to merge 1 commit into from
+158 −0

Conversation

@shrutipatel31
Copy link
Contributor

@shrutipatel31 shrutipatel31 commented Jan 6, 2026 •
edited
Loading

Summary:
Adds the estimate_hypothetical_early_stopping_savings() function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341

@meta-cla meta-cla bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Jan 6, 2026
@meta-codesync
Copy link

meta-codesync bot commented Jan 6, 2026

@shrutipatel31 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D90150341.

shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 6, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
db5e818 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 6, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
d1b20e5 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 6, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
596be05 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
@codecov-commenter
Copy link

codecov-commenter commented Jan 6, 2026 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.72%. Comparing base (c8a5280) to head (4aa249c).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #4744      +/-   ##
==========================================
- Coverage   96.72%   96.72%   -0.01%     
==========================================
  Files         582      583       +1     
  Lines       60809    60857      +48     
==========================================
+ Hits        58818    58864      +46     
- Misses       1991     1993       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 7, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
6489da6 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 7, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
d0dde1d 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
1a5d934 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
9f40eac 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
1f1b98f 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
33de530 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
177cfd2 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
a38f721 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Key changes:

- Added `estimate_hypothetical_early_stopping_savings()` to `experiment_replay.py` which combines `get_default_ess_or_none()`, `replay_experiment()`, and `estimate_early_stopping_savings()` into a single utility
- Added constants `MAX_REPLAY_TRIALS`, `REPLAY_NUM_POINTS_PER_CURVE`, and `MAX_PENDING_TRIALS` to `experiment_replay.py`
- Added optional `minimize` parameter to `replay_experiment()` to explicitly control optimization direction
- Updated `ax_sweep_orchestrator.py` to use the new `estimate_hypothetical_early_stopping_savings()` function
- Added unit tests for the new function in `test_experiment_replay.py`

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
a514410 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
7568d2d 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
4c9e8d4 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
43bff93 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 9, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
3294d10 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 12, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
c1287d1 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 12, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
530e212 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 13, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
a4b4209 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 13, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
380c028 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 13, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
91b0c73 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 14, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
98e662f 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 pushed a commit to shrutipatel31/Ax that referenced this pull request Jan 14, 2026
@meta-codesync
Extract early stopping replay utilities to OSS (facebook#4744)
444c507 
Summary:
Pull Request resolved: facebook#4744

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 14, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
365498d 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Reviewed By: bernardbeckerman

Differential Revision: D90150341
shrutipatel31 pushed a commit to shrutipatel31/Ax that referenced this pull request Jan 14, 2026
@meta-codesync
Extract early stopping replay utilities to OSS (facebook#4744)
cc0f1af 
Summary:
Pull Request resolved: facebook#4744

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341

Reviewed By: bernardbeckerman
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 14, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
b460457 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Reviewed By: bernardbeckerman

Differential Revision: D90150341
shrutipatel31 pushed a commit to shrutipatel31/Ax that referenced this pull request Jan 14, 2026
@meta-codesync
Extract early stopping replay utilities to OSS (facebook#4744)
d5fbcb6 
Summary:
Pull Request resolved: facebook#4744

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341

Reviewed By: bernardbeckerman
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 15, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
e0b3a74 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Reviewed By: bernardbeckerman

Differential Revision: D90150341
shrutipatel31 pushed a commit to shrutipatel31/Ax that referenced this pull request Jan 15, 2026
@meta-codesync
Extract early stopping replay utilities to OSS (facebook#4744)
eed3423 
Summary:
Pull Request resolved: facebook#4744

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Differential Revision: D90150341

Reviewed By: bernardbeckerman
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 15, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
49ac9f3 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Reviewed By: bernardbeckerman

Differential Revision: D90150341
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
4aa249c 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Reviewed By: bernardbeckerman

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 15, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
138ad57 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Reviewed By: bernardbeckerman

Differential Revision: D90150341
shrutipatel31 added a commit to shrutipatel31/Ax that referenced this pull request Jan 15, 2026
@shrutipatel31 @facebook-github-bot
Extract early stopping replay utilities to OSS (facebook#4744)
206b2c6 
Summary:

Adds the `estimate_hypothetical_early_stopping_savings()` function to the OSS module. This function estimates potential compute savings by replaying an experiment with a default early stopping strategy.

Reviewed By: bernardbeckerman

Differential Revision: D90150341
@meta-codesync meta-codesync bot closed this in 10dd3e7 Jan 15, 2026
@meta-codesync
Copy link

meta-codesync bot commented Jan 15, 2026

This pull request has been merged in 10dd3e7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

CLA Signed Do not delete this pull request or issue due to inactivity. fb-exported Merged meta-exported

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shrutipatel31 @codecov-commenter @facebook-github-bot