Skip to content

Change online training verifier #1371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
jacklanchantin wants to merge 64 commits into
base: online_training
Choose a base branch
from
Draft

Change online training verifier #1371

jacklanchantin wants to merge 64 commits into from

Conversation

@jacklanchantin
Copy link
Contributor

@jacklanchantin jacklanchantin commented Oct 14, 2025 •
edited
Loading

What does this PR do? Please describe:

  • Adds GrpoLossConfig adv_std_normarlization (for DrGRPO)
  • Adds GrpoLossConfig loss_token_mean for normalizing over all tokens
  • Adds new if statement to skip ref logprob computation for kl if beta == 0
  • Adds tis_imp_ratio_cap to use truncated importance sampling correction

Fixes #{issue number}

Does your PR introduce any breaking changes? If yes, please list them:
List of all backwards-incompatible changes.

Check list:

  • Was the content of this PR discussed and approved via a GitHub issue? (no need for typos or documentation improvements)
  • Did you read the contributor guideline?
  • Did you make sure that your PR does only one thing instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests?
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (no need for typos, documentation, or minor internal changes)

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 14, 2025
@jacklanchantin jacklanchantin changed the title drgrpo Importance Sampling Correction, and DrGRPO args Oct 22, 2025
@jacklanchantin jacklanchantin changed the title Importance Sampling Correction, and DrGRPO args Change online training verifier Oct 22, 2025
jacklanchantin and others added 3 commits November 13, 2025 17:25
@jacklanchantin
tokenizer
5cc0610
@lydiadli @jacklanchantin
set VLLM_ALLOW_INSECURE_SERIALIZATION=1 for newer vllm versions and a…
7f7b50d 
…dd a comment for calling newer vllm generate function (#1396)

* set VLLM_ALLOW_INSECURE_SERIALIZATION=1 for newer vllm versions

* update generate function to align with newer vllm version
@jacklanchantin
comment out unused
14d2571
jacklanchantin and others added 23 commits November 14, 2025 18:45
@jacklanchantin
logging
fcb24b2
@jacklanchantin
logging
8d40e31
@jacklanchantin
logging
424338a
@jacklanchantin
fix src_key_text
45cd0aa
@jacklanchantin
logging
214be19
@jacklanchantin
maybe sync fix
fe061c3
@uralik
do not throttle client-server port
7bbd503
@jacklanchantin
Merge branch 'kulikov/init_fix' of github.com:facebookresearch/fairse…
58b1df7 
…q2 into jacklanchantin/drgrpo
@jacklanchantin
maybe_sync_fix
1641e5c
@jacklanchantin
move tokenizer
9068b24
@lydiadli @uralik @jacklanchantin
ppl/logp reward (#1415)
a39ea52 
* do not throttle client-server port

* maybe_sync_fix

* move tokenizer

* ppl/logp reward

* remove prepare_preference_batch func (intended for online dpo

* keep an noop prepare_preference_batch method to instantiate

* make logging clearer.

* 1. add additional logic to add whitespace if needed. 2. not reusing prefix token from fs2.

* pass prefix text rather than tokens in rm (it would also support rm tokenizer different from policy model)

* clear up string_input flag.

---------

Co-authored-by: uralik <kulikov@cs.nyu.edu>
Co-authored-by: jacklanchantin <jacklanchantin@gmail.com>
@jacklanchantin
move tokenizer
399464d
@jacklanchantin
Merge branch 'jacklanchantin/drgrpo' of github.com:facebookresearch/f…
ba27796 
…airseq2 into jacklanchantin/drgrpo
@jacklanchantin
comment
a03edf6
@jacklanchantin
divide by 0 protection in ppl loss
c57b70d
@jacklanchantin
stability
5adcb1d
@jacklanchantin
clip if no think
ba349a6
@jacklanchantin
print prompt text
c5f6cf6
@jacklanchantin
fixes
d14b610
@jacklanchantin
entropy stabilization
4d61714
@jacklanchantin
clipping
fdef776
@jacklanchantin
clean
6c6048c
@jacklanchantin
start nll loss
a6013ce
@jacklanchantin jacklanchantin force-pushed the jacklanchantin/drgrpo branch 2 times, most recently from f967068 to a6013ce Compare December 2, 2025 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cbalioglu cbalioglu Awaiting requested review from cbalioglu cbalioglu will be requested when the pull request is marked ready for review cbalioglu is a code owner

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jacklanchantin @lydiadli @uralik