feature(nyz&dcy): add LLM/VLM RLHF loss (PPO/GRPO/RLOO) #857
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PaParaZz1
force-pushed
the
dev-rlhf-loss
branch
from
6965fd3 to
2e49437
Compare
PaParaZz1
changed the title
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #857 +/- ##
==========================================
+ Coverage 75.44% 75.52% +0.07%
==========================================
Files 689 698 +9
Lines 56360 56679 +319
==========================================
+ Hits 42523 42807 +284
- Misses 13837 13872 +35
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Berit-chengyi
force-pushed
the
dev-rlhf-loss
branch
from
d3f6f3f to
9cb6ca3
Compare
- Add test_grpo_rlhf.py for GRPO unit tests - Add test_rloo_rlhf.py for RLOO unit tests - Update GRPO implementation - Update RLOO implementation
Berit-chengyi
force-pushed
the
dev-rlhf-loss
branch
from
9cb6ca3 to
8d34eac
Compare
Berit-chengyi
force-pushed
the
dev-rlhf-loss
branch
from
eba91a1 to
2cbd9fb
Compare
Berit-chengyi
force-pushed
the
dev-rlhf-loss
branch
from
7bcd64d to
17a7a71
Compare
PaParaZz1
commented
Feb 19, 2025
ding/rl_utils/grpo.py
Outdated
| ) -> Tuple[namedtuple, namedtuple]: | ||
| """Calculate the policy loss for GRPO | ||
| Args: | ||
| data (grpo_policy_data): Data containing the following fields: |
There was a problem hiding this comment.
polish comment formats
ding/rl_utils/grpo.py
Outdated
| clip_ratio: float = 0.2, | ||
| beta: float = 0.1, # Weight coefficient for KL divergence | ||
| ) -> Tuple[namedtuple, namedtuple]: | ||
| """Calculate the policy loss for GRPO |
ding/rl_utils/rloo.py
Outdated
| } | ||
|
|
||
| # Create return namedtuples | ||
| loss_info = namedtuple('LossInfo', ['policy_loss'])(policy_loss=loss) |
There was a problem hiding this comment.
if there is just one field, you can directly return it rather than use namedtuple
ding/rl_utils/rloo.py
Outdated
|
|
||
| # Create return namedtuples | ||
| loss_info = namedtuple('LossInfo', ['policy_loss'])(policy_loss=loss) | ||
| metric_info = namedtuple('MetricInfo', list(metrics.keys()))(**metrics) |
There was a problem hiding this comment.
you can define namedtuple at the beginning of this file
ding/rl_utils/grpo.py
Outdated
| data (grpo_policy_data): Data containing the following fields: | ||
| - logit_new: Current policy logits [B, L, V] | ||
| - logit_old: Old policy logits [B, L, V] | ||
| - logit_ref: Reference policy logits [B, L, V] |
ding/rl_utils/grpo.py
Outdated
| - logit_ref: Reference policy logits [B, L, V] | ||
| - action: Actions taken [B, L] | ||
| - adv: Advantage values [B] | ||
| - weight: Attention mask [B, L] |
There was a problem hiding this comment.
use the extra Shapes part
ding/rl_utils/grpo.py
Outdated
| - adv: Advantage values [B] | ||
| - weight: Attention mask [B, L] | ||
| clip_ratio (float): PPO clipping ratio, default 0.2 | ||
| beta (float): Weight coefficient for KL divergence, default 0.1 |
There was a problem hiding this comment.
add a period to the end of each sentence.
ding/rl_utils/grpo.py
Outdated
| - logit_old: Old policy logits [B, L, V] | ||
| - logit_ref: Reference policy logits [B, L, V] | ||
| - action: Actions taken [B, L] | ||
| - adv: Advantage values [B] |
Berit-chengyi
force-pushed
the
dev-rlhf-loss
branch
from
7a82a7b to
5358b8d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Related Issue
TODO
Check List