add cli dict parsing for grpo_config

[Tavish9]

What does this PR do?

Adds dict parsing logic to model_init_kwargs in grpo_config, enabling dynamic configuration via CLI. Users can now pass dictionary-like strings (e.g., --model_init_kwargs '{"torch_dtype":"bfloat16") through command-line arguments, which are automatically parsed into Python dicts for the target fields.

Logic is the same as TrainingArguments.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[qgallouedec]

Thanks! Does it work is you directly modify transformers.training_args._VALID_DICT_FIELDS instead?

[Tavish9]

Yes, but both of transformers.training_args and trl.GRPOConfig should have their independent _VALID_DICT_FIELDS , as private attribute does.

In GRPOConfig.__post_init__, it first post-inits it's _VALID_DICT_FIELDS and then transformers.training_args's

[qgallouedec]

It seems to work:

from transformers.training_args import _VALID_DICT_FIELDS
from trl import GRPOConfig

_VALID_DICT_FIELDS.append("model_init_kwargs")

args = GRPOConfig("output_dir", model_init_kwargs='{"num_labels": 2}')
print(args.model_init_kwargs)  # {"num_labels": 2}

[qgallouedec]

To do this properly, the first step would be to convert _VALID_DICT_FIELDS into a class attribute of TrainingArguments in transformers. Are you ready to open such a PR in Transformers?

Then we could do:

# in transformers
class TrainingArguments:
    _VALID_DICT_FIELDS = [...]

# in trl
class GRPOConfig(TrainingArguments):
    _VALID_DICT_FIELDS = TrainingArguments._VALID_DICT_FIELDS + ["model_init_kwargs"]

which eliminates the need to duplicate the post init

[Tavish9]

To do this properly, the first step would be to convert _VALID_DICT_FIELDS into a class attribute of TrainingArguments in transformers. Are you ready to open such a PR in Transformers?

Then we could do:

# in transformers
class TrainingArguments:
    _VALID_DICT_FIELDS = [...]

# in trl
class GRPOConfig(TrainingArguments):
    _VALID_DICT_FIELDS = TrainingArguments._VALID_DICT_FIELDS + ["model_init_kwargs"]

which eliminates the need to duplicate the post init

Yes, that was my initial thought as well. However, considering that the transformers defines _VALID_DICT_FIELDS as semi-private, I decided against submitting a PR to their repository. If we follow the semi-private variable approach, each config should ideally have its own variable, even though this might lead to some code duplication in the __post__init__ logic. That said, I’m also open to the idea of modifying the semi-private variable in the transformers to make it a class attribute. However, I’m not sure if the maintainers would be receptive to this change in philosophy.

What's your suggestions?

[qgallouedec]

Yes I think first modifying transformers is the way to go.

[Tavish9]

okay, I would notify you when pr merged. :)

[Tavish9]

Hi, @qgallouedec, the PR in Transformers is merged. 🥳

[Tavish9]

Any potential issues for merging?

[qgallouedec]

I just need to review it carefully and ensure backwards compatibility
I'll do it asap.