Add auto_insert_empty_system_msg config flag

[BramVanroy]

Currently there is functionality to automatically insert an empty system message if system occur in the Jinja template.

alignment-handbook/src/alignment/data.py

Lines 27 to 38 in 87cc800

	def maybe_insert_system_message(messages, tokenizer):
	if messages[0]["role"] == "system":
	return
	# chat template can be one of two attributes, we check in order
	chat_template = tokenizer.chat_template
	if chat_template is None:
	chat_template = tokenizer.default_chat_template
	# confirm the jinja template refers to a system message before inserting
	if "system" in chat_template:
	messages.insert(0, {"role": "system", "content": ""})

This is not foolproof method: GEMMA (SFT) for instance has an explicit part in its chat template that says that if the role is "system" that an error must be raised. So this leads to a conflict with the alignment handbook. Therefore I suggest giving all control to the user but doing so in a backwards-compatible manner: by adding a flag that, if True, adds the system message with the same behavior as before, and when False no system message will be added. To this end, auto_insert_empty_system_msg is added as a DataArguments argument (defaulting to True).

Note that there is one small breaking change: the maybe_insert_system_message is currently implemented only for sft, generation and rm, but it was not used for dpo. There, a system message was ALWAYS added if the first message was not system, even if the Jinja template did not mention system. I changed that to be in line with the other tasks so the default behavior will be a litttle bit different when a tokenizer's chat template does not contain 'system' in DPO.

[lewtun]

Thanks for the fix, this LGTM! I think the small issue about DPO not being 100% consistent with the previous behaviour is fine for now

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[WeitaoLu]

A small bug solution, if you encounter AttributeError: 'DataArguments' object has no attribute 'auto_insert_empty_system_msg'
when running dpo , you can commit line 99 in run_dpo.py to disable this flag
#"auto_insert_empty_system_msg": data_args.auto_insert_empty_system_msg,

[BramVanroy]

A small bug solution, if you encounter AttributeError: 'DataArguments' object has no attribute 'auto_insert_empty_system_msg' when running dpo , you can commit line 99 in run_dpo.py to disable this flag #"auto_insert_empty_system_msg": data_args.auto_insert_empty_system_msg,

If you get the attribute error you'll likely just need to update the repo to the newer version.