[NewOp] Add generate_challenging_qa_mapper based on MindGYM principles

[Bat-Reality]

​​Introduces a novel QA generation module based on self-challenging mechanisms, designed to autonomously synthesize high-quality reasoning-focused question-answer pairs, inspired by the MindGYM paper.​​

[cyruszhang]

can import from model_utils: torch, vllm

[cyruszhang]

can use existing third party retry library or put retry_on_error in a util

[HYLcool]

Please merge the latest main branch and run pre-commit locally.

[lingzhq]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a new generate_challenging_qa_mapper operator, a significant feature for generating reasoning-focused QA pairs using a multi-turn conversation with a language model. The implementation is well-structured. However, I've identified a few issues that need attention. There's a critical error handling gap that could lead to a crash if the model's output is malformed. Additionally, the GPU configuration is hardcoded, which could cause runtime errors or suboptimal performance in different environments. I've also provided suggestions to improve code clarity and enhance the new test case with assertions.

[gemini-code-assist]

If extract_json returns None because the model output is malformed, the subsequent line qa["thinking"] = ... will raise a TypeError, causing the process to crash. It's crucial to check if qa is None and handle this case gracefully, for instance, by raising a ValueError to trigger the retry mechanism with a more informative error message.

Suggested change

	qa = self.extract_json(qa[0].outputs[0].text)
	qa["thinking"] = multihop[0].outputs[0].text
	qa = self.extract_json(qa[0].outputs[0].text)
	if qa is None:
	raise ValueError("Failed to extract valid JSON from model output.")
	qa["thinking"] = multihop[0].outputs[0].text

[gemini-code-assist]

This line is redundant as self.model_key is immediately overwritten on line 116. It should be removed to avoid confusion and unnecessary model preparation.

[gemini-code-assist]

This test runs the operator but doesn't have any assertions to verify the output. A test should validate the behavior of the code. Please add assertions to check that the returned result dictionary contains the expected keys from the generated QA pair. This will make the test more meaningful and robust.

Suggested change

	result = op.process(deepcopy(sample))
	print(f'Output results: {result}')
	result = op.process(deepcopy(sample))
	self.assertIn('background_document', result)
	self.assertIn('reasoning_category', result)
	self.assertIn('sub_questions', result)
	self.assertIn('relationship_category', result)
	self.assertIn('multihop_question', result)
	self.assertIn('multihop_answer', result)
	self.assertIn('thinking', result)
	print(f'Output results: {result}')

[lingzhq]

I ran some tests locally and have a few points of feedback.

ERRORs Found During Testing:

• Mixed Languages in Output: The model's output sometimes contains mixed Chinese and English, for example, returns a sample containing a new field named '背景文档'.

• No Clear Termination Condition: The generation process doesn't seem to respect the requested number of outputs. I tried to generate 3 samples, but the log showed at least 50 "Processed prompts" iterations without stopping.

Other suggestions for Improvement:

• JSON Parsing Robustness: The process still requires many retries (always up to 6 times) due to frequent JSON parsing failures.
• Generation Configs: Consider exposing generation configurations as kwargs to the user instead of hardcoding them.
• Performance: The generation speed is a bit slow, currently averaging around 45 toks/s for the output.