-
Notifications
You must be signed in to change notification settings - Fork 325
Description
Describe the bug
#9228 introduced a regression.
The following description leads to this invalid TypeSpec definition after being imported.
@extension("x-oaiMeta", #{name: "Python Grader", group: "graders", example: """
{
"type": "python",
"name": "Example python grader",
"image_tag": "2025-05-08",
"source": """
def grade(sample: dict, item: dict) -> float:
\"""
Returns 1.0 if `output_text` equals `label`, otherwise 0.0.
\"""
output = sample.get("output_text")
label = item.get("label")
return 1.0 if output == label else 0.0
""",
}When in fact it should result in the following.
@extension(
"x-oaiMeta",
#{
name: "Python Grader",
group: "graders",
example: "{\n \"type\": \"python\",\n \"name\": \"Example python grader\",\n \"image_tag\": \"2025-05-08\",\n \"source\": \"\"\"\ndef grade(sample: dict, item: dict) -> float:\n \\\"\"\"\n Returns 1.0 if `output_text` equals `label`, otherwise 0.0.\n \\\"\"\"\n output = sample.get(\"output_text\")\n label = item.get(\"label\")\n return 1.0 if output == label else 0.0\n\"\"\",\n}\n",
}
)Reproduction
components:
schemas:
GraderPython:
x-oaiMeta:
name: Python Grader
group: graders
example: |
{
"type": "python",
"name": "Example python grader",
"image_tag": "2025-05-08",
"source": """
def grade(sample: dict, item: dict) -> float:
\"""
Returns 1.0 if `output_text` equals `label`, otherwise 0.0.
\"""
output = sample.get("output_text")
label = item.get("label")
return 1.0 if output == label else 0.0
""",
}Here the example property in the extension should be considered a literal string, and not JSON that needs to be parsed.
Checklist
- Follow our Code of Conduct
- Check that there isn't already an issue that request the same bug to avoid creating a duplicate.
- Check that this is a concrete bug. For Q&A open a GitHub Discussion.
- The provided reproduction is a minimal reproducible example of the bug.
Copilot
Metadata
Metadata
Labels
bugSomething isn't workingSomething isn't working