Skip to content

Commit

Permalink
feat(data): Smaller extract for human eval demo
Browse files Browse the repository at this point in the history
Add an additional, limited JSONL evaluation job input manifest for
running shorter demonstrative human evaluation jobs (where 20
examples take quite a while to go through)
  • Loading branch information
athewsey committed Jun 12, 2024
1 parent 59c788e commit 6883bae
Show file tree
Hide file tree
Showing 2 changed files with 36 additions and 0 deletions.
26 changes: 26 additions & 0 deletions datasets/Prepare-SQuAD.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -441,6 +441,32 @@
" )"
]
},
{
"cell_type": "markdown",
"id": "edd8abd8-3837-48f0-bad5-ade36e39f4bc",
"metadata": {},
"source": [
"For **demonstrating human evaluation** jobs, even 20 examples might be a bit tedious to wade through in a workshop: So we'll create an even shorter dataset for that as a separate file:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ee0b1c50-b6b8-4406-9eb7-b732c55c9f3e",
"metadata": {},
"outputs": [],
"source": [
"with open(\"question-answering/eval-job-input-qa.manifest.jsonl\") as fin:\n",
" with open(\"question-answering/human-demo-input-qa.manifest.jsonl\", \"w\") as fout:\n",
" for ix, line in enumerate(fin):\n",
" if ix >= 10:\n",
" break\n",
" if ix > 0:\n",
" fout.write(\"\\n\")\n",
" fout.write(line.strip())\n",
"print(f\"Wrote {ix} records to human demo file\")"
]
},
{
"cell_type": "markdown",
"id": "b0e0d33e-aac0-41c0-b2b2-8ae40e9195f9",
Expand Down
Loading

0 comments on commit 6883bae

Please sign in to comment.