Instruction tuning GPT-2 video, github code and dataset

[cpgrant]

Keep up the AI for good work. Would like to suggest:

• a video about instruction tuning gpt-2 including github code and dataset
• a video about finetuning gpt-2 with github code + domain specific dataset

Watched and ran your train_gpt2.py from the build-nanogpt repository on a desktop 4090 GPU.
It ran perfectly after adjusting a few variables suggested by ChatGPT. Initially it ran into a memory error. Thank you 😊

[dustinwloring1988]

If you wanted to train on github code would it just be better to add it to your pretraining?

[ViswanathaReddyGajjala]

I think @cpgrant might be interested in understanding and learning about instruction tuning in general. This knowledge can help in building private applications.