[QUESTION] Sample idx, bin files in public domain for trying out pretrain_gpt.py?

[sambar1729]

Your question
Ask a clear and concise question about Megatron-LM.

Can we have a sample idx + bin files as required by the pretrain_gpt.py ?

Running tools/preprocess_data.py on some sample data like

{"src": "www.nvidia.com", "text": "The quick brown fox", "type": "Eng", "id": "0", "title": "First Part"}
{"src": "The Internet", "text": "jumps over the lazy dog", "type": "Eng", "id": "42", "title": "Second Part"}

needs transformer_engine and on an A100 this takes a long time to build from source (the pip install

pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable

also fails).

This is just too much work to get some training data to run pretrain_gpt.py with. Can some sample idx, bin files as required by the pretraining be provided in a public place?

Thanks.

[elliottnv]

@ashors1 Can you please help with this question? Thank you!

[ashors1]

Hi, we've made some changes to Megatron recently to remove the required dependency on Transformer Engine. You should no longer need to install Transformer Engine to run this script. The following works for me:

conda create --name megatron python==3.10.12
conda activate megatron
conda install pytorch==2.2.0 torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
git clone https://github.com/NVIDIA/Megatron-LM.git && cd Megatron-LM && pip install .
pip install six regex
python3 tools/preprocess_data.py --input your_input.json --tokenizer-type GPT2BPETokenizer --workers 1 --output-prefix test --vocab-file your_bpe/vocab.json --merge-file your_bpe/merges.txt