RuntimeError: Dataset scripts are no longer supported

[mlang]

When running

python data/openwebtext/prepare.py

python throws the following RuntimeError:

RuntimeError: Dataset scripts are no longer supported, but found openwebtext.py

[wrmedford]

Found a solution here: huggingface/datasets#7693 (comment)

Just need to downgrade your datasets installed version.

[lhoestq]

I opened a PR here to remove the dataset script: https://huggingface.co/datasets/Skylion007/openwebtext/discussions/22

Once it's merged it will work again with recent versions of datasets :)

[VinodHatti7019]

For anyone encountering this issue, here are the current solutions:

Quick Fix (Temporary):

pip install datasets==2.14.0

Downgrading to datasets 2.14.0 or earlier will resolve this immediately.

Permanent Fix:
A PR has been submitted to remove the dataset script from the HuggingFace repository: https://huggingface.co/datasets/Skylion007/openwebtext/discussions/22

Once merged, the issue will be resolved for all recent versions of the datasets library.

Alternative:
You can also manually download the dataset and preprocess it locally without relying on the HuggingFace script.

Hope this helps! 👍