Check for probably-hallucinated references in arxiv papers

Some vibe-coded scripts (thanks Claude-Code) for finding references in arxiv papers that:

1. match the title of a real arxiv paper
2. have a completely different author list

Some manual checks suggest that most of these references are LLM-hallucinated.

See this blog post for some analysis of the data.

Dependencies

• AWS CLI
• pdftotext++
• anystyle
• anthropic-sdk-python

Steps to acquire the data

1. Download metadata for all arxiv papers (about 1.5GB as of September 2025)

foo@bar:~$ ./dl_arxiv_oai.sh

2. build a much smaller DB (arxiv_papers.json) containing the metadata we actually need

foo@bar:~$ python build_arxiv_db.py

3. download all arxiv PDFs since 2000 and extract refs (about 30GB). WARNING: THIS BILLS YOUR AWS ACCOUNT

foo@bar:~$ python dl_pdfs_and_extract_refs.py

4. find potential hallucinated refs with a bunch of heuristics

foo@bar:~$ python find_hallucinated_refs.py

5. filter false positives using Claude. WARNING: THIS BILLS YOUR ANTHROPIC ACCOUNT

foo@bar:~$ python filter_hallucinated_refs.py

6. plot the results

foo@bar:~$ python show_results.py