This is a small package to make parsing robots.txt rules easier.
-
Updated
Mar 31, 2017 - PHP
This is a small package to make parsing robots.txt rules easier.
Go robots.txt parser
Robots.txt parser / generator
Replaces the default virtual robots.txt generated by WordPress with an editable one, and deletes any physical robots.txt file that may already exist.
python binding for Google robots.txt parser C++ library
A sane, minimal robots.txt file (for the western world)
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
A java clone of Google's robotst.txt parser: https://github.com/google/robotstxt
A native Rust port of Google's robots.txt parser and matcher C++ library.
Robots Scanner
Stream decoders for sitemap.xml data and link feeds.
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.
Determining bias to search engines from Robots.txt
Workshop materials for creating the Center City Sips District 2022 interactive map
A Web Scraping project using robotstxt & rvest in R programming.
Add a description, image, and links to the robotstxt topic page so that developers can more easily learn about it.
To associate your repository with the robotstxt topic, visit your repo's landing page and select "manage topics."