robots.txt Checker
Verify if a URL is allowed to be crawled by specific User-Agents. Test your scraper against a site's `robots.txt` rules instantly.
Enter a URL to check if it's safe to scrape.
Respect the Rules
Web scraping has a "code of honor", and rule #1 is respecting `robots.txt`. Before you write a single line of scraper code, use this tool to check if the website owner explicitly forbids automated access to the content you want.
Test Custom Bots
Some sites block `*` (everyone) but allow `Googlebot`. Others block AI bots like `GPTBot` or `CCBot` specifically. Use this tool to see exactly which agents are blocked. If you are building a commercial scraper, you should check if your custom User-Agent string would be allowed.
Frequently asked questions
Learn about scraping permissions
Level up your
data gathering
See why ManyPI is the data extraction platform of choice for
modern technical teams.