Free tool
robots.txt Checker
Verify if a URL is allowed to be crawled by specific User-Agents. Test your scraper against a site's `robots.txt` rules instantly.
Enter a URL to check if it's safe to scrape.
Respect the Rules
Web scraping has a "code of honor", and rule #1 is respecting `robots.txt`. Before you write a single line of scraper code, use this tool to check if the website owner explicitly forbids automated access to the content you want.
Test Custom Bots
Some sites block `*` (everyone) but allow `Googlebot`. Others block AI bots like `GPTBot` or `CCBot` specifically. Use this tool to see exactly which agents are blocked. If you are building a commercial scraper, you should check if your custom User-Agent string would be allowed.
Questions
We already know your next customer
Describe your ideal customer. ManyPI finds validated leads, reaches out, and turns emails into sales.