To celebrate leap day we are excited to announce the launch of "Can I Crawl?"
This is a new tool that we've launched to help webmasters and robots with interpreting and understanding robots.txt files.
The tool features color coding for each type of directive, in-line links for URL's in the robots.txt file, descriptions for each directive, and a way to switch from an interactive view to a raw view.
It even divides the file into sections by User-Agent directive to help readability.
If you're setting up a robots.txt file to instruct bots on how to crawl (or not crawl) your site, this tool is great for checking your instructions for accuracy.
Or maybe you have a bot and need to check a site's robots.txt file for instructions and sitemaps to help your crawl - this tool is perfect for that as well.
We hope you find the tool useful - Enjoy!