✊🏿 Black Lives Matter. Please consider donating to Black Girls Code today.
🐇 Announcing Dash VTK for 3d simulation graphics. Check out the March webinar.

Robots.txt Tester (Show and Tell)

This is a simple robots.txt tester for anyone interested in analyzing robots files especially SEOs.

Once you specify a set of paths or absolute URLs, it extracts all User-agents (and their respective rules) in the file, runs the checks for all combinations of (URL-useragent), and produces a DataFrame in a Dash DataTable.

It can be useful for testing on a large scale, before deploying new robots.txt files, or before publishing a new set of pages.

Source code: https://github.com/eliasdabbas/robotstxt_app

Feedback, suggestions more than welcome :slight_smile:

3 Likes