Enter a website to fetch and validate its robots.txt file, then test whether a specific URL is allowed or blocked for a crawler.
