Build a valid robots.txt file with allow/disallow rules, sitemap reference, and crawl-delay. Supports Googlebot, Bingbot, and all user-agents.
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Sitemap: https://toolreign.com/sitemap.xml
More free utilities you might find useful
Quick answers to common questions
robots.txt is a file at your site root that tells search engine crawlers which pages they can or cannot access. It does not block pages from being indexed if linked externally.
Upload it to your website root: https://yoursite.com/robots.txt. It must be accessible at that exact path.
Yes. Disallow /admin/, /api/, staging areas, and internal search results. Do not use robots.txt to hide sensitive data — use authentication instead.
Yes. Generated locally in your browser.
Your data is processed entirely in your browser. Nothing is sent to any server.