Web Tools tools.bryantsmith.com

SEO & Performance

Robots.txt Generator

Start from a common crawling setup, then fine-tune the user-agents, rules and sitemap. The robots.txt preview updates as you type — nothing leaves your browser.

Runs in your browser v1.0.0

Start from a common setup

1-click preset

Presets load an editable starting point. Change anything below and the preview follows.

Rules

editable

Added as a Sitemap: line at the end of the file. Leave blank to omit it.

robots.txt

live 


                

                

                    
                    
                    
                    
                

            

        

    


    

        

            
How it works

        

        

            

                Each user-agent group becomes a User-agent: line plus its
                Allow:, Disallow: and Crawl-delay: directives, in the
                order shown. Paths are normalised to start with /, and one Sitemap:
                line is appended when you provide a URL. The whole file is assembled locally with plain
                JavaScript, so it can be copied straight into your site root or downloaded.
                Copy share link packs the whole configuration into the URL fragment so a
                setup can be bookmarked or sent to a teammate.
            

        

    




Related tools
  • Robots.txt TesterCheck whether a URL is allowed or blocked for a chosen user-agent under pasted robots.txt rules.
  • Sitemap XML GeneratorTurn a list of URLs into valid sitemap XML with live validation, copy, and download — entirely in your browser.
  • Schema.org GeneratorBuild common schema.org JSON-LD structured data with guided fields and copy it straight into your pages.
  • Canonical URL CheckerCompare a page URL with its canonical URL and flag common canonicalization problems, entirely in your browser.
  • Hreflang GeneratorGenerate hreflang link elements for locale and region combinations from a base URL and URL pattern — entirely in your browser.
  • JSON-LD Validator HelperValidate JSON-LD syntax and flag common structural omissions in your structured data — entirely in your browser.
More SEO & Performance tools