You can create the 'robots.txt' file in your root directory. Copy above text and paste into text file.
How Does Robots.txt Generator?
Robots.txt Generator generates a file that is very opposed to the sitemap that indicates the pages to be included, so the robots.txt syntax is of great importance to any website. Whenever a search engine crawls any site, it always looks for the robots.txt file that is located at the root level of the domain. When identified, the crawler will read the file and then identify the files and directories that can be blocked.
Why is it important to use our Robots.txt generator tool?
It is a very useful tool that has made many webmasters' lives easier by helping them make their Googlebot sites friendly. It is a robot.txt file generator tool that can generate the necessary file by executing the difficult task in no time and by absolutely free. Our tool comes with a friendly interface that offers the options to include or exclude things in the robots.txt file.
Using our Robots.txt Generator tool:
Using our amazing tool, you can generate the robots.txt file on your site by following these simple and easy steps:
By default, all robots are allowed to access files on your website, you can choose which robots want to allow or deny access.
Choose tracking delay which indicates which delay should be traces, allowing you to choose between the preferred delay time of 5 to 120 seconds. It is defined as "without delay" by default.
If you already have a site map for your website, you can paste it into the text box. On the other hand, you can leave the field empty if you do not.
Search the list of robots is given, you can select those you want to crawl your site and you can deny robots that you do not want to scan your files.
The last step is to limit the directories. The path must contain a "/" bar as the path is relative to the root.
In the end, when you have finished the Googlebot friendly robots.txt file generation with the help of our robots txt tool generator, you can download to the root directory of the site.