SpiderMonkey SRobots Text Generator and Fetcherrobots.txt Fetcher and Generator

User Agent:* Recommended setting.
Disallow:* Recommended setting.
Help:
Fetch Your robots.txt File
Below is a resource for fetching your existing robots.txt file or checking that you have one.
Create a robots.txt file
This resource will generate a "robots.txt" file for you to place in the root ("/") directory of your web site. To the right you will type the directory names --virtual or real-- you wish to forbid to robots.
Format is "/directory/". You can also enter a specific page like "mysecretpage.htm"
Why robots.txt? Your web pages should all contain meta tags (<meta name="robots" content="[no]follow, [no]index">) to instruct web crawlers. Objects like images or java class files cannot be so encoded. The first thing a web crawler looks for when it reaches your site is the robots.txt file. If it doesn't find it, a "404 Not Found" error is generated and logged. What waste. Next it seeks its own name within the file. Thirdly, it reads the robots.txt entires under its name to see where it can go. If there is no robots.txt file in your root directory (/) then it will a) simply abort or b) crawl everything, including your secret files, your images, your "worker" scripts and so on. More details on robots.txt; how robots work; writing meta tags...
Allow full access Allow some robots full access? Enter their "User Agent" below.
User Agent examples: SpiderMonkey, Googlebot, Slurp, ZyBorg
 

Valid XHTML 1.1!