robot help: SpiderMonkey robots.txt Fetcher It is important that every web site have a robots.txt file in the root directory to avoid the numerous 404 errors and to make the site more "robot-friendly". To manage a robot's crawl of your site, you can use this simple file (robots.txt) in the top-level domain (i.e.: www.mobrien.com/robots.txt) as an adjunct to properly written META tags.
|# EXAMPLE robots.txt|
User-agent: *# Enter specific user-agent but "*" is best.
|# EXAMPLE robots.txt to exclude a single robot|