Power Sellers Unite

Earthfragments.com · Posted: Fri Feb 10, 2006 6:04 pm Post subject: robots.txt

Can anyone explain to me in plain english how I can create and implement this file? Since my domain was stolen by yugoslavic porn spammers, I've been using frames for my website, and need all the SEO help I can get now :roll:

psunite · Joined: 27 Jan 2005 Posts: 6213 Location: Columbus OH

First off I think frames isn't really good for SEO

The robots.txt file doesn't help you to get SEO it actually is used to prevent search engine spiders not to crawl or index your site.

Example

User-agent: googlebot
Disallow: /

will prevent googlebot from indexing your whole site.

User-agent: *
Disallow: /

will prevent all bots from indexiing your whole site.

You don't want either but if you have information on your site that you don't want to get index and say its in the directory called private you would use

User-agent: *
Disallow: /private

Some examples of directories you might want to block are

Disallow: /cgi-bin/
Disallow: /_borders/
Disallow: /_derived/
Disallow: /_fpclass/
Disallow: /_overlay/
Disallow: /_private/
Disallow: /_themes/
Disallow: /_vti_bin/
Disallow: /_vti_cnf/
Disallow: /_vti_log/
Disallow: /_vti_map/
Disallow: /_vti_pvt/
Disallow: /_vti_txt/