-SpongeBobb-
New Member
Eh, i read this last night and went to bed so did not have a chance to look at my robots.txt file to see how messed up it is. This blog post is from the SEOmoz blog.
Quote: Some of the Internet's most important pages from many of the most linked-to domains, are blocked by a robots.txt file. Does your website misuse the robots.txt file, too? Find out how search engines really treat robots.txt blocked files, entertain yourself with a few seriously flawed implementation examples and learn how to avoid the same mistakes yourself.
The robots.txt protocol was established in 1994 as a way for webmasters to indicate which pages and directories should not be accessed by bots. To this day, respectable bots adhere to the entries in the file... but only to a point.
Your Pages Could Still Show Up in the SERPs
Bots that follow the instructions of the robots.txt file,
Quote: Some of the Internet's most important pages from many of the most linked-to domains, are blocked by a robots.txt file. Does your website misuse the robots.txt file, too? Find out how search engines really treat robots.txt blocked files, entertain yourself with a few seriously flawed implementation examples and learn how to avoid the same mistakes yourself.
The robots.txt protocol was established in 1994 as a way for webmasters to indicate which pages and directories should not be accessed by bots. To this day, respectable bots adhere to the entries in the file... but only to a point.
Your Pages Could Still Show Up in the SERPs
Bots that follow the instructions of the robots.txt file,