WaftWermrit
New Member
Ok, I've heard good and bad things about having a robots.txt file on your server. Is there a benefit to having this file, or does it hinder the google bots from searching your site. Does anyone know?It should only hinder robots if you tell it to... and it could potentially be bad if you don't have one. For instance, if your server automatically redirects you to another (html) page on a 404 error, the robot will get that as well when it looks for the robot.txt file, which can't really be good...The forums part of my website has a robots.txt file (it's phpBB), but the main part of my website does not have one. You think I should add one?the first thing a search spider looks for when visiting your site is the robots.txt file. It's worth having one even if its just to tell the spider that it can access all your files (if you dont want to block anything)Dubya wrote:May I recommend this great tutorial from Ian McAnerin? It's a great guide to robots.txtCheers,Johann