Please help with Robots.txt

[dadju]

Hi GuysHope you can help.I have just registered to use the google webmaster tools.Under Robots.txt analysis (Analysis of cached robots.txt)robots.txt URL http://www.alpine-connection.com/robots.txt Last downloaded August 18, 2007 11:12:44 PM PDT Status 404 (Not found) [?] Is Status 404 a problem. I have nothing in my HTML code about robots should i place some code and if so what should it sayRegards PaoloHi,Add below to your site<meta name="robots" content="index,follow" />Hi Many thanks for the info Regards PaoloHi, If you want to exclude certain pages being indexed by google then you should put <meta name="robots" content="noindex,nofollow" />Its always better to put one robots.txt and controll the crawl of the search engines. thanks,batu544,http://www.justclick2go.comHi ThereWhat do you mean that it better to controll the search of the engines Regards PaoloYou should do a quick google search for robots.txt because it is a text file that you place in the main directory of your webserver, same place as your default.htm fileIt is used for controling the spiders that crawl your site so that you can stop crawlers accessing information they shouldnt and also to help guide the spiders around your site, especially showing spiders not from yahoo or google where to find your sitemap.xml file.put this in the meta tags<meta name="robots" content="index,follow" />then in your webserver directory make a file called robots.txt and put something like this inside itUser-agent: *Sitemap: http://www.yourdomain.com/sitemap.xmlThis tells spiders where to find your sitemap file, you can also allow / disallow certain directories like soDisallow: /PDF/But do a search and see how it works, you can specify different rules for different spider bots.I cant find the link in may favourites, but a google search will show you, there is this site that will automatically check your robots.txt file and make sure that you dont have any errors in it. make sure you get the one that can recognise the sitemap.xml command.also, that google error that you have is because you havnt made a robots.txt file its telling you that it cant find one, error 404 means doesnt exist.hope this helps.Hi Guys Thanks for all your helpI placed the robots meta tag about 6 days ago,but i have just been to the google webmasters tools and it still showing 404 not found ( Last caculated 20 August) Is this still a problem .Can some one check to make sure i have placed the robots in the right place http://www.alpine-connection.comRegards PaoloSteven D wrote:Hi SteveThanks for your advice , is this straight forward to do to set up a Robots.txt.fileIf you can point me to the right direction or point me to a website.Having a 404 error does it stop you from getting rank higher in the search enginesRegards Paolono i doubt it will stop you, your PR and SERP is determined by quality of information and incoming links and a few others, I think what happens is that robots check it to see if they are not allowed in certain directories, and if there isnt a robots.txt file they prob just go hard.Just make a text file in your root directory, call it robots.txt and slap this in itUser-agent: *Sitemap: http://www.yourdomainname.com/sitemap.xmlAllow: /This will let every bot search every page and fdirectory of your site, this means that your images may appear in search engines, same with your flash objects and PDF files.If there is a directory or directories you dont want to be searched just swap the allow line to Disallow: /pics/orDisallow: /pics/Disallow: /PDF/What ever you need to block.then go herehttp://tool.motoricerca.info/robots-checker.phtmland it will check if your file is up to scratchHi Steve Thanks for the info Regards Paolonp budy, just like to try help and get things right, since I uploaded my site about 2 weeks ago I have spent over 1 full week changing it to be more SEO compatible, I have had to do the following,Title Tag is not first 65 characters of the heading of an articleMeta Tags & Description Tags are now stored in the database for each articleRobots.txt filePut all my JS code into seperate files and call when neededRemove heaps of my tables, I had so many that it couldn't find parts of my siteAdded a huge blurb at the bottom of important pages that explains what we do and the areas we help, so will get better SERPHave listed in like 250 web directories, only 15 had accepted my link so farHave uploaded 1 article to link 30 places, accepted in like 5 places so farRegistered for like 3 sets of forums and posted fanatically, but helpful *plum* so my posts dont get deleted.Help some other guy set up some forums in return for featured linkand the list goes on and on, and I reakn I have like another 5 months of doing this before I get even close to a good ranking