I read robots.txt is used to allow , or disallow crawler to particular page....so i dont add robot.txt file to my site .
robots.txt file is important for all websites ? thanks in advance It is important when you have some pages which you don't want Google to crawl like payment page or login page. @angleabrow1 and born star thanks for replying ....
plz check this is correct robot.txt format ?
user-agent:googlebot
disallow:/login.php
allow:/service.php Yes it is one of important thing. Each website should add Robots.txt file. Usually it is used for allow or disallow to a particular page. yes,if you are dealing with Google or any search engine than you have to upload robots.txt on your server. It will be helpful in crawling purpose Quote: Originally Posted by seocatalysts http://www.v7n.com/forums/seo-forum/278578-robots-txt-file-important-all-websites.html#post1767778 Quote: Originally Posted by snakeair http://www.v7n.com/forums/seo-forum/278578-robots-txt-file-important-all-websites.html#post1769334
Thanks for help what is * stands for ? and then why don't use allow ?
like
allow:/php-development-service
allow:/webdesign-service
only disallow is enough ? Quote: Originally Posted by pocketknife
Hi, egsindia
The robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file. Robots.txt help in rankings??? can you please elaborate how??
Its true. Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file. here an Example:
1. How to allow all search engine spiders to index all files
Use the following content for your robots.txt file if you want to allow all search engine spiders to index all files of your Web site:
User-agent: *
Disallow:
2. How to disallow all spiders to index any file
If you don't want search engines to index any file of your Web site, use the following:
User-agent: *
Disallow: /
robots.txt file is important for all websites ? thanks in advance It is important when you have some pages which you don't want Google to crawl like payment page or login page. @angleabrow1 and born star thanks for replying ....
plz check this is correct robot.txt format ?
user-agent:googlebot
disallow:/login.php
allow:/service.php Yes it is one of important thing. Each website should add Robots.txt file. Usually it is used for allow or disallow to a particular page. yes,if you are dealing with Google or any search engine than you have to upload robots.txt on your server. It will be helpful in crawling purpose Quote: Originally Posted by seocatalysts http://www.v7n.com/forums/seo-forum/278578-robots-txt-file-important-all-websites.html#post1767778 Quote: Originally Posted by snakeair http://www.v7n.com/forums/seo-forum/278578-robots-txt-file-important-all-websites.html#post1769334
Thanks for help what is * stands for ? and then why don't use allow ?
like
allow:/php-development-service
allow:/webdesign-service
only disallow is enough ? Quote: Originally Posted by pocketknife
Hi, egsindiaThe robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file. Robots.txt help in rankings??? can you please elaborate how??
1. How to allow all search engine spiders to index all files
Use the following content for your robots.txt file if you want to allow all search engine spiders to index all files of your Web site:
User-agent: *
Disallow:
2. How to disallow all spiders to index any file
If you don't want search engines to index any file of your Web site, use the following:
User-agent: *
Disallow: /