W Wrythm New Member Apr 25, 2012 #1 I have a one site which is online but it is in construction phase, so I add robots.txt file to restrict crawlers for whole site. Is it OK to add robot file when your site is under construction and allow crawling when site is full functional?
I have a one site which is online but it is in construction phase, so I add robots.txt file to restrict crawlers for whole site. Is it OK to add robot file when your site is under construction and allow crawling when site is full functional?
