Can You Use Robots.txt to Influence Which Part of Your Site Gets Crawled?
Please watch and listen to Matt Cutts response and suggestions. Robots.txt should not be used to try and optimize or manipulate google’s crawl.
Please watch and listen to Matt Cutts response and suggestions. Robots.txt should not be used to try and optimize or manipulate google’s crawl.