Twitter's robots.txt, It shows everything is disallowed, but surprisingly search engines are crawling and indexing everybody's profiles pages, Why? This is how Google handles the robots.txt file: Controlling crawling and indexing now documented on code.google.com robot.txt just a guide . but actually google will index what they want . so dont confused about it
