Understanding the Importance of the robots.txt file