smithmary01 Posted January 14, 2020 Share Posted January 14, 2020 What is the main roll of robots.txt file? Quote Link to comment Share on other sites More sharing options...
linneasandel Posted January 24, 2020 Share Posted January 24, 2020 Robots. txt file is also known as the robots exclusion protocol which is a text file and tells web robots which pages on your site to crawl. It is very important to have as it tells them to not crawl the parts of your site that aren't displayed to the public. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.