smithmary01 Posted January 14, 2020 Posted January 14, 2020 What is the main roll of robots.txt file? Quote
linneasandel Posted January 24, 2020 Posted January 24, 2020 Robots. txt file is also known as the robots exclusion protocol which is a text file and tells web robots which pages on your site to crawl. It is very important to have as it tells them to not crawl the parts of your site that aren't displayed to the public. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.