Robots Tag, nofollow and robots.txt
Contents
The robots tag allows you to specify that a particular page should NOT be indexed by a search engine. The specific tag is:
This should be used if you have a page that you do not want search engines to include in search results. Website platforms such as WordPress, Wix, Shopify, SquareSpace, Weebly and others have options in the admin to keep pages out of search engine indexes, and when these are selected the noindex tag is included in the source code.
Nofollow links tells search engines that it is not a natural link and not to use it as a criteria for SEO. This is most often used on banner ads and other paid links. In the link code, a rel=”nofollow” HTML tag is added.
A robots.txt file can be added which provides search engine crawling spiders with specific instructions about which sections they can crawl or not crawl. These crawl instructions are specified by “disallowing” or “allowing” the behaviour of certain (or all) search spiders. See more about robots.txt in this article on moz.com