SEARCH ENGINE WIKI

    ROBOTS META TAG DEFINITION

ROBOTS META TAG DEFINITION

The Robots Tag - or RobotsMetaTag - allows a page designer or editor to tell search engines whether or not the page and/or its links should be indexed and/or followed by the visiting search engine spider (see SearchEngineRobot).

While a robots.txt file is often used in the root directory of the site to tell search engine robots what pages and links to index, sometimes webmasters will set this information at the page level instead, using the robots META tag. A RobotsMetaTag is not generally required if a robots.txt file is used.

The robots META tag has the following options:

This tag means the same as "noindex, nofollow". Robots are told not to index this page and not to follow the links on this page.

This tag means the same as "index, follow". Robots are told that they can index this page and follow any of the links on this page. This tag is actually unnecessary, as search engines assume that they can index and follow the links anyway, without an express command in the robots META tag to do otherwise.

This tag tells search engines not to index this page BUT that they can follow the links. Why? Both "index" and "follow" are default setting for the robots META tag, so if your tag doesn't say otherwise, search engines will assume that they can index and follow the links on a given page. In this instance, your robots META tag has told engines that they couldn't index the page but not that they couldn't follow the links, so it is assumed that they are permitted to do so.

This tag tells search engines not to follow the links on this page but (for the reasons explained just preceeding) that they can index the page.

This tag is unnecessary, as index and follow are the default actions for search engine robots.

This tag is unnecessary, as index and follow are the default actions for search engine robots.

Same is "none".

Same as "all" and, strictly speaking, unnecessary.

Categories
CategoryGlossary

PageStatus: OpenPage