Meta Robots

When the crawler crawls our website directly first the crawler read the robots.txt information then exclude the Disallowed pages. But still the disallowed page will be crawled if an external link comes from the other website to that page. In such cases to avoid indexing of the disallowed pages in the searche engines we use meta robots in the head section of that webpage.

We can this meta robots directly instead of writing the robots.txt file. This will give the exact result as we want.

meta robots tag is written the head section of a webpage.

Meta  Noindex:

The syntax for writing meta robots is:

<meta name="robots" content="noindex" />

This guide the robots not to index the page. This is very important for banking accounts, social networking sites like orkut etc. Disallowing a page through Robots.txt will not make secure but by adding meta noindex makes a page secure from crawling.

To index the page. we write meta content as "index". This tell the robot to index this page.

<meta name="robots" content="index" />

This is also a good way of guiding the search engine robots to index the page. It is suggested to use meta index if we want the page to be indexed by search engines.

Meta  Noindex & Nofollow:

The syntax for writing meta noindex,nofollow

<meta name="robots" content="noindex,nofollow" />

The syntax for writing meta index,follow

<meta name="robots" content="index,follow" />