What exactly is a Robot?
A Robot in Search engine optimization services terminology
is a search engine software program which visits a
page on a website and follows all the links of the
website from that page and indexes some or all of
the pages in the website.
Why do we need robots?
Everyday search engines receive hundreds of new website
submissions. It is quite cumbersome and time consuming
for a human to review the whole of the website and
judge whether that particular website meets the search
engine optimization standards and index the same.
Here is where our friend robot comes into picture.
Robots are highly intelligent software programs which
crawl the entire website, checking the relevancy,
consistency and significance of the site thereby effectively
indexing the website into the search engine database
reducing the amount of time consumed per site. In
this way a robot can quickly index more sites per
day. Though a robot is not a very critical aspect
of search engine optimization services technology,
it is advisable to include it.
Controlling a Robot
Normally, a robot visits the home page of the site
and follows the links present in the page, scanning
each link and page. Sometimes we do not prefer a robot
to index a particular page(s) in our site. For instance,
you might want a series of pages to be viewed in sequence
and would like to index only the page one. To achieve
this, we have a special kind of Meta tag known as
Robots Meta tag. The robots Meta tag is similar to
other Meta tags and is placed in the head of the document.
This tag dictates the robot which pages to be indexed,
which pages not be indexed, which links should be
followed and which links should not be followed.
A typical meta robots tag would resemble as follows,
<meta name="robots" content ="index,
The meta tag describes us that the robot should index
the page visited but to not follow the links in the
The other most and significant part of controlling
robots is the robots.txt file. The robots.txt file
is used primarily to control areas or portions of
the website by excluding those portions being visited
by the robot. Whenever a robot visits a site, it first
checks the robots.txt file.
The robots.txt file,
Finally, we can conclude that the robots.txt file
basically acts as a filter thereby providing total
control over the search engine robot.
While talking of robots I would like to mention the
revisit tag. This tag is an important tag in the realm
of search engine optimization services technology.
The revisit tag tells the search engine the time duration
after which it should visit your site again. If you
change your site's contents frequently then your revisit
time should be say a week else it can be higher. Your
search engine rankings dip if the search engine visits
you a second time and finds that the content has not
been altered significantly. Though not all search
engines honor the Revisit tag, it is advisable to
include the tag. If you are keen on top search engine
rankings use the revisit tag judiciously.