Please visit our new web site

Search Engine Optimization Services

Elements of Search Engine


Requesting a Quote on a Project


seedot.com is a web design and web development studio Bulgaria, website design company Bulgaria, flash web design, graphic design company in Bulgaria, flash website design studio Bulgaria, flash web design, and graphic design company in Bulgaria for utmost Internet success. An Multimedia Agency specialising in Internet Services, Web Design, CD Rom, Video and Presentations based in Sofia Bulgaria. We combine a broad range of services including business management, marketing, graphic design and IT in order to provide you with a professional and personal service, Web development and design,asp, php and delphi projects, download programs, info and resume, professional sites and programing


seedot.com » Search Engine Optimization Services » Handling Robots


Handling Robots

What exactly is a Robot?

A Robot in Search engine optimization services terminology is a search engine software program which visits a page on a website and follows all the links of the website from that page and indexes some or all of the pages in the website.

Why do we need robots?

Everyday search engines receive hundreds of new website submissions. It is quite cumbersome and time consuming for a human to review the whole of the website and judge whether that particular website meets the search engine optimization standards and index the same. Here is where our friend robot comes into picture. Robots are highly intelligent software programs which crawl the entire website, checking the relevancy, consistency and significance of the site thereby effectively indexing the website into the search engine database reducing the amount of time consumed per site. In this way a robot can quickly index more sites per day. Though a robot is not a very critical aspect of search engine optimization services technology, it is advisable to include it.

Controlling a Robot

Normally, a robot visits the home page of the site and follows the links present in the page, scanning each link and page. Sometimes we do not prefer a robot to index a particular page(s) in our site. For instance, you might want a series of pages to be viewed in sequence and would like to index only the page one. To achieve this, we have a special kind of Meta tag known as Robots Meta tag. The robots Meta tag is similar to other Meta tags and is placed in the head of the document. This tag dictates the robot which pages to be indexed, which pages not be indexed, which links should be followed and which links should not be followed.

A typical meta robots tag would resemble as follows,

<meta name="robots" content ="index, nofollow">

The meta tag describes us that the robot should index the page visited but to not follow the links in the page.

The other most and significant part of controlling robots is the robots.txt file. The robots.txt file is used primarily to control areas or portions of the website by excluding those portions being visited by the robot. Whenever a robot visits a site, it first checks the robots.txt file.

The robots.txt file,

  • is a text file created in notepad or any text editor
  • should be placed in the top level directory or root of the website or server space
  • should include all lower case letters

Through the robots.txt file we can,

  • Exclude all robots from visiting the server
  • Allow complete access to all robots
  • Exclude robots from accessing a portion of the server
  • Exclude a specific robot
  • Exclude certain type of files from accessing by specifying the file extensions.

Finally, we can conclude that the robots.txt file basically acts as a filter thereby providing total control over the search engine robot.

While talking of robots I would like to mention the revisit tag. This tag is an important tag in the realm of search engine optimization services technology. The revisit tag tells the search engine the time duration after which it should visit your site again. If you change your site's contents frequently then your revisit time should be say a week else it can be higher. Your search engine rankings dip if the search engine visits you a second time and finds that the content has not been altered significantly. Though not all search engines honor the Revisit tag, it is advisable to include the tag. If you are keen on top search engine rankings use the revisit tag judiciously.


Seedot.com web design, web development, b2c, b2b web system

Profile :: Careers :: FAQ :: Partners :: What We Do :: How We Do It
Awards :: Modules :: Technologies :: Hosting :: Corporative Design
Brand Development :: Digital Solutions :: Administration :: Search engine
Site Map :: Offshore :: Client List :: Case Studies :: Development Process
Pricing Reference
:: Contact :: News :: Requesting a Quote on a Project

Bulgarian :: English language :: Deutsch

Add this page to your favorites! | Print this page |
Make seedot.com Your Homepage!

Copyright 1997-2003 Seedot.com, Sofia, Bulgaria.
All Rights Reserved. e-mail:
Last update: