The Axandra newsletter archive

14 January 2003

This week we're telling you what a robots.txt file is and why you should use it on your Web site so that search engines can correctly index your site.

Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file.

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

The robots.txt file is a simple text file (it contains no HTML code). It must be placed in your root directory, for example "http://www.your-web-site.com/robots.txt".

How do I create a robots.txt file?

The robots.txt file is a simple text file. Open a simple text editor to create it. The content of a robots.txt file consists of so-called "records".

A record contains the information for a special search engine. Each record consists of two fields: the user agent line and one or more Disallow lines. Here's an example:

User-agent: googlebot
Disallow: /cgi-bin/

This robots.txt file would allow the "googlebot", which is the spider software program of Google, to retrieve every page from your site except for the files from the "cgi-bin" directory. All files in the "cgi-bin" directory will be ignored by googlebot.

The Disallow command works like a wild card. If you enter

User-agent: googlebot
Disallow: /logs

both "/logs.html" and "/logs/index.html" as well as all other files in the "logs" directory would not be indexed by search engines.

If you leave the Disallow line blank, you're telling the search engine that all files may be indexed. In any case, you must enter a Disallow line for every User-agent record.

There's much more to know about robots.txt files than we have space for in this newsletter. For that reason, we've created a special article about this topic on our Web site. Among other things, that article includes these topics:

• Where to find user-agent names
• 7 things you should avoid when designing your robots.txt
• Tips and tricks for a good robots.txt file
• Examples for easy and complex robots.txt files

http://www.Axandra.com/news/all-about-robots-txt.htm

* Google has replied to SearchKing lawsuit

As you might remember, the small online advertising network company SearchKing filed a lawsuit against Google last October.

SearchKing claimed that Google had lowered the PageRank ratings for the SearchKing Web pages. In addition to asking for damages, SearchKing filed a motion with its complaint asking for a preliminary injunction to force Google to restore its PageRank ratings.

Now Google has replied to the complaint. Google claims that it has done nothing wrong and asks the court not to grant an injunction. In addition, Google has filed its own motion to dismiss the lawsuit.

* AskJeeves has dropped banner ads

The search engine company Ask Jeeves said that it has stopped using banner advertising on their Web site. They would annoy users and they'd be unappealing to advertisers.

Ask Jeeves, which also operates the search engine Teoma, plans to use text-based ads that appear only when a user searches for a specific search term (similar to Google's AdWords program).

* New Google search site country

Google has added a new country to its set of search sites: Peru.

* Just released: ARELIS 3.7 with email filter check

Among featuring many user suggestions, ARELIS 3.7 now offers a built in email filter check. This will help webmasters not to unintentionally send email messages that trigger spam filters by mistake.

By the way, ARELIS cannot be used to send mass emails and it does not create a list of email addresses.

Other new features include an improved database management and a better email client. All reported bugs have been fixed.

Registered users can download the trial version and install it over their current version. It will be automatically registered.

To download the trial version of ARELIS or to see a list of changes, click here.

* Increase your link popularity eBook

We have just finished a new PDF book called "Increase your link popularity and get targeted traffic". It has more than 50 pages and is full of tips and tricks about link popularity and getting targeted traffic to your site.

Best of all, it doesn't cost you anything. Download it here.

* Web sites we recommend

• Take care of filter words and stop words
  
  
• Happy New Year
  
  
• Yahoo to acquire Inktomi
  
  
• Google Froogle - Froogle Google
  
  
• How to write keyword rich Web pages

The Search Engine Facts newsletter is free. Please recommend it to someone you know.

You may publish one of the articles above on your Web site. However, you must not change the contents in any way. Also, you must keep all links and you must add the following two lines with a link to www.Axandra.com: "Copyright by Axandra.com. Internet marketing and search engine ranking software ."

Back issues:
http://www.Axandra.com/news

Copyright 2002-2003 Axandra / Voget Selbach Enterprises GmbH