Why Server Logs Are Important Than Traffic Counters

This article tries to explain why use server log files for traffic report instead of popular statistics counters. While everyone will agree that traffic report, visitor information and their query is extremely important for websites to run. For example what exactly users are looking for can be found more precisely. In case if you use service like overture and wordtracker then they may not display result specific to your site.

Statistics counters

Counter seems to be easy to use just create an account copy and paste the JavaScript into your pages and it’s done. You are ready to get users demographic and system information easily. But the down side is most of the time there is lack of detailed page access information by search engine bots and human. I was using a popular counter that wasn’t showing much about search engine crawling also using Google’s webmaster account it appears to be Google was visiting my site after 3~4 days. Which was inaccurate as the crawling frequency of search engines are so high to drive crazy the web server.

Benefits of server log files

Now look at the server log files which are bit difficult to understand. But once when get used to it it will definitely change the way you look at the search engines. Server log files are generated and updated when somebody tries to access the web pages from server. They are easy to use as well, just by FTP account you can download all the data starting from the beginning. The last lines of this document explain how to read log files.

When I was not using log files and completely dependent on counter for my traffic reports. I had impression that search engines are reluctant to crawl sites regularly and also my traffic bandwidth limit would be enough for my visitors. Now after looking at log files I was shocked that search engines are daily coming and downloading the files many. times per day. Also my agony with certain search engines were gone as they visiting my pages daily though they don’t index them.

You may argue that why can’t I use counter and log files together. Well to use counter on each pages you need to put a back link of counter service provider. Remember search engine love pages with enough content and comparatively smaller number of outbound links. By adding each new link in your pages you are decreasing the value of your partners back links kept in your pages. This is obvious when you understand the basic Page rank calculation formula.

The following lines explains how to read log files

203.145.159.44 - - [30/May/2007:03:37:24 -0400] "GET /indexhtml HTTP/1.1" 404 2294 example.com "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1) Gecko/20061010 Firefox/2.0" "-"

The above lines represent the single file access log here;

(203.145.159.44) - IP Address of visitor that may be human or search engine bot.
(- -) – The first dash will be replaced by login name and second dash will be replaced by full name of visitors but because of security reasons most of the ISP will hide it.
[30/May/2007:03:37:24 -0400] - Is obvious it’s the visit time and date the, last parameter is the GMT offset.
GET – Command for http request.
/indexhtml – File requested.
HTTP/1.1 - Protocol here http
404 – Status code returned by server.
2294 – File size in bytes.
Example.com – File residing under the domain.
Mozilla/5.0 – Name of browser or search engine bots for example for Google it is GoogleBot/1.2
Other parameters are system information.