How The Robot Report is Calculated

Robots are detected from the logfile based on their user agent, not the IP. This works well for the large, public search engine spiders. The list of the user agent strings is maintained in the text file SearchEngineRobots.txt within the data directory under the installed location.

The concept of session and session timeout parameters are not used with robots because a single crawl from Google might take several hours to crawl the site. The start of the session is measured from the request for robots.txt instead.