Robots & Spiders

Many of the requests to your web server are from 'robots' or 'spiders'. These are programs that browse the web automatically, for example to compile indexes for search engines.

Although these requests are important for your site's visibility on search engines, they're not relevant for visitor behavior reports, and so ClickTracks automatically removes sessions that appear to be generated from robots and spiders from these reports and isolates them into a single special robot report. It does this in two ways:

If the session begins by requesting 'robots.txt' this all subsequent pages from this session removed from the behavior reports.

If the user agent contains any string from the file 'UserAgentsToIgnore' (using partial matching) the session is removed from the behavior reports.

Finally a visitor session that repeatedly requests the same pattern of pages is removed from the behavior reports.

The robot report (available in ClickTracks Optimizer and Pro only) details robot reports from certain specific search engines. See Robot Report for details.