Robot Report

The robot report is new in ClickTracks Optimizer and Professional, version 5.5. It's not available in Analyzer. Note also that the report is not available with JavaScript data sources (ClickTracks Hosted and JDC) since this method of data collection does not track robots.

The robot report shows how frequently search engine robots are visiting your site. Search engines use automatic software known as robots, spiders, crawlers or just bots, to crawl millions of pages on the web. In so doing they extract the text and build the index of pages and search keywords. The first step in getting your site well ranked in the engines for the keywords that matter is to get the search robots to crawl your site, and from then on you'll want to pay close attention to which pages and how frequently the robots visit. The robot report shows this is single view that is quick and intuitive.

The report shows a list of pages on your site and adjacent to this a column for each search engine robot. The value in each cell is a specific metric for that page or a total for the engine in the column header. Note that many sites share a robot so you won't see all the individual search engines listed, but instead the robot name. For example Ask Jeeves and Teoma use the same robot but are distinct search engines when viewed in the search engine report.

The pages are sorted top-to-bottom in the order of popularity of visits by humans. In general you will be most concerned with robots indexing the pages that are most frequently seen by people visiting the site, since those are the pages where you will focus your efforts in creating compelling text and page layout. A special case is made for robots.txt which is not normally seen by human visitors. This file is very important for SEO work and is therefore displayed at the top of the list.

The metrics in the report are:

Recency

       Number of days elapsed since robot visited. Most recent pages are darker blue, pages visited weeks ago are lighter blue while those visited never are the lightest

       Actual date of last robot visit.

Frequency

       Number of times in the selected time interval the robots have visited (see below)

       How frequently robots visit each page.

Date range

The top of the report shows the date range under analysis for robots. This range is fixed to the most recent 10 weeks from the end date of the date range selected, or the range of dates imported, whichever is sooner.

Page errors

Note that ClickTracks does not show pages with 404 errors in the robot report