We Counted the Log, Not the Crawler

On 6 August we published that one of Google’s crawlers had requested 381 pages here in a ten minute window, and that every one of them was a filter page. We used that number to explain why we had just written new rules for it. The number was wrong, and the mistake is an ordinary one.

Our host writes a separate line in its log each time a request passes through a layer of the server, and again when a client retries under a different version of the web protocol. So one visit can appear two or three times. We counted the lines. Counted as requests, that window holds 146, across 127 different addresses.

Still every one a filter page, still roughly one every four to six seconds. The rules we wrote were the right rules. The number we gave for why is 2.6 times too high, and we published it.

The rules worked. In a 22 minute window on 8 August, after they went live, that crawler made no requests at all. Same crawler, same site, the same filter pages still there to fetch. We waited for a window wide enough to say that honestly, because a few seconds of quiet is not proof of anything.

What we are not claiming: the filter pages are still being fetched heavily, just not by that crawler, and not by anything that identifies itself as a crawler at all. That is a different problem, and it is ours to solve rather than something a rule file can ask nicely about.