August 8, 2026
The Crawler The Rules Never Named
Every gift shelf on this site can be narrowed by occasion, relationship, price, category and vibe. Because those filters combine, the number of possible filtered pages has no practical end, so our robots file has long carried rules telling automated visitors to read the twelve hundred or so real pages and skip the endless combinations.
The rules worked. We had been checking them the wrong way around. Google’s search crawler obeys them provably: its own reporting shows thousands of filter addresses blocked exactly as instructed, and an inspection of one such address came back never crawled. Every dashboard we normally read said the protection was doing its job.
Then we opened the server’s raw access log instead of the dashboards. In one ten minute window on 6 August 2026, a different Google crawler, one that fetches for Google products other than Search, requested 381 pages here. All 381 were filter pages, the exact addresses the rules forbid. It had been at this for weeks, a request every few seconds, hundreds of thousands of fetches against a site with about twelve hundred real pages.
Correction, August 8, 2026
The figure of 381 above counts lines in a server log, not requests. One visit can be logged more than once: the log gains a line at each server layer a request crosses, and another if the client falls back to an older protocol. Counted properly, that ten minute window holds 146 requests from 127 addresses. What follows from it is unchanged: all of them were filter pages, and the rules written in response were the right rules.
The measurement this entry promises below has since been made. On 8 August, with the new rules live, we watched for 22 minutes and that crawler fetched nothing. That result and the arithmetic behind the corrected figure are published as the corrected count and the follow-up measurement.
The explanation is small and unforgiving. Rules in a robots file bind the visitor they name, and a visitor named nowhere falls through to the catch-all section. This crawler carries a different name from the search crawler, and the log shows it fetching exactly what our catch-all forbids while its sibling touched none of it. A rule can be correct, obeyed by the crawler you watch, and still miss the crawler doing the damage.
We had even chased this symptom before. An earlier version of the file saw the same leak in the crawl reports and added the blocking rules to the image crawler’s section, a guess at which visitor was responsible. The guess was wrong, the leak continued, and the fix measured as nothing. The real name was sitting in the access log the entire time, and the log had never been opened.
The file now names that crawler and its image and video siblings directly and hands each one the full set of filter blocks. While we were in it we also cleaned up what search tooling had been flagging for a while: six pointer lines written as if they were directives, which no robots standard defines, are now plain comments carrying the same pointers, and a rate limit addressed to a crawler that ignores rate limits by policy is gone.
This entry is written before we know whether the fix worked. The earlier attempt looked exactly like a fix and changed nothing, so this one does not get to claim success just for shipping. Once the change is live we read the access log again. The filter requests from that crawler either fall to zero, or this entry gets a correction naming what we got wrong.