"All you can eat scan? No, we read 256 KB of your page, and we are full"
A scanner that downloads whatever a server sends is a scanner that is one bad target away from trouble. A "favicon" that's actually a 5 MB js app file, a robots.txt that's a 500 MB log, a gzip body that expands a hundredfold -- these aren't hypotheticals, they're just unusual responses until they're pointed at an unguarded fetch.
The cap was always there, but it was just too late
I was working on a new feature for promo codes, when Boris Grozev pinged me with a question about the scan limits. He hinted on a scenario with a scanner eating up its bandwidth budget because of a stray request for something as trivial as a favicon. So I was on to investigate.
Every check that reads a response body has always had a size limit: 256 KB for the page HTML, 32 KB for discovery files like robots.txt and favicon.ico, 8 KB for batch probes. But the old implementation asked the HTTP client for the whole response and then cut it down. The cap bounded what we stored, not what we downloaded. A 10 MB favicon arrived in full and we kept 32 KB of it. What a waste, eh?
Now the cap lives in the transfer itself. The moment the buffer fills, the fetch stops consuming -- the connection closes mid-body and we keep the prefix. I tested it with a raw socket server pushing 10 MB: with a 32 KB cap, roughly 192 KB crossed the wire (the cap plus normal socket buffering), and exactly 32 KB was stored. Because compression is transparent, the limit applies to the decompressed bytes -- a gzip bomb can't sneak around it.
Doesn't that break checks on big-but-legit pages?
Less than you'd think, because of where the information lives:
- SEO checks read
<title>, meta tags, canonical, hreflang, Open Graph -- all of which must sit in<head>, the first few KB of any conforming page. A page can be a megabyte of body content and every head-derived check sees all of it. - Favicon detection only needs the file's magic bytes -- PNG/GIF/WebP/SVG signatures all sit at byte zero. A 33 KB icon and a 33 MB icon look identical to the check.
- robots.txt larger than 32 KB is vanishingly rare, and sitemaps report "N URLs, complete" vs "at least N" -- the format itself tells us when we got the whole file.
- Status, headers, timing, TLS (the majority of every scan) never read a body at all. They're untouched.
The median real webpage is well under 100 KB of HTML. The cap only bites on genuinely huge pages, and even then the first 256 KB covers the head and most of the body.
A prefix is not a document and the cards admit it
Here's the part that matters most: a truncated body can produce false confidence if a check interprets "didn't find it" as "isn't there". An insecure http:// script in the unread tail of a page would hide behind a "no mixed content found" verdict.
So every fetch result now carries a flag: this body is a prefix, not the whole document. Presence findings are still solid -- if we found it, it's there. But absence-based verdicts now qualify themselves:
- Mixed content and SRI cards that find nothing add (large page -- only the first 256 KB was analyzed).
- A truncated
robots.txtmarks the file as partially read -- the "indexable" leg notes it instead of claiming a clean full-file pass. - Sitemap counts already distinguished "N URLs" from "at least N".
Well, we can't fix the whole world, now can we?
Remote scans from the worker bots get the same honesty: when a scanner node ships a body that exactly fills its byte budget, the result is treated as capped there too. Local and remote tell the same truth.
Why bother?
Two reasons, really. Politeness and safety on our side -- we don't want our crawler pulling megabytes per check on thousands of domains, or a hostile target inflating our memory and bandwidth.
And correctness -- a scan that can't prove a negative shouldn't claim one. "Nothing found in the first 256 KB" is a smaller, truer sentence -- and we'd rather print the smaller, truer sentence, than risk misleading you.