TECH FLOW Svět Androida
← Back to the stream
patronview.com · picked by Petr Mišák · 57d ago

Web owner battles bot invasion: 99% of traffic are automated crawlers

Source preview: Web owner battles bot invasion: 99% of traffic are automated crawlers
AI summary

A website owner running a database of philanthropists found that 99% of traffic comes from bots. After battling scrapers for a year, including a massive wave from China that generated 3.6 million requests in a single day, he documented his defense strategies against both general web scrapers and AI crawlers. AI bots from companies like Anthropic and Amazon consumed tens of thousands of pages per day without sending any actual visitors.

The summary is written by AI from the source; it isn’t the newsroom’s opinion. For details, read the source.
Read source

4 people have already opened the source

Tip author’s note

Cloudflare recently reported that bots have exceeded human activity online, and AI is accelerating this trend further. If someone runs a website where traffic is predominantly generated by entities—organic or synthetic—that consume resources without providing value, how long can such a site realistically operate? The internet is fundamentally changing, so let's hope it evolves in a direction we actually want.

AI questions & answers
How can website administrators distinguish between bot and human traffic?

Bots typically lack referrer information, visit only one page with high bounce rates, and don't execute JavaScript. Administrators should monitor raw server logs since JavaScript-based analytics only capture human visitors. Cloudflare provides bot-specific traffic insights and enables edge-level filtering rules.

What are typical page crawl-to-visitor ratios for different crawlers?

Quality search engines like Googlebot maintain ratios around 46 pages crawled per visitor referred. AI training crawlers are much worse: Claude's crawler achieved 35,000:1 and Amazon's crawler sent no visitors at all. Bing sits at 406:1, worse than Google but far better than AI-focused crawlers.

Are geographic blocks an effective solution against bot traffic?

Geographic blocks can help against mass waves from specific countries like China, but they're not foolproof. Scrapers adapt and can route through other regions, such as AWS datacenters in America. Combining geographic blocks with other measures like CAPTCHA, referrer checking, and user-agent filtering is more effective.

Should website owners block AI training crawlers entirely?

Polite AI companies respect HTTP 403 responses, making simple firewall rules effective. Since many AI crawlers consume significant bandwidth while sending zero traffic or attribution, blocking them is reasonable. The key metric is pages crawled per visitor referred—AI crawlers score terribly on this metric.

Questions and answers are written by AI about the topic, not taken from the source; they aren’t the newsroom’s opinion.
Related from the stream
Mentions