LLM bots are the easiest to spot. Rate limit them to 16kb/s and 1 request every 5 seconds. Give them a friendly multilingual WCAG status 429 page that says to slow down or status 444 and drop them into nftables ipsets if they continue to be aggressive. Ask your favorite LLM how to set this up. If your LLM gets session paused, ask Gemini or Grok. Tell it if you are behind a CDN. Enable logging of ssl curves, ciphers, etc... Show your LLM some examples from your access logs.
What I still don't understand is who is doing it? I assume it's not the big companies like OpenAI or Google or Anthropic, right? So there are a bunch of small things out there doing the scraping and then they sell it to the big places? Or are these just smaller AI models doing the scraping and using it for themselves? Is there an underground ecosystem where people buy and sell this stuff? I understand why it's valuable to the AI companies I'm curious about how all this work, the economics of it all.
Behind nearly every AI bot is some human who will see real benefit from scraping my site.
I just make sure all my pages are super lightweight to cache and serve from cloudflare and let the bots do what they like.
If I put something on the internet for all to see, I want bots to be able to see it too.
I've been on the other side of bot blocks - when I ask my LLM to buy me X and it comes back and says the only place it could buy from is some Bulgarian seller because everywhere else has anti-bot scripts, I make sure to give that Bulgarian site my business.
Same thing on the tiny scale. My side project site gets maybe 2 real visitors a month and a steady stream of 0-second "visits" from US datacenters. For a library with real users that load must be brutal.
In what world is 11k searches a lot? The story here is that AI bots are exposing University Libraries bad setups. I haven't heard any institution here in northern europe having any trouble.
While I am fully sympathetic to the bot scraping problem and support high quality public data on an open internet, I can't help but think the whole system needs a re-architecting.
Can we re-deploy the internet as statically hosted data available through a defined api? Something like DNS for data; eventually consistent and evenly distributed. Either the data is open and ubiquitous, or closed and inaccessible.
It’s alpha but it should be ready for real-world testing in the next couple of days. I’m going to drop it in front of my Gitea and home-grown api servers.
I’m super tired of Anubis spamming me with furry fetish art on its pointless interstitials 20 times a day. As someone pointed out in another comment, LLM scraper bots are super easy to detect with basic heuristics. PoW and session cookies and an interstitial are massive overkill and annoying spam.
"Open Access" means supporting users. Now your users are also bots, and the people that deploy those bots. You need to be able to serve those users too.
These days, AI attacks are indiscriminately targeting personal websites, and they've become far too severe. They even come in disguised through referrer spam[1], so they're tricky to block. In my opinion, I suspect such scraped data is being used to classify and train AI models.
It's a canonical DOI on what was a download server. It says exactly which file is being requested. (I chose a random recent one.) At the very most, they should serve a static page with a bibtex entry, and a few other metadata fields, and a link to the PDF and whatever other formats are available.
Instead, we get piles of captchas, login-gate spam about how they're "open access" now and have some initiative to be LLM friendly.
Getting back to the problem of people doing stuff that requires access to library materials: What sort of server-side cost multiplier do you suppose this ACM page has vs. just serving a few S3-compatible URLs that point to a few mirrors? (Or, got forbid, Anna's Archive / libgen, since ACM is open access, and those sites would be doing nothing wrong by mirroring ACM content.)
University libraries are being swarmed by AI bots, putting public access at risk
(cbc.ca)117 points by blakesterz 6 hours ago | 53 comments
Comments
I just make sure all my pages are super lightweight to cache and serve from cloudflare and let the bots do what they like.
If I put something on the internet for all to see, I want bots to be able to see it too.
I've been on the other side of bot blocks - when I ask my LLM to buy me X and it comes back and says the only place it could buy from is some Bulgarian seller because everywhere else has anti-bot scripts, I make sure to give that Bulgarian site my business.
Can we re-deploy the internet as statically hosted data available through a defined api? Something like DNS for data; eventually consistent and evenly distributed. Either the data is open and ubiquitous, or closed and inaccessible.
https://git.eeqj.de/sneak/smallwebwaf
It’s alpha but it should be ready for real-world testing in the next couple of days. I’m going to drop it in front of my Gitea and home-grown api servers.
I’m super tired of Anubis spamming me with furry fetish art on its pointless interstitials 20 times a day. As someone pointed out in another comment, LLM scraper bots are super easy to detect with basic heuristics. PoW and session cookies and an interstitial are massive overkill and annoying spam.
RIGHT?
https://en.wikipedia.org/wiki/Internet2
"Open Access" means supporting users. Now your users are also bots, and the people that deploy those bots. You need to be able to serve those users too.
[1]https://www.makonea.com/en-US/casual/scraping-bots-are-destr...
If anything, libraries should be becoming more AI friendly, not less. Instead, we get this sort of bullshit from the ACM:
https://dl.acm.org/doi/10.1145/3832783.3834511
It's a canonical DOI on what was a download server. It says exactly which file is being requested. (I chose a random recent one.) At the very most, they should serve a static page with a bibtex entry, and a few other metadata fields, and a link to the PDF and whatever other formats are available.
Instead, we get piles of captchas, login-gate spam about how they're "open access" now and have some initiative to be LLM friendly.
Getting back to the problem of people doing stuff that requires access to library materials: What sort of server-side cost multiplier do you suppose this ACM page has vs. just serving a few S3-compatible URLs that point to a few mirrors? (Or, got forbid, Anna's Archive / libgen, since ACM is open access, and those sites would be doing nothing wrong by mirroring ACM content.)