University libraries are being swarmed by AI bots, putting public access at risk

Kevin Maimann · CBC News

· Posted: Nov 05, 2026 4:00 AM EST | Last Updated: 10 minutes ago

Academic libraries are fighting off swarms of data-scraping bots — that are slowing or crashing sites meant for students, researchers and the public — as AI companies continue hunting for content to feed their large language models.

University libraries say entire collections are being knocked offline by persistent streams of unruly bots trying to access their archives, which include academic, historical and government documents. 

"They're just out there on the internet, just bombarding us with trying to grab up as much material as possible," Jill Strykowski, cataloguing and government documents lead at San Jose State University, told CBC News.

The bots are used by both AI and third-party companies to scour websites and retrieve data, often to feed LLMs like Claude, Gemini and ChatGPT.

Libraries, museums and other cultural heritage institutions are prime targets because they have quality, vetted information that is generally not hidden behind paywalls or logins. 

The bots are "causing an incredible amount of work and headache for every institution that does digitization work and every institution that is trying to make information openly available," Strykowski says.

Researchers found such reports started appearing in late 2024 across North America, Europe and Oceania and experts tell CBC News the problem continues to get worse.

LISTEN | Even smarter, with even fewer guardrails:

And new challenges keep arising, says Dean Seeman, director of collections and digital infrastructure at University of Victoria Libraries.

"Sometimes the bot traffic gets so bad that it causes the sites to go down completely for a little while," he said. 

Seeman says this creates a "real problem" for public access. 

Regular people using library websites are now, effectively, in competition with infinite traffic from bot farms, says Peter Lewis, Canada Research Chair in Trustworthy Artificial Intelligence at Ontario Tech University.

"This is another instance of large, private companies making use of common-good public interest institutions in a way that's very extractive and just assumes that that stuff is available and is free — and then using it to be able to produce highly commercial models that then they can go and sell back to people," he said. 

Workers enter data from an old document belonging to the McMillan Memorial Library during a digitization exercise at the Eastland Library in Nairobi on March 24, 2022. Libraries and other cultural heritage institutions are prime targets for AI bots. (Patrick Meinhardt/AFP/Getty Images)

Good bots and bad bots

Eric Hellman, executive director of Project Gutenberg — a U.S.-based volunteer organization that digitizes and archives cultural works and distributes eBooks — wrote in a blog last year that these bots, which he characterizes as poorly programmed "monstrosities," are "destroying open access" on the internet. 

He tells CBC News that "everything's gotten worse" since he wrote that.

Hellman says the problem comes down to good bots and bad bots. 

He explains that search engines like Google, and archiving sites like Internet Archive, use bots that follow robots.txt — voluntary sets of instructions on websites that tell them what they can and can't do, so they don't disrupt the site's functioning while gathering information.

AI scraping bots, on the other hand, sometimes ignore these rules and overwhelm websites — in some cases requesting thousands of documents a second and downloading them in every available format.

WATCH | Young AI researchers in Ontario:

As an example, the University of North Carolina at Chapel Hill reported last year that it received more than 11,000 searches for Finnish music from thousands of different internet addresses on the same day. The university said its library catalogue was periodically shutting out students, faculty and staff due to bot traffic.

And because one company or bot farm can work from thousands of IP addresses, Hellman says the old fix of blocking problematic IPs is no longer useful.

Michael Weinberg, co-director of GLAM-E, a collaborative effort focused on open-access policies for cultural institutions, is working to protect public access to information. (Submitted by Michael Weinberg)

He says Project Gutenberg is happy for its materials to be used as AI training data and even offers a one-click download of its entire library of books, "but the bots are too stupid to use it." 

"Our material is almost all public domain material. It's exactly the sort of material that they should be training on," he said. "We just wish they would do it in a respectful way that doesn't take down websites."

Open access a 'huge benefit'

Some libraries have set up CAPTCHA challenges or other verification systems to weed out bots. 

The problem is, those measures can also weed out the good bots, says Michael Weinberg, co-director of GLAM-E, a collaborative academic initiative at NYU Law School and the University of Warwick focused on open-access policies for cultural institutions.

Some libraries have also begun to require login credentials, a controversial move that those who spoke to CBC News say they want to avoid because it puts barriers on human access to things they believe should be easily accessible.

And as the scraping bots get more advanced, experts say they could also learn to work around these strategies.

The Anthropic logo is displayed at during the Dreamforce technology summit in San Francisco, on Sept. 17. AI firms have broadly argued that scraping publicly available data is permissible under fair use and in the public's best interest. (Carlos Barria/Reuters)

Major AI firms have argued that scraping publicly available data to train their models is permissible under fair use and in the public's best interest.

But faced with widespread criticism and copyright lawsuits, Anthropic, OpenAI, Google and Microsoft expressed support last week for honouring a single, standardized opt-out signal in a website's robots.txt file that could block their web crawlers from harvesting data.

Compliance would be voluntary, however, and the agreement leaves out problematic bots from third-party companies and smaller AI startups.

• Used bookstores navigate 'suspicious' bulk orders amid AI book-shredding fears

• How a small Canadian publisher is resisting the AI book wave

• AI is making it harder for voters to get accurate information: Elections Canada chief

Weinberg says the digitization of valuable information has been a "fantastic" development for humanity — giving researchers, teachers and curious citizens easy access to resources that, 20 years ago, "would have required driving for flying or a special permission to get access."

Organizations like GLAM-E are trying to figure out long-term solutions to preserve that access for humans and helpful bots alike.

Right now, the bots create "a disincentive for libraries and other institutions to make parts of their collection available online," Weinberg said. "I think it's been a huge benefit to society that libraries and museums and other cultural institutions have put a huge amount of work into making their collections available."

People look at their laptops and in the main reading room of the New York Public Library on Oct. 5, 2016, in New York City. Increased bot traffic means regular people using library websites are effectively competing with infinite traffic from bot farms, says one expert. (Dominick Reuter/AFP/Getty Images)