A swarm of AI “crawlers” is running rampant on the internet, scouring billions of websites for data to feed algorithms at leading tech companies — all without permission or payment, upending the online economy.
Before the rise of AI chatbots, websites allowed search engines to access their content in return for increased visibility, a system that rewarded them with traffic and advertising revenues.
But the rapid development of generative AI has allowed tech giants like Google and OpenAI to harvest information for their chatbots with web crawlers,…
