The rise of artificial intelligence (AI) has prompted significant shifts in website traffic dynamics, particularly due to the activity of AI crawlers. According to a report from Computer World, these web-scraping bots are becoming a focal point of scrutiny as they increasingly consume vast amounts of data for training large language models (LLMs).
A pressing concern surrounding these crawlers is their tendency to access and gather data without authorisation. This behaviour contrasts sharply with "verified" bots, including those from search engines like Google and Bing, which are transparent about their operations. Cloudflare, a web performance and security company, has been monitoring AI bot traffic to discern patterns, identifying the most aggressive bots, their request volumes, and the frequency of their crawls.
In their findings, researchers at Cloudflare noted that "facebookexternalhit" has emerged as the leading source of bot traffic, accounting for a substantial 27.16% of the total volume. This particular bot has garnered a reputation for generating excessive traffic, raising alarms among website operators. Following closely behind is Bytespider, associated with TikTok's parent company ByteDance, which contributed 23.35%. Other notable players include Amazonbot, responsible for 13.34% of traffic, Anthropic's ClaudeBot at 8.06%, and GPTBot, which made up 5.60%.
The report also highlights intriguing trends over the year regarding bot traffic. While Bytespider's activity significantly declined, seeing a drop of approximately 80% to 85% by year's end, ClaudeBot experienced a surge mid-year before stabilising. In contrast, GPTBot exhibited consistent traffic levels throughout 2024.
Businesses are increasingly considering the implications of such bot activity, especially as the competition for data intensifies. Understanding the behaviour and impact of these AI crawlers is vital for companies as they navigate an evolving digital landscape marked by the proliferation of AI technologies.
Source: Noah Wire Services