Data study
How often AI crawlers read software pages
Every page request a crawler made to AppBreakup over 13 days (Oct 26 – Nov 7, 2026), counted from our own server logs. AppBreakup is a software directory with 5,498 apps, their alternatives, pricing and comparisons, so these are the reading habits of AI and search crawlers on one site about software.
By AppBreakup editorial · Data: to (UTC) · Updated · Method · Download CSV
We started logging crawler requests on Oct 26, 2026, so this edition covers 13 days rather than 28. The window grows each day until it reaches 28 days.
Key findings
- AI crawlers made 82 requests for every Googlebot request (628,883 against 7,695), and 15 for every request from all search engines combined.
- OpenAI was the busiest AI crawler, with 269,232 requests (42.8% of AI crawler traffic) across 12,739 pages, about 21 requests per page.
- By number of pages read, the widest AI crawler was Amazonbot (16,718 distinct pages). Googlebot read 2,694.
- Alternatives pages drew 51.5% of AI crawler requests, against 58.6% of search engine requests.
- Google's non-Search crawler, GoogleOther, made 440,816 requests: 57 times as many as Googlebot. We count it under other bots, not AI crawlers, because Google doesn't say what it's used for beyond research and products other than Search.
Data for Oct 26 – Nov 7, 2026. Free to quote with a link to this page; see how to cite.
Headline numbers
- Crawler and bot requests
- 1,503,725
- to 23,929 distinct pages
- AI crawlers
- 628,883
- 41.8% of requests · 21,432 pages
- Search engines
- 41,852
- 2.8% of requests · 12,788 pages
- Other bots
- 832,990
- 55.4% of requests · 23,384 pages
- AI crawler requests per Googlebot request
- 82
- 7,695 Googlebot requests
- AI crawler requests per day
- 48,376
- average over 13 days
Requests per day
AI crawlers and search engines on separate charts with their own scales: on one shared scale, search engine traffic would be too small to see. Hover a bar for the exact count.
AI crawlers
Search engines
Show the daily numbers as a table
| Day (UTC) | AI crawlers | Search engines | Other bots |
|---|---|---|---|
| 32,869 | 2,586 | 189,556 | |
| 46,953 | 289 | 46,120 | |
| 47,181 | 103 | 7,681 | |
| 34,150 | 166 | 3,169 | |
| 26,888 | 179 | 8,180 | |
| 24,073 | 305 | 16,454 | |
| 72,232 | 2,613 | 104,692 | |
| 46,765 | 3,026 | 46,858 | |
| 52,344 | 3,194 | 138,183 | |
| 57,469 | 8,649 | 91,260 | |
| 69,949 | 9,250 | 114,349 | |
| 75,231 | 3,028 | 34,996 | |
| 42,779 | 8,464 | 31,492 |
Requests by crawler
Share is of all crawler and bot requests. Requests per page is requests divided by distinct pages: how often a crawler came back to the same page.
| Crawler | Group | Requests | Share | Distinct pages | Requests per page |
|---|---|---|---|---|---|
| GoogleOtherGoogle's general-purpose crawler for research and products other than Search. It doesn't feed Google Search. | Other bots | 440,816 | 29.3% | 10,457 | 42.2 |
| Other botAny other user agent that calls itself a bot, crawler or spider. | Other bots | 312,882 | 20.8% | 21,253 | 14.7 |
| OpenAITraining (GPTBot), ChatGPT search (OAI-SearchBot) and fetches a ChatGPT user asked for (ChatGPT-User). | AI crawlers | 269,232 | 17.9% | 12,739 | 21.1 |
| AmazonbotUser agents: Amazonbot | AI crawlers | 129,162 | 8.6% | 16,718 | 7.7 |
| AnthropicTraining (ClaudeBot), Claude search (Claude-SearchBot) and fetches a Claude user asked for (Claude-User). | AI crawlers | 101,669 | 6.8% | 9,379 | 10.8 |
| PerplexityPerplexity's index (PerplexityBot) and fetches a Perplexity user asked for (Perplexity-User). | AI crawlers | 80,740 | 5.4% | 14,510 | 5.6 |
| Meta AIUser agents: meta-externalagent, Meta-ExternalFetcher, FacebookBot | AI crawlers | 48,059 | 3.2% | 5,998 | 8.0 |
| ApplebotUser agents: Applebot | Search engines | 31,354 | 2.1% | 12,017 | 2.6 |
| SemrushUser agents: SemrushBot, SiteAuditBot | Other bots | 25,050 | 1.7% | 13,215 | 1.9 |
| Other SEO toolUser agents: Screaming Frog, SEOkicks, serpstatbot, DataForSeoBot, BLEXBot, barkrowler | Other bots | 21,306 | 1.4% | 17,238 | 1.2 |
| MajesticUser agents: MJ12bot | Other bots | 16,465 | 1.1% | 7,907 | 2.1 |
| AhrefsUser agents: AhrefsBot, AhrefsSiteAudit | Other bots | 13,010 | 0.9% | 11,565 | 1.1 |
| GooglebotThe crawler that feeds Google Search. | Search engines | 7,695 | 0.5% | 2,694 | 2.9 |
| Script or libraryHTTP libraries and command-line tools (curl, Python, Go and similar). | Other bots | 2,458 | 0.2% | 1,437 | 1.7 |
| YandexBotUser agents: YandexBot, YandexImages | Search engines | 1,295 | <0.1% | 1,132 | 1.1 |
| Performance testerUser agents: HeadlessChrome, Lighthouse, PageSpeed, GTmetrix, Chrome-Lighthouse | Other bots | 822 | <0.1% | 60 | 13.7 |
| Other search engineUser agents: Qwantbot, SeznamBot, Yeti, Sogou, Exabot, PetalBot, MojeekBot, Bravebot | Search engines | 817 | <0.1% | 760 | 1.1 |
| BingbotUser agents: bingbot, BingPreview, adidxbot | Search engines | 550 | <0.1% | 517 | 1.1 |
| DuckDuckBotUser agents: DuckDuckBot, DuckAssistBot | Search engines | 137 | <0.1% | 53 | 2.6 |
| Link previewUser agents: facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Discordbot, TelegramBot, WhatsApp, Pinterest, redditbot, Embedly, SkypeUriPreview, Iframely | Other bots | 133 | <0.1% | 20 | 6.7 |
| No user agent | Other bots | 16 | <0.1% | 3 | 5.3 |
| Other AIUser agents: cohere-ai, Diffbot, YouBot, MistralAI-User, Timpibot | AI crawlers | 16 | <0.1% | 16 | 1.0 |
| Search Console testUser agents: Google-InspectionTool | Other bots | 16 | <0.1% | 6 | 2.7 |
| Google AdsUser agents: AdsBot-Google, Mediapartners-Google | Other bots | 11 | <0.1% | 11 | 1.0 |
| BaiduspiderUser agents: Baiduspider | Search engines | 4 | <0.1% | 4 | 1.0 |
| Common CrawlUser agents: CCBot | AI crawlers | 3 | <0.1% | 3 | 1.0 |
| MozUser agents: DotBot, rogerbot | Other bots | 3 | <0.1% | 2 | 1.5 |
| BytespiderUser agents: Bytespider | AI crawlers | 2 | <0.1% | 2 | 1.0 |
| Uptime monitorUser agents: UptimeRobot, Pingdom, StatusCake, Better Uptime, monitor | Other bots | 2 | <0.1% | 2 | 1.0 |
Which pages they read
Each crawler's requests split by page type. App pages describe one app; alternatives pages list the apps people switch to; other pages include the home page, A–Z lists and site pages such as sign-in and corrections.
| Crawler | App pages | Alternatives | Comparisons | Pricing | Categories | Other pages |
|---|---|---|---|---|---|---|
| All AI crawlers | 6.2% | 51.5% | 3.2% | 1.3% | 3.9% | 33.8% |
| All search engines | 19.7% | 58.6% | 5.3% | 3.4% | 3.1% | 10.1% |
| GoogleOther | 6.4% | 20.5% | 5.7% | 0.1% | 9.0% | 58.3% |
| OpenAI | 7.5% | 6.9% | 5.8% | 1.0% | 8.4% | 70.4% |
| Amazonbot | 3.8% | 88.8% | 1.5% | 1.2% | 0.5% | 4.1% |
| Anthropic | 3.1% | 87.6% | 0.2% | 1.6% | 0.2% | 7.2% |
| Perplexity | 10.6% | 77.1% | 2.9% | 2.9% | 1.2% | 5.4% |
| Meta AI | 4.4% | 82.1% | 0.6% | <0.1% | 0.3% | 12.6% |
| Applebot | 13.1% | 71.2% | 4.6% | 2.8% | 1.1% | 7.2% |
| Googlebot | 41.7% | 22.9% | 4.6% | 2.2% | 5.4% | 23.1% |
| YandexBot | 35.5% | 16.6% | 15.9% | 13.4% | 16.9% | 1.7% |
| Other search engine | 22.3% | 4.0% | 17.1% | 21.5% | 28.2% | 6.9% |
| Bingbot | 38.7% | 31.5% | 12.7% | 0.0% | 12.4% | 4.7% |
| DuckDuckBot | 46.0% | 6.6% | 1.5% | 0.0% | 4.4% | 41.6% |
| Other AI | 87.5% | 6.3% | 6.3% | 0.0% | 0.0% | 0.0% |
| Baiduspider | 75.0% | 0.0% | 0.0% | 0.0% | 0.0% | 25.0% |
| Common Crawl | 0.0% | 0.0% | 0.0% | 0.0% | 33.3% | 66.7% |
| Bytespider | 100.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
App categories AI crawlers read most
AI crawler requests to pages about a single app (its app, alternatives and pricing pages), grouped by the app's category. Top 15 of 365,858 such requests. Categories with more apps get more requests, so requests per app is the fairer comparison.
| # | Category | AI crawler requests | Share | Apps read | Requests per app |
|---|---|---|---|---|---|
| 1 | Monitoring and error tracking tools | 6,362 | 1.7% | 65 | 98 |
| 2 | Note-taking apps | 5,763 | 1.6% | 52 | 111 |
| 3 | Database clients | 5,403 | 1.5% | 47 | 115 |
| 4 | Code editors | 5,378 | 1.5% | 58 | 93 |
| 5 | Remote desktop software | 5,341 | 1.5% | 44 | 121 |
| 6 | 3D modeling and CAD software | 5,166 | 1.4% | 51 | 101 |
| 7 | CI/CD and code hosting platforms | 5,031 | 1.4% | 41 | 123 |
| 8 | SEO tools | 5,009 | 1.4% | 51 | 98 |
| 9 | Music streaming services | 5,003 | 1.4% | 48 | 104 |
| 10 | Product analytics tools | 4,944 | 1.4% | 40 | 124 |
| 11 | Customer support and live chat software | 4,940 | 1.4% | 48 | 103 |
| 12 | Terminal emulators | 4,936 | 1.3% | 49 | 101 |
| 13 | Cloud storage and file sync services | 4,905 | 1.3% | 47 | 104 |
| 14 | HR and people software | 4,849 | 1.3% | 49 | 99 |
| 15 | AI chatbots and assistants | 4,774 | 1.3% | 48 | 99 |
Methodology
- Source. AppBreakup's own server logs. When a request for a page arrives, our server checks the user agent string against a list of known crawlers and, if it matches, records the day, the crawler's name, its group and the path. Nothing else is stored: no IP address, no full user agent, no cookies and no personal data.
- Identified by user agent only. We don't verify crawlers by IP address or reverse DNS, so some requests may come from scrapers using a crawler's name. Treat the numbers as what crawlers claimed to be.
- What counts as a request. One full page load (an HTML document request) by a recognised bot. Requests are counted whatever the response status, including redirects and pages that weren't found. Background navigation data and prefetches aren't counted; only browsers make them.
- Excluded. Files rather than pages (robots.txt, sitemaps, llms.txt, RSS feeds, images, this report's CSV, and scanners probing for files such as /.env), paths beginning with a dot such as /.well-known, share images, the admin area, our API and redirect links, and private stats links.
- Groups. AI crawlers are OpenAI, Anthropic, Perplexity, Meta AI, Amazonbot, Common Crawl (widely used to train AI models), Bytespider and a few smaller AI companies. Each company's training crawler, search crawler and user-triggered fetcher share one label, because we store the label rather than the user agent. Search engines are Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider and smaller engines. Everything else, including GoogleOther, SEO tools, link previews, uptime monitors and scripts, is in other bots. Google-Extended and Applebot-Extended are robots.txt settings, not crawlers, so they never appear in logs.
- Window. The last 28 complete days in UTC, ending yesterday; today is left out because it's still filling in. This edition covers Oct 26 – Nov 7, 2026, 13 days, because logging began on Oct 26, 2026.
- Distinct pages counts unique paths; query strings aren't logged, so /search?q=a and /search?q=b are one page. App categories come from the app's current category on AppBreakup.
- Limits. This is one site: a young software directory. Crawl rates depend on a site's size, age, links and robots.txt, so other sites will see different numbers. Our robots.txt lets every crawler read all public pages.
- Updates. The numbers refresh daily over the rolling window. We review the method and the findings monthly; when you cite a number, cite the date range shown with it.
How to cite
You're welcome to use these numbers and charts under CC BY 4.0. Please credit AppBreakup with a link to this page, and include the date range.
Suggested citation
AppBreakup editorial (2026). "How often AI crawlers read software pages." AppBreakup, data for Oct 26 – Nov 7, 2026. https://appbreakup.com/reports/ai-crawlers (accessed Nov 8, 2026).
Link
<a href="https://appbreakup.com/reports/ai-crawlers">AppBreakup AI crawler report</a>
Download CSVCheck if AI crawlers can read your site
Questions about the data: [email protected]