Can AI assistants read your site?

What we check

• Crawler access. Your robots.txt rules for 12 crawlers, including OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic’s ClaudeBot, Claude-SearchBot and Claude-User; PerplexityBot; Google-Extended; Googlebot and Bingbot.
• Content without JavaScript. How many words a crawler that doesn’t run JavaScript sees on your homepage. Many vibe-coded apps show close to none.
• The basics machines rely on. Title, meta description, structured data, sitemap, llms.txt, link previews, noindex tags and canonical links.

Questions

Why would AI assistants not know about my app?

Usually one of three reasons: robots.txt blocks their crawlers, the homepage is an empty JavaScript shell that crawlers can't read, or nothing on the web links to the site, so crawlers never find it.

Do AI crawlers run JavaScript?

Most don't. GPTBot, ClaudeBot and PerplexityBot generally read the HTML your server sends. If your app is a single-page React, Vite, Lovable or Bolt build, they may see an empty page. Server rendering or pre-rendering fixes it.

Should I block AI training crawlers?

It's your call. Blocking training crawlers such as GPTBot or ClaudeBot doesn't remove you from AI search answers, as long as the search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) can still read the site.

What is llms.txt?

An optional Markdown file at /llms.txt that summarises what your product is and links to your key pages, so AI tools can understand it quickly. It's a young convention, so treat it as a nice-to-have.

What does the checker read?

Only public files: robots.txt, llms.txt, your sitemap and your homepage, fetched once as AppBreakupBot. It doesn't store the address you check, and it never pretends to be another crawler.