Why Your Website Isn't Showing Up in AI Search Results

GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and OAI-SearchBot need explicit permission to crawl your site. Many websites block these crawlers either through restrictive robots.txt rules inherited from templates or through WAF (Web Application Firewall) rules designed to block bots broadly.

Fix: Add these rules to robots.txt:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

Structured data is the single highest-ROI GEO investment available. FAQPage schema gives AI engines a machine-readable list of every question your page answers and the complete text of each answer. Pages with FAQPage schema are cited at roughly 3× the rate of equivalent pages without it, because the AI can extract a complete, citable answer directly, no synthesis required.

Fix: Add FAQPage JSON-LD to your homepage and service pages with at least 8 questions covering methodology, timeline, deliverables, and client profile. Add Article schema to all blog posts and guides.

AI engines are attempting to answer specific user questions. Pages that are structured as marketing copy ("We help ambitious businesses reach their potential") give AI engines nothing to cite. Pages that directly answer common questions ("Our typical website redesign project takes 3 to 6 weeks and includes four phases: Discovery, Architecture, Design, and Delivery") are highly citable.

Fix: Audit your key pages for direct question-answer pairs. Add an FAQ section to your homepage and services pages. Rewrite your About and Process pages in declarative, fact-forward language.

AI engines favor content that makes specific, verifiable claims. Vague brand language ("world-class results") carries no citation value. Specific statistical claims (for example "Lighthouse scores moved from 48 to 94 within one week of launch") give AI engines something concrete to include in a synthesized answer. Pages under 400 words rarely get cited at all.

Fix: Introduce statistical anchors throughout key pages. Use real client data, percentage improvements, timeline specifics, and named methodologies. Every page that matters should have at least 800 words of substantive content.

AI engines build a "knowledge graph" of entities on the web. Your business is an entity. If your Organization schema is missing or incomplete, no foundingDate, no location, no knowsAbout array, no sameAs links to LinkedIn and Twitter, AI engines have low confidence in who you are and may avoid citing you for fear of providing inaccurate information.

Fix: Implement comprehensive Organization schema with name, url, description, foundingDate, email, areaServed, and a knowsAbout array listing your actual domains of expertise. Add sameAs links to every established profile (LinkedIn, Twitter/X, Crunchbase, etc.).

Many modern websites render their key content client-side in JavaScript frameworks. While Google's crawler has improved at rendering JavaScript, most AI crawlers use simpler HTTP-only fetching, they receive the raw HTML and see nothing if your content is loaded dynamically. A website that appears complete in a browser may look nearly empty to an AI crawler.

Fix: Ensure all key content, especially your service descriptions, FAQ section, and About information, exists in the server-rendered HTML, not only in client-side JavaScript. Test by disabling JavaScript in Chrome DevTools and checking if your content still appears.

The llms.txt standard is an emerging convention, analogous to robots.txt, but designed for language models rather than crawlers. Placing an llms.txt at your domain root gives AI models a structured, concise overview of your business, services, and key pages in a format optimized for machine reading. It's a declaration that says "here's what we are and what you should know about us."

Fix: Create /llms.txt at your domain root with your business description, service overview, page index, and contact information. Keep it under 2,000 characters and use the standard Markdown-light format with # headings and > descriptions.