What AI Systems Actually See When They Crawl Your Website

The question "what does Google see when it crawls my website?" has been answered exhaustively. Rich results testers, mobile-friendly testers, and Core Web Vitals tools have made Googlebot's view of your content relatively transparent. The equivalent question for AI systems — "what does ChatGPT see? What does Claude see? What does Perplexity actually extract from my pages?" — has almost no tooling and far less documentation. The answer is fundamental to AI visibility strategy, and it starts with understanding that AI systems do not read your website the way a human or a search crawler does.

Stage 1: Rendering and Boilerplate Stripping

An AI retrieval system's first pass at your content is aggressive filtering. Navigation menus, headers, footers, cookie banners, sidebars, and repeated site chrome are identified and removed before any semantic processing occurs. The system is not interested in your brand name appearing in the navigation on every page. It is interested in the body content. On most websites, this first-pass filter removes 25–45% of the total HTML text. What remains is the extraction candidate pool.

Stage 2: Chunking

The filtered text is split into chunks — discrete semantic units of approximately 300–600 tokens. Chunking is where most content strategies fail invisibly. If your content is structured so that meaning spans paragraph boundaries (a claim in paragraph one that is only meaningful in the context of paragraph three), the chunk boundary may fall between the claim and its context. The extracted chunk contains an incomplete thought. When that chunk is retrieved for a query, the AI model receives a fragment. Fragments produce lower-quality answers and lower citation probability.

Stage 3: Entity Extraction

Within each chunk, the AI system identifies named entities — organisations, people, products, places, concepts — and maps their relationships. Entities that are named explicitly, described with consistent attributes, and connected to external validation sources receive high confidence scores. Entities that are implied rather than named, described inconsistently across chunks, or lacking external references receive low confidence scores. Low-confidence entities are effectively invisible to AI recommendation systems, regardless of how prominently they feature in your content.

Stage 4: Trust Signal Evaluation

After entity extraction, AI systems evaluate trust signals on each chunk and at the domain level. Does the schema markup match the body text claims? Are the entity attributes consistent across chunks? Are there sameAs links that validate entity identity against external authority sources? Do any claims contradict claims from higher-authority sources in the retrieval index? Each trust check either increases or decreases the confidence weight applied to the chunk during retrieval ranking. High-trust chunks are retrieved preferentially. Low-trust chunks are deprioritised even when they are semantically relevant to a query.

What Gets Discarded

• Navigation, footer, and sidebar content — stripped before chunking
• Boilerplate text repeated across multiple pages — low signal-to-noise, suppressed
• Chunks with dangling references requiring cross-chunk context to be meaningful
• Claims that contradict external authority sources — trust-penalised out of retrieval
• Entities with no external validation and no sameAs links — low confidence, low retrieval weight
• Content past the context window limit on long pages — simply not processed