Crawl Fidelity: The Gap Between What You See and What AI Reads
Crawl fidelity measures the completeness of content extraction. A fidelity gap means content exists on your page but was not captured in the crawl — and will not be seen by AI systems.
Crawl fidelity is the ratio of content that was successfully extracted in a crawl to the content that is visually present on the page. A fidelity of 1.0 means everything you can read on the page was captured. A fidelity of 0.5 means half the content was lost — invisible to any AI system that relies on the crawled representation of the page.
Common Sources of Fidelity Gaps
The most frequent causes of crawl fidelity gaps: JavaScript-rendered content that the crawl timeout does not capture (JS execution takes longer than the 15-second limit), content inside iframes (excluded by default from DOM extraction), content inside Shadow DOM (requires special handling), content behind authentication (login gates, paywalls), content inside images (text embedded in graphics is not extractable), and content inside non-standard HTML elements that the parser does not traverse.
Measuring the Fidelity Gap
SiteNexis estimates crawl fidelity by comparing the extracted text length against the expected content length based on page structure signals: the number of heading elements, the DOM tree depth, and the page load timing. A page with 20 H2 sections that produces only 400 words of extracted text has a probable fidelity gap — 20 sections at 400 words suggests approximately 20 words per section, which is unusually sparse for a real content page.
●Crawl fidelity is marked as "estimated" in the SiteNexis report because it is computed from signals rather than a direct comparison against a ground-truth content inventory. The user knows the actual content of their pages — the fidelity estimate tells them where to investigate, not what the precise gap is.
Fixing Fidelity Gaps
The correct fix depends on the source of the gap. For JS rendering gaps: implement SSR or SSG for content sections so they appear in the initial HTML payload. For iframe content: move content out of iframes into the main document. For authentication gaps: ensure pages intended for AI visibility are not behind a login wall. For Shadow DOM gaps: provide a main-document text alternative for content in Web Components.
DOM Forensics — Part 5 of 10