Rendered vs Source HTML: Where AI Extraction Diverges
Source HTML and rendered HTML are two different documents. Understanding the difference between them reveals why content that exists "on the page" may be invisible to AI systems.
Source HTML is the HTML delivered by the server in the HTTP response. Rendered HTML is the DOM state after the browser has parsed the source HTML, executed all JavaScript, applied CSS, and completed all asynchronous data fetches. For modern web applications, these two documents can be dramatically different — and different AI crawlers read one or the other.
The Two Crawler Types
HTTP-based crawlers fetch the source HTML only. They receive the server response and parse it without JavaScript execution. Headless browser crawlers (like SiteNexis) launch a browser, navigate to the URL, and read the rendered DOM after JS execution. The content gap between these two methods depends entirely on how much of the page's content is JavaScript-rendered versus server-rendered.
What Exists Only in Rendered HTML
Content that typically exists only in rendered HTML (invisible to HTTP crawlers): product descriptions and pricing loaded from an API after page mount, review and rating content loaded asynchronously, personalized content (recommendations, recently viewed), tab content not in the active tab on initial load in CSR applications, infinite scroll content, and modal content loaded on user interaction.
What Exists Only in Source HTML
Content that sometimes exists only in source HTML (may disappear in rendered HTML): server-side rendered data that is overwritten by client-side hydration (hydration errors that clear SSR content), meta tags written by the server that client-side routing overwrites without proper SPA meta management, and structured data in script tags that dynamic rendering removes in certain configurations.
●The SiteNexis crawl captures rendered HTML. If you want to compare source vs rendered for a specific page: view-source: the URL (source HTML) and then use browser DevTools Elements panel (rendered HTML). Content that appears in DevTools but not in view-source is JavaScript-rendered. Content that appears in view-source but not in DevTools may have been overwritten during hydration.
DOM Forensics — Part 8 of 10