DOM Crawling Methodology: What SiteNexis Actually Reads
SiteNexis does not scrape HTML source. It crawls the rendered DOM — the state of the page after JavaScript has executed. Here is why that distinction matters and how the crawl works.
There are two versions of any web page: the HTML source sent by the server, and the rendered DOM — the document object model after JavaScript has been parsed, executed, and applied. For pages built with React, Next.js, Vue, or similar frameworks, these two versions can be dramatically different. SiteNexis crawls the rendered DOM using a headless Chromium browser (via Puppeteer), not the HTML source.
Why Rendered DOM Crawling Matters
An HTML source scraper sees only the initial HTML payload from the server — everything that JavaScript renders after load is invisible. For server-side rendered (SSR) pages, this difference is small. For client-side rendered (CSR) pages — where a React component tree builds the DOM after the page loads — the HTML source may contain only a div with id="root" and a script tag. The entire content is invisible to source-based crawlers.
The Crawl Pipeline
The SiteNexis crawl pipeline: (1) Launch headless Chromium browser, (2) Navigate to target URL, (3) Wait for network idle (no pending network requests) to ensure JS-rendered content has loaded, (4) Extract the serialized DOM after execution, (5) Pass to Cheerio for HTML parsing and text extraction, (6) Run boilerplate removal and chunk extraction. The 15-second page timeout balances completeness against crawl speed — pages that do not reach network idle in 15 seconds are crawled with whatever DOM state exists at that point.
What the Crawl Captures
After the rendered DOM is captured, the crawler extracts: all text content (cleaned of HTML tags), all anchor elements with their href and text attributes, all meta elements (title, description, canonical, robots directives), all structured data (JSON-LD script elements), all heading elements (h1 through h6) with their text and DOM position, and the page load timing data for performance analysis.
●robots.txt is parsed before any page is crawled. Pages with a Disallow directive for the SiteNexis crawler user-agent are not crawled. Schema markup in `<script type="application/ld+json">` is extracted as a separate structured data record, not treated as page text.
Crawl Fidelity Measurement
Crawl fidelity is the proportion of a page's content that was successfully extracted in the crawl. A fidelity of 1.0 means all text content was captured. A fidelity below 0.7 indicates significant extraction failure: either JavaScript rendering issues, content behind authentication walls, or content in non-text formats (images, embedded PDFs). Low crawl fidelity is flagged as a critical issue in the Machine Readability analysis.
DOM Forensics — Part 1 of 10