Boilerplate Ratio: Measuring Extraction Noise
The boilerplate ratio is the proportion of a page's extracted text that is template noise rather than content signal. High boilerplate ratios directly reduce Machine Readability and extraction quality.
Every page has two types of text: content text (unique to this page, written to convey specific information) and boilerplate text (shared across many pages, part of the template rather than the content). For AI extraction, only content text carries signal. Boilerplate text is noise that dilutes the content signal and reduces the quality of the chunks and embeddings derived from the page.
How Boilerplate Is Identified
SiteNexis uses cross-page frequency analysis to identify boilerplate. Every distinct text block across all crawled pages is tracked. A text block that appears on 70% or more of pages is classified as boilerplate. This threshold catches navigation menus, headers, footers, cookie consent text, promotional banners, and legal disclaimers — all of which repeat across the site. Text blocks appearing on fewer than 70% of pages are treated as content text.
Computing the Boilerplate Ratio
The boilerplate ratio for a page is: (boilerplate text words) / (total extracted text words). A page with 2,500 total words where 900 words are classified as boilerplate has a boilerplate ratio of 36%. The Machine Readability Score penalises high boilerplate ratios because they indicate that a large proportion of the "content" that AI systems would extract is actually template noise.
▲The pages with the highest boilerplate ratios are often short pages: a 300-word blog post with the same 200-word navigation and footer as a 2,000-word guide has a 40% boilerplate ratio versus the guide's 9% boilerplate ratio. Short pages are disproportionately harmed by high boilerplate text volumes.
Reducing Boilerplate Without Reducing Navigation
The goal is not to remove navigation — it is to ensure navigation text is correctly excluded from content extraction. The most effective architectural approach: use clear semantic HTML sectioning (nav, header, footer, main, aside) so that boilerplate elements are easily identifiable and excludable by extraction pipelines. Content in `<main>` with boilerplate in `<nav>` and `<footer>` produces near-zero boilerplate contamination in the extracted text.
DOM Forensics — Part 4 of 10