Boilerplate Detection and Why It Kills Retrieval
Boilerplate is the navigation, footers, sidebars, and template text that appears on every page. AI systems try to remove it — but when they fail, it contaminates the chunks that reach retrieval.
Every page on a typical website contains a significant proportion of text that is not the page's content: navigation menus, header CTAs, footer links, cookie notices, sidebar widgets, email signup forms. For a human reader, these elements are invisible — the eye skips them. For an AI extraction pipeline, they are text — and text that is not correctly identified as boilerplate gets included in the content chunks.
What Happens When Boilerplate Reaches the Chunks
When boilerplate text enters the chunk extraction phase, it corrupts the chunks in two ways. Dilution: the semantic content of a chunk is diluted by navigation text, reducing the embedding quality and making the chunk less likely to match a relevant query. Contamination: specific boilerplate phrases ("Home / Products / Solutions / About") can be mistaken for topical signals, leading the AI to associate the content with topics the page does not actually cover.
How Boilerplate Detection Works
Effective boilerplate detection uses element-pattern classification, not element-type classification. Element type classification says: strip all `<nav>` and `<footer>` elements. This misses boilerplate in `<div>` elements and incorrectly removes navigation that is part of the content (table of contents, in-page navigation). Element pattern classification identifies boilerplate by its structural pattern: text that appears verbatim or near-verbatim across more than a threshold proportion of pages on the domain.
●SiteNexis boilerplate detection compares extracted text blocks across all crawled pages. A text block that appears on 80% or more of pages is classified as boilerplate regardless of its DOM position. This catches text that moves between DOM locations across templates (e.g., a promotional banner that appears in different wrapper elements on different page types).
Boilerplate Ratio and Machine Readability
The boilerplate ratio is the proportion of total extracted text that is classified as boilerplate. A page with 2,000 words of text where 800 words are navigation, footers, and sidebar copy has a 40% boilerplate ratio. This directly reduces the Machine Readability Score: the effective content density is much lower than the raw word count suggests, and the chunks extracted from this page carry proportionally more noise.
AI Visibility Engineering — Part 2 of 10