Information Gain Engine: What Your Content Adds Beyond the SERP Cohort
The Information Gain Engine measures what your page contributes that the existing top-ranking pages do not already cover. It is the only metric that answers: does this page deserve to exist?
Every piece of content on the web is evaluated by AI retrieval systems against competing content. A page that repeats what the top 10 SERP results already say provides no information gain — the AI system already has that information from higher-authority sources. A page that introduces unique entities, answers questions the SERP cohort does not address, or provides evidence not available elsewhere contributes measurable information gain. SiteNexis measures this delta with the Information Gain Engine.
The Ten-Phase IGE Pipeline
The IGE runs ten sequential phases. First: the Serper API fetches the top organic results for the page's primary keyword. Second: concurrent crawling collects body text from each SERP result. Third: entity extraction identifies the named entities discussed across the cohort. Fourth: question extraction identifies the questions the cohort answers. Fifth: evidence extraction identifies specific data, statistics, and citations used. Sixth: entity gap detection identifies entities the target page covers that the cohort does not. Seventh: question gap detection identifies questions the target page answers that the cohort does not. Eighth: shared knowledge detection measures overlap between the target page and the cohort. Ninth: retrieval value estimation scores the marginal value of the target page given what the cohort already provides. Tenth: the IGE Score is computed.
Entity Gap Detection
Entity gaps are the most actionable IGE finding. If the SERP cohort covers entities A, B, C, and D, and your page covers A, B, and E — entity E is a gap you fill. But the inverse is also important: if your page covers only A and B while the cohort covers A through F, your entity coverage is thin and your page adds very little. The entity gap detector runs regex and structural pattern matching across both the target page and the cohort to identify these asymmetries.
◆High entity gap score (your page covers entities the cohort does not) is the strongest predictor of IGE score. Focus content expansion on entities that are semantically related to the primary topic but absent from current top-ranking pages.
Question Gap Detection
AI systems are trained to answer questions. A page that answers questions the SERP cohort does not answer provides clear information gain for question-triggered retrieval. The question gap detector extracts implied and explicit questions from all sources — phrased as interrogatives, or implied by headings and structural patterns. Questions answered by the target page but not by the cohort are scored as high-value information gain signals.
The IGE Score
The final Information Gain Score is a 0–100 composite of entity gain, question gain, evidence uniqueness, and shared knowledge overlap (inverted — more overlap reduces the score). A page with score below 40 is adding little beyond what existing sources cover. A page above 70 has genuine unique value that AI retrieval systems will recognize as differentiating. The IGE Score feeds into Citation Probability — pages that add information are more likely to be cited.