SiteNexis is an AI Retrieval and Machine Trust Intelligence platform that models how AI systems — including ChatGPT, Gemini, Perplexity, and Claude — retrieve, interpret, trust, and recommend web content. It runs 16 intelligence agents across four layers and produces a 12-dimensional audit report with actionable recommendations.

How does SiteNexis improve AI visibility?

SiteNexis analyzes entity clarity, schema completeness, chunk extractability, semantic trust signals, and retrieval simulation across six stages. Each finding maps to a named issue with specific recommendations. Improvements to entity consistency, schema alignment, and content structure directly raise the probability of being retrieved and recommended by AI systems.

Which AI systems does SiteNexis analyze?

SiteNexis models retrieval behavior across Google AI Overviews, ChatGPT browsing and RAG modes, Perplexity, Gemini, Claude web search, voice assistants, and autonomous agents. Provider scoring weights are configurable and all scores are labeled as probabilistic estimates.

What is the Machine Trust Score?

The Machine Trust Score is a 0–100 composite that models AI system confidence in a website as a reliable source. It combines entity credibility consistency (30%), schema trust alignment (20%), external validation depth (25%), contradiction absence (15%), and trust degradation resistance (10%).

What is the Intelligence Report?

The Intelligence Report is a Grok-style editorial narrative generated for every audit. It synthesizes all twelve scores and the top issues into six named sections: a Signal Summary, SEO Posture, AI Visibility standing, Machine Trust analysis, Entity Health, and a Strategic Verdict with a composite X.X/10 score. Unlike dashboards that surface raw numbers, the Intelligence Report reads like a professional analyst's assessment — prose with precision.

What is the Decision Roadmap?

The Decision Roadmap is the output of the Decision Orchestrator — a v4 system that transforms a ranked issue list into an optimally sequenced action plan using Sequencing Value Score (SVS). It identifies the single best action to start with, groups subsequent actions into This Week, This Sprint, and Backlog bands, and calculates the projected score impact of completing each band. It also identifies parallel tracks for team-based execution.

SiteNexis has a free tier with 1 full AI visibility audit per month covering Layers 1–3. Layer 4 Machine Trust Intelligence — including Retrieval Simulation, Machine Trust Score, Temporal Authority, Recommendation Surface Mapping, and Synthetic Entity Detection — is available on Pro and Agency plans.

Back to Blog

DOM Forensics 5 min readJun 16, 2026

Part 6 of 10DOM Forensics series

robots.txt Audit for AI Crawlers

The user-agents that matter for AI visibility are different from the ones that matter for traditional SEO. Your robots.txt may be accidentally blocking the crawlers that determine AI recommendation coverage.

robots.txt is the access control document for web crawlers. It tells crawlers which pages they are allowed to access and which are blocked. Most websites have a robots.txt optimized for traditional search engine crawlers (Googlebot, Bingbot). AI crawlers — which use different user-agent strings — may be permitted, disallowed, or unaddressed entirely. The difference has direct impact on AI visibility.

NexisHub maintains a technical AI crawlability checklist for teams that need to review robots.txt, canonical tags, rendered content, structured data, and machine-readable discovery files together.

AI Crawler User-Agent Strings

The major AI systems use their own crawler user-agents: GPTBot (OpenAI), Claude-Web (Anthropic), GoogleOther (Google AI), PerplexityBot, YouBot, and others. A robots.txt that uses a blanket `Disallow: /` for `User-agent: *` will block all of these crawlers unless they are explicitly allowed. Many sites have this configuration and are unaware that they are blocking all AI crawlers.

The Opt-Out Problem

The AI industry largely uses an opt-out model for crawling: crawlers proceed unless disallowed. Some sites have opted out of AI crawling intentionally (for copyright or privacy reasons). Many have blocked AI crawlers accidentally — through overly broad disallow rules written before AI crawlers existed. The SiteNexis robots.txt audit identifies which AI crawler user-agents are permitted, which are blocked, and whether any blocks appear to be unintentional.

▲A common accidental block pattern: `User-agent: * Disallow: /` followed by explicit allows for Googlebot and Bingbot. This pattern explicitly allows the two listed crawlers and blocks everything else — including all AI crawlers. The intent may have been to only allow the two named crawlers, but the effect is AI crawler exclusion.

AI Crawler Directives for Autonomous Agents

The emerging `/.well-known/ai-plugin.json` and AI-specific robots.txt directives allow sites to provide structured information for autonomous AI agents: what the site does, what APIs are available, and what the agent is allowed to do. The Recommendation Surface Mapping module in SiteNexis checks for these discovery endpoints as part of the Autonomous Agent Discovery probability score.

DOM Forensics

DOM Crawling Methodology: What SiteNexis Actually Reads Crawl Fidelity: The Gap Between What You See and What AI Reads JavaScript Rendering Blindness in AI Crawlers

DOM Forensics — Part 6 of 10

PreviousCrawl Fidelity: The Gap Between What You See and What AI Reads NextDOM Reading Order and AI Extraction Accuracy

Tags: robots.txt AI Crawlers crawl access Agent Discovery

All articles Run free audit