How SiteNexis Runs 16 Autonomous Agents in a Single Audit
A detailed look at the BullMQ-backed agent architecture that runs a complete AI Retrieval and Machine Trust audit — sequenced across seven phases, with graceful partial failure built into every agent.
When you submit a domain to SiteNexis, you trigger a 16-agent pipeline. These agents do not run as a monolith — they operate as independent, stateless workers communicating exclusively through a BullMQ message bus. No agent calls another directly. No agent knows what other agents are doing. The Infrastructure Agent sequences their execution, monitors their progress, and assembles partial results if any agent fails. The entire audit completes in 3–8 minutes, depending on domain size.
The Seven Execution Phases
Phase 1 runs the Crawl Agent sequentially — everything else depends on having pages to analyze. Phase 2 runs the SEO Agent and Schema Agent in parallel: technical signals first, since later agents consume schema data. Phase 3 runs the AI Retrieval Agent, Entity Agent, and Performance Agent in parallel — all depend on the crawled page set but not on each other. Phase 4 runs Citation and Semantic Trust in parallel, both consuming the entity intelligence output from Phase 3. Phase 5 is the Layer 4 payload: Retrieval Simulation, Machine Trust, Temporal Authority, Recommendation Mapping, and Synthetic Entity Detection run in parallel, all depending on the combined output of Phases 2–4. Phase 6 runs the Visualization Agent to pre-compute graph layouts. Phase 7 is the Reporting Agent, which generates and uploads the PDF report.
●The Infrastructure Agent is the only agent that writes audit.status to the database. Every other agent emits agent:progress and agent:completed events via BullMQ — the Infrastructure Agent listens and aggregates. This constraint prevents conflicting status writes and ensures the audit state machine remains consistent.
The 16 Agents and Their Responsibilities
- Crawl Agent — Full-site crawl, HTML parsing, chunk extraction, entity pre-extraction. Max 500 pages.
- Technical SEO Agent — Title, meta, canonical, robots.txt, sitemap, broken links. Fully programmatic.
- AI Retrieval Agent — Machine readability, chunk quality, AI extractability via Claude API.
- Entity Intelligence Agent — Entity detection, consistency, coverage, disambiguation, Perception Graph.
- Citation Intelligence Agent — Citation probability using weighted formula. No AI API.
- Semantic Trust Agent — Authorship, organisational, content, and structural trust. Contradiction detection on top 20 pages only.
- Schema Agent — Schema detection, validation, field audit, snippet auto-generation.
- Performance Agent — Lighthouse on top 5 pages by PageRank. Never all pages.
- Retrieval Simulation Agent — 6-stage simulation on top 30 pages. Deterministic.
- Machine Trust Agent — Entity credibility, schema alignment, external validation, contradiction, decay signals.
- Temporal Authority Agent — Authority velocity, semantic drift, decay modeling, freshness scoring.
- Recommendation Mapping Agent — Surface coverage across AI Overviews, chat, voice, agent discovery.
- Synthetic Entity Agent — Pattern detection for fake entities, manufactured authority, schema manipulation.
- Visualization Agent — Pre-computes D3 graph layouts. Caches in Redis for 24h.
- Reporting Agent — PDF generation via @react-pdf/renderer, S3 upload, Report record creation.
- Infrastructure Agent — Orchestration, sequencing, error recovery, status management.
Graceful Partial Failure
Every Layer 4 agent implements partial failure handling. If the Claude API fails mid-audit for the Machine Trust Agent's contradiction detection, the agent logs a warning, sets contradictionAbsenceScore to null, and continues with the remaining four sub-scores. The audit does not fail. The partial result is preserved. The Infrastructure Agent marks the audit complete with a partial status flag on the affected module. Ten minutes of compute should not disappear because one API call returned a 429.
The Message Protocol
Every agent emits a standardized AgentMessage: auditId, agentId, event (started | progress | completed | failed), optional payload, optional errorMessage, retryCount, and timestamp. The Infrastructure Agent subscribes to all agent queues and builds a real-time picture of audit progress. SSE events are forwarded to the frontend, where the dashboard streams live progress — phase by phase, agent by agent — without polling.
◆Layer 4 agents gate behind the layer4Analysis plan limit. If a user on the Starter plan submits a domain, phases 1–4 run normally. Phase 5 is skipped entirely — the Infrastructure Agent checks plan limits before enqueuing Layer 4 jobs.
See the full agent pipeline in action on your own domain.
Start free audit