Google says llms.txt not required for AI SEO but Lighthouse still

Google’s John Mueller told the Search Off the Record podcast that LLMs cannot rely on an llms.txt file to decide which site to surface for a query, underscoring that the file offers no ranking advantage and is only useful once a user‑agent is already on the site [1]. This clarification comes as Google’s official AI‑search guidance, updated May 15 2026, reiterates that sites do not need llms.txt to appear in AI Overviews or AI Mode, while Chrome’s Lighthouse tool now audits the file under its experimental Agentic Browsing category [2].

Mueller’s stance on discovery

Mueller argued that discovery still hinges on traditional HTML pages and internal links, describing llms.txt as a “dead end” for LLM‑driven site selection because the file is self‑reported and offers no differentiating signal [1]. He likened the situation to the historic decline of meta‑keywords, which lost value once every site began stuffing them, leaving search engines without a reliable ranking cue [1]. The only scenario he endorsed for llms.txt is post‑discovery navigation—e.g., an AI agent that has already landed on a site could use the file as a “store directory” to locate a purchase flow [1].

Lighthouse audit and agentic browsing

Google Search’s guidance confirms that no new machine‑readable files, including llms.txt, are required for a page to be eligible for AI Overviews or AI Mode; the core criteria remain indexability, quality, and relevance [2]. However, Chrome’s Lighthouse now includes an llms.txt audit within its Agentic Browsing checks, treating the file as an optional orientation aid for software agents that need to understand a site’s purpose and navigation [2]. A missing file returns a “Not Applicable” status, while server errors are flagged as technical problems—indicating that the audit is about agent readiness, not a ranking penalty [2].

What to watch

• Lighthouse updates – Monitor future releases of the Agentic Browsing category for any shift from “optional” to “recommended” status for llms.txt.
• Industry adoption – Track how many high‑traffic sites begin publishing llms.txt files and whether third‑party AI crawlers start to reference them.
• Google’s next guidance – Watch for any clarification from the Search Relations team on whether llms.txt could ever influence discovery or ranking signals.

The split between Google Search’s “no‑file‑needed” stance and Lighthouse’s optional audit highlights a growing dual audience for web content: traditional crawlers that rank pages and emerging agents that need clear, machine‑readable site summaries. Whether llms.txt evolves from a niche navigation aid to a broader SEO signal remains an open question.