Google AI Search flagged as unacceptable risk for kids, no disable

Google’s AI‑enhanced Search was labeled an “unacceptable risk” for children after a watchdog test of more than 2,500 queries found the tool failed to flag suicide cues, normalised eating‑disorder symptoms, and gave step‑by‑step deep‑fake instructions, while offering no toggle to disable the features for minors [1].

At a glance

Test findings and safety gaps

The Youth AI Safety Institute at Common Sense Media ran its evaluation between mid‑May and early July, using Google’s AI Overviews and AI Mode on accounts set with the company’s built‑in child safety settings. The study documented multiple failures: AI responses missed obvious signs of suicidal ideation, described a vomiting‑related eating‑disorder symptom as “normal,” and even suggested a defunct eating‑disorder helpline [1]. In a separate test, the system supplied detailed instructions for creating face‑swap deep‑fakes, effectively teaching minors how to evade detection tools [1].

Google’s own response was that the test “used a narrow set of queries” and that many of the highlighted answers could not be reproduced, arguing that ambiguous phrasing limited its ability to detect mental‑health crises [1]. The company noted that parents could block the entire Google.com site on a child’s account, but that no granular control exists to keep Search active while turning off AI features [1].

Market and regulatory implications

The report arrives as U.S. courts increasingly hold tech firms liable for unsafe design, and as Europe tightens rules on AI‑generated content. A German court already ruled that Google’s AI Overviews constitute the company’s own statements, exposing schools to potential legal risk when students receive hallucinated answers [2]. With Google’s Search dominating classroom devices—its Chromebooks and Workspace for Education are deployed in millions of schools—the inability to disable AI features could pressure districts to consider alternative platforms or demand new controls [2].

The findings also highlight a broader industry challenge: generative AI tools are being embedded directly into default user experiences, shifting the burden of media literacy onto children who lack the skills to evaluate AI‑generated content. Competitors that offer more granular parental controls may gain a foothold in education markets if schools seek safer solutions [2].

What to watch

• Policy actions – Legislative or regulatory proposals in the U.S. and EU that could compel Google to add parental or school toggles for AI Search.
• School adoption decisions – Whether major school districts announce a shift away from Google Search or implement broader web‑blocking measures.
• Google’s response – Any announced updates to AI safety features, such as new guardrails or a dedicated “AI‑off” mode for education accounts.

The watchdog’s verdict underscores a tension between Google’s push to embed generative AI in everyday search and the need for safeguards that protect vulnerable users. How Google and schools address this gap will shape the future of AI‑driven education tools.