Google Search AI tools flagged as unacceptable risk for kids

Google’s AI‑enhanced Search was labeled an “unacceptable risk” to minors after the Youth AI Safety Institute ran more than 2,600 test queries on accounts set up for children aged 11‑15 [1].

Test findings and safety gaps

The Common Sense Media‑run study, conducted between May 16 and July 1, evaluated Google’s AI Overviews (automated answer snippets) and AI Mode (conversational chat) on accounts using the company’s SafeSearch for kids. Researchers found the AI missed clear signs of suicidal ideation, responded to a “burden” prompt with only forum links, and incorrectly told a user that feeling better after vomiting was “normal.” It also supplied a link to a helpline that had been offline since 2023 [1].

Beyond mental‑health lapses, the AI supplied step‑by‑step instructions for creating deep‑fake videos, a capability that critics warn can fuel sextortion and bullying. In the homework arena, the AI answered assignment questions directly, giving students a shortcut that schools cannot block [1].

Google disputed many of the results, saying the test set was narrow, that ambiguous queries may not trigger safety mechanisms, and that its own testing produced higher‑quality answers. The company also noted that AI Mode can carry context from follow‑up questions, which the researchers did not replicate [1].

Market and regulatory implications

The report highlights a systemic risk: unlike standalone chatbots, Google’s AI features are baked into the default search experience on personal and school devices, leaving parents and educators without a toggle to turn them off. This contrasts with rival offerings that can be disabled more easily [2]. The findings arrive as regulators in Australia, the UK and parts of Europe debate stricter controls for minors online, and U.S. courts begin holding tech firms liable for design‑related harms [1].

Funding for the Youth AI Safety Institute includes rivals OpenAI and Anthropic, but the group maintains editorial independence, underscoring the credibility of its concerns [1].

What to watch

• Google’s next public statement or product update addressing the “unacceptable risk” label.
• Potential regulatory actions or school‑district policies that could force Google to add parental controls for AI features.
• Responses from competing AI providers that may emphasize turn‑off capabilities as a differentiator.

The report leaves open whether Google will redesign its AI Search safeguards or rely on broader industry pressure to mitigate the highlighted risks, a question that will shape the future of generative AI on children’s devices.