California governor vetoes landmark AI bill that would have mandated government oversight of powerful frontier models

California governor Gavin Newsom has vetoed a state artificial intelligence (AI) safety bill that was seen as a landmark that could have had global significance in regulating Silicon Valley’s development of the technology.

The bill, which had been passed nearly unanimously in both houses of the state legislature, required safety testing of the most advanced AI systems, called frontier models, before they could be released to the public.

It gave the state’s attorney general the right to sue companies over serious harm caused by their technologies, such as death or property damage, and mandated a kill switch to turn off AI systems in extreme circumstances.

Newsom faced heavy pressure to veto the bill from tech companies as well as influential members of his own party.

Risks

He said the bill was “well-intentioned” but focused too much on regulating advanced models without ways of measuring risk or harms.

The bill “does not take into account whether an AI system is deployed in high-risk environments, involves critical decision making or the use of sensitive data”, Newsom said.

He said it should be rewritten with more input from AI experts in academia and business leaders for a science-backed analysis of potential risks of frontier models.

With federal legislation having made no meaningful progress, the California bill would have been one of the US’ first major steps toward regulating AI, similar to moves that have been undertaken in the EU and China.

The EU’s AI Act became law this year after lengthy negotiations, which included paring back controls on frontier models.

The final law includes mandatory self-regulation of frontier models through codes of conduct, rather than formal regulation, a position that had been supported by France, Germany and Italy over concerns that too much oversight of advanced models could affect the bloc’s competitiveness in the emerging industry.

Oversight

US tech companies and researchers voiced similar concerns over the California bill, with Li Feifei, known as the “godmother of AI” and co-founder of AI start-up World Labs, saying in an August opinion piece that the bill would “harm our budding AI ecosystem”.

OpenAI, Google, Meta and Microsoft also said the bill would restrict innovation.

Some 50 other academics, including “godfather of AI” Geoffrey Hinton, wrote to Newsom that the act was needed to ensure a responsible approach to the technology’s development.

“Decisions about whether to release future powerful AI models should not be taken lightly, and they should not be made purely by companies that don’t face any accountability for their actions,” the academics wrote.

Senator Scott Wiener, who authored the bill, said the veto leaves companies free to develop “extremely powerful technology” with no government oversight.