How do you know when AI is powerful enough to be dangerous? Regulators try to do the math

How do you know if an artificial intelligence system is so powerful that it poses a security danger and shouldn't be unleashed without careful oversight?

For regulators trying to put guardrails on AI, it's mostly about the arithmetic. Specifically, an AI model trained on 10 to the 26th floating-point operations per second must now be reported to the U.S. government and could soon trigger even stricter requirements in California.

Say what? Well, if you're counting the zeroes, that's 100,000,000,000,000,000,000,000,000, or 100 septillion, calculations each second, using a measure known as flops.

What it signals to some lawmakers and AI safety advocates is a level of computing power that might enable rapidly advancing AI technology to create or proliferate weapons of mass destruction, or conduct catastrophic cyberattacks.

Those who've crafted such regulations acknowledge they are an imperfect starting point to distinguish today's highest-performing generative AI systems — largely made by California-based companies like Anthropic, Google, Meta Platforms and ChatGPT-maker OpenAI — from the next generation that could be even more powerful.

Critics have pounced on the thresholds as arbitrary — an attempt by governments to regulate math.

“Ten to the 26th flops,” said venture capitalist Ben Horowitz on a podcast this summer. “Well, what if that’s the size of the model you need to, like, cure cancer?”

An executive order signed by President Joe Biden last year relies on that threshold. So does California's newly passed AI safety legislation — which Gov. Gavin Newsom has until Sept. 30 to sign into law or veto. California adds a second metric to the equation: regulated AI models must also cost at least $100 million to build.

Fol