On
Why You Can't Recall AI Like a Defective Drug: The Global Governance Challenge

At a recent AI summit in New Delhi, OpenAI's Sam Altman made a sobering prediction: the first versions of artificial superintelligence could arrive as soon as 2028. He went further, warning that AI could be weaponized to engineer new pathogens—and that democratic societies need to act now before the technology they create outpaces their ability to control it.

Altman isn't alone in sounding the alarm. Geoffrey Hinton, widely regarded as the godfather of AI, has repeatedly cautioned that developing machine intelligence surpassing human capability could represent a genuine existential threat.

Mustafa Suleyman makes a similar case in his book The Coming Wave, arguing that when AI combines with synthetic biology, a single individual could potentially engineer a lethal pandemic.

This isn't some distant hypothetical anymore. Just last week, a clash over who controls AI and under what conditions caused a major rift between a tech company and the U.S. Department of Defense.

Why People Keep Comparing AI to Pharmaceuticals

As politicians and business leaders grapple with these challenges, they naturally look for precedent. Many are turning to the pharmaceutical industry as a governance model.

U.S. Senator Richard Blumenthal, one of the few lawmakers actively pushing for AI regulation, has suggested that the way the FDA oversees drugs could serve as a template for AI oversight.

On the surface, the comparison makes sense. The pharma industry demonstrates that rigorous licensing and strict monitoring can manage dangerous technologies without stifling innovation.

Tech companies are already operating this way—often without realizing it. They manage AI risk through familiar steps: pre-deployment testing, staged rollouts, and post-launch monitoring. In other words, the pharma playbook has become the default governance framework across much of the AI industry.

Here's the problem: it's the wrong framework. And the difference isn't just technical—it's fundamental.

The pharma model works because three foundational conditions exist: high barriers to entry, physical products, and slow development cycles. AI has none of these.

1. Barriers to Entry

Bringing a new drug to market costs roughly $1.1 billion, according to a 2020 study in the Journal of the American Medical Association.

The labs, clinical trials, and manufacturing plants mean only a handful of players can compete—and regulators can easily track them.

AI is completely different. A powerful model can be built at a fraction of that cost, fine-tuned on consumer hardware, and deployed globally from a laptop.

This means regulators aren't watching a few companies. They're trying to monitor essentially anyone, anywhere.

2. The Physical Nature of Products

Drugs are physical goods. Manufacturing and distribution require materials, equipment, and logistics—natural checkpoints for regulators.

AI models? Entirely different.

Once released, a model's weights can be duplicated perfectly and distributed globally in minutes, at near-zero cost.

More critically, you can't issue a recall for software the way you can for a defective batch of medicine. Once it's on the internet, it's there forever.

Even cloud-based models aren't bulletproof. Anthropic recently revealed that three Chinese AI labs—DeepSeek, Moonshot AI, and MiniMax—used 24,000 accounts to generate over 16 million interactions with Claude to extract model capabilities through a technique called "distillation."

They didn't need to breach supply chains or build factories. Just API access and cleverly crafted prompts.

There's no pharmaceutical equivalent.

3. Development Speed

Drug approval typically takes years. AI evolves at software speed.

Model capabilities improve not just from hardware upgrades, but also from:

  • novel training methods
  • software updates
  • continuous version releases

Anthropic released two major Claude versions in just 10 weeks.

By the time a pharma-style approval process finishes evaluating a model, that model is already obsolete—replaced by a more powerful version.

Why "Test-Deploy-Monitor" Falls Short

The pharmaceutical mindset isn't limited to government. It's baked into corporate culture too.

In pharma, the familiar risk is side effects. Test first, release second, monitor afterward, recall if problems emerge.

Many organizations are applying the same logic to AI. It sounds responsible. But it creates a false sense of security.

Testing and staged rollouts have real value. They catch bugs, enforce operational discipline, and demonstrate due diligence to boards.

But they only address familiar risks like product defects or technical failures.

AI risks operate on a completely different level. They involve irreversibility, rapid propagation, and abuse potential.

Unlike a defective product, when AI causes serious harm, you can't issue a recall notice.

A New Framework for AI Risk Management

To address this gap, researchers propose the CARE governance framework.

CARE consists of four steps:

  • Catastrophize—identify worst-case scenarios
  • Assess—evaluate actual risk levels
  • Regulate—establish control measures
  • Exit—plan for when controls fail

Applied to business, this framework suggests several concrete actions.

First, leaders need to identify "shadow AI"—the tools employees are using that the company didn't authorize.

Next: pinpoint irreversible decision points. Examples include automated customer emails, AI code deployed directly to production systems, or algorithm-driven hiring platforms.

Organizations must also control data flows carefully. Every AI tool is essentially a two-way data pipeline. When proprietary data enters a third-party system, it's nearly impossible to fully retrieve.

Red team exercises matter too—but not just for finding technical bugs. Test for abuse scenarios.

Finally, one executive should own AI risk, the way a CFO owns financial risk.

For decades, the pharma model represented one of governance's greatest successes: protecting the public while enabling innovation.

But with AI, that model isn't enough.

Altman's proposal at the New Delhi summit—creating an international AI agency modeled on the IAEA—reflects a more realistic view. AI needs oversight mechanisms proportional to actual risk, not borrowed frameworks from entirely different industries.

Business leaders should think similarly. The governance problems governments face internationally mirror the challenges inside your organization right now.

So design your systems around the technology you're actually using—not the technology you wish it resembled.

Related Articles