Extinction Bounties

Policy-based deterrence for the 21st century.

Policy-Research Disclaimer (click to close)

Extinction Bounties publishes theoretical economic and legal mechanisms intended to stimulate scholarly and public debate on catastrophic-risk governance. The site offers policy analysis and advocacy only in the sense of outlining possible legislative or contractual frameworks.

  • No Legal or Financial Advice. Nothing here should be treated as a substitute for qualified legal counsel, financial due-diligence, or regulatory guidance. Stakeholders remain responsible for ensuring their actions comply with the laws and professional standards of their own jurisdictions.
  • Exploratory & Personal Views. All scenarios, numerical examples and opinions are research hypotheses presented by the author in an academic capacity. They do not represent the views of the author’s employer, funding bodies, or any governmental authority.
  • Implementation Caveats. Any real-world adoption of these ideas would require democratic deliberation, statutory authority, and robust safeguards to prevent misuse. References to enforcement, penalties, or “bounties” are illustrative models, not instructions or invitations to engage in private policing or unlawful conduct.
  • No Warranty & Limited Liability. Content is provided “as is” without warranty of completeness or accuracy; the author disclaims liability for losses arising from reliance on this material.

By continuing beyond this notice you acknowledge that you have read, understood, and accepted these conditions.

Our 2-minute elevator pitch

[alpha] How large should an extinction bounty be?

TL;DR

A lower bound on the harm of risking human extinction

Robin Hanson’s original fine-insured-bounty (FIB) proposal says:
Fine = expected social harm ÷ detection probability.

When the “harm” is human extinction, though, this formula goes “kaboom” even for very low chances of it happening. A simple formula that gives us a lower bound:
Expected social harm = (cost of a human live) * (number of humans currently alive) * (chance of extinction).

Some stats:

  1. Value of a Statistical Life (VSL).
    Major US agencies cluster between about $7 – 14 million per life:

    • EPA keeps a central $7.4 M (2006 dollars, ≈$10 M today).
    • FEMA’s 2023 Standard Values table uses $12.5 M.
    • USDOT’s 2025 guidance pegs VSL at $13.7 M (base-year 2024).
    • Let’s use $10 million as a nice, easy-to-work with round number.

  2. How many lives?
    As of April 2025, this planet is home to 8.2 billion people.

  3. Expected social harm by chance of extinction event:

    • 1‑in‑1,000,000 ( 1 × 10⁻⁶ ) → ≈ 8.2 × 10¹⁰ USD — about $82 billion
    • 1‑in‑100,000 ( 1 × 10⁻⁵ ) → ≈ 8.2 × 10¹¹ USD — about $820 billion
    • 1‑in‑10,000 ( 1 × 10⁻⁴ ) → ≈ 8.2 × 10¹² USD — about $8.2 trillion
    • 1‑in‑1,000  ( 1 × 10⁻³ ) → ≈ 8.2 × 10¹³ USD — about $82 trillion
    • 1‑in‑100  ( 1 × 10⁻² ) → ≈ 8.2 × 10¹⁴ USD — about $820 trillion
    • 1‑in‑10   ( 1 × 10⁻¹ ) → ≈ 8.2 × 10¹⁵ USD — about $8.2 quadrillion

And that ignores the value of any future generations lost!

These numbers are far, far too large to impose on people, or even entire companies, as credible bounties, even if that’s what falls out of the math. Such is the nature of such gambling with, well, everything.

Deterrence shouldn’t need cosmic-scale checks

Let’s do a reality check.

I honestly do not think that a smart young teenager, even an IMO gold medalist, would look at a fine of say “$1 million dollars if caught developing new frontier AI models, payable to the bounty hunter” and think to themselves “That’s far below the $82 billion floor my P(doom) expects such a fine should be. I should go do that and just make $1 million fast enough every day that I can afford to pay it as many times as I get caught, which will probably be daily.” That is an insane way to make career decisions.

For the purposes of deterrence, fines far lower than even the 1-in-a-million lower bound social harm estimate should get us the deterrence we are after and then some. These people have many, many other options for jobs they could take and live full, rich lives with. And these crimes are like, the opposite of crimes of passion - the only way someone could possibly even get to the point where they might be facing down the barrel of an extinction bounty is by making targeted career decisions, almost certainly pursuing tertiary education up through the PhD level, etc., etc. Smart, conscientious, capable people with a lot of options simply wouldn’t chooe the one that is abundantly likely to bankrupt them a few weeks into their first job at best and send them to jail for their working lives as well as bankrupt them on average.

The importance of no refractory period

I allude to this in the previous section, but: It’s absolutely possible someone tries to “tank” a $1 million fine if they think that buys them months or years of runway to continue working on their AI in secret. If on the other hand we allow bounty hunters to name a date or time period during which the actiosn took place, then other bounty hunters (more likely the same group since they already have collected all the evidence) can just file for another fine if the shop keeps its doors open even after that. In the extreme case this could turn into a $1 million fine, per person, per day situation, which is definitely enough to strike terror into even the most deep-pocketed of backers.