I wrote our proposal back when GPT-4 was released. It has been updated over time, but it deserves more consideration. Our proposal is the core of what we want to achieve. It is the most important document that we have, and it needs to be good.
I've also hear people say things in the line of "we're an activist group - we're not in the business of writing policy", but I think we should have a well-formulated and executable ask.
What does a it mean for the proposal to be good? I can think of a couple of aspects:
- Prevents AI catastrophe. Any proposal that does not significantly reduces the chance of AI catastrophe does not suffice.
- Clear. Non-vague. Easy to understand. No jargon unless necessary.
- Implementable. You can give it to willing leader and it becomes policy.
- Compact. The longer the document is, the less people will read it. It can be information dense, but not overly wordy.
What can we improve
@orpheuslummis sent an email and listed some criticisms on it:
- Our benchmarks (e.g. MMLU) & numbers (flop counts) are outdated, and algorithmic efficiency keeps pushing capabilities towards lower hardware thresholds. Hardware thresholds may not prevent AI catastrophe.
- We should consider a tiered system / staged implementation / easier pathway to join. Going straight for the big ask could be too much
- "Guaranteed safety" is perhaps too vague.
- No escalation framework.
- "IAEA-like agency" is too vague, role of AISI not discussed
- It ignores inference scaling and open weights. o-series models circumvent training FLOP thresholds. Open weights make post-release control difficult. The proposal doesn't mention either.
- Domain-specific dangers ("jagged frontier"): the proposal exempts "narrow AI systems", but optimized bio-design or cyber-offense model are dangerous below reasonable compute threshold.
- Limiting publication of algorithms and limiting hardware capability advances may be weak and difficult. MIRI's technique whitelist for above-threshold training may be a better answer.
Literature & other proposals
How will we work towards a new proposal
- We discuss the merits of changes here, in public. Please post comments, criticisms and other relevant thoughts below.
- I think I should take the lead on drafting a next version, and work towards something concrete. That will probably be a PR to our existing proposal. I will mention that here.
- I don't think my personal opinion on this will determine what we'll get, because I'm no longer CEO of PauseAI (@Maximophone is).
I wrote our proposal back when GPT-4 was released. It has been updated over time, but it deserves more consideration. Our proposal is the core of what we want to achieve. It is the most important document that we have, and it needs to be good.
I've also hear people say things in the line of "we're an activist group - we're not in the business of writing policy", but I think we should have a well-formulated and executable ask.
What does a it mean for the proposal to be good? I can think of a couple of aspects:
What can we improve
@orpheuslummis sent an email and listed some criticisms on it:
Literature & other proposals
How will we work towards a new proposal