ACX Open Thread Comment, Dec 24 2023

Estimates for AI-Extinction of >90%¹ are unreasonable because it would mean that every scenario in its complement is <10%. If one were to take aggressive, conservative priors for the path to PCM (paperclip maximizer), at least a few of them would be greater than 10% on their own², or at the very least, some combination would sum up to more than 10%.

I. Precondition failure

1. Moore's Law plateaus
2. the scaling hypothesis fails
3. appropriate warning shots arrive
4. controls arrive without warning shots
5. AI-safety theorists succeed

II. Known doubts about PCMs come true

6. instrumental convergence doesn't work as planned
7. the orthogonality hypothesis doesn't pan out
8. (I'm not an expert on PCMs, but those who are can fill in these blanks)
9. ...
10. ...

III. And of course, there are several unknown unknowns about PCMs, given how theoretical they are

11. ...
12. ...
13. ...
14. ...
15. ...

I forget who, but some professor who is a hero of this community, went back and calculated that estimates for atmospheric ignition pre-Trinity Test didn't appropriately factor in the probability of getting the maths/physics wrong (i.e., unknown unknowns) and that estimates should have been much higher. Meanwhile, the PCM story has less empirical support than atmospheric ignition, so the same "unknown unknowns" concern should also apply.

How is Eliezer's forecast for PCMs not an example of bad faith or a failure of imagination³?

---

[1]: https://www.astralcodexten.com/p/why-i-am-not-as-much-of-a-doomer

[2]: As an example, even a reasonable, well-informed Moore's Law hawk would concede a ~10% chance that it stops short somewhere. And by well-informed, I mean someone knowledgeable about Rock's Law, basic physics, etc.

[3]: It doesn't take much creativity to come up with scenarios where the PCM story comes off the rails: https://www.astralcodexten.com/p/open-thread-305/comment/44770931