Better Alignment Exercises
Created Thursday 03 August 2023


2) Existing Problem Sets
	2a) MATS Problems
		2a1) Vivek Hebbar's MATS Problems
		https://docs.google.com/document/d/1Hl57Oxf7gb0x-3KreMTJwxPoVWEqcd6ybBXPKpZUkVI/edit
		https://docs.google.com/document/d/1NVVtdsfz7HiseVFSk3jYly4sPG4dG03wFFDrD8rBXU0/edit
		2a2) Janus Mats questions
		https://docs.google.com/document/d/1g0s_QipWHIQxLDlC_q0sM_D-PdiJ__0PDLiz2_ALfyk/edit
	2b) Richard Ngo Alignment Research Exercises
	https://www.lesswrong.com/posts/kj37Hzb2MsALwLqWt/alignment-research-exercises
	2c) AGI Safety Fudamentals Curriculum
	(Out-dated) https://forum.effectivealtruism.org/posts/BpAKCeGMtQqqty9ZJ/agi-safety-fundamentals-curriculum-and-application
	2023 course (https://course.aisafetyfundamentals.com/alignment)
	2d) Some Conceptual Alignment Research Projects
	2e) Going through Zvi's Crux List at the outer level of questions: https://www.lesswrong.com/posts/agv26XfXfKfKiKwDm/the-crux-list
		2e1) The outer level is
		What worlds count as catastrophic versus non-catastrophic?
		What would or would not count as a catastrophe caused by losing control over AGIs?
		If AGIs control the future, what would it take to make this non-catastrophic?
		Is it possible that most meaningfully capable AGIs could be outside of human control, yet humans continue to control the future?
		How difficult would uncontrolled AGIs be to shut down? Would we do it?
		What pivotal acts are available to prevent the construction of additional AGIs?
		Could AI systems increasingly take control even without being AGIs?
		What comparative advantages if any might humans retain over AGIs?
		Is intelligence a big deal?
		What is physically possible to achieve with greater intelligence, with or without iteration and experiment and time?
		Will we be able to achieve human cognitive enhancement in time to matter?
		To what extent will it be a competitive advantage to have less control over AGIs?
		To what extent would an AGI on the internet be able to seek resources and power?
		At what stage of development does AI become an AGI with what probability? If it does become an AGI, at what point does it have dangerous affordances? At what point in training, testing and deployment are you at risk of becoming doomed later down the line, whether or not you yet have an AGI? (Note that if AGI ends up being trained by a substantially different process, these questions might become ill-formed).
		What kind of architecture will the first AGIs have?
		How fast or slow a takeoff should we expect? What does that imply?
		Alignment questions and difficulties. There are so many alignment questions, anything listed here would only be a sampling, even if one stays within what we have of a paradigm – this is an especially non-exhaustive list.
		Will AGIs necessarily be agents in various senses?
		General questions about human decision making, values, civilizational capacity, cognitive abilities, coordination mechanisms.
		What is the proper mode of reasoning for thinking about what might happen in the future?
		What are the most useful and relevant intuition pumps and metaphors?
		General questions about our ability to impact the future.
		How does this interact with concerns of value drift, change and interstellar travel?
		Is it possible for people to have widespread access to AGIs under their personal control, without having the ability to set that AGI free from their control?
		Timelines. People have different timelines, timelines have implications for chances of good outcomes by interacting in various ways with different dynamics here.
		Do we have a meaningful agent overhang, or other important overhang in ability to convert a base model into an AGI, and is this likely to continue?
		Warning shots.
		What should we expect in terms of national regulation?
		What should we expect in terms of global coordination and regulation?
		Lab, government and other actions in the endgame when an AGI is close, either in a pending transition to when trained base models become capable of being turned into AGIs, or when this is already the case or AGI is close by some other method. What dynamics and outcomes are likely to be in play? What will people choose to do? There is lots of disagreement here, this section is not meant to be complete or SOTA, only a sketch of some potential sub-questions, even more so than elsewhere in this list.
			2e1a) IDEA Go through Zvi's crux list and think of an exercise for each crux.
			5d

	2f) Deep Learning Curriculum - AI Alignment by Jacob Hilton
	2g) Seri Mats Curriciulm, though I can't find it.
	2h) Turntrout most frequently includes exercises in his posts on alignment. See his https://www.lesswrong.com/posts/xCxeBSHqMEaP3jDvY/reframing-impact seequence.


3) Existing AI safety Courses
	3a) https://users.cs.utah.edu/~dsbrown/cs6960.html
	3b) https://inst.eecs.berkeley.edu//~cs294-149/fa18/
	3c) https://dorsa.fyi/cs521/
4) What exercises actually caused you to change your mind about alignment?
	4a) Trying to create a corribile utility function, though that was over an extended period of time.
	4b) Looking at the type of agency and thinking of what theorems you could prove about that which would solve AI safety w/o disguising the hard parts of the problem
	4c) Specifying an outer reward function and breaking it iteratively.
	4d) Reading about how little we understand infinite-compute intelligences. See this paper https://arxiv.org/abs/1510.04931 and what I've learnt about TDT bots for Prisonner's Dilemma.
	4e) Thinking about what worlds extremize various utility functions, and what worlds extremize human utility functions.
		4e1) Thinking about how much the world is optimized for humans already.




5) Exercise Ideas
	5a) I've been thinking about exercises for alignment, and I think going through a list of lethalities and applying them to an alignment propsal would be a good one. Doing the same with Paul's list would be a bonus challenge. If I had some pre-written answer sheet for one proposal, I could try the exercise my self to see how useful it would be. This post, which I haven't read yet, looks like it would serve for the case of RLHF. I'll try it tomorrow and report back here.
		5a1) Could do this with ELK and look at the prize-winning submissions.
	5b) Any builder breaker setup
	5c) Tuning your ontology for alignment feels like it would be helpful. See the comment on tuning ontologies on this LW 	post: https://www.lesswrong.com/posts/bbB4pvAQdpGrgGvXH/?commentId=L39KgqRasbRFvJYHK
	5d) 2e1a
	Search for alignment cruxes and create exercises which help you decide what side of the crux you fall on.