List of disagreements in AI safety - Revision history

Issa: /* Probability of doom */

2022-02-26T11:00:44Z

‎Probability of doom

Issa at 23:37, 18 June 2021

2021-06-18T23:37:15Z

Issa: /* Takeoff dynamics */

2021-02-24T01:37:20Z

‎Takeoff dynamics

Issa: /* Meta-philosophy */

2020-12-01T01:03:07Z

‎Meta-philosophy

Issa: /* Takeoff dynamics */

2020-07-06T21:35:28Z

‎Takeoff dynamics

Issa: /* Probability of doom */

2020-07-06T21:27:55Z

‎Probability of doom

Issa: /* AI timelines */

2020-07-06T21:23:49Z

‎AI timelines

Issa at 21:21, 6 July 2020

2020-07-06T21:21:37Z

Issa at 21:20, 6 July 2020

2020-07-06T21:20:20Z

Issa: /* Highly reliable agent designs */

2020-06-10T08:57:05Z

‎Highly reliable agent designs

@@ Line 33: / Line 33: @@
 * civilizational adequacy
 * probability of doom without any special EA/longtermist intervention
-* prior on difficulty of alignment (there is [https://arbital.com/p/alignment_difficulty/] but it looks like the page only defines what "difficulty" means, and doesn't actually go on to discuss whether we should expect it to be difficult), and ideas like "if ML-based safety were to have any shot at working, wouldn't we just go all the way and expect the default (no EA intervention) approach to AGI to just produce basically ok outcomes?"
+* [[Difficulty of AI alignment]]: prior on difficulty of alignment (there is [https://arbital.com/p/alignment_difficulty/] but it looks like the page only defines what "difficulty" means, and doesn't actually go on to discuss whether we should expect it to be difficult), and ideas like "if ML-based safety were to have any shot at working, wouldn't we just go all the way and expect the default (no EA intervention) approach to AGI to just produce basically ok outcomes?"
 * how likely optimization daemons/mesa-optimizers are or what they will look like
 * how solvable are coordination problems? e.g. how avoidable is a "[[race to the bottom]]" on safety?

@@ Line 9: / Line 9: @@
 part of what i find frustrating about many of these discussions is that there are very few examples that they use (humans vs chimps, alphago, and a couple others might be the main sources of concrete evidence?), since the disagreement is about future superintelligent systems/the nature of intelligence (which, if we understood very well, means we could build AGI today). the disagreements often seem to boil down to differences in intuition, which is hard to tinker with from the outside (i.e. if i don't have access to eliezer/paul's brain, it's not too clear to me what i can even do to make progress on these disagreements, besides going out and learning a ton of technical fields to build intuition). This train of thought led me to ask this question: [https://www.greaterwrong.com/posts/bDwQddhqaTiMhbpPF/what-are-some-exercises-for-building-generating-intuitions]
-meta point (HT david manheim): some/most of these disagreements might just be definitional confusions (e.g. people disagree on what they mean by "transformative AI" or whatever).
+meta point (HT [[David Manheim]]): some/most of these disagreements might just be definitional confusions (e.g. people disagree on what they mean by "transformative AI" or whatever).
 ==AI timelines==

@@ Line 65: / Line 65: @@
 * [[Resource overhang]]
 * what lessons can we learn by looking at the [[Evolution|evolutionary history]] of chimps vs humans?
 * what lessons can we learn from [[AlphaGo]]?
 * to what extend "recursive self-improvement" is a distinct thing, as compared with just "AIs getting better and better at doing AI research" [https://arbital.com/p/KANSI/?l=1fy#subpage-1gp] "If you imagine steady improvement in the self-improvement, that doesn't give a local team a strong advantage." [https://docs.google.com/document/pub?id=17yLL7B7yRrhV3J9NuiVuac3hNmjeKTVHnqiEa6UQpJk]

← Older revision		Revision as of 01:03, 1 December 2020
Line 133:		Line 133:

	* How big of a deal are the things [[Wei Dai]] worries about? (meta philosophy, meta ethics, [[human safety problem]]s)		* How big of a deal are the things [[Wei Dai]] worries about? (meta philosophy, meta ethics, [[human safety problem]]s)
		+	* [[Will it be possible for humans to detect an existential win?]]

	==See also==		==See also==

@@ Line 56: / Line 56: @@
 * [[Will there be significant changes to the world prior to some critical AI capability threshold being reached?]]
 ** what precursors/narrow systems we will see prior to AGI
-** how short is the window between "clearly infrahuman" and "vastly superhuman" for important real-world tasks like "doing AI research" and "building nanobots"? See also [[Kasparov window]]. For example rohin says [https://www.greaterwrong.com/posts/uKbxi2EJ3KBNRDGpL/comment-on-decision-theory/comment/pNrynCrozQPj3tFws] "I think we will first build powerful AI systems that are not that much more powerful than humans, and that direct alignment of ML techniques will be sufficient to make that safe (even though they do pose an x-risk)." (which there is some significant window where we have superhuman-but-not-vastly-superhuman AI)
+** how short is the window between "clearly infrahuman" and "vastly superhuman" for important real-world tasks like "doing AI research" and "building nanobots"? See also [[Kasparov window]]. For example rohin says [https://www.greaterwrong.com/posts/uKbxi2EJ3KBNRDGpL/comment-on-decision-theory/comment/pNrynCrozQPj3tFws] "I think we will first build powerful AI systems that are not that much more powerful than humans, and that direct alignment of ML techniques will be sufficient to make that safe (even though they do pose an x-risk)." (which mean he believes there is some significant window where we have superhuman-but-not-vastly-superhuman AI)
 * Is there a [[secret sauce for intelligence]]? how many/how "[[lumpy]]" are insights for creating an AGI? I think this has also been described as "whether AGI is a late-paradigm or early-paradigm invention" [https://intelligence.org/2017/12/01/miris-2017-fundraiser/#3] (if I'm understanding what they mean by "early/late-paradigm")
 ** "the degree of complexity of useful combination, and the degree to which a simple general architecture search and generation process can find such useful combinations for particular tasks" [https://www.facebook.com/yudkowsky/posts/10155848910529228?comment_id=10155848951264228&reply_comment_id=10155849315174228]

← Older revision		Revision as of 21:27, 6 July 2020
Line 28:		Line 28:

	==Probability of doom==		==Probability of doom==
		+
		+	(Will the world end because of unaligned AI? Will the future be controlled by completely alien values, or values resulting from an "ideal" deliberation process, or something else?)

	* civilizational adequacy		* civilizational adequacy

← Older revision		Revision as of 21:23, 6 July 2020
Line 12:		Line 12:

	==AI timelines==		==AI timelines==
		+
		+	(When will humans create AGI? When will some really crazy things start to happen in the world?)

	* will current ML techniques scale to AGI?		* will current ML techniques scale to AGI?

@@ Line 5: / Line 5: @@
 organization of this list: the most decision-relevant questions appear as sections, and in each section, the top level of questions together answer the question. sub-level questions answer the top level question, and so forth.
-an alternative organization for these disagreements (which this page doesn't follow) would be to collect all the most basic (innermost) sub-questions; these are the "cruxes", i.e. the minimal set of of questions which determine the answers to all the other questions.
+an alternative organization for these disagreements (which this page doesn't follow) would be to collect all the most basic (innermost) sub-questions; these are the "cruxes", i.e. the minimal set of questions which determines the answers to all the other questions.
 part of what i find frustrating about many of these discussions is that there are very few examples that they use (humans vs chimps, alphago, and a couple others might be the main sources of concrete evidence?), since the disagreement is about future superintelligent systems/the nature of intelligence (which, if we understood very well, means we could build AGI today). the disagreements often seem to boil down to differences in intuition, which is hard to tinker with from the outside (i.e. if i don't have access to eliezer/paul's brain, it's not too clear to me what i can even do to make progress on these disagreements, besides going out and learning a ton of technical fields to build intuition).

@@ Line 92: / Line 92: @@
 * How useful will HRAD work be for thinking about AGI? It's not clear to me whether the disagreement is mainly about which goalposts are the relevant ones (i.e. MIRI people have lower standards for what count as useful insights for AGI) vs an actual disagreement about how directly applicable MIRI work will be:
-** [[Goalpost for usefulness of HRAD work]]
+** [[Goalpost for usefulness of HRAD work]]; [[List of success criteria for HRAD work]]
 ** [[Something like realism about rationality]]. if there turns out to be no [[simple core algorithm for agency]], or if understanding agency better doesn't help us build an AGI, then we might not be in a better place wrt aligning AI.
 ** What will AGI look like?