My understanding of how IDA works - Revision history

Issa at 00:27, 6 October 2020

2020-10-06T00:27:07Z

Issa: /* Analysis */

2020-10-05T23:45:44Z

‎Analysis

Issa: /* Analysis */

2020-04-29T04:21:26Z

‎Analysis

Issa: /* Analysis */

2020-04-29T01:30:52Z

‎Analysis

Issa: /* Analysis */

2020-04-29T01:28:50Z

‎Analysis

Issa at 00:13, 29 April 2020

2020-04-29T00:13:31Z

Issa: /* Analysis */

2020-04-28T23:58:06Z

‎Analysis

Issa: /* Analysis */

2020-04-28T23:49:54Z

‎Analysis

Issa at 03:58, 26 April 2020

2020-04-26T03:58:22Z

Issa: /* Analysis */

2020-03-07T08:05:58Z

‎Analysis

← Older revision		Revision as of 00:27, 6 October 2020
Line 45:		Line 45:

	ok i think i've identified the most important and most confusing difference between evans paper vs paul's/ajeya's explanations: in the paul version, the amplified system has a human stuck in it, so it makes it seem like you can't produce many samples/examples for the supervised learning to do its job. but in the evans paper, you learn all the decompositions ''beforehand'', so the amplified system is itself completely automatic. So then my question becomes something like, if the number of samples is the problem, then why does paul/ajeya explain it like they do?		ok i think i've identified the most important and most confusing difference between evans paper vs paul's/ajeya's explanations: in the paul version, the amplified system has a human stuck in it, so it makes it seem like you can't produce many samples/examples for the supervised learning to do its job. but in the evans paper, you learn all the decompositions ''beforehand'', so the amplified system is itself completely automatic. So then my question becomes something like, if the number of samples is the problem, then why does paul/ajeya explain it like they do?
		+
		+	==See also==
		+
		+	* [[Short-term preferences-on-reflection]]

	==References==		==References==

@@ Line 27: / Line 27: @@
 <p>Note that, logically speaking, "human ability" in the above sentence should refer to the ability of humans working in concert with other genies. This really seems like a key fact to me (it also doesn't seem like it should be controversial).</p></blockquote>
-The precise details of "human ability" seems important here. As I see it, IDA can only remain competitive if the AIs are running the show eventually (and humans are only "running the show" to the extent that they seeded the whole recursive process). In other words, it seems like Sovereigns would also be described as "primarily relying on humans working in concert with AIs".
+The precise details of "human ability" seem important here. As I see it, IDA can only remain competitive if the AIs are running the show eventually (and humans are only "running the show" to the extent that they seeded the whole recursive process). In other words, it seems like Sovereigns would also be described as "primarily relying on humans working in concert with AIs".
 Some expositions of IDA depend on the human to decompose tasks which are arbitrarily complicated (into slightly less complicated tasks), so that the AI can be trained via supervised learning to learn how to decompose such tasks. For example: "This depends on the strong assumption that humans can decompose all tasks <math>T_n</math> into tasks in <math>T_{n−1}</math>".<ref>https://owainevans.github.io/pdfs/evans_ida_projects.pdf</ref> Doesn't this mean the AI can't be competitive? When the world is crazy, we can't rely on the humans having enough time to train such decompositions.

@@ Line 44: / Line 44: @@
 also wouldn't learning the "breaking down" process itself require too many samples?
-ok i think i've identified the most important and most confusing difference between evans paper vs paul's/ajeya's explanations: in the paul version, the amplified system has a human stuck in it, so it makes it seem like you can't produce many samples/examples for the supervised learning to do its job. but in the evans paper, you learn all the decompositions ''beforehand'', so the amplified system is itself completely automatic.
+ok i think i've identified the most important and most confusing difference between evans paper vs paul's/ajeya's explanations: in the paul version, the amplified system has a human stuck in it, so it makes it seem like you can't produce many samples/examples for the supervised learning to do its job. but in the evans paper, you learn all the decompositions ''beforehand'', so the amplified system is itself completely automatic. So then my question becomes something like, if the number of samples is the problem, then why does paul/ajeya explain it like they do?
 ==References==

← Older revision		Revision as of 01:30, 29 April 2020
Line 43:		Line 43:

	also wouldn't learning the "breaking down" process itself require too many samples?		also wouldn't learning the "breaking down" process itself require too many samples?
		+
		+	ok i think i've identified the most important and most confusing difference between evans paper vs paul's/ajeya's explanations: in the paul version, the amplified system has a human stuck in it, so it makes it seem like you can't produce many samples/examples for the supervised learning to do its job. but in the evans paper, you learn all the decompositions ''beforehand'', so the amplified system is itself completely automatic.

	==References==		==References==

← Older revision		Revision as of 01:28, 29 April 2020
Line 41:		Line 41:

	if we break down tasks in T_n, will they really be in T_{n-1}? by default i sort of expect that the task will be broken down in such a way that one of the pieces basically contains all of the difficulty, and that that piece will be just as difficult as the original problem. ok so then what happens if you try to break apart ''that'' piece into further pieces? i'm not sure. maybe once you keep doing this, you get to a point where you can't break it down further, or the ''act of breaking things down'' itself requires high intelligence (so you can't just learn this from humans).		if we break down tasks in T_n, will they really be in T_{n-1}? by default i sort of expect that the task will be broken down in such a way that one of the pieces basically contains all of the difficulty, and that that piece will be just as difficult as the original problem. ok so then what happens if you try to break apart ''that'' piece into further pieces? i'm not sure. maybe once you keep doing this, you get to a point where you can't break it down further, or the ''act of breaking things down'' itself requires high intelligence (so you can't just learn this from humans).
		+
		+	also wouldn't learning the "breaking down" process itself require too many samples?

	==References==		==References==

@@ Line 31: / Line 31: @@
 Some expositions of IDA depend on the human to decompose tasks which are arbitrarily complicated (into slightly less complicated tasks), so that the AI can be trained via supervised learning to learn how to decompose such tasks. For example: "This depends on the strong assumption that humans can decompose all tasks <math>T_n</math> into tasks in <math>T_{n−1}</math>".<ref>https://owainevans.github.io/pdfs/evans_ida_projects.pdf</ref> Doesn't this mean the AI can't be competitive? When the world is crazy, we can't rely on the humans having enough time to train such decompositions.
-I guess another thing I don't really understand: eventually, the distillation tries to solve the hardest tasks directly (<math>T_N</math> in the Evans IDA document). But like, these are things humans can already solve directly, right? So then why didn't we train on human solving these problems directly? Does it take humans too much time to produce enough samples? Or is there a safety reason (like, if M knows how to do all these decompositions at the time when it gets trained to solve the hardest tasks, then it will try to reason in a human-like manner)? There seems to be a conflict here with what Paul says about IDA, namely: Paul says that after the first round, the distilled agent will be infrahuman in some says. This seems to imply that the distill procedure isn't powerful enough to mimic a human. But in the Evans document, eventually we train on the amplified system's data directly in order to produce a distilled agent that can do everything a human can. So here it seems like the distill process ''is'' powerful enough to mimic a human directly. The only ways out that I can think of:
+I guess another thing I don't really understand: eventually, the distillation tries to solve the hardest tasks directly (<math>T_N</math> in the Evans IDA document). But like, these are things humans can already solve directly, right? So then why didn't we train on human solving these problems directly? Does it take humans too much time to produce enough samples? (The paper does say IDA is intended when "it is infeasible to provide large numbers of demonstrations or sufficiently dense reward signals for methods like imitation learning or RL to work well.") Or is there a safety reason (like, if M knows how to do all these decompositions at the time when it gets trained to solve the hardest tasks, then it will try to reason in a human-like manner)? There seems to be a conflict here with what Paul says about IDA, namely: Paul says that after the first round, the distilled agent will be infrahuman in some says. This seems to imply that the distill procedure isn't powerful enough to mimic a human. But in the Evans document, eventually we train on the amplified system's data directly in order to produce a distilled agent that can do everything a human can. So here it seems like the distill process ''is'' powerful enough to mimic a human directly. The only ways out that I can think of:
 * in the Evans document, IDA doesn't actually get to human tasks in the end
@@ Line 39: / Line 39: @@
 Related confusion: Paul says after the first round of IDA, the AI is superhuman in some ways and infrahuman in other ways. So there must be something that the AI ''can't'' learn to do from supervised learning by looking at human examples. OTOH, it can eventually learn to do everything humans can by looking at a sufficiently amplified system's examples. So what is the difference here? Why does having access to the amplified system's examples allow it to do things it couldn't from human examples alone?
 if we break down tasks in T_n, will they really be in T_{n-1}? by default i sort of expect that the task will be broken down in such a way that one of the pieces basically contains all of the difficulty, and that that piece will be just as difficult as the original problem. ok so then what happens if you try to break apart ''that'' piece into further pieces? i'm not sure. maybe once you keep doing this, you get to a point where you can't break it down further, or the ''act of breaking things down'' itself requires high intelligence (so you can't just learn this from humans).

← Older revision		Revision as of 23:58, 28 April 2020
Line 41:		Line 41:

	so evans paper does say "More precisely: it is infeasible to provide large numbers of demonstrations or sufficiently dense reward signals for methods like imitation learning or RL to work well."		so evans paper does say "More precisely: it is infeasible to provide large numbers of demonstrations or sufficiently dense reward signals for methods like imitation learning or RL to work well."
		+
		+	if we break down tasks in T_n, will they really be in T_{n-1}? by default i sort of expect that the task will be broken down in such a way that one of the pieces basically contains all of the difficulty, and that that piece will be just as difficult as the original problem. ok so then what happens if you try to break apart ''that'' piece into further pieces? i'm not sure. maybe once you keep doing this, you get to a point where you can't break it down further, or the ''act of breaking things down'' itself requires high intelligence (so you can't just learn this from humans).

	==References==		==References==

← Older revision		Revision as of 23:49, 28 April 2020
Line 39:		Line 39:

	Related confusion: Paul says after the first round of IDA, the AI is superhuman in some ways and infrahuman in other ways. So there must be something that the AI ''can't'' learn to do from supervised learning by looking at human examples. OTOH, it can eventually learn to do everything humans can by looking at a sufficiently amplified system's examples. So what is the difference here? Why does having access to the amplified system's examples allow it to do things it couldn't from human examples alone?		Related confusion: Paul says after the first round of IDA, the AI is superhuman in some ways and infrahuman in other ways. So there must be something that the AI ''can't'' learn to do from supervised learning by looking at human examples. OTOH, it can eventually learn to do everything humans can by looking at a sufficiently amplified system's examples. So what is the difference here? Why does having access to the amplified system's examples allow it to do things it couldn't from human examples alone?
		+
		+	so evans paper does say "More precisely: it is infeasible to provide large numbers of demonstrations or sufficiently dense reward signals for methods like imitation learning or RL to work well."

	==References==		==References==

← Older revision		Revision as of 03:58, 26 April 2020
Line 45:		Line 45:

	[[Category:Iterated amplification]]		[[Category:Iterated amplification]]
		+	[[Category:AI safety]]