Something like realism about rationality - Revision history

Issa at 20:24, 26 June 2020

2020-06-26T20:24:47Z

Issa: /* Topics of disagreement */

2020-06-17T08:10:13Z

‎Topics of disagreement

Issa: /* Topics of disagreement */

2020-06-17T08:09:56Z

‎Topics of disagreement

Issa: /* Topics of disagreement */

2020-06-17T01:29:52Z

‎Topics of disagreement

Issa: /* Topics of disagreement */

2020-06-17T01:11:36Z

‎Topics of disagreement

Issa: /* Topics of disagreement */

2020-06-16T19:35:26Z

‎Topics of disagreement

Issa: /* Topics of disagreement */

2020-06-16T19:27:53Z

‎Topics of disagreement

Issa: /* Topics of disagreement */

2020-06-16T19:25:27Z

‎Topics of disagreement

Issa: /* Topics of disagreement */

2020-06-16T19:23:57Z

‎Topics of disagreement

Issa: /* Topics of disagreement */

2020-06-16T19:21:02Z

‎Topics of disagreement

@@ Line 65: / Line 65: @@
 * [[Goalpost for usefulness of HRAD work]]
 * [[List of success criteria for HRAD work]]
 ==References==

@@ Line 13: / Line 13: @@
 ** I think a more precise version of the above is the following: Will mathematical theories of embedded rationality always be the relatively-imprecise theories that can’t scale to “2+ levels above” in terms of abstraction?<ref>https://www.greaterwrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality/comment/aNHTM9iwfrCeAcHtn</ref>
-From Rohin's comment [https://www.greaterwrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality/comments/iSubbXvKW7uM6rus6 here], we can build the case against "rationality realism" using three premises:
+From Rohin's comment [https://www.greaterwrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality/comment/iSubbXvKW7uM6rus6 here], we can build the case against "rationality realism" using three premises:
 # It's very hard to use relatively-imprecise theories to build things "2+ levels above". (If MIRI people disagree with this premise, then it means they're fine with aiming for relatively-imprecise theories as the final output product of their research agenda.)

@@ Line 13: / Line 13: @@
 ** I think a more precise version of the above is the following: Will mathematical theories of embedded rationality always be the relatively-imprecise theories that can’t scale to “2+ levels above” in terms of abstraction?<ref>https://www.greaterwrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality/comment/aNHTM9iwfrCeAcHtn</ref>
-From Rohin's comment [https://www.lesswrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality?commentId=iSubbXvKW7uM6rus6 here], we can build the case against "rationality realism" using three premises:
+From Rohin's comment [https://www.greaterwrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality/comments/iSubbXvKW7uM6rus6 here], we can build the case against "rationality realism" using three premises:
 # It's very hard to use relatively-imprecise theories to build things "2+ levels above". (If MIRI people disagree with this premise, then it means they're fine with aiming for relatively-imprecise theories as the final output product of their research agenda.)

@@ Line 15: / Line 15: @@
 From Rohin's comment [https://www.lesswrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality?commentId=iSubbXvKW7uM6rus6 here], we can build the case against "rationality realism" using three premises:
-# It's very hard to use relatively-imprecise theories to build things "2+ levels above". (If MIRI people disagree with this premise, then it means they're fine with aiming for relatively-imprecise theories as the final output product of their research agenda. So in that case I would consider MIRI vs critics to have different [[List of success criteria for HRAD work|success criteria for HRAD work]] (actually, success criteria are kind of all over the place so maybe this phrasing isn't so good).)
+# It's very hard to use relatively-imprecise theories to build things "2+ levels above". (If MIRI people disagree with this premise, then it means they're fine with aiming for relatively-imprecise theories as the final output product of their research agenda.)
 # Real AGI systems are "2+ levels above" mathematical theories of embedded rationality.
 # Mathematical theories of embedded rationality will always be the relatively-imprecise theories that can't scale to "2+ levels above". (If MIRI people reject this premise, that means they agree on precise theories as a final output product of HRAD work. So in this case the actual disagreement is about whether MIRI can use this output product to achieve an agreed-upon aim, namely helping to detect/fix/ensure the safety of AGI systems.)

← Older revision		Revision as of 01:11, 17 June 2020
Line 35:		Line 35:

	Actually Rohin's argument is like (1 AND 2 AND 3) OR 4.		Actually Rohin's argument is like (1 AND 2 AND 3) OR 4.
		+
		+	----
		+
		+	Another way to state it.
		+
		+	1. There's potentially a disagreement about what output miris research will have. Like what the final product looks like. E.g. Richard might think there can't be a nice theory as output, so there's a restriction on what the output can look like.
		+
		+	2. For some of those final products, there's a disagreement about whether it's sufficiently helpful for aligning ai.
		+	You could also turn this around and ask what output would be sufficient to be helpful.
		+
		+	so Richard is (1), and Rohin is (2).
		+
		+	like, richard might think a mathematical theory of embedded rationality is impossible; whereas rohin just seems to think that having such a theory wouldn't help you align AGI.
		+
		+	What about Daniel Dewey? It's a little unclear, as nate pointed out: "I’m unclear on whether we miscommunicated about what sort of thing I mean by “basic insights”, or whether we have a disagreement about how useful principled justifications are in modern practice when designing high-reliability systems."
		+
		+	Daniel could be (1) if he misunderstood what MIRI's aims are, i.e. nate's first option; and daniel could be (2) if it's about the latter option nate presents.

	==notes==		==notes==

@@ Line 29: / Line 29: @@
 ...or something. Actually, maybe what's confusing me is what "relatively imprecise" vs "precise" actually means.
-Ok now, out of Rohin's three-step argument + the fourth step above, I think we can construct a ''positive'' case for HRAD work by inverting the negative case appropriately.
+Ok now, out of Rohin's three-step argument + the fourth step above, I think we can construct a ''positive'' case for HRAD work by inverting the negative case appropriately. Something like:
 ==notes==

← Older revision		Revision as of 19:27, 16 June 2020
Line 28:		Line 28:

	...or something. Actually, maybe what's confusing me is what "relatively imprecise" vs "precise" actually means.		...or something. Actually, maybe what's confusing me is what "relatively imprecise" vs "precise" actually means.
		+
		+	Ok now, out of Rohin's three-step argument + the fourth step above, I think we can construct a ''positive'' case for HRAD work by inverting the negative case appropriately.

	==notes==		==notes==

@@ Line 23: / Line 23: @@
 I think maybe [[Abram]] will reject premise (3) (since he did agree to the crux presented by Rohin). But I think Nate/Eliezer will rather reject premise (1), based on what they've been saying?
-In <ref>https://www.greaterwrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality/comment/3f6fadtcv8FsgwKsz</ref> Abram gives two possible meanings for "model agency exactly" (which I take to be the what a mathematical theory of embedded rationality must do), namely building agents from the ground up vs being able to take an arbitrary AI system (e.g. one built with ML) and then being able to predict roughly what it does. Abram's claim seems to be that doing the former will help with the latter, even if the predictions in the latter context won't be super precise. So this seems to be a denial of Rohin's (3) above. So wait, does Rohin reject that we can build agents from the ground up, or that doing so won't help with predicting AI systems even in broad strokes?
+In <ref>https://www.greaterwrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality/comment/3f6fadtcv8FsgwKsz</ref> Abram gives two possible meanings for "model agency exactly" (which I take to be the what a mathematical theory of embedded rationality must do), namely building agents from the ground up vs being able to take an arbitrary AI system (e.g. one built with ML) and then being able to predict roughly what it does. Abram's claim seems to be that doing the former will help with the latter, even if the predictions in the latter context won't be super precise. So this seems to be a denial of Rohin's (3) above. So wait, does Rohin reject that we can build agents from the ground up, or that doing so won't help with predicting AI systems even in broad strokes? If the latter, then I think there should be another step in Rohin's argument, something like:
-I think there could be maybe another step in Rohin's argument, something like:
+:4. even a precise/mathematical theory is difficult to scale "2+ levels above" if this precise theory is being applied to AI systems that aren't built using this theory
 ...or something. Actually, maybe what's confusing me is what "relatively imprecise" vs "precise" actually means.