Late 2021 MIRI conversations - Revision history

Issa at 21:27, 1 August 2022

2022-08-01T21:27:44Z

Issa at 21:25, 1 August 2022

2022-08-01T21:25:30Z

Issa at 21:18, 1 August 2022

2022-08-01T21:18:27Z

Issa at 21:18, 1 August 2022

2022-08-01T21:18:13Z

Issa at 23:50, 31 July 2022

2022-07-31T23:50:59Z

Issa at 23:42, 31 July 2022

2022-07-31T23:42:52Z

Issa at 23:39, 31 July 2022

2022-07-31T23:39:42Z

Issa at 23:30, 31 July 2022

2022-07-31T23:30:07Z

Issa at 23:24, 31 July 2022

2022-07-31T23:24:28Z

Issa at 23:22, 31 July 2022

2022-07-31T23:22:26Z

@@ Line 3: / Line 3: @@
 {| class="wikitable"
 ! Title !! Summary !! My thoughts !! Further reading/keywords
 |-
 | [https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/hwxj4gieR7FWNwYfa Ngo and Yudkowsky on AI capability gains]

@@ Line 6: / Line 6: @@
 | [https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/hwxj4gieR7FWNwYfa Ngo and Yudkowsky on AI capability gains]
 | This conversation jumps between a lot of different topics, including both object level points and meta points (about how one ought to reason in these kinds of situations). Some of the main covered topics are: (1) recursive self-improvement and consequentialism (Eliezer defends the position that these are useful abstractions that apply to the real world, whereas Richard thinks they are wrong, or at least messier to apply to the real world than Eliezer believes or at the very least that the validity of these concepts has not been comprehensively argued for by Eliezer); (2) a meta-disagreement about what Eliezer got wrong in the [[Hanson-Yudkowsky debate]] (Eliezer thinks he made a 'correct argument about a different subject' rather than an 'incorrect argument about the correct subject', and thinks he didn't properly take into account the [[law of earlier failure]] -- Hanson's arguments failed much more prosaically than Eliezer expected; Richard on the other hand believes Eliezer's error is about the abstraction on consequentialism/recursive self-improvement being messier to apply to the real world than Eliezer expected); (3) the concept of utility (Eliezer defends the concept being useful and coherent and applicable to analyzing advanced AI; Richard is skeptical and goes on and on about advance predictions and such); (4) competence of government response on handling the AI situation (Eliezer is generally pessimistic, analogizing to COVID and the [[wikipedia:Subprime mortgage crisis]]; Richard thinks Eliezer is cherry-picking the negative data points); (5) whether there will be a period of rapid economics progress from "pre-scary" AI before "scary" cognition appears (Eliezer doesn't think this is likely, but also that there is no single principle that prohibits "hanging around" at a level for like 5 years where GDP goes up dramatically but the world doesn't end, that multiple smaller things would have to go unexpectedly for it to happen but it can still happen; Richard doesn't really say what his opinion is, but a lot of his hope seems to be here?). Eliezer also reveals the least impressive technology that his model rules out before the end of the world, namely [[copy-pasting strawberries]].
-| I found this conversation frustrating to read. As the participants themselves noted, the conversation spends too much time debating meta principles and not enough time on object level disagreement. Richard also didn't really state his opinions most of the time, making it difficult to see where his questions are coming from/why he is choosing particular lines of questioning. This makes the conversation feel "jerky". I actually thought [[Nate Soares]] provided a lot of good framing/moderation, and wished he would have been more heavy-handed in proposing topics and getting the main participants to stick to the object level.
+| I found this conversation frustrating to read. As the participants themselves noted, the conversation spends too much time debating meta principles and not enough time on object level disagreement. Richard also didn't really state his opinions most of the time, making it difficult to see where his questions are coming from/why he is choosing particular lines of questioning. This makes the conversation feel "jerky". I actually thought [[Nate Soares]] provided a lot of good framing/moderation, and wished he would have been more heavy-handed in proposing topics and getting the main participants to stick to the object level. The main cruxes seem to be the stuff Nate brought up: can "non-scary" cognition end the acute risk period? (Eliezer thinks no, that you need a really powerful AI, whereas Richard seems to think yes, which is why he asks what Eliezer thinks is the hardest thing you can do with a "non-scary" AI) and will there be a longish regime of "pre-scary" cognition that we can study and learn from to help better align "scary" cognition? (Eliezer would be surprised to see such a period where we just "hang around", whereas Richard seems to think it is likely)
+|
 |-

@@ Line 6: / Line 6: @@
 | [https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/hwxj4gieR7FWNwYfa Ngo and Yudkowsky on AI capability gains]
 | This conversation jumps between a lot of different topics, including both object level points and meta points (about how one ought to reason in these kinds of situations). Some of the main covered topics are: (1) recursive self-improvement and consequentialism (Eliezer defends the position that these are useful abstractions that apply to the real world, whereas Richard thinks they are wrong, or at least messier to apply to the real world than Eliezer believes or at the very least that the validity of these concepts has not been comprehensively argued for by Eliezer); (2) a meta-disagreement about what Eliezer got wrong in the [[Hanson-Yudkowsky debate]] (Eliezer thinks he made a 'correct argument about a different subject' rather than an 'incorrect argument about the correct subject', and thinks he didn't properly take into account the [[law of earlier failure]] -- Hanson's arguments failed much more prosaically than Eliezer expected; Richard on the other hand believes Eliezer's error is about the abstraction on consequentialism/recursive self-improvement being messier to apply to the real world than Eliezer expected); (3) the concept of utility (Eliezer defends the concept being useful and coherent and applicable to analyzing advanced AI; Richard is skeptical and goes on and on about advance predictions and such); (4) competence of government response on handling the AI situation (Eliezer is generally pessimistic, analogizing to COVID and the [[wikipedia:Subprime mortgage crisis]]; Richard thinks Eliezer is cherry-picking the negative data points); (5) whether there will be a period of rapid economics progress from "pre-scary" AI before "scary" cognition appears (Eliezer doesn't think this is likely, but also that there is no single principle that prohibits "hanging around" at a level for like 5 years where GDP goes up dramatically but the world doesn't end, that multiple smaller things would have to go unexpectedly for it to happen but it can still happen; Richard doesn't really say what his opinion is, but a lot of his hope seems to be here?). Eliezer also reveals the least impressive technology that his model rules out before the end of the world, namely [[copy-pasting strawberries]].
-| I found this conversation frustrating to read. As the participants themselves noted, the conversation spends too much time debating meta principles and not enough time on object level disagreement. Richard also didn't really state his opinions most of the time, making it difficult to see where his questions are coming from/why he is choosing particular lines of questioning. This makes the conversation feel "jerky". I actually thought Nate Soares provided a lot of good framing/moderation, and wished he would have been more heavy-handed in proposing topics and getting the main participants to stick to the object level.
+| I found this conversation frustrating to read. As the participants themselves noted, the conversation spends too much time debating meta principles and not enough time on object level disagreement. Richard also didn't really state his opinions most of the time, making it difficult to see where his questions are coming from/why he is choosing particular lines of questioning. This makes the conversation feel "jerky". I actually thought [[Nate Soares]] provided a lot of good framing/moderation, and wished he would have been more heavy-handed in proposing topics and getting the main participants to stick to the object level.
+|
 |-

@@ Line 6: / Line 6: @@
 | [https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/hwxj4gieR7FWNwYfa Ngo and Yudkowsky on AI capability gains]
 | This conversation jumps between a lot of different topics, including both object level points and meta points (about how one ought to reason in these kinds of situations). Some of the main covered topics are: (1) recursive self-improvement and consequentialism (Eliezer defends the position that these are useful abstractions that apply to the real world, whereas Richard thinks they are wrong, or at least messier to apply to the real world than Eliezer believes or at the very least that the validity of these concepts has not been comprehensively argued for by Eliezer); (2) a meta-disagreement about what Eliezer got wrong in the [[Hanson-Yudkowsky debate]] (Eliezer thinks he made a 'correct argument about a different subject' rather than an 'incorrect argument about the correct subject', and thinks he didn't properly take into account the [[law of earlier failure]] -- Hanson's arguments failed much more prosaically than Eliezer expected; Richard on the other hand believes Eliezer's error is about the abstraction on consequentialism/recursive self-improvement being messier to apply to the real world than Eliezer expected); (3) the concept of utility (Eliezer defends the concept being useful and coherent and applicable to analyzing advanced AI; Richard is skeptical and goes on and on about advance predictions and such); (4) competence of government response on handling the AI situation (Eliezer is generally pessimistic, analogizing to COVID and the [[wikipedia:Subprime mortgage crisis]]; Richard thinks Eliezer is cherry-picking the negative data points); (5) whether there will be a period of rapid economics progress from "pre-scary" AI before "scary" cognition appears (Eliezer doesn't think this is likely, but also that there is no single principle that prohibits "hanging around" at a level for like 5 years where GDP goes up dramatically but the world doesn't end, that multiple smaller things would have to go unexpectedly for it to happen but it can still happen; Richard doesn't really say what his opinion is, but a lot of his hope seems to be here?). Eliezer also reveals the least impressive technology that his model rules out before the end of the world, namely [[copy-pasting strawberries]].
-| I found this conversation frustrating to read. As the participants themselves noted, the conversation spends too much time debating meta principles and not enough time on object level disagreement. Richard also didn't really state his opinions most of the time, making it difficult to see where his questions are coming from/why he is choosing particular lines of questioning. This makes the conversation feel "jerky".
+| I found this conversation frustrating to read. As the participants themselves noted, the conversation spends too much time debating meta principles and not enough time on object level disagreement. Richard also didn't really state his opinions most of the time, making it difficult to see where his questions are coming from/why he is choosing particular lines of questioning. This makes the conversation feel "jerky". I actually thought Nate Soares provided a lot of good framing/moderation, and wished he would have been more heavy-handed in proposing topics and getting the main participants to stick to the object level.
+|
 |-

@@ Line 5: / Line 5: @@
 |-
 | [https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/hwxj4gieR7FWNwYfa Ngo and Yudkowsky on AI capability gains]
-| This conversation jumps between a lot of different topics, including both object level points and meta points (about how one ought to reason in these kinds of situations). Some of the main covered topics are: (1) recursive self-improvement and consequentialism (Eliezer defends the position that these are useful abstractions that apply to the real world, whereas Richard thinks they are wrong, or at least messier to apply to the real world than Eliezer believes or at the very least that the validity of these concepts has not been comprehensively argued for by Eliezer); a meta-disagreement about what Eliezer got wrong in the [[Hanson-Yudkowsky debate]] (Eliezer thinks he made a 'correct argument about a different subject' rather than an 'incorrect argument about the correct subject', and thinks he didn't properly take into account the [[law of earlier failure]] -- Hanson's arguments failed much more prosaically than Eliezer expected; Richard on the other hand believes Eliezer's error is about the abstraction on consequentialism/recursive self-improvement being messier to apply to the real world than Eliezer expected); (3) the concept of utility (Eliezer defends the concept being useful and coherent and applicable to analyzing advanced AI; Richard is skeptical and goes on and on about advance predictions and such); (4) competence of government response on handling the AI situation (Eliezer is generally pessimistic, analogizing to COVID and the [[wikipedia:Subprime mortgage crisis]]; Richard thinks Eliezer is cherry-picking the negative data points); (5) whether there will be a period of rapid economics progress from "pre-scary" AI before "scary" cognition appears (Eliezer doesn't think this is likely, but also that there is no single principle that prohibits "hanging around" at a level for like 5 years where GDP goes up dramatically but the world doesn't end, that multiple smaller things would have to go unexpectedly for it to happen but it can still happen; Richard doesn't really say what his opinion is, but a lot of his hope seems to be here?). Eliezer also reveals the least impressive technology that his model rules out before the end of the world, namely [[copy-pasting strawberries]].
+| This conversation jumps between a lot of different topics, including both object level points and meta points (about how one ought to reason in these kinds of situations). Some of the main covered topics are: (1) recursive self-improvement and consequentialism (Eliezer defends the position that these are useful abstractions that apply to the real world, whereas Richard thinks they are wrong, or at least messier to apply to the real world than Eliezer believes or at the very least that the validity of these concepts has not been comprehensively argued for by Eliezer); (2) a meta-disagreement about what Eliezer got wrong in the [[Hanson-Yudkowsky debate]] (Eliezer thinks he made a 'correct argument about a different subject' rather than an 'incorrect argument about the correct subject', and thinks he didn't properly take into account the [[law of earlier failure]] -- Hanson's arguments failed much more prosaically than Eliezer expected; Richard on the other hand believes Eliezer's error is about the abstraction on consequentialism/recursive self-improvement being messier to apply to the real world than Eliezer expected); (3) the concept of utility (Eliezer defends the concept being useful and coherent and applicable to analyzing advanced AI; Richard is skeptical and goes on and on about advance predictions and such); (4) competence of government response on handling the AI situation (Eliezer is generally pessimistic, analogizing to COVID and the [[wikipedia:Subprime mortgage crisis]]; Richard thinks Eliezer is cherry-picking the negative data points); (5) whether there will be a period of rapid economics progress from "pre-scary" AI before "scary" cognition appears (Eliezer doesn't think this is likely, but also that there is no single principle that prohibits "hanging around" at a level for like 5 years where GDP goes up dramatically but the world doesn't end, that multiple smaller things would have to go unexpectedly for it to happen but it can still happen; Richard doesn't really say what his opinion is, but a lot of his hope seems to be here?). Eliezer also reveals the least impressive technology that his model rules out before the end of the world, namely [[copy-pasting strawberries]].
 | I found this conversation frustrating to read. As the participants themselves noted, the conversation spends too much time debating meta principles and not enough time on object level disagreement. Richard also didn't really state his opinions most of the time, making it difficult to see where his questions are coming from/why he is choosing particular lines of questioning. This makes the conversation feel "jerky".
+|