Missing gear vs secret sauce - Revision history

Issa at 21:16, 9 June 2020

2020-06-09T21:16:46Z

2020-06-09T21:15:06Z

2020-06-09T21:06:30Z

2020-06-09T21:05:37Z

2020-06-09T20:58:00Z

2020-06-09T20:55:50Z

2020-06-09T07:13:50Z

2020-06-09T07:13:10Z

2020-06-09T07:12:14Z

2020-06-09T07:09:43Z

@@ Line 19: / Line 19: @@
 | Payoff thresholds || Not necessarily || Does not specify ||
 |-
-| One algorithm || Yes || || || || Small number, possibly one? ||
+| One algorithm<ref>https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/#One_algorithm</ref> || Yes || || || || Small number, possibly one? ||
 |-
 | Lumpy AI progress || Yes ||

@@ Line 11: / Line 11: @@
 | Missing gear || Not necessarily. I think this term is somewhat ambiguous about whether the final piece is expected to be big vs small. ||
 |-
-| Secret sauce || Yes ||
+| Secret sauce || Yes || || || || Small number, possibly one? ||
 |-
 | One wrong number function / Step function ||
@@ Line 19: / Line 19: @@
 | Payoff thresholds || Not necessarily || Does not specify ||
 |-
-| One algorithm || Yes ||
+| One algorithm || Yes || || || || Small number, possibly one? ||
 |-
 | Lumpy AI progress || Yes ||

@@ Line 27: / Line 27: @@
 | Simple core algorithm || Yes ||
 |-
-| Small number of breakthroughs needed for AGI || Yes ||
+| Small number of breakthroughs needed for AGI || Yes || || || || Small number (up to around 10?) ||
 |-
-| Good consequentialist reasoning has low Kolmogorov complexity<ref>https://agentfoundations.org/item?id=1228</ref> || Yes || || I think MIRI wants humans to discover this, for the sake of being able to align the AI. But this core of good consequentialist reasoning can also be discovered by a search process (e.g. resulting in a mesa-optimizer). ||
+| Good consequentialist reasoning has low Kolmogorov complexity<ref>https://agentfoundations.org/item?id=1228</ref> || Yes || || I think MIRI wants humans to discover this, for the sake of being able to align the AI. But this core of good consequentialist reasoning can also be discovered by a search process (e.g. resulting in a mesa-optimizer). || || Small number? ||
 |}

@@ Line 29: / Line 29: @@
 | Small number of breakthroughs needed for AGI || Yes ||
 |-
-| Good consequentialist reasoning has low Kolmogorov complexity<ref>https://agentfoundations.org/item?id=1228</ref> || Yes ||
+| Good consequentialist reasoning has low Kolmogorov complexity<ref>https://agentfoundations.org/item?id=1228</ref> || Yes || || I think MIRI wants humans to discover this, for the sake of being able to align the AI. But this core of good consequentialist reasoning can also be discovered by a search process (e.g. resulting in a mesa-optimizer). ||
 |}

@@ Line 9: / Line 9: @@
 ! Term !! Is the final piece a big breakthrough? !! Nature of final piece !! Found by humans or found by AI? !! Length of lead time prior to final piece !! Number of pieces !! Explanation
 |-
-| Missing gear || Not necessarily ||
+| Missing gear || Not necessarily. I think this term is somewhat ambiguous about whether the final piece is expected to be big vs small. ||
 |-
 | Secret sauce || Yes ||
@@ Line 15: / Line 15: @@
 | One wrong number function / Step function ||
 |-
-| Understanding is discontinuous || Not necessarily ||
+| Understanding is discontinuous || Not necessarily || Restricts the final piece to be about understanding, where the AI goes from "not understanding" to "understanding" something. ||
 |-
-| Payoff thresholds || Not necessarily ||
+| Payoff thresholds || Not necessarily || Does not specify ||
 |-
 | One algorithm || Yes ||

@@ Line 7: / Line 7: @@
 {| class="sortable wikitable"
-! Term !! Is the final piece a big breakthrough? !! Nature of final piece !! Found by humans or found by AI? !! Length of lead time prior to final piece !! Explanation
+! Term !! Is the final piece a big breakthrough? !! Nature of final piece !! Found by humans or found by AI? !! Length of lead time prior to final piece !! Number of pieces !! Explanation
 |-
 | Missing gear ||

@@ Line 31: / Line 31: @@
 | Small number of breakthroughs needed for AGI ||
 |-
-| Good consequentialist reasoning has low Kolmogorov complexity ||
+| Good consequentialist reasoning has low Kolmogorov complexity<ref>https://agentfoundations.org/item?id=1228</ref> ||
 |}
 [[Category:AI safety]]