Difference between revisions of "Narrow window argument against continuous takeoff"
(Created page with " "When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else...") |
|||
| Line 1: | Line 1: | ||
"When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else go FOOM. An ''exactly right law of diminishing returns'' that lets the system fly through the ''soft takeoff keyhole'' is unlikely - ''far'' more unlikely than seeing such behavior in a system with a roughly-constant underlying optimizer, like evolution improving brains, or human brains improving technology. Our present life is no good indicator of things to come."<ref>https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff</ref> | "When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else go FOOM. An ''exactly right law of diminishing returns'' that lets the system fly through the ''soft takeoff keyhole'' is unlikely - ''far'' more unlikely than seeing such behavior in a system with a roughly-constant underlying optimizer, like evolution improving brains, or human brains improving technology. Our present life is no good indicator of things to come."<ref>https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff</ref> | ||
| + | |||
| + | <blockquote><p>A secret of a lot of the futurism I'm willing to try and put any weight on, is that it involves the startling, amazing, counterintuitive prediction that something ends up in the not-human space instead of the human space - humans think their keyholes are the whole universe, because it's all they have experience with. So if you say, "It's in the (much larger) not-human space" it sounds like an amazing futuristic prediction and people will be shocked, and try to dispute it. But livable temperatures are rare in the universe - most of it's either much colder or much hotter. A place like Earth is an anomaly, though it's the only place beings like us can live; the interior of a star is much denser than the materials of the world we know, and the rest of the universe is much closer to vacuum.</p> | ||
| + | <p>So really, the whole hard takeoff analysis of "flatline or FOOM" just ends up saying, "the AI will not hit the human timescale keyhole." From our perspective, an AI will either be so slow as to be bottlenecked, or so fast as to be FOOM. When you look at it that way, it's not so radical a prediction, is it?<ref>https://www.lesswrong.com/posts/tjH8XPxAnr6JRbh7k/hard-takeoff?commentId=PDeciAspyhH5eHWhc</ref></p></blockquote> | ||
narrow window argument against soft takeoff: eliezer says at some points that you need the parameter k that controls the growth to be in a really narrow range for it to NOT go into either a FOOM or petering out. in contrast, i think buck/paul has said something like, if you try to model it with math, you typically get a soft takeoff? what's going on here? | narrow window argument against soft takeoff: eliezer says at some points that you need the parameter k that controls the growth to be in a really narrow range for it to NOT go into either a FOOM or petering out. in contrast, i think buck/paul has said something like, if you try to model it with math, you typically get a soft takeoff? what's going on here? | ||
Revision as of 03:49, 22 May 2020
"When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else go FOOM. An exactly right law of diminishing returns that lets the system fly through the soft takeoff keyhole is unlikely - far more unlikely than seeing such behavior in a system with a roughly-constant underlying optimizer, like evolution improving brains, or human brains improving technology. Our present life is no good indicator of things to come."[1]
A secret of a lot of the futurism I'm willing to try and put any weight on, is that it involves the startling, amazing, counterintuitive prediction that something ends up in the not-human space instead of the human space - humans think their keyholes are the whole universe, because it's all they have experience with. So if you say, "It's in the (much larger) not-human space" it sounds like an amazing futuristic prediction and people will be shocked, and try to dispute it. But livable temperatures are rare in the universe - most of it's either much colder or much hotter. A place like Earth is an anomaly, though it's the only place beings like us can live; the interior of a star is much denser than the materials of the world we know, and the rest of the universe is much closer to vacuum.
So really, the whole hard takeoff analysis of "flatline or FOOM" just ends up saying, "the AI will not hit the human timescale keyhole." From our perspective, an AI will either be so slow as to be bottlenecked, or so fast as to be FOOM. When you look at it that way, it's not so radical a prediction, is it?[2]
narrow window argument against soft takeoff: eliezer says at some points that you need the parameter k that controls the growth to be in a really narrow range for it to NOT go into either a FOOM or petering out. in contrast, i think buck/paul has said something like, if you try to model it with math, you typically get a soft takeoff? what's going on here?
