Architecture

In discussions about AI alignment (especially AI takeoff), the term architecture is used to mean ... something like the basic design of the AI system (like what kind of machine learning is being used in what way, what the high-level organization of the components is). Architecture is used in contrast with content, which means something like all the low-level details, the specific knowledge that's hand-coded in, ...? I'm honestly not sure I can distinguish these two very well, and I wish people would define these terms more formally. This might be pointing to the same distinction Richard Sutton makes in his Bitter Lesson post.

see https://www.lesswrong.com/posts/jd9LajtGWv93NC8uo/source-code-size-vs-learned-model-size-in-ml-and-in-humans?commentId=9ifYwKyyZmS8ZEyvi for an example (namely, specialized data structures for a task) where the content/architecture split is unclear.

synonyms? "mental architecture", "cognitive architecture", the "architecture of the AI"

"Hanson defines the “content” of a system to be its small modular features, while its “architecture” is its most important, least modular features." [1] [2]

"For similar reasons, I’m skeptical of a blank-slate AI mind-design intelligence explosion. Sure if there were a super mind theory that allowed vast mental efficiency gains all at once, but there isn’t. Minds are vast complex structures full of parts that depend intricately on each other, much like the citizens of a city. Minds, like cities, best improve gradually, because you just never know enough to manage a vast redesign of something with such complex inter-dependent adaptations." [3] -- how does Hanson reconcile this with how small the human genome is?

"Hanson mentions Peter Norvig’s recent paper, where Norvig was arguing with Noam Chomsky and saying that it’s wrong to expect there to be a simple elegant theory of linguistics. Instead there are just many messy details that one has to get right, with no key architecture." [4] -- I think this is conflating two levels. There's the source code level or top level [5] and then there's the learned model or subsystem level. The top level can be very simple even if the learned model is not simple (and has lots of messy details). As far as I can understand, the Chomsky position is that the top level is simple, and that there is no subsystem level (because it's not necessary, because the top level can directly do things without statistical learning).

Robin Hanson: "I think our dispute in part comes down to an inclination toward architecture or content. That is, one view is that there's just a clever structure and if you have that basic structure, you have the right sort of architecture, and you set it up that way, then you don't need very much else, you just give it some sense organs, some access to the Internet or something, and then it can grow and build itself up because it has the right architecture for growth. Here we mean architecture for growth in particular, what architecture will let this thing grow well? [...] My opinion [...] is that it's largely about content. There are architectural insights. There are high-level things that you can do right or wrong, but they don't, in the end, add up to enough to make vast growth. What you need for vast growth is simply to have a big base. [...] I think that for minds, what matters is that it just has lots of good, powerful stuff in it, lots of things it knows, routines, strategies, and there isn't that much at the large architectural level." [6]

"There are things like databases of stored games of chess, usable with chess-playing programs, but that is not the same as having databases of actual cognitive content." [7] -- So it looks like "content" refers only to knowledge that's hard-coded in, rather than what is learned during training.

"Hanson says that an effective AI system cannot just be created by building the right architecture and feeding it a lot of raw data; it also needs a considerable amount of content to make sense of it." [8] -- Right, the key question is quantitative: exactly how much initial content (in bytes or some other unit of measure of content) is required?

"Hanson replies that, in addition to good architecture, a human baby also has large amounts of genetically encoded content about the kind of information to pay attention to, and human babies are also explicitly taught." [9] -- How big does Hanson think the genetic content of the human brain is?

is the difference between humans and chimpanzees mostly about architecture or content? there's also the question of humans now vs humans thousands of years ago, where it seems clear that the difference is culture/"content".

if progress happens via compute, is that architecture or content? if progress happens via larger/better datasets, is that architecture or content?

if you come up with a simple learning algorithm that then has to spend a lot of time interacting with the world and using compute, in order to become smart, is that architecture or content? I feel like we need to distinguish the "top level" algorithm from the "secondary level/discovered-by-top-level" algorithm. The top level architecture could be very important, even if the secondary level is basically lots and lots of content. "It seems to me for this particular argument to carry, it's not enough to say you need content. There has to be no master trick to learning or producing content." [10]

"Intelligence is mostly about architecture, or “knowledge” along the lines of knowing to look for causal structure (Bayes-net type stuff) in the environment; this kind of knowledge will usually be expressed procedurally as well as declaratively. Architecture is mostly about deep insights. This point has not yet been addressed (much) on Overcoming Bias, but Bayes nets can be considered as an archetypal example of “architecture” and “deep insight”. Also, ask yourself how lawful intelligence seemed to you before you started reading this blog, how lawful it seems to you now, then extrapolate outward from that." [11]

potential synonym and term to search for in the Hanson-Yudkowsky debate: "The Atari case was an example of an event that Robin Hanson called "architecture" and doubted, and that I called "insight"." [12]

See also

• Content sharing between AIs