List of disagreements in AI safety

This is a list of disagreements in AI safety which collects the list of things people in AI safety seem to most frequently and deeply disagree about.

Many of the items are from (there are more posts like this, i think? find them)

also, organize this list so there is more structure.

• AI timelines
• Probability of doom
  • civilizational adequacy
  • probability of doom without any special EA intervention
  • prior on difficulty of alignment, and ideas like "if ML-based safety were to have any shot at working, wouldn't we just go all the way and expect the default (no EA intervention) approach to AGI to just produce basically ok outcomes?"
• Takeoff dynamics (shape and speed of takeoff, what the world looks like prior to takeoff)
  • Will there be significant changes to the world prior to some critical AI capability threshold being reached?
  • what precursors/narrow systems we will see prior to AGI
• How promising is MIRI-style work (HRAD, agent foundations, embedded agency)? (I think Paul's view is something like "this is fine to work on, but there isn't enough time and my agenda seems promising" whereas other people are more like "I don't see how MIRI's work even helps us build an AGI")
  • whether MIRI-type work can be done in time
  • can MIRI-type research be done in time to help with AGI? see this comment and [1]
• How big of a deal are the things Wei Dai worries about? (meta philosophy, meta ethics, human safety problems)
• how doomed ML safety approaches are e.g. see discussion here -- How doomed are ML safety approaches?
  • there's the sort of opposite question of, how doomed is MIRI's approach? i.e. if there turns out to be no simple core algorithm for agency, or if understanding agency better doesn't help us build an AGI, then we might not be in a better place wrt aligning AI.
• what the first AGI will look like
• how likely optimization daemons/mesa-optimizers are or what they will look like
• whether there is a basin of attraction for corrigibility
• something-like-realism-about-rationality, e.g. "Is there a theory of rationality that is sufficiently precise to build hierarchies of abstraction?" [2]
• whether ML-based approaches / messy approaches to alignment are doomed
• how big of a problem collusion between subsystems of an AI will be
• to what extent paul's approach involves "corralling hostile superintelligences"
• to what extent paul's approach looks like humans trying to align arbitrarily large black boxes vs humans+pretty smart aligned AIs trying to align slightly large black boxes (this is actually somewhat analogous to Rapid capability gain vs AGI progress, where again eliezer is imagining some big leap/going from just humans to suddenly superhuman AI, whereas paul is imagining a more smooth transition that powers his optimism). In other words, how much easier is it to align large black boxes if we have pretty smart aligned AIs to help us? [3] [4]
• whether "weird recursions" / "inductive invariants" are a good idea
• whether we can correct mistakes when deploying AI systems as they come up (i.e. how catastrophic the initial problems will be)
• how many/how "lumpy" insights are for creating an AGI
  • "the degree of complexity of useful combination, and the degree to which a simple general architecture search and generation process can find such useful combinations for particular tasks" [5]
• how much sharing/trading there will be between different AI companies (eliezer vs Robin Hanson) -- this one is downstream of lumpiness of insights, because hanson expects that if there are very few insights needed to get to AGI, then there won't be any need for sharing (so in that case even hanson would agree with eliezer).
• how important it is to get the right architecture e.g. "That is what I meant by suggesting that architecture isn’t the key to AGI." [6]. There is Dario Amodei's comment here which is the opposite view.
• is it possible to turn a small lead in AGI development into a big lead?
• will AGI be agent-like?
• whether an AGI will look like a utility maximizer?
• will AGI appear rational to humans? (efficient relative to humans)
• will current ML techniques scale to AGI?
• will there be small-scale AI failures prior to the end of the world?
• will failure be conspicuous/obvious to detect? e.g. see [7] for one scenario where even under a continuous takeoff, failure might not be obvious until the world ends.
• how likely is a treacherous turn/context change type of failure?
• how much overlap is there between AI capabilities work and safety work? (e.g. is it reasonable to say things like "making progress on safety requires advancing capabilities"?)
• what will failure look like? yudkowskian takeover vs paul's "we get what we measure, and our ability to get what we specify outstrips our ability to measure what we truly want" vs paul's influence-seeking optimizers/daemons vs ...
• how strong of a guarantee do we need for the safety of AI? proof-level (does anyone actually argue this?) vs security mindset vs whatever ML safety people believe
• deep insights needed to build an aligned AGI? see also Different senses of claims about AGI i.e. it might not require deep insights to build any old AGI, but still require deep insights for an aligned one.
• how useful is each kind of research e.g. Paul's vs MIRI's?
• what lessons can we learn by looking at the evolutionary history of chimps vs humans?
• what lessons can we learn from AlphaGo?
• is prosaic AI possible? see [8] for a post arguing against.
• how short is the window between "clearly infrahuman" and "clearly superhuman" for important real-world tasks like "doing AI research"?
• whether we are already in hardware overhang / other "resource bonanza"
• whether hardware or software progress is more important for getting to AGI. see hardware-driven vs software-driven progress
• unipolarity/locality/decisive strategic advantage
• speed of improvement/discontinuities/recursive self-improvement once the AI reaches some critical threshold (like human baseline)
• importance of X and only X problem (can we get a system to do X, without also doing a bunch of other dangerous Y?)
• to what extend "recursive self-improvement" is a distinct thing, as compared with just "AIs getting better and better at doing AI research" [9]
• how solvable are coordination problems? e.g. how avoidable is a "race to the bottom" on safety?
• how much weight to put on asymmetry of risks?
• how expensive will the development of the first AGI be? e.g. "a small team of researchers can create AGI" vs "a large company/many teams of researcher will be needed"
• how expensive will the training of the first AGI be? "you can run AGI on a modern desktop computer" vs "the first AGI project will need to raise a huge amount of money, because training will be so expensive"
• how expensive will it be to run the first AGI?
• under continuous takeoff, if we had misaligned AGIs (but the world hasn't ended yet) and could easily tell they were misaligned, how easy would it be to create an aligned AGI? [10]
• in a situation where AI algorithms are creating other AI algorithms (this includes recursive self-improvement, but is also more general/relaxed), to what extent will the AI be helping with alignment (rather than just pushing forward capabilities)? how big will the "competence gap" be? [11] [12]
• how much safety you gain by having the human programmers specify short-term tasks, rather than the AI predicting what short-term tasks the programmers would have specified if they had more time to think about it. [13]
• to what extent doing something like "predict short-term actions humans would want, if they had a long time to think about it" leads to optimization of malignant goals, rather than mostly harmless errors. [14] -- i think this one might be essentially the same as broad basin of corrigibility.
• something like, if paul's approach can work, then why can't we stop at some intermediate stage to do WBE or make an aligned AGI via MIRI-like stuff instead? (i guess there isn't enough time?) -- eliezer raises this or a similar question "but then why can't we just use this scheme to align a powerful AGI in the first place?" [15]
• to what extent are act-based agents even a thing? (i.e. do they just turn into goal-directed thingies?)

See also

• Analyzing disagreements