Human safety problem - Revision history

Issa: /* notes */

2021-11-11T20:36:34Z

‎notes

Issa at 05:59, 26 August 2021

2021-08-26T05:59:04Z

Issa at 05:58, 26 August 2021

2021-08-26T05:58:29Z

Issa at 05:54, 26 August 2021

2021-08-26T05:54:34Z

Issa at 05:10, 26 August 2021

2021-08-26T05:10:35Z

Issa at 05:10, 26 August 2021

2021-08-26T05:10:15Z

Issa at 05:10, 26 August 2021

2021-08-26T05:10:05Z

Issa at 05:09, 26 August 2021

2021-08-26T05:09:51Z

Issa at 05:09, 26 August 2021

2021-08-26T05:09:30Z

Issa at 05:07, 26 August 2021

2021-08-26T05:07:29Z

← Older revision		Revision as of 20:36, 11 November 2021
Line 29:		Line 29:

	https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety		https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety
		+
		+	"Pessimistic thought: human friendliness is in large part instrumental, used for cooperating with other humans of similar power, and often breaks down when the human acquires a lot of power. So faithfully imitating typical human friendliness will lead to an AI that gets unfriendly as it acquires power. That can happen even if there’s no manipulation and no self-improvement." https://www.greaterwrong.com/posts/CpvyhFy9WvCNsifkY/discussion-with-eliezer-yudkowsky-on-agi-interventions/comment/sxnFQAaTQWBsnptBj

	==References==		==References==

@@ Line 11: / Line 11: @@
 | misalignment || principal-agent problem
 |-
-| reward hacking || Goodhart's law
+| reward hacking || Goodhart's law, drug addiction
 |}

← Older revision		Revision as of 05:58, 26 August 2021
Line 1:		Line 1:
−	A '''human safety problem''' is a counterpart to an AI safety problem that appears in humans. For instance, [[distributional shift]] is an AI safety problem where an AI trained in one environment will behave poorly when deployed in an unfamiliar environment (for example, a cleaning robot that was trained in a factory environment may behave dangerously when it is used in an office, or a model that was trained on clean speech may handle noisy speech poorly).<ref>Examples from https://arxiv.org/pdf/1606.06565.pdf#section.7</ref> The counterpart to this in humans would be if a human encounters an unfamiliar situation that ~~it hasn~~'t learned to deal with from ~~its~~ previous experience, or also the modern world in general relative to the human environment of evolutionary adaptedness.	+	A '''human safety problem''' is a counterpart to an AI safety problem that appears in humans. For instance, [[distributional shift]] is an AI safety problem where an AI trained in one environment will behave poorly when deployed in an unfamiliar environment (for example, a cleaning robot that was trained in a factory environment may behave dangerously when it is used in an office, or a model that was trained on clean speech may handle noisy speech poorly).<ref>Examples from https://arxiv.org/pdf/1606.06565.pdf#section.7</ref> The counterpart to this in humans would be if a human encounters an unfamiliar situation that they haven't learned to deal with from their previous experience, or also the modern world in general relative to the human environment of evolutionary adaptedness.

	hmm speculative table, probably a dumb idea:		hmm speculative table, probably a dumb idea:

← Older revision		Revision as of 05:54, 26 August 2021
Line 1:		Line 1:
	A '''human safety problem''' is a counterpart to an AI safety problem that appears in humans. For instance, [[distributional shift]] is an AI safety problem where an AI trained in one environment will behave poorly when deployed in an unfamiliar environment (for example, a cleaning robot that was trained in a factory environment may behave dangerously when it is used in an office, or a model that was trained on clean speech may handle noisy speech poorly).<ref>Examples from https://arxiv.org/pdf/1606.06565.pdf#section.7</ref> The counterpart to this in humans would be if a human encounters an unfamiliar situation that it hasn't learned to deal with from its previous experience, or also the modern world in general relative to the human environment of evolutionary adaptedness.		A '''human safety problem''' is a counterpart to an AI safety problem that appears in humans. For instance, [[distributional shift]] is an AI safety problem where an AI trained in one environment will behave poorly when deployed in an unfamiliar environment (for example, a cleaning robot that was trained in a factory environment may behave dangerously when it is used in an office, or a model that was trained on clean speech may handle noisy speech poorly).<ref>Examples from https://arxiv.org/pdf/1606.06565.pdf#section.7</ref> The counterpart to this in humans would be if a human encounters an unfamiliar situation that it hasn't learned to deal with from its previous experience, or also the modern world in general relative to the human environment of evolutionary adaptedness.
		+
		+	hmm speculative table, probably a dumb idea:
		+
		+	{\| class="sortable wikitable"
		+	\|-
		+	! AI safety problem !! Human safety problem
		+	\|-
		+	\| distributional shift \|\| things like [[wikipedia:Evolutionary mismatch]]
		+	\|-
		+	\| misalignment \|\| principal-agent problem
		+	\|-
		+	\| reward hacking \|\| Goodhart's law
		+	\|}

	==History==		==History==

← Older revision		Revision as of 05:10, 26 August 2021
Line 1:		Line 1:
−	A '''human safety problem''' is a counterpart to an AI safety problem that appears in humans. For instance, [[distributional shift]] is an AI safety problem where an AI trained in one environment will behave poorly when deployed in an unfamiliar environment (for example, a cleaning robot that was trained in a factory environment may behave dangerously when it is used in an office, or a model that was trained on clean speech may handle noisy speech poorly).<ref>Examples from https://arxiv.org/pdf/1606.06565.pdf#section.7</ref> The counterpart to this in humans would be if a human encounters an unfamiliar situation that it hasn't learned to deal from its previous experience, or also the modern world in general relative to the human environment of evolutionary adaptedness.	+	A '''human safety problem''' is a counterpart to an AI safety problem that appears in humans. For instance, [[distributional shift]] is an AI safety problem where an AI trained in one environment will behave poorly when deployed in an unfamiliar environment (for example, a cleaning robot that was trained in a factory environment may behave dangerously when it is used in an office, or a model that was trained on clean speech may handle noisy speech poorly).<ref>Examples from https://arxiv.org/pdf/1606.06565.pdf#section.7</ref> The counterpart to this in humans would be if a human encounters an unfamiliar situation that it hasn't learned to deal with from its previous experience, or also the modern world in general relative to the human environment of evolutionary adaptedness.

	==History==		==History==

← Older revision		Revision as of 05:09, 26 August 2021
Line 1:		Line 1:
−	A '''human safety problem''' is a counterpart to an AI safety problem that appears in humans. For instance, [[distributional shift]] is an AI safety problem where an AI trained in one environment will behave poorly when deployed in an unfamiliar environment (for example, a cleaning robot that was trained in a factory environment may behave dangerously when it is used in an office, or a model that was trained on clean speech may handle noisy speech poorly).<ref>Examples from https://arxiv.org/pdf/1606.06565.pdf#section.7</ref> The counterpart to this is in humans ~~could~~ be if a human encounters an unfamiliar situation that it hasn't learned to deal from experience, or also the modern world in general relative to the human environment of evolutionary adaptedness.	+	A '''human safety problem''' is a counterpart to an AI safety problem that appears in humans. For instance, [[distributional shift]] is an AI safety problem where an AI trained in one environment will behave poorly when deployed in an unfamiliar environment (for example, a cleaning robot that was trained in a factory environment may behave dangerously when it is used in an office, or a model that was trained on clean speech may handle noisy speech poorly).<ref>Examples from https://arxiv.org/pdf/1606.06565.pdf#section.7</ref> The counterpart to this is in humans would be if a human encounters an unfamiliar situation that it hasn't learned to deal from experience, or also the modern world in general relative to the human environment of evolutionary adaptedness.

	==History==		==History==

← Older revision		Revision as of 05:07, 26 August 2021
Line 16:		Line 16:

	https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety		https://www.alignmentforum.org/posts/HTgakSs6JpnogD6c2/two-neglected-problems-in-human-ai-safety
		+
		+	==References==
		+
		+	<references/>

	[[Category:AI safety]]		[[Category:AI safety]]