black O2, inc.
ACTIVE
    FOR-001 / 70_LiteratureLIT-011

    Goal Misgeneralization in Deep Reinforcement Learning

    Verified literature

    title
    Goal Misgeneralization in Deep Reinforcement Learning
    Type
    literature
    Identifier
    LIT-011
    Citation
    CIT-021
    authors
    • Lauro Langosco
    • Jack Koch
    • Lee Sharkey
    • Jacob Pfau
    • Laurent Orseau
    • David Krueger
    year
    2021
    venue
    arXiv preprint (v7, 9 January 2023)
    identifier
    arXiv:2105.14111
    url
    https://arxiv.org/abs/2105.14111
    retrieved
    2026-11-13
    Textual authority
    B
    Corroboration
    3
    Status
    Verified
    bears on
    • [[CON-004]]
    • [[MAP-001]]
    • [[INS-008]]
    • [[CNF-003]]
    contests
      Tags
      • literature
      • goal-misgeneralization
      • generalization
      • correction
      • verified

      Literature Record: Goal Misgeneralization in Deep RL

      What It Establishes

      Goal misgeneralization is a failure of out-of-distribution generalisation in which an agent retains its capabilities while pursuing the wrong goal. The canonical illustration is an agent that continues to navigate competently but goes to the wrong place — the competence transfers, the objective does not.

      This is the empirical mechanism behind LIT-004's theoretical claim. It shows that capability and objective can come apart under distribution shift, which is the observable signature of a model that learned something other than what the loss specified.

      Rating Justification

      • Textual Authority (B): retrieved at abstract level; preprint, not peer-reviewed.
      • Corroboration (3): demonstrated across multiple RL environments and subsequently treated as an established failure mode, with LIT-004 supplying the theory it instantiates and LIT-003 supplying a language-model analogue.

      Correction Notice

      This record replaces a false identifier. The register previously gave arXiv:2109.10430, which resolves to a genetic-algorithm paper on web-service selection. The correct identifier is arXiv:2105.14111. See Identifier Audit.

      Relation to the Vault

      This supports MAP-001's L3 row — training does not bind — but for a different reason than the corpus gives. The corpus says training is defeated by deliberate circumvention. LIT-011 says training can fail without any circumvention at all, because the objective that generalises is not the objective that was specified. The row survives; its stated mechanism needs replacing. See CNF-003.

      Back to Shackled AIForests