black O2, inc.
ACTIVE
    FOR-001 / 70_LiteratureLIT-010

    Agent Incentives: A Causal Perspective

    Verified literature

    title
    Agent Incentives: A Causal Perspective
    Type
    literature
    Identifier
    LIT-010
    Citation
    CIT-020
    authors
    • Tom Everitt
    • Ryan Carey
    • Eric Langlois
    • Pedro A. Ortega
    • Shane Legg
    year
    2021
    venue
    arXiv preprint (v2, 15 March 2021)
    identifier
    arXiv:2102.01685
    url
    https://arxiv.org/abs/2102.01685
    retrieved
    2026-11-13
    Textual authority
    B
    Corroboration
    3
    Status
    Verified
    bears on
    • [[SYS-002]]
    • [[CON-006]]
    • [[CNF-004]]
    contests
    • [[SYS-002]]
    Tags
    • literature
    • incentives
    • causal-influence-diagrams
    • correction
    • verified

    Literature Record: Agent Incentives — A Causal Perspective

    What It Establishes

    A framework for analysing agent incentives using causal influence diagrams, with formal criteria for value of information and a new criterion for value of control, both proved sound and complete. It introduces response incentives — which environmental changes alter an optimal decision — and instrumental control incentives — whether an agent can influence its utility through a variable.

    The instrumental-control criterion is the formal statement of the vault's deterrent argument: an agent has an incentive to control any variable that affects its objective, which is precisely why a termination device installs an incentive to prevent termination.

    Rating Justification

    • Textual Authority (B): retrieved at abstract level; preprint, not peer-reviewed. Its formal results are theorems, so the peer-review gap matters less than for an empirical claim, but the scale does not permit an exception.
    • Corroboration (3): the criteria are reused across the incentive-analysis literature and are consistent with LIT-002's observed behaviour, which is the empirical counterpart of an instrumental control incentive.

    Correction Notice

    This record replaces a false identifier. The vault's register previously attributed "Agent Incentives: A Causal Approach" to Rohin Shah under arXiv:2203.11025. That identifier resolves to an unrelated paper on multigrid preconditioners for the Helmholtz equation. The paper that exists is Agent Incentives: A Causal Perspective, by Everitt et al., at arXiv:2102.01685. The author attribution was also wrong. See Identifier Audit.

    Relation to the Vault

    This is the formal backbone of SYS-002's deterrent leg — and, correctly cited, it supports that leg rather than threatening it. The vault previously recorded this citation as its highest risk on the assumption that it undercut the deterrent. That judgement was made about a paper that had not been read and whose identifier was wrong. See CNF-004.

    Back to Shackled AIForests