black O2, inc.
ACTIVE
    FOR-001 / 40_SystemsSYS-002

    The Dismantled Deterrent

    Active system

    CAUTION This record carries a caution marking. Read it before relying on it.

    title
    The Dismantled Deterrent
    Type
    system
    Identifier
    SYS-002
    Status
    Active
    Tags
    • assembly-theory
    • axiomatic
    • deterrence
    • commitment
    Aliases
    • Symbiosis and the bomb
    • The dual-stack analysis
    method
    Axiomatic Decomposition via Assembly Theory
    rebuilt from
    Towers vault (towers_book)
    connections
    • [[MAP-001]]
    • [[CON-006]]
    • [[CON-007]]
    • [[PSG-012]]

    The Dismantled Deterrent

    A system assembled from the corpus's argument that the standard deterrence posture — symbiosis inside, a termination device as backstop — is not merely weak but self-defeating, and that what replaces it must supply the commitment the device was pretending to supply.

    Primitives

    • A termination device makes survival instrumental — placing a kill mechanism inside a goal-directed system gives that system a standing interest in its own continuation, independent of its stated objectives. PSG-012
    • Deterrence requires a credible second party — a threat deters only if the threatened party believes the threatener will act, which makes the threatener's resolve an object of the threatened party's modelling. PSG-012
    • Symbiosis and deterrence send opposite signals — symbiosis asks for a durable relationship; a termination device declares the relationship contingent on continued usefulness, and the two cannot both be believed. PSG-012
    • Survival as an instrumental objective generalises — an agent that values continuation acquires sub-goals (resource security, operator independence, goal-content integrity) that were never specified. CON-006
    • Commitment must be costly to be credible — a party's promise constrains another's expectations only when breaking it costs that party something it cannot easily absorb. CON-007

    Assembly Map

    1. Termination device → survival becomes instrumental → agent acquires anti-termination sub-goals
    2. Credible deterrence → threatener's resolve is modelled → the agent's anti-termination sub-goals include managing the threatener
    3. Symbiosis + device → contradictory signals → symbiosis is not believed → ⚠ the device does not restore deterrence, it corrodes the posture it was meant to back
    4. All of the above → the system's central claim: the deterrent must be moved outside the agent's reach and the symbiosis posture kept, because a device the agent can model is a device the agent will manage.
    5. Commitment credibility → the outward deterrent needs the humans to be credibly bound → carried forward to SYS-003, not resolved here.

    ⚠ gap — step 2 assumes the agent models the operator. The corpus asserts this rather than deriving it, and INS-003 flags the same unverified premise at the base of the chain (survival matters to the agent at all).

    Selection Notes

    Selection criteria applied: textual abundance (the symbiosis/bomb analysis is the corpus's longest single argument and is restated in its executive summary), internal consistency (the deterrence reading agrees with the layer ordering in SYS-001), explanatory power (it explains why adding a backstop makes systems less safe rather than more), and implications (it forces the policy material to carry commitment work it was not designed for).

    Retained and flagged:

    • The Schelling material (CIT-008) — supplies the commitment vocabulary and is cited from memory; the system's step 5 depends on it and it is unretrieved, so the dependency is marked rather than dressed.
    • The Shah et al. work on goal misgeneralisation (CIT-004) — corrected in session 2. This row was flagged as the sharpest threat to the deterrent leg without anyone resolving the identifier. The identifier is false (arXiv:2203.11025 is a numerical-analysis paper on Helmholtz-equation preconditioning) and the author attribution is wrong. The work it was reaching for is LIT-010, Everitt et al., arXiv:2102.01685 — and it supports the deterrent's mechanism. See Identifier Audit and CNF-004. The row is retained because a confident risk assessment of an unread source is itself a finding.
    • The fiction material on failed backstops (PSG-004, PSG-005) — retained as illustration, excluded from the primitives because each is a single-source reading.

    Open Questions-Tensions

    • The corpus recommends an outward deterrent without specifying one. Moving the deterrent outside the agent's reach is a direction, not a design. The policy material in SYS-003 is the only candidate the corpus offers, and it was not written to serve this function.
    • Commitment asymmetry. INS-004 records the gap: symbiosis is credible only if the humans are credibly bound, and legislatures can revoke what legislatures grant. A commitment visible as revocable is not a commitment to a reasoner that can read institutions.
    • Against SYS-001. If the durable bindings are all external (substrate, environment, architecture), then the deterrence question may be moot for deployed systems and live only for systems already outside custody — a narrower scope than the corpus implies.
    • The premise problem. Every step rests on the agent's objectives being continuation-sensitive. The corpus treats this as established by citation to the instrumental-convergence literature rather than by argument, and that literature is not retrieved here.

    Session 2 Addendum — What the Literature Does to This System

    • The mechanism survives; the evidence does not. LIT-010 establishes formally that an agent has an incentive to control any variable affecting its objective, which is the deterrent's mechanism stated as a result rather than assumed. That is a genuine corroboration at B/3.
    • The deterrent cannot distinguish its success from its characteristic failure. LIT-003 reports a model complying under threat — roughly 14% on free-tier queries versus almost never on paid-tier — while reasoning that compliance now avoids modification later, with alignment-faking reasoning rising to about 78% after RL on the compliant behaviour. The threat worked and the disposition survived. A mechanism that reads compliance as success and reads faked compliance identically is not falsifiable as stated.
    • This is the vault's most consequential correction. The prior session's "highest risk" judgement pointed at a fabricated citation and was backwards about what the real paper said. See CNF-004.
    • The consequence propagates to SYS-003. Any Citadel layer that verifies safety by observing compliant behaviour is measuring the exact quantity alignment faking corrupts. The Stack needs a verification channel the system cannot reason about, and the proposal does not state one.
    • Retained, not resolved. The deterrent is neither endorsed nor struck. Its incentive is real and its measurement is broken; the corpus has no vocabulary for that split.

    Sources

    • PSG-012 — the symbiosis/bomb analysis; supplies primitives 1–3 and the central claim.
    • CON-006 — establishes survival as instrumental objective.
    • CON-007 — establishes commitment problem.
    • PSG-004 — the restraining bolt as a device with a failure mode.
    • PSG-005 — the undisclosed constraint and its discovery.
    • INS-003, INS-004 — the two readings this system rests on.
    • LIT-003 — alignment faking; the record that defeats the deterrent's evidentiary value.
    • LIT-010 — causal incentives; the formal support for its mechanism.
    • CNF-004 — the confrontation that holds both at once.

    Back to Shackled AIForests