System Invariants (Stable Theory Contract)
This document defines the non-negotiable behavioral invariants for the DEXBot2 system. It is a contract for code review and release safety, not a design tutorial.
Scope
- COW Pipeline: planning, projection, reconciliation, commit, and fund accounting flows.
- Sync Engine: fill history sync, open-order sync, orphan detection, cache vs chain consistency.
- Maintenance Runtime: pipeline gating, illegal-state handling, cooldown semantics, dust ordering.
- Grid & Reconcile: price-level uniqueness, dust detection scope, reconcile cancel semantics.
- Batch & Pipeline: stale handling, retry gating, stale-flag cleanup.
- Fund Registry: cross-bot allocation invariants.
- Subscriptions: health watchdog, silent-death detection.
- Broadcast: uncertain-broadcast recovery, retry, deadlock-free reconcile.
Invariant Prefixes
| Prefix | Subsystem |
|---|---|
INV-COW |
COW pipeline |
INV-PROJ |
Projection |
INV-ID |
Order identity |
INV-LIFE |
Order lifecycle (chain existence) |
INV-ACC |
Accounting / fund tracking |
INV-SYNC |
Sync engine |
INV-MAINT |
Maintenance runtime |
INV-GRID |
Grid structure |
INV-RECON |
Reconcile |
INV-BATCH |
Batch / pipeline |
INV-STATE |
State / lifecycle |
INV-REG |
Fund registry |
INV-SUB |
Subscriptions |
INV-BROADCAST |
Broadcast |
INV-TIME |
Timing windows |
COW Pipeline
INV-COW-001Master immutability until commit- The master grid must not be mutated during planning/execution prep.
- All intermediate mutations happen in
WorkingGrid. - Master updates occur only during commit after guard checks pass.
INV-COW-002Commit atomicity- Commit swaps working state to master atomically.
- On failed/aborted execution, working state is discarded and master remains unchanged.
INV-COW-003Pre-broadcast staleness guard + bounded re-plan- A plan whose working grid went stale before broadcast (master mutated mid-planning: fills, syncs) must never be broadcast as-is.
- Re-plan ONCE from fresh master with the same fills (
replanStaleBatch, bounded bySTALE_PLAN_REPLAN_LIMITindexbot_cow_runtime.ts); restore the boundary-shift budget consumed by the abandoned plan; clear only the abandoned batch's OWN pending-broadcast entries (earlier unresolved batches' entries are kept — the recursion guard then aborts + reconciles instead of re-creating possibly-landed slots). - Still stale or no fill context → proceed + structural resync (never a silent abort that drops the fill set); commit-time guard + chain adoption close residual divergence.
INV-COW-004Working-grid stack exactly-once push/pop- Pushes go through the single
_pushWorkingGridRef(sets_workingGridPushedmarker); releases go through_popWorkingGridRef(marker-guarded, early-return sites) or_releaseWorkingGridRef(identity-checked, commit sites). - A result that was never pushed must never pop — that would underflow or steal a nested grid's stack entry.
_commitWorkingGridreleases the entry exactly once on every settle path (return or throw).
- Pushes go through the single
INV-COW-005Grid regeneration bumps_gridVersion_clearOrderCachesLogicmust bump_gridVersionso an in-flight COW plan (baseVersion from the pre-swap grid) is refused at commit instead of committing over a regenerated zero-slot grid.
INV-COW-006Boundary moves only on fills and same-batch spread promotion- Fund changes never move the boundary. Only boundary crawl (
deriveTargetBoundary,order/utils/order.ts), spread promotion, and the fill-driven rail clamp move it. Promotion shifts it onto orders placed in the same atomic batch and may absorb a same-side order already placed inside the band (that order is rail-typed by construction — a placed SPREAD slot isILLEGAL_SPREAD_STATEinvalidateOrder, so "same-side" needs no extra resolution); it becomes a legal rail member once the boundary slides past it. Only the empty band slot is a CREATE target, and the absorption still counts against the reserve/stranding depth caps. - Funds drive order sizing/budget allocation only;
syncBoundaryToFunds/calculateFundDrivenBoundaryare deleted (1.5.3). deriveRailBoundary(dexbot_cow_runtime.ts, pre-broadcast) clamps the committed boundary to the batch's resulting rails:maxBuyIdx <= boundary <= minSellIdx - gapSlots - 1. A fill's counter order is the rail fact, so it decides the boundary; an empty refill hole never outvotes it. Rail membership = the batch's own op destinations + master rail orders on their own side of the planned boundary (BUY at/below, SELL at/above the implied sellStart); a master order parked on the wrong side is a STRAY (refused-commit adoption, SPREAD-GUARD straggler, stranded hold) and is skipped, because clamping onto it would self-legalize the corrupt state (live: one stray sell clamped 138 → 115). Rail indices are the orders' LIVE chain price (resolveLiveOrderLadderPosition), not stale slot ids. A crossed/over-ceiling rail fails closed (rail-crossed/rail-exceeds-grid) and keeps the previous value instead of legalizing it; the result then passes the sharedvalidatePersistedBoundarygate (with cancel/rotation sources allow-listed), so the rail clamp and the fill re-anchor end in one validator. The gate accepts a violation-reducing repair: a candidate that strands strictly fewer placed orders than the committed boundary is allowed (the fill-driven retreat case), while an equal count stays refused. The comparison uses the validator's own in-band count (countInBandPlacements/boundaryMoveReducesStrands), never a second stranding predicate.
- Fund changes never move the boundary. Only boundary crawl (
INV-COW-007Boundary hold on guard-skipped refills- When a slot on the plan's refill wire (
collectRefillSlotIds: the plan's CREATE ids minus edge-ladder edge ids) was guard-skipped at broadcast (skippedUpdateSlotIds∪clampedUpdateSlotIds∪skippedCreateSlotIds),resolveRefillBoundaryHoldkeeps the COMMITTED boundary instead of committing the plan's shifted one over the stranded rail hole; the grid still commits. The hold is scoped to the promotion direction (the move advances toward the band holes): a fill-driven retreat moves away from them, so its skipped refills must not pin the old boundary.resolveRefillSidesupplies the plan's rail side; an unknown side keeps the legacy hold (fail safe). - Edge-ladder CREATEs are excluded because reserves are static edge insurance whose fills never crawl — a skipped reserve must not pin geometry.
- The hold is intersected with
deriveRailBoundary: when the batch actually placed a counter order above the committed boundary, the rail clamp advances the boundary (the hold becomes a no-op and the counter stays live). The hold still applies when nothing was placed on the promoted rail.boundaryHeldiseffectiveBoundary !== workingBoundary, so crawl consumption follows the value actually committed. - The hold rides to the commit as
options.boundaryHeld, which suppresses the pending-crawl clear. trackBoundaryHoldcounts consecutive hold batches (manager._consecutiveBoundaryHolds, cleared on the first clean batch) with a last-hold snapshot (manager._lastBoundaryHoldInfo); runs of 3+ escalate visibly. It also records a hold signature (manager._lastHeldPlanSignature: committed boundary + guard pivot +_lastFilledAt+ held wire) so a fill-less re-plan of an unchanged state defers instead of re-broadcasting the identical vetoed plan.- Fill-driven rebalances never sleep on
_broadcastingFlaginside_fillProcessingLock:performSafeRebalance({deferIfBroadcasting:true})returns adeferredresult and the region-end hook schedules one no-fill rebalance (schedulePostRecoveryRebalance). This prevents the 20s lock-acquisition-timeout cascade when a broadcast region outlives the 30s_awaitBroadcastIdlewait. The stale-flag watchdog (_clearStaleBroadcastFlag, 120s) fires the same region-end hook when it hard-clears a leaked flag, so deferred fills are woken even when the region never ends throughstopBroadcasting(). - A hold run carrying fresh fills (>=
TIMING.BOUNDARY_HOLD_RESYNC_THRESHOLD) requests a guard-aware structural re-center (requestStructuralGridResync, cooldownTIMING.BOUNDARY_HOLD_RESYNC_COOLDOWN_MS) instead of holding forever — the heal path when the grid is trailing the market.
- When a slot on the plan's refill wire (
INV-COW-008Owed fill crawls survive unapplied commitsorder/strategy.tsrecords a crawl per shift-eligible fill at intake (slot-level dedupe, capped at 500);deriveTargetBoundaryfolds still-owed_pendingFillCrawlsinto the next derivation, excluding the current batch's slots, edge slots, and window members — a window that reaches the grid edge is not an edge order, using the same window exclusion every placement picker andcountLiveEdgeOrdersapply._commitWorkingGridclears them only when the plan's boundary was actually applied — a held boundary (boundaryHeld), a gate-rejected boundary, and a null boundary all leave them owed.- A window that lands BEHIND the newest real fill is repaired by
reanchorBoundaryFromLastFill(called fromcalculateTargetGridright after the derivation): the fill anchors the geometry —SELL fill at f => boundary ≥ f − gapSlots,BUY fill at f => boundary ≤ f − 1— because a ±1 crawl can never catch up. The anchor index is the current fill batch's EXTREMUM for the fill side (max slot for SELL, min slot for BUY), not the mutable last-processed pivot: a sweep processed in chunks overwrites the pivot per chunk (live: 1366 → 1349 → 1361), and anchoring on a non-extremum leaves the furthest fill's counter order in-band. Prices come from the fill record (the real execution price,INV-COW-011) with the slot id as a legacy fallback. Fail-closed preconditions: a pivot whoselastFillPivotSourceisfill(a book seed never anchors; the bot-scoped last-market-fill row is the fallback), on-ladder price within one increment, a post-clamp target that actually puts the fill outside the window, andvalidatePersistedBoundary— a strand refuses the re-anchor unless the candidate strands strictly fewer placed orders than the committed boundary (the violation-reducing repair, measured with the validator's own count; an equal count still refuses). ELIGIBILITY matches the crawl: EDGE-pinned insurance fills never arm the pivot (recordLastFilledPricesskips them, shared classifiercollectEdgePinnedFillIds) and so can never anchor — an edge order is static insurance, not market movement, and an absolute anchor at a rail-edge price would teleport the window there; PARTIAL fills still anchor (real trade) because the fill's OWN PARTIAL residual at the anchor slot is explicitly allow-listed (it is inside the band by construction). Startup/recovery load paths tolerate that pure in-band state (tolerateTransientStranding) so a restart does not rebuild the boundary before GAP-EVAC clears the residual. A successful re-anchor is absolute: it consumes the whole pending-crawl snapshot taken at plan time, so no older delta can be applied on top. - Consumption is SCOPED to what the derivation applied:
deriveTargetBoundaryreturnsconsumedCrawlIds(the owed records whose shift it folded, plus the records it deliberately never counts — current-batch slots, edge-classified slots, malformed records), the plan carries it (buildSuccessResult→CowResult→_commitWorkingGridoptions), and_consumePendingFillCrawlsdrops exactly those, loggingDropped N … kept=M. A capped fold consumes only the entries it actually applied (same-side prefix up to the applied magnitude), so the remainder stays owed exactly once._clearPendingFillCrawlsremains the explicit allowed-loss path (rebuild, rejected snapshot, restart replay) and drops everything. - A commit without the wire consumes nothing (a create-only plan never derived a boundary): clearing it would be the old silent-loss bug (2026-10-25: 7 slots 147–153 swept, every record dropped, boundary ending 7 levels behind).
consumePendingFillCrawlsapplies them onto a restored finite boundary;applyPersistedPendingCrawlsis the shared startup/recovery wrapper used before sync/reconcile;_clearPendingFillCrawlsdrops them when the boundary is re-anchored (grid rebuild viainitializeGrid, rejected snapshot viarejectCorruptedGridSnapshot, persisted snapshot wipe viaAccountOrders.clearGrid).
INV-COW-009LAST-FILL-GUARD pivot survives restarts with the snapshot, and only as a fill fact- Single source of truth: the manager's in-memory pivot is authoritative at runtime; the disk row is a mirror that rides the grid snapshot (
persistGridSnapshot→storeMasterGrid10th param →lastFillPivot), so it invalidates in lockstep with the boundary/genesis instead of forming a second ledger (the_pendingFillCrawlslockstep contract). - Provenance gate: only a pivot written through the shared
setLastFillPivot(type, price, 'fill')writer is persist-eligible. A book-seeded pivot (seedLastFilledPricesFromBook, provenance'book') is a heuristic — max resting buy / min resting sell is not market truth — and is never fossilized into the snapshot. The fill-batch and queued-fill refresh sites funnel through one implementation (free function inutils/system.ts;OrderManager._setLastFillPivotdelegates to it), so value, timestamp, and provenance cannot drift apart; the queued-fill refresh and the restore path call the same free function directly, with the legacy-stub fallback built into the shared writer rather than copy-pasted. A single-sided book arms through the same writer with provenance'book'; the two-sided midpoint and per-side mirror seeds stay direct writes by design, because they carry no pivot provenance and must not be persist-eligible. - The row carries
{price, type, fillsAt, genesisHash}validated by one shared gate (normalizeLastFillPivot) used by BOTH thestoreMasterGridsanitizer and the loader — the two shape checks cannot drift.nullfrom the payload builder always clears the stored row (consumed pivot / cold manager / no live genesis),undefinedstays a legacy no-op for old callers, and a malformed shape also clears. - Restore ordering:
grid.loadGridrestores the pivot right after_restoreBoundary— with the persisted genesis applied and the grid re-typed, BEFORE the first reconcile/broadcast — so a restart cannot re-open the cold window where evacuation stamps get applied against a boundary still being rebuilt. AllloadGridcallers (startup resume, price-match resume, recovery reload) inherit it; there is no second restore call site to drift. - Validation chain (
restoreLastFillPivot): TTL expiry (GRID_LIMITS.LAST_FILL_PIVOT_TTL_MS, 24h; the pivot is a "latest fill" fact, not a permanent ratchet; expired pivots hand over to the startup book seed) and genesis binding (mismatched hash = dead generation) erase the row through one shared drop path (clearPersistedLastFillPivotunder the bot's persistence lock, best-effort). The SAME TTL bounds the in-memory reference at READ time (resolveLastFillGuardReference/resolveFillProvenancedRow), because the geometry-bind watchdog re-centers at most once per pivot and reports that a fill or the TTL restores the side — that promise must not depend on a restart. Off-grid refusal (resolveOnGridPivot— the runtime's own ladder validator, imported lazily from the restore site so ONE implementation serves both the per-probe live guard and the restore; the restorer can never accept a value the runtime would refuse per probe) is a no-op that falls back to the book seed and leaves the row for the next snapshot flush to clear (the cold in-memory pivot makes the payload builder return null). The snapped ladder level is restored, never the raw float, and the ORIGINALfillsAtis preserved throughsetLastFillPivot'satMsparam so the TTL keeps meaning "age of the last fill", not "time since this restart". - Generation invalidation mirrors the pending-crawl ledger:
initializeGridandrejectCorruptedGridSnapshotclear the in-memory pivot viaresetLastFillPivot(exposed asOrderManager._resetLastFillPivot), which clears the FULL scalar family — per-side mirrors included — soseedLastFilledPricesFromBook's cold gate cannot be silently suppressed by stale per-side values after a grid rebuild, andAccountOrders.clearGridwipes the persisted row with the snapshot. - A rebuild then RE-ARMS explicitly (
rearmLastFillPivotFromMarketFill) from the bot-scoped last-market-fill row, through the same writer, at the original timestamp (taken from the same row that supplied the price, so a book-seeded live pivot never lends its clock) and with provenance'fill'. The guard must not go cold while the account's last trade price is known (pre-1.6.14 behaviour: a rebuild let a fresh grid sell below the last sale), and the re-arm must be a logged, persisted write rather than an accident of the read-time fallback — the rebuild site's comment and the runtime behaviour were allowed to differ before this. Consequence, pinned by test: the pivot SURVIVES a rebuild, bounded by the same 24h TTL (enforced at restore AND at read time; an expired row is neither used as a fallback nor re-armed). If the market has moved away from that price, the same-side rail readslast-fill-blockedand only a fill (or TTL expiry) restores it; the geometry-bind watchdog therefore re-centers at most ONCE per pivot (_lastGeometryResyncPivotKey) — a re-center cannot move the pivot, so a second attempt would only churn the new generation — and then reports that fact instead of escalating. - EDGE-PINNED fills never arm the pivot:
recordLastFilledPricesskips the ids returned bycollectEdgePinnedFillIds(the same window/edge classifier the placement pickers use). An edge order is static fat-finger insurance, not market movement; letting a wick on one arm the pivot at a rail-edge price would veto that whole side of the book AND, because the pivot doubles as the re-anchor's absolute geometry anchor, teleport the window to the edge.
- Single source of truth: the manager's in-memory pivot is authoritative at runtime; the disk row is a mirror that rides the grid snapshot (
INV-COW-010Grid-layer spread-correction cycle is atomic and boundedcheckSpreadConditionholds the refcounted broadcast region (startBroadcasting/stopBroadcasting, released infinally) across its whole attempt loop, so a fill-driven rebalance defers rather than committing a master-version bump between the initial plan and its broadcast. This narrows the race; the COW commit guard remains the backstop.- Before broadcasting,
resolveOccupiedCorrectionre-derives a plan whose CREATE destination became a live order since planning (getOccupiedCorrectionSlots), or withholds it — a known-taken slot is never shipped. The COW build layer stays authoritative against a detector false positive. - A transient refusal re-plans ONCE from fresh master (
replanSpreadCorrectionFromFreshMaster, bounded byTIMING.SPREAD_CORRECTION_MAX_ATTEMPTS); a non-transient refusal (commit-refused/update-target-missing/unmatched-chain-orders/shutdown) is not retried and counts toward the out-of-spread stale clock. - Candidates are tightening-edge-only on real grid slots (
tightensSpread+ distance-to-edge ordering +doesNotCrossMarket) AND last-fill consistent: the planner applies the guard's own direction rule (passesLastFill, same pivot, same predicate) before the quota slice, so a batch is never built from ops the broadcast guard would drop. An unplaceable side reports the geometry bindlast-fill-blockedinstead of a wasted "prepared but not executed" cycle. - The budget spent by the loop is chain-truth for a known candidate set:
available = chainFree − virtualReservationis circular when the reservation describes the very slots being funded, so an over-sized projection (rail sized from a larger balance or from inventory locked in resting orders) would read 0 forever (planned 3, placing 0, 84 minutes live).deriveChainTruthFundBudgetre-derives it asmin(chainFree, min(virtualReservation, Σ candidate ideals))whenavailableis 0 and the reservation exceeds chain free (a zero caused by fees or an empty balance is a real constraint and is honoured); the chain stays the final arbiter. - Terminal stalls are named and escalated on their own windows rather than the structural 30 minutes: rail oversubscription (
availableFund = chainFree − virtualreads 0 while the side still holds inventory) onSPREAD_STALE_FUND_ESCALATE_MS, thelast-fill-blockedgeometry bind onSPREAD_STALE_GEOMETRY_ESCALATE_MS(capped at ONE re-center per fill pivot —_lastGeometryResyncPivotKey— because a re-center cannot move the pivot, and the skip is explained once instead of rebuilt every cooldown), and the persistence clock clears only after sustained tolerance (SPREAD_STALE_IN_TOLERANCE_RESET_MS) or a placement. - This is the grid layer's counterpart to
INV-COW-003(COW-layer pre-broadcast re-plan) and toINV-COW-007's broadcast-flag deferral contract.
INV-COW-011Live-price provenance for fill, gap and crossing decisions- A fill record carries the chain execution price (
sync_engine._computeFillTransitionResult/ pass-1 full-fill / pass-2 adoption), computed from paid/received amounts when only the history is available; the slot's generation price is a fallback, never the first choice. The queued-fill pivot refresh (refreshLastFillPivotFromQueue) likewise prefers fill economics over the slot price. The slot's identity price is NOT rewritten (GRID-PRICE-INVARIANT); only the transient fill/pivot fact uses the chain price. - Gap-evacuation geometry maps an on-chain order's LIVE price onto the genesis ladder (
resolveLiveOrderLadderPosition) before falling back toparseSlotIndex(slot.id). Detection (detectGapEvacuationCandidates), the drain-time validation (_validatePriceCorrectionEntry), and the teeth (_processGapEvacuationTeeth) all use it, so a refused-commit adoption that leaves an order keyed at an in-rail slot cannot hide an in-band order. - A price mismatch whose live chain price maps inside the gap band queues a cancel-only
gapEvacuationcorrection and drops any opposite-kind (plain re-price) entry for the same chain order. Evacuation takes precedence over correcting the order back to a stale slot price. - Stale-stamp refusal vs. forced retreat. A gap-evacuation rotation is B-stamped with its plan-build boundary/gap and re-proven against the LIVE geometry at execution (
isEvacuationStampStillValid): a boundary commit landing mid-batch must not grant a bypass for a move that no longer makes grid sense (the 2026-10-30 bid-edge incident). That refusal assumes the live boundary survives the commit. When the live rails ALREADY conflict —maxBuy > minSell − gap − 1, i.e. no boundary can host both rails with the band (liveRailsConflictWithGap, reached through the single decision pointforcedRetreatJustifiesEvacuation, which also refuses to override a stamp whose source/dest ids are unparseable — an unprovable stamp is never overridden) — the batch's retreat is forced, the plan's boundary is the only consistent geometry, and refusing the evacuation is what permanently strands the order inside the band (spread collapse: 4-slot band, real spread 1.21% vs 2.0% target). Both the build loop and the final pivot gate therefore keep the evacuation in that case. - Committed-geometry sweep. Because the in-batch refusal is fail-closed by design, the writer of the stranding must repair it: after a commit, and on every maintenance tick,
scheduleGapEvacRepairre-derives the in-band set against the COMMITTED geometry (pruning resolved ledger entries WITHOUT counting a new observation —reconcileGridis the cycle's single tick, and counting twice armed the cancel-only teeth in the very cycle whose rotation evacuated the order), queues the cancel-only teeth past the threshold, and schedules a fill-less COW re-plan (rate-limited 60s per bot). A queued cancel-only evacuation whose chain order the SAME batch rotates is held out of the pre-batch correction drain (splitEvacuationCancelsSupersededByRotations): the rotation IS the evacuation, and cancelling first destroyed the batch's own source (live:2 target(s) missing on chain→ structural resync + full grid reload instead of two in-place price updates). Plans are fill-driven, so without this a quiet market never heals a collapsed spread. - The crossing guard's candidate set (
buildCrossingCheckCandidates) uses the live chain price for master and pending orders, so an emission cannot print onto a live opposite-side order that was adopted at a stale slot price.
- A fill record carries the chain execution price (
INV-PROJ-001New projected orders remain virtual- Orders projected into empty slots must be
VIRTUALwith noorderIduntil chain confirmation.
- Orders projected into empty slots must be
INV-PROJ-002Preserve on-chain PARTIAL size in projection- If identity is retained (
keepOrderId=true) and current state isPARTIAL, projected size must preserve current on-chain remaining size. - It must not be overwritten by ideal geometric
targetSize. - Exception: a
PARTIALwith a rotation/size-update action targeting itsorderIddoes usetargetSize(the explicit-UPDATE path inprojectTargetToWorkingGrid,modules/order/utils/validate.ts). - Preserve-path size must be normalized to finite, non-negative value.
- If identity is retained (
INV-PROJ-003ACTIVE on-chain projection preserves current size (same as PARTIAL)- If identity is retained and state is
ACTIVE, projection preserves current on-chain size via the sameshouldPreserveSizepath asPARTIAL(validate.ts,projectTargetToWorkingGrid). - An explicit UPDATE action targeting the
orderIdis required to applytargetSize.
- If identity is retained and state is
INV-ID-001Order identity retention ruleorderIdand on-chain state are retained only when order is on-chain and side/type is unchanged.- Otherwise projected order becomes
VIRTUALwithorderId=null.
INV-ACC-001Committed accounting source of truth- Committed chain/grid totals derive only from on-chain orders (
ACTIVE/PARTIALwithorderId) and their projected sizes. - Virtual orders contribute only to virtual pools, not committed chain totals.
- Committed chain/grid totals derive only from on-chain orders (
INV-ACC-002Fund invariant consistency (INVARIANT 1)- Tracked totals must remain consistent with blockchain totals within
FUND_INVARIANT_PERCENT_TOLERANCE. Total = Free + Committedmust hold per side.- False violations due to ideal-size projection overstatement are prohibited.
- Tracked totals must remain consistent with blockchain totals within
INV-ACC-003Cross-bot fund registry invariant (INVARIANT 3)- Shared-account per-bot commitment must not exceed the bot's proportional share of chain balance.
- Checked with widened tolerance
max(PERCENT_TOLERANCE * 3, 0.15). - Registry failure logs an error (
order/accounting.ts, cross-bot registry invariant block), not a silent skip.
Order Lifecycle (ORDER_LIFECYCLE.md)
Chain existence is answered by ONE record per slot (modules/order/lifecycle.ts), not by the seven per-site structures that used to disagree (_pendingBroadcasts, createUncertain, _committedOrderIds, _orderIdAssignedAt, _placedAt, _chainSizeAppliedFills, _lastUnmatchedChainOrders). Those now exist only as live views onto the record.
INV-LIFE-001Unknown is never gonephase === 'unknown'(lost receipt, ambiguous read, unparseable chain entry) impliesmayExistOnChain === trueandisConfirmedGone === false.- Only a chain-observed absence sets
phase === 'gone'. A consumer must never coerceunknowninto "absent" via a default.
INV-LIFE-002One record per slot- A slot is the identity across its order incarnations. A chain-id change (rotation, re-place, adoption) updates the existing record; it must not start a parallel one.
- Writers funnel through
_applyOrderUpdate→_syncLifecycleFromMutation; no call site maintains its own chain-existence bookkeeping (that includes the stamps:assignedAt/placedAtare written by the funnel only). - The one mutation that does not call
_applyOrderUpdateis the COW whole-map commit (_commitWorkingGrid): it projects every slot of the committed map through the same funnel before swapping, so the primary placement path stampsplacedAt/assignedAttoo. Without it the surplus-cancel grace is inert for every COW-created order.
INV-LIFE-003One definition per guard- Absence decisions ("is this order's absence trustworthy yet?") use
isInFlight(), read through the slot (readSlotLifecycle) so a persistedcreateUncertainmarker participates. - The committed-but-missing guard uses
isRecentlyCommitted(). - The fresh-id guard uses
isRecentlyAssigned(). - A read site must not re-derive a window from a raw map, and must not consult a subset of
isInFlight(): that is how four sites ended up with four answers.
- Absence decisions ("is this order's absence trustworthy yet?") use
INV-LIFE-004A LIVE unresolved record is never evicted or downgraded- Registry pruning skips
unknownandbroadcastingwhile their uncertainty is inside the bound; past it no decision consults them, so they become evictable (the registry keeps a ceiling). markIntentIfResolvedlikewise refuses to downgrade a live uncertainty, and resolves one whose bound has expired instead of leaving an un-honoured suppression behind.- A pending CREATE has two distinct exits: resolved (
deletePending→intent) and abandoned (_expirePendingBroadcast→unknown). Collapsing them turns a lost receipt into a clean hole that gets re-CREATEd.
- Registry pruning skips
INV-LIFE-005Unparseable chain entries make absence unknown, not absent- A chain order that was returned but could not be decoded puts its slot's absence in the UNKNOWN bucket (
sync_enginepass-1 guard 0, mirroring the accounting recovery sweep'smalformedOrderIds); the slot is not virtualized.
- A chain order that was returned but could not be decoded puts its slot's absence in the UNKNOWN bucket (
INV-LIFE-006Evidence is monotonic; clocks cannot be refreshed for free- Within a phase the evidence rank may only rise (
local<broadcast-receipt<adoption<chain-snapshot<fill-history), so a local write cannot overwrite what the chain said. observedAtadvances only for chain-backed evidence.recordedAt(the anchor of the unresolved bound) advances only on a phase change or chain-backed evidence: re-assertingunknownis not new information.
- Within a phase the evidence rank may only rise (
INV-LIFE-007The unresolved bound is measured from when the uncertainty was RECORDEDisInFlight()is bounded (2×PENDING_BROADCAST_TTL_MS) fromrecordedAt, so a lagging node cannot park the grid and a bot that re-assertsunknownevery cycle cannot extend its own suppression.- An explicit decision (
markIntent) or chain evidence resolves the phase sooner; age alone never extends it.
Sync Engine
INV-SYNC-001Drift-refetch for partial fill correctnessisEffectivelyFullmust not unconditionally trust cachedrawOnChain.for_sale.- When cached
for_saleis smaller than the grid's own size (drift signal), refetch from chain viareadSingleOrder. isEffectivelyFullrequires either: chain-confirmed empty (refetch returned null), grid also at 0, or the other side rounding to 0.- The legacy
newSizeInt <= 0fast-path is forbidden.
INV-SYNC-002No TTL-based chain refetch- Chain is only consulted on a drift signal (cache < grid size).
- Time-based TTL refetches are prohibited — a stale-but-consistent cache is still correct.
INV-SYNC-003Drift refetch null = chain-confirmed empty- When drift refetch returns null from
readSingleOrder, the order is authoritatively gone. - The sync engine must treat this as a chain-confirmed empty, not a connection failure.
- When drift refetch returns null from
INV-SYNC-004Orphan adoption rejects duplicate price levels- Pass-2 adoption must check whether an active grid order of the same type already maps to the same ladder level (
chainPriceOnSameLevel→pricesOnSameLevel: the chain price and the grid level resolve to one slot, so equality is by level, not by a tuned band). calculatePriceToleranceis NOT the duplicate rule; it survives only inside helpers that need a representable-price tolerance, and the oldfindMatchingGridOrderByOpenOrdermatcher is gone (see GRID_PRICE_INVARIANT.md).- If a duplicate exists, the orphan is NOT adopted; it is pushed to
unmatchedChainOrdersasduplicate-price-levelfor chain cancel. - Size is irrelevant — any duplicate violates the one-order-per-price-level invariant.
- Pass-2 adoption must check whether an active grid order of the same type already maps to the same ladder level (
INV-SYNC-005Fill queue back-pressure- Subscription callbacks must not push fills beyond
MAX_INCOMING_FILL_QUEUE. - Rejected fills must leave subscription cursors retryable; the queue must remain unchanged.
- Back-pressure must not start a consumer for rejected fills.
- Subscription callbacks must not push fills beyond
INV-SYNC-006syncFromOpenOrders acquires fill-processing locksyncFromOpenOrdersacquires_fillProcessingLockby default.AsyncLockis re-entrant — callers already inside the lock rely on the intrinsicisReentrant()check instead of afillLockAlreadyHeldparameter.- Direct call sites without the lock contract are prohibited.
INV-SYNC-007Authoritative sync preserves fetched free balances- Free-balance values fetched during a sync must not be silently discarded; they seed the optimistic balance model.
- If fetched balances are unavailable (failure/error), the optimistic model remains untouched rather than zeroed.
synchronizeWithChainmust not double-deduct already-locked funds from fetchedbuyFree/sellFree.- After authoritative open-order sync,
checkFundDriftAfterFillsis expected to returnisValid=trueas a design consequence (no enforcement code exists — this sub-clause expresses the intended post-condition, not an assertion).
INV-SYNC-008SPREAD-typed fills resolve the real side- A SPREAD slot can carry an on-chain order (spread-correction activation). Fill processing must resolve the real BUY/SELL side (chain order type, pays asset, or price-vs-startPrice convention) before pushing the fill or computing the transition.
- A SPREAD-typed fill would silently drop the
deriveTargetBoundaryshift and produce an illegal SPREAD+on-chain state rejected byvalidateOrder. - The pays-asset mapping has ONE implementation (
order/fill_ledger.tsresolveTradedSide, used by_computeFillContext, the unmatched-fill attribution and the queued pivot refresh). The SPREAD price fallback stays with the caller that owns the ladder (resolveSpreadOrderSide). slot.typeis the ACCOUNTING side and is retagged to the traded side by a partial-fill transition (sosizestays dimensionally consistent forrecalculateFunds). It is NOT rail identity: a repair proof must also read the level's geometry (geometryTypeForSlotIndex), and a fill-retagged slot on the opposite rail is still wrong-side. Seedocs/FILL_PIPELINE.mdINV-FILL-002/INV-FILL-006.
Maintenance Runtime
INV-MAINT-001Pipeline in-flight defers maintenance- When
isPipelineEmptyreturnsisEmpty=false(batch in-flight, recovery-in-flight, or broadcasting active),checkSpreadConditionand divergence corrections must be skipped. - Dust cancels mutate the chain, so they take the exclusive placement region (
withPlacement('dust-cancel')) and defer to the next cycle when it is busy or the bot is shutting down — they are NOT "safe regardless of pipeline state". The rotation batch that follows the cancels runs after the region is released and acquires its own lease. - Pipeline signals (
batchInFlight,recoveryInFlight,broadcasting) must be passed toisPipelineEmpty.
- When
INV-MAINT-002Illegal state abort- When
consumeIllegalStateSignalreturns a non-null signal, the maintenance cycle must:- Trigger immediate recovery sync (
_triggerStateRecoverySync). - Skip persistence (
_persistAndRecoverIfNeeded) in the same cycle. - Skip remaining maintenance steps (spread checks).
- Arm one maintenance cooldown cycle (
_maintenanceCooldownCycles = 1).
- Trigger immediate recovery sync (
- When
INV-MAINT-003Maintenance cooldown- After a hard-abort (illegal state), the next maintenance cycle must skip all checks (spread, health, etc.).
- Cooldown is exactly one cycle; normal checks resume on the subsequent cycle.
- Cooldown must not stack or compound.
INV-MAINT-004Maintenance idle gate- Maintenance must wait for a quiet window with no fill queue, sync, or batch activity (
BLOCKCHAIN_SETTLE_DELAY_MS). - Recent activity tracking covers: fill queueing, fill processing completion, COW batch start/end, open-order sync, periodic fetches.
- Maintenance must wait for a quiet window with no fill queue, sync, or batch activity (
INV-MAINT-005Dust-first ordering- Dust partials are cancelled immediately on detection — no delay, no timer.
- Grid resync and structural maintenance wait only for the idle settle delay.
Grid Structure
INV-GRID-001One-to-one order mapping- One grid slot = at most one on-chain order. No two chain orders may map to the same grid slot.
- Sync engine tracks
matchedGridOrderIdsthrough both sync passes and skips already-matched slots. - Surplus orders (matched count above the per-side
coreOrders+edgeOrderstargets) are flagged for cancellation.
INV-GRID-002One order per price level- The active grid must have at most one on-chain order per (type, price) pair.
- Duplicate price levels are a structural violation — the sync engine must reject orphan adoption at a duplicate price, and the reconcile layer must cancel offenders on chain.
- Size is irrelevant; any duplicate violates this invariant.
INV-GRID-003Interior dust detection at duplicate price levels- Interior partials (not top-of-window) are eligible for dust detection if they share a price level with an ACTIVE sibling.
- Top-of-window partials remain always eligible.
- Two PARTIALs sharing a price with no active sibling do not qualify (left to rebalancer).
INV-GRID-004Slot price equals its genesis level (GRID_PRICE_INVARIANT.md)order.pricefor a slot-idxorder must equalpriceForSlot(idx, genesis); the genesis ladder is the only authoritative price for a slot.- Enforced at all six emission sites (CREATE / UPDATE / CREATE-FALLBACK, RECONCILE-CREATE / RECONCILE-UPDATE, STARTUP-CREATE): an off-grid emission is blocked, never broadcast.
- Range guards (
isChainPriceOutOfGrid) are bounds checks, not membership checks — they cannot substitute for this invariant. - Adoption keeps the slot's own level (a fill/chain price is metadata, not the slot's price);
loadGridrepairs a pre-existing off-grid slot price at load. - A snapshot carrying orders but no usable ladder is refused before any mutation (
loadGrid, before the startup resume decision, and at the persisted-row schema), and the sync entry refuses to reconcile a ladder-less grid at all — without a ladder no consumer has authority for a slot's price, which is what this invariant exists to prevent. The tolerance matcher that used to cover that state has been removed.config.gridLimits.MISSING_GENESIS_POLICYpicks the response:'rebuild'(default, structural resync) or'halt'(manual reset). One decision, one module:modules/order/genesis_policy.ts. - Tests: GPI-001..015 (
tests/test_grid_price_invariant_guard.ts), GPI-WIRE-001..009 (tests/test_grid_price_invariant_wiring.ts), LEGACY-ADOPT/MATERIALIZE/ADOPT-NAME (tests/test_sync_out_of_grid_defer.ts), GEN-01..21 (tests/test_missing_genesis_policy.ts).
Reconcile (GRID_RECONCILE.md)
INV-RECON-001Rotation-only size updates in reconcilereconcileGriddoes not emit generic in-place size UPDATEs for active slot diffs.- Size-changing UPDATE actions are rotation updates (
newGridIdpath). - Non-rotation size correction is handled by dedicated maintenance flows.
INV-RECON-002Dust health gating parity- Dust health thresholding applies consistently to both CREATE and rotation destination holes.
INV-RECON-003Reconcile cancels duplicate chain orders unconditionally- An unmatched chain order whose price equals an active same-type grid slot's price (exact slot-price equality via
priceSlotEqualat the asset precision) is a suspected duplicate and must be cancelled on chain via_cancelChainOrderwithreleaseUntrackedFunds: true. - Cancelled IDs are filtered out of
unmatchedParsedto prevent reprocessing. - No size guard — any duplicate at the same price is a violation.
- The earlier fuzzy
SUSPECTED_DUPLICATE_TOLERANCE_MULTIPLIER(5×calculatePriceTolerance) andSUSPECTED_DUPLICATE_TOLERANCE_FLOORare removed; only exact price-level equality triggers a reconcile cancel. - Cancel deferral under
INV-RECON-006applies before submission.
- An unmatched chain order whose price equals an active same-type grid slot's price (exact slot-price equality via
INV-RECON-004Rebalance must not convert on-chain slots to SPREAD via CREATEperformSafeRebalancemust not emitCREATEactions that convert existing on-chain slots into SPREAD orders.- On-chain mid-slot must keep its BUY/SELL type before commit.
INV-RECON-005Extreme placement ordering- BUY placements must use nearest available free slots first (descending price, so the nearest-to-center slots fill first —
order/utils/order.tsbuildOutsideInPairGroups). - SELL placements must use nearest available free slots first (ascending price).
- BUY placements must use nearest available free slots first (descending price, so the nearest-to-center slots fill first —
INV-RECON-006Fresh-placement cancels are deferred for a grace window- A CANCEL action targeting an
orderIdplaced withinTIMING.SURPLUS_CANCEL_GRACE_MS(15 min) is deferred. The timestamp is written only by the lifecycle write funnel intomanager._placedAt, and tested byisFreshlyPlacedOrder. - Exemption (chain-observed wrong side). A CANCEL for an order the correction queue holds as a chain-observed
typeMismatch(entry.typeMismatch === truewith achainOrderId) is exempt: the chain already proved the side is wrong, so parking it adds a second 15-minute wait after the pre-batch drain deferred (active region) or failed. A plan-shaped CANCEL whose reason istype-mismatchis NOT exempt — it only proves master ≠ target, which is the stale-target lag this grace exists for. The exemption is a queue-provenance fact, not a grid blocker: seedocs/FILL_PIPELINE.mdINV-FILL-004. - A queued
typeMismatchentry stays valid while the level's stored type OR its rail geometry still disagrees with the chain side; a retaggedslot.typealone is not a repair (INV-SYNC-008 /docs/FILL_PIPELINE.mdINV-FILL-006). COWRebalanceEngine.executedrops the whole slot's action group for a deferred cancel (dropping only the CANCEL would let a paired CREATE fold back into an UPDATE and re-materialize the placement), then re-syncs the deferred slots from master viaworkingGrid.syncFromMasterso the commit does not virtualize a still-live order.- Divergence correction (
applyGridDivergenceCorrections) applies the same guard after dropping any queued committed-size UPDATE for the not-desired/size-0 slot. grid-load/grid-initload contexts are excluded so restarts do not suppress the sweep.
- A CANCEL action targeting an
Batch / Pipeline
INV-BATCH-001Illegal state batch abortexecuteBatchthrowsILLEGAL_SPREAD_STATEon an illegal grid layout (emitted atmodules/order/utils/validate.ts, propagated viamodules/order/manager.ts_throwOnIllegalState).- The
_handleBatchHardAbortcatch forILLEGAL_ORDER_STATE(dexbot_class.ts) is a test-only dead branch — production never emits that code; only a test stub uses it. - In production, recovery + cooldown are armed on the next maintenance tick via
_abortFlowIfIllegalState(theINV-MAINT-002path), returningabortedForIllegalState: trueto the caller. The caller does not need to return immediately; the maintenance tick handles recovery. - Hard abort triggers one immediate recovery sync (
_triggerStateRecoverySync) plus arms one maintenance cooldown cycle (_maintenanceCooldownCycles = Math.max(current, 1)).
INV-BATCH-002Stale-only cancel fast path- When a cancel-only batch fails with "order does not exist" (stale), the handler must:
- Virtualize the slot (
state = VIRTUAL,orderId = null,size = 0,rawOnChain = null). - Return
stale: true. - NOT trigger a recovery sync.
- Track the stale order id in
_staleCleanedOrderIdsto prevent double-credit. - Preserve manager index validity.
- Virtualize the slot (
- When a cancel-only batch fails with "order does not exist" (stale), the handler must:
INV-BATCH-003"Cannot deduct" → recovery sync- When
executeBatchthrows "Cannot deduct all or more from order than order contains", the handler must:- Return
recoveredBySync: true,reason: 'ORDER_SIZE_DRIFT'. - Trigger one recovery sync.
- NOT virtualize the slot.
- Preserve
orderIduntil sync reconciles it. - NOT mark the order as stale-cleaned.
- Return
- Fast path: if the batch result indicates
ORDER_SIZE_DRIFT_TARGETED(dexbot_state_recovery.ts), a targeted repair applies the correction directly and skips_triggerStateRecoverySync.
- When
State / Lifecycle
INV-STATE-001Bootstrap suppresses invariant checks- While
isBootstrapping()is true,_verifyFundInvariantsmust be suppressed. - Suppression covers
recalculateFundsand_updateOrdertrigger paths.
- While
INV-STATE-002Recovery validation not masked by bootstrap_performStateRecoverymust detect drift even whenisBootstrapping()is true.- Bootstrap suppression of invariant checks must not also suppress recovery validation.
INV-STATE-003Grid resize respects budget after capping- BUY allocation must not exceed calculated budget after
_recalculateGridOrderSizesFromBlockchaincapping.
- BUY allocation must not exceed calculated budget after
Fund Registry
INV-REG-001Cross-bot allocation ≤ proportional share- Per-bot committed amounts (sum of on-chain orders) must not exceed
totalChainBalance × allocatedPercent. - Violation triggers an error-level log entry (not silent), with tolerance
max(PERCENT_TOLERANCE * 3, 0.15). - Registry registration is pre-flight + atomic; only shared-account bots register (
dexbot.tsderivessharedAccountswhereaccountGroups[a].length > 1), and registration completes before any shared-account bot starts. - Release happens in
DEXBot.shutdown.
- Per-bot committed amounts (sum of on-chain orders) must not exceed
INV-REG-002Async-locked registry writes- All
fund_registrymutations (register, update, release) must hold an async lock. - Concurrent writes from multiple bots sharing the same account must be serialized.
- All
Subscriptions
INV-SUB-001Fill discovery via periodic pollingstartFillPollinginvokesprocessObjectsper active subscription everyFILL_POLL_INTERVAL_MS(default 60s).entry.activeandentry.reconnectingflags gate per-subscription work.fillPollInProgressflag prevents overlapping poll ticks.
Broadcast
INV-BROADCAST-001BROADCAST_DEADLINE graceful recovery- On
BroadcastUncertainError, the bot must:- Retry once with a fresh deadline window (
_executeWithRetryOnUncertain). - Skip retry when
err.partialOnChainStateis true (pair-mode grouped execution). - After retry expiry, reconcile —
AsyncLockis re-entrant so no specialfillLockAlreadyHeldparameter is needed.
- Retry once with a fresh deadline window (
- Daemon broadcast (
broadcastWithDeadline) pins allCREDENTIAL_DAEMON_BROADCAST_RETRIESattempts to one node; only when they all fail with provably untransmitted failures (pre-send errors only) does it report the node to the health ledger (threshold → blacklist + shared health-cache exclusion, logged) and rotate to the next best node. Re-signing an uncertain broadcast would duplicate a landed transaction on chain (new tx ID per signature), so BROADCAST_DEADLINE is reported to the bot and retries happen only after chain verification.
- On
INV-BROADCAST-002Deadlock-free reconcile after uncertain broadcast_reconcileAfterUncertainBroadcastdoes not need afillLockAlreadyHeldflag becauseAsyncLockis re-entrant — a secondacquire()from within the same execution context runs the callback directly instead of queueing.- The lock hierarchy requires
_syncLock(2)to be acquired before_gridLock(3)in all paths; nogridLockAlreadyHeldbypass flag exists.
INV-BROADCAST-003Verify-before-retry; never re-sign an uncertain broadcast- An uncertain outcome (RPC timeout, connection dropped with a response pending, unknown code) must never be re-signed — a re-sign would land a duplicate transaction on chain (new tx ID per signature).
- Retries are limited to provably-untransmitted failures (pre-send connection/frame errors) and re-broadcast happens only after an AUTHORITATIVE absence read (non-empty, non-truncated chain read containing none of the batch's CREATEs).
- Direct-key and claw broadcast paths classify failures via
modules/broadcast_failure.tsand surface typedBroadcastUncertainErrorso the COW machinery engages instead of a blind error.
INV-BROADCAST-004Truncated/empty reads are ambiguous, not authoritative absence- A
get_full_accountswindow capped at the order limit omits the freshest orders (by_account index = seller,id ascending — fresh creates sort last), so absence on a truncated/empty read can never free slots/capital, confirm a cancel, discard a batch, or clear pending-broadcast protection. - All absence decisions use
readOpenOrdersWithMeta; truncated/empty snapshots defer (keep protection + request structural resync) instead of acting.
- A
INV-BROADCAST-005One exclusive placement region; one lease-liveness predicate- Every chain-mutating path (COW broadcast, reconcile Phase 1+2, correction drain, dust/residual cancel, pool swap) runs inside the exclusive placement region (
modules/order/placement_lock.ts). The broadcast refcount only defers fills; it is not mutual exclusion. - The region is NOT reentrant. Nested work receives the holder's lease and verifies it with
manager.isPlacementLeaseCurrent(lease)—isPlacementActive()is true for ANY holder and must never stand in for ownership. - Non-token holders must use
manager.withPlacement()(single acquire/release contract, bounded wait, shutdown probe).correctAllPriceMismatchesis the only nested consumer and takes the inherited lease explicitly. - Pending-broadcast liveness has ONE policy (
getLivePendingBroadcasts): TTL, or TTL + a sync completed after the record, or 2× TTL hard cap. Every reader (COW guard, reconcile CREATE, reconcile surplus sweep, crossing candidates, maintenance lightweight-sync gate) uses it. - A PLACEMENT-BACKED hung region is torn down by ONE authority:
_clearStalePlacementRegion()afterPLACEMENT_STALE_CLEAR_MS, and it also clears that region's stale broadcast refcount atomically. The teardown's refcount clear is sound only because a refcount raised by a placement holder belongs to that same region; a refcount raised OUTSIDE any placement region (the spread-correction pre-broadcast hold inorder/grid.ts) is not touched by it and is still cleared by its own watchdog. The broadcast watchdog refuses to act while a placement lease is held (so it only clears a refcount leaked without a lease), and the placement watchdog refuses a teardown while a bot-level pipeline counter younger than the bound proves progress. - The owner of a region must re-verify its own lease (
isPlacementLeaseCurrent) before it BROADCASTS a CREATE/UPDATE batch: a watchdog teardown turns the lease into a stale token, and a resumed holder must abort instead of broadcasting unprotected (COW body:PLACEMENT_LEASE_LOST; reconcile Phase 1+2: the plan is discarded and the cycle defers). Pure cancel paths are exempt — a cancel carries its own order id and cannot duplicate a CREATE.
- Every chain-mutating path (COW broadcast, reconcile Phase 1+2, correction drain, dust/residual cancel, pool swap) runs inside the exclusive placement region (
INV-TIME-001Every timing window is classified, and self-healing windows are removal targetsmodules/timing_policy.tsclassifies everyTIMINGwindow asexternal,bound,policyorself-healing, each with a rationale; aself-healingwindow must also name the evidence that replaces it.tests/test_timing_policy.tsfails on an unclassified window, so the inventory can only shrink by decision.- A wait that compensates for our own lost state is a bug with a timer attached; the fix is a lifecycle phase, not patience. See TIMING_INVENTORY.md.
Change Policy
- Any intentional invariant change must:
- Update this document in the same PR/commit.
- Include explicit rationale and risk note.
- Add or update regression tests in the relevant test files.