Grid Reconciliation: Distributed State Mismatch in a Single-Threaded Runtime

The Core Problem

DEXBot2 holds an intended grid state — which orders should be on-chain, at what prices and sizes, each assigned a grid slot with an orderId. The blockchain holds the actual state — the limit orders that physically exist. Fills, partial cancellations, race conditions, and external cancellations or manual order edits cause these to diverge.

Reconciliation aligns the bot's model with on-chain reality. It runs at startup when the gap is widest (bot was offline, fills happened, grid may have been regenerated).

Why Not Cancel Everything

• No atomic cancel+create on BitShares — limit_order_cancel cancels the full order; there is no partial size reduction
• A full teardown leaves the bot unable to trade during the rebuild window

Architecture: 3-Phase Plan-then-Execute

The reconcile runs from recalculateGrid during startup full-resync. There is no per-attempt wall-clock race around the reconcile itself — an outer timeout would fire mid-batch and orphan in-flight broadcasts (the duplicate-accumulation death spiral). The reconcile is bounded only by the 10-minute total resync safety net, and every internal chain read follows the shared guarded-read standard.

Phase 1 does all reasoning in memory under _gridLock (fast); Phases 2 and 3 execute outside that planning lock — holding it across RPC calls would block fills, sync, and divergence checks. Phase 2 holds the broadcasting single-flight marker, while the sync/apply paths it invokes follow the canonical lock hierarchy. No planning-phase grid lock spans a network operation.

Phase 2 and 3 both respect the dryRun flag: when true, no on-chain mutations are attempted — plans are logged but not executed.

targetCount (per side, targetSell/targetBuy) is sourced from bot config (activeOrders window plus reserveOrders edge ladder) and determines how many live orders each side should maintain. The internal planOnly flag controls whether _reconcileStartupSide records plans for Phase 2 or executes inline — Phase 1 always calls with planOnly=true.

                    Grid generated
                            │
                            ▼
┌──────────────────────────────────────────────────┐
│  PHASE 1: Planning (under _gridLock)             │
│                                                  │
│  • Sanitize phantom orders (ACTIVE/PARTIAL with  │
│    orderId absent on-chain → VIRTUAL, skip);     │
│    defer freshly-assigned orderIds, ghost pass   │
│  • Detect suspected duplicates (exact price) →   │
│    queue for Phase 2 cancel                      │
│  • Match unmatched chain orders to virtual slots │
│    → plan updates                                │
│  • Detect grid-edge lockup → plan largest-order  │
│    cancel to free funds                          │
│  • Detect excess chain orders → plan cancels     │
│  • No on-chain RPC calls inside this phase       │
└──────────────────┬───────────────────────────────┘
                   ▼
   returns { plannedCreates, plannedUpdates, plannedCancels }
┌──────────────────────────────────────────────────┐
│  PHASE 2: Execution (outside _gridLock)          │
│                                                  │
│  1. Cancellations — revalidate each plan against │
│     live ownership, geometry, and chain state    │
│  2. Updates — batch (3 retries), then sequential │
│     fallback with per-failure recovery sync      │
│  3. Creates — outside-in pairing (outermost grid │
│     slots first, BUY desc / SELL asc), batched   │
│     where the DEX supports batch creates;        │
│     uncertain lands adopt; created IDs tracked   │
│     for Phase-3 protection                       │
└──────────────────┬───────────────────────────────┘
                   ▼
┌──────────────────────────────────────────────────┐
│  PHASE 3: Fresh Count + Stale Surplus Cleanup    │
│                                                  │
│  • Re-fetch chain state after Phase 2 (guarded)  │
│  • Adopt uncertain-landed creates into VIRTUAL   │
│    slots                                         │
│  • Protect Phase-2 created IDs from surplus      │
│    cancel                                        │
│  • Cancel orders exceeding per-side target not   │
│    tracked by any grid slot's orderId (orphans)  │
└──────────────────────────────────────────────────┘

Phase 1 — Pure Planning Under _gridLock

Source: reconcileGridOrders() Phase 1 in grid_reconcile.ts.

1. Phantom order sanitization (lines 216-245): For each isOrderPlaced() order whose orderId is not in the chain snapshot, reset it to VIRTUAL with skipAccounting to prevent fund inflation. Two absence-decision guards make this safe:

   • Freshly-assigned deferral (lines 226-235): an orderId assigned within TIMING.SYNC_LOCK_TIMEOUT_MS (via manager._orderIdAssignedAt) may be an in-flight create/adopt whose broadcast has not landed or is not yet visible to a lagging/truncated read — virtualizing it and re-creating would duplicate a real live order (the reconcile-timeout death-spiral root cause). It is skipped (deferred) rather than virtualized.
   • Ghost heuristic (line 229): an order with size <= 0 && PARTIAL (a known filled ghost) still passes through so known fills get cleaned up.
   • Virtualization always uses { skipAccounting: true } so startup cleanup never inflates ChainFree.

2. Duplicate detection (lines 258-331): For each unmatched chain order, find the nearest active same-side grid order. If its price equals that grid order's slot price exactly (priceSlotEqual at the asset precision), flag it as a suspected duplicate and queue for a Phase 2 cancel (never cancelled under lock). Non-equal neighbours are only logged with nearest-same-side diagnostics and continue into per-side reconciliation. Duplicate IDs are removed from the unmatched set so they aren't also paired for updates/creates.

3. Per-side reconciliation via _reconcileStartupSide(planOnly=true) (lines 343-372):

   • Count matchedOnGrid (active grid orders with orderId)
   • neededSlots = targetCount - matchedOnGrid; pick virtual slots to activate
   • Match sorted unmatched chain orders to virtual slots → plannedUpdates
   • Detect grid-edge lock and plan a largest-order cancel
   • Plan creates for remaining slots
   • Plan excess cancellations (guarded by matchedOnGrid > 0): orphans first, then matched surplus (chainCount - targetCount, reserve edge slots last) — the matched-excess selection is shared by the planOnly and execute branches so planning can never drift from execution; planOnly omits releaseUntrackedFunds for matched slots (their funds are tracked on the grid slot)
   • Fork-kept shelf orders (live non-slot-N ids below the rail, e.g. deep-*) are never cancel candidates or reserve members — see Shelf Orders
   • Vacated-rail refill: each PROCEEDING update whose vacated price exactly matches (priceSlotEqual) an empty, sized, in-rail slot of the same side queues a refill CREATE in the same plan (source startupVacatedRailRefill) — skipped updates, ghost prices (lattice moved), in-band slots, and already-desired slots never refill; refill targets require VIRTUAL state with no orderId

Returns { plannedCreates, plannedUpdates, plannedCancels, chainSellCount, chainBuyCount }.

Phase 2 — Blockchain Execution Outside Lock

Source: reconcileGridOrders() Phase 2 in grid_reconcile.ts.

Phase 1 releases _gridLock before network I/O. Phase 2 runs under the broadcasting single-flight marker so a fill-driven COW rebalance cannot plan the same slots concurrently; individual sync/apply paths follow the canonical lock hierarchy. No planning-phase _gridLock is held across RPC calls.

Cancellations — v1.6.6 stale-plan guard: Before submitting any plannedCancels, reconcile performs one guarded pre-cancel read and builds a per-order signature (type, price, size) from both the Phase-1 and current snapshots. For every plan, _startupCancelPlanStillCurrent() requires:

• the target chain order still exists with its exact Phase-1 signature;
• a matched plan still owns the same chain order in the same live grid slot;
• planned type, slot price, boundary, and gap geometry still match; and
• an originally unmatched surplus is still not owned by any live slot.

Validation is per-plan: an earlier cancellation or unrelated order change does not invalidate an otherwise untouched plan. A truncated or failed pre-cancel read skips all cancellation submissions. A complete empty read is authoritative but likewise causes no plans to execute because no target remains present. This closes stale-cancellation replays without requiring the whole book to remain unchanged.

Surplus settlement uses the same live-ownership decision. Only an order that is still untracked and signature-unchanged reaches _cancelChainOrder; a now-owned or geometry-changed order is skipped rather than released from the stale snapshot.

Updates:

• Batch via _executeStartupUpdateBatch when supportsBatchUpdate is available
• Retry up to 3× (maxBatchAttempts = 3)
• On each failure: _recoverStartupSyncFailure() re-fetches open orders from chain (guarded read) and re-syncs manager state via manager.syncFromOpenOrders(), then _refreshStartupUpdatePlans() rebuilds plans against the fresh chain state
• If retries are exhausted or batch helpers are unavailable → _executeStartupSequentialUpdateFallback() one-by-one with per-failure recovery

Creates: _executePlannedStartupCreates groups the outermost grid slots toward the center (BUY descending / SELL ascending), so the most price-critical orders are placed first. BitShares DEX batch-create operations are used where supported. Every created chain ID is captured into phase2CreatedOrderIds so Phase 3 cannot later cancel the freshly-created orders.

Phase 3 — Fresh Re-read, Adoption, Stale Surplus Cleanup

Source: Phase 3 in reconcileGridOrders() (guarded by if (!dryRun)).

1. Guarded fresh re-read: readOpenOrdersGuarded re-fetches all open orders. A truncated result returns null and Phase 3 defers, keeping the pre-Phase-2 counts for the summary log; a capped window omits exactly the freshest Phase-2 creates.

2. Adopt uncertain-landed creates: For any fresh chain order not matching a grid orderId and not created by a slot, it attempts targeted slot adoption — matching a VIRTUAL slot by type+price+size (within tolerance) and registering it via _applySync(..., 'createOrder') with the create-fee deduction. Full syncFromOpenOrders is deliberately not used here (its pass-1 virtualizes ACTIVE slots missing from the snapshot, and a lagging read right after the Phase-2 broadcast would destroy the confirmed grid). If adoption fails, the ID is still protected from surplus-cancel; the next sync loop's orphan adoption registers it.

3. Stale surplus cancellation: Per side, count orders exceeding targetCount that no grid slot holds via orderId (including the Phase-2 created IDs). Cancel only these untracked surplus orders, sorted by chain ID for determinism. This catches orphans lost during grid reinitialization — on-chain orders with no corresponding grid slot.

Partial Failure State

If Phase 2 partially succeeds (some cancels, some creates fail), there is no rollback. The bot proceeds with the resulting state. Because the reconcile runs at startup before the fill pipeline activates, no fills are missed during this window. Remaining mismatches are caught by the next maintenance/structural divergence cycle or the next startup reconcile.

Timeouts and Read Coverage

• No per-attempt race around the reconcile itself — the 1.4.8 change removed it to avoid orphaning mid-batch broadcasts (see the recalculateGrid call site in modules/order/grid.ts).
• The whole resync is bounded by a 10-minute total timeout (PIPELINE_TIMING.TIMEOUT_MS * 2) and applied with Promise.race in recalculateGrid().
• Every internal chain read goes through readOpenOrdersGuarded with the 30s / 3-retry / node-failover standard. Truncated reads are always ambiguous and return null; empty reads are ambiguous only when a caller explicitly sets deferEmpty: true. Phase-2 cancellation uses the default empty-aware policy, then validates each plan against the fresh per-order signature before any mutation.

Edge Cases (All Hit in Production or Code Review)

Fresh Grid Guard (matchedOnGrid > 0)

_reconcileStartupSide() in grid_reconcile_internal.ts

When a brand-new grid is generated, every slot is VIRTUAL — matchedOnGrid = 0. Without a guard, every on-chain order appears "unmatched" and would be cancelled as excess:

if (matchedOnGrid > 0 || neededSlots === 0) {
    cancelCount = Math.max(0, chainCount - targetCount);
}

When matchedOnGrid === 0 AND scaling up (neededSlots > 0), excess cancellation is skipped — the guard covers both the fresh-grid scenario and the scale-down case (neededSlots === 0). Stale duplicates are still caught by the SUSPECTED DUPLICATE detection in Phase 1 of reconcileGridOrders.

Grid-Edge Lock

_isGridEdgeFullyActive() in grid_reconcile_internal.ts detects when the grid boundary is fully active (all slots on-chain) before cancelling excess orders.

When all outermost orders of a side are ACTIVE with orderId, all balance is committed to the edges. Cancel the largest order among the update candidates (_cancelLargestOrder()) to free maximum funds with minimum operations, since the DEX does not expose partial-reduce in one operation. The cancelled slot gets a replacement create.

Detection sorts orders by price (BUY descending, SELL ascending) and checks that the outermost ones are all isOrderPlaced().

Duplicate Cancellation

reconcileGridOrders() in grid_reconcile.ts

An unmatched chain order whose price equals an active same-type grid slot's price — exact slot-price equality via priceSlotEqual at the asset precision — is a suspected duplicate → queued for Phase 2 cancellation (not cancelled under lock). Non-equal neighbours are only logged with nearest-same-side diagnostics, never cancelled here. The earlier fuzzy SUSPECTED_DUPLICATE_TOLERANCE_MULTIPLIER (5× calculatePriceTolerance) was removed.

Batch Update Failure Recovery

Up to 3 batch attempts (maxBatchAttempts = 3). Each failure triggers a recovery sync + plan refresh. If all plans are empty → resolved early. After 3× → sequential fallback with per-plan recovery (each individual failure triggers a recovery sync + queue refresh).

Phantom Orders via Reconcile

Reconcile's role in the defense-in-depth: during Phase 1, any ACTIVE/PARTIAL order whose orderId is not found on-chain is reset to VIRTUAL with skipAccounting. The freshly-assigned deferral protects in-flight broadcasts, and the ghost heuristic lets known fills pass. See developer_guide.md for the full 3-layer defense.

Shelf Orders (Fork-Kept Manual Orders)

Live on-chain orders with non-slot-N ids below the rail (e.g. deep-* manuals kept across a fork) are shelf orders: they sit outside the grid contract and must survive every startup path untouched. Three gates enforce this (all no-ops on grids that only mint slot-N ids):

• Reserve classification/placement (order.ts, grid_reconcile_internal.ts, manager.ts): reserveEdgeIdSet, the Tier-2 live-anchor scan, _pickEdgeReserveSlots, and pickEdgeReserves all gate to parseSlotIndex(id) !== null, so a shelf can never count as the reserve edge (which would mask a real reserve deficit) nor be activated as a reserve it would never be counted as. The live-reserve count additionally excludes window members via liveWindowIdSet, so an edge-reaching window cannot masquerade as dedicated reserves.
• Startup excess cancels (grid_reconcile_internal.ts): the matched-excess selection filters to slot-N ids in both planOnly and execute branches — the cheapest-first sort would otherwise wipe the shelf on the next boot.
• Geometric size recalc (grid.ts): _recalculateGridOrderSizesFromBlockchain skips non-slot-N slots in the per-slot loop, so divergence-triggered resizing never overwrites manual shelf sizes on-chain (the shelf stays in the budget denominator, so allocation math is unchanged).

COW Interaction

Reconcile Phase 1 runs under _gridLock with no side effects on the frozen master Map. The working grid is not involved — reconcile is a startup operation that runs before the COW pipeline is active. See COPY_ON_WRITE_MASTER_PLAN.md and COW_INVARIANTS.md for COW rules.

Slot-Price Invariant at the Reconcile Emission Sites

Three of the six guarded emission sites are reconcile sites (RECONCILE-CREATE, RECONCILE-UPDATE, STARTUP-CREATE in grid_reconcile_internal.ts): every op reconcile emits is checked against the slot's genesis level and an off-grid emission is skipped, not broadcast — see GRID_PRICE_INVARIANT.md. The coupling is bidirectional: the invariant guard's persistent-rejection escalation and the deferred-hold escalation both exit through the same structural resync (debounced reload → full reset) described here, and a full reset's update-first reconcile emits the rail's genesis level, so the guard does not block its own resolution.

Truncated-Read Ambiguity (since 1.4.8)

Every chain read feeding an absence/surplus decision goes through readOpenOrdersGuarded. A truncated snapshot is always unreadable. An empty snapshot is also unreadable at callers that set deferEmpty: true; the Phase-2 cancellation pre-read intentionally accepts a complete empty result, executes no plan, and leaves all remaining state for the normal reconcile paths:

• _recoverSyncFromChain() — plus its recovery sites in _createOrderFromGrid / _cancelChainOrder — defers on empty/truncated reads (deferEmpty: true). A pass-1 phantom cleanup would otherwise virtualize live slots from a partial window.
• _adoptPossiblyLandedCreate() defers to an uncertain outcome on truncated reads, and the startup group batch uncertain verification follows the same rule.
• Phase 3 final refresh skips adoption/surplus-cancel on a truncated read, keeping the pre-phase-2 counts for the summary log.
• Adoption paths (_adoptPossiblyLandedCreate, grouping path, reconcile adoption loop) apply the create-fee deduction via _applySync for accounting parity.

The underlying rule is INV-BROADCAST-004: a capped get_full_accounts window omits the freshest orders (fresh creates sort last), so absence can never be authoritative on a truncated read.

Lock Hierarchy

manager.ts lock declaration — canonical reference in developer_guide.md.

Level 0: _fillProcessingLock    Level 1: _divergenceLock
Level 2: _syncLock              Level 3: _gridLock
Level 4: _fundLock

Acquire in ascending level order only. AsyncLock is re-entrant (nested acquire() run directly, not queued).

Historical Correction (1.4.6)

Before 1.4.6, _syncLock was Level 3 and _gridLock was Level 2, causing ABBA deadlock when reconcile needed _gridLock (old Level 2) while holding _syncLock (old Level 3). The workaround flag gridLockAlreadyHeld patched 8 call sites.

Commit 705cde9c fixed it: swapped levels (_syncLock → 2, _gridLock → 3), eliminated the flag, and restructured Phase 1 to be purely in-memory so no RPC calls run under _gridLock (developer_guide.md §Startup Sequence).

Nesting Safety (1.4.6)

Commit e64db685 replaced 6 single-value boolean state fields with refcounts/stacks to prevent premature resume from re-entrant nested acquisitions.

Key Constants

Testing Coverage

File Reference