Broker disconnected
Bad keys, a broker outage and the circuit breaker are three different things. What halts, what keeps running, how to reconnect, and how to resume.
- Who it is for
- Owners & operators
- Reading time
- 4 min read
- Updated
The is the only part of Bellwether that talks to your broker, and it trusts the broker over its own books. When that connection breaks, it stops sending and says so. Three different things hide behind "broker disconnected".
Three different problems
| What happened | What you see | What to do |
|---|---|---|
| Bad or revoked keys. The broker answered "not authorised" | An immediate (broker_auth_error) and a critical alert on Overview reading "Broker connection needs attention" | Reconnect the keys (below), then Resume |
| A blip. A timeout, a 5xx or a rate limit | Nothing, if it recovers: the call is tried up to 3 times with a short backoff. If not, that tick fails closed (nothing is submitted) and an ops alert lands on Activity › Alerts | Nothing |
| The circuit breaker. 5 consecutive failed broker attempts, usually within seconds of each other | A halt (broker_circuit_open) and a critical alert "Broker connection paused after repeated failures" | Check the broker's status page; Resume when it is back |
The breaker never re-arms itself: probing the broker and flipping in and out of a halt during an outage would bury the first cause. Resume is manual: pressing Resume ( required) clears the halt and resets the breaker on the Executor's next tick. If the broker is still down, the breaker opens again within seconds and the halt returns; nothing was sent in between. A broker that says "not authorised" is never retried: retrying bad keys is wrong, so it halts at once.
What halts and what keeps running
A halt blocks new orders, and only that.
- Stops: every submission. the PM (shown as on ) makes while halted are blocked when the Executor reaches them (in the Inbox they read with the halt as their note), approved-but-unsent orders too.
- Keeps running: the Executor loop (it keeps checking the broker), the PM's sessions and briefs, , and your open positions with their resting day orders, which live at the broker and are untouched.
- Goes stale: equity marks and position values, which come from the broker; expect Overview to freeze until the connection is back.
The reason is on Risk and under Risk › Halts with its time; the banner on every screen only says that trading is halted and how long ago.
The env pill and executor health
The PAPER or LIVE pill in the top bar names which account the book runs on; it does not report the connection. The Executor writes a health row after every tick, failed ones included, but the console does not draw it as a pill yet; the operator sees it on Admin under Health. You notice a stalled Executor as frozen equity, proposals that stay queued, and the alerts above; after 5 failed ticks in a row (about 25 seconds) it raises one critical "Executor failing" alert for the streak.
Orders found at the broker
At the start of every tick, before anything expires, the Executor looks for broker orders missing from its books.
- An order it placed but did not record (a crash right after the submit) is adopted and tracked as if nothing happened; a warning names it. It is never resubmitted, so there is never a duplicate: resubmission is idempotent on
client_order_id. - An adopted order whose proposal has since expired or been blocked is cancelled at the broker: the books no longer want it.
- An order you placed yourself in the broker's app is tracked so its fills reach your books, with the alert "Found an order at the broker that Bellwether did not place". Bellwether never cancels it.
Reconnecting keys
Keys are replaced, never edited in place, and replacing them asks for your authenticator code.
- Open Connect your broker. It shows the last error the Executor recorded for the old connection.
- Paste a new paper key pair. The Executor detects the environment from the keys; a live pair is labelled live, refused, and leaves the book parked.
- Wait while the page says it is verifying: the Executor probes the new keys within about 30 seconds and the connection turns verified (or shows the broker's reason if they are wrong too). The old connection is disabled.
- Press Resume in the halt banner and enter your code.
For the house book, whose keys live in the deployment rather than in the console, Accounts › Your connection says the connection is configured by the operator on the server and offers no Reconnect: the operator rotates the key (halt, new trading-only key at the broker, store it in the deployment, restart the Executor, confirm a heartbeat, revoke the old key, Resume).
When the database is unreachable
If Bellwether's own database goes away, the Executor pauses: nothing is submitted or half-done, a halt it cannot write is held and retried every tick, and alerts it cannot write are kept in memory (the last 50) and written once the database is back, marked as replayed, with one "Executor db recovered" alert naming the blind stretch. Queued proposals resume on their own. In the first 45 seconds after a deploy, transient connection errors are logged rather than alerted.