TBS run queue - the 3 Nov 2026 stuck queue and capacity to-do list
Written 3 Nov 2026 after an incident on the VPS. Code:
backend/app/api/tbs.py(queue),backend/tbs/runner.pyandsweep.py(workers),backend/tbs/pool.py(worker pool, new). Related:knowledge-base/tbs-backtester.md(queue and limits),knowledge-base/vps-deploy-busy-guard.md.
1. What happened (3 Nov 2026)
A user's new run (#518) stayed "Queued: 2 runs ahead of it". Checked over SSH, read only:
| Observation | Value |
|---|---|
| Server load | 0.00, nothing computing |
Runs marked RUNNING in bt_run | #499 (started 17:59 IST, 494 of 494 trades done), #517, #518 |
| Processes under the uvicorn process | one idle worker, one defunct (dead, never reaped) |
| Runs cancelled in the same hour | about 18 (a user trying runs and cancelling them) |
How a run takes the slot: one lock lets only one backtest execute at a time. A run releases it only when it completely finishes (_execute -> store.finish_run). #499 computed every trade but never got to finish_run, so the lock was never released and every later run waited for it. The cancelled runs left the queue normally; #517 and #518 were the two "ahead".
Most likely cause (not proven; no stack dump was possible): with more than one worker, run_backtest used with ProcessPoolExecutor(...). Leaving that block waits for every worker process to exit. On Linux the pool forked the threaded web server, which copies locks held by other threads (logging, database, DuckDB) into the child; such a child can hang at start or at exit. Windows starts workers fresh, which is why a local run never showed it. The leftover idle and defunct processes fit this.
2. What was fixed (commit 44fdbf6)
- Workers are spawned, not forked (
tbs/pool.py,make_pool), for backtests and bulk sweeps. Cost: a few seconds of imports at the start of each run. - The pool is closed with a time limit (
close_pool): pending work is dropped, a worker still alive after 15 s is killed. The lock is always released, whatever a worker does. - Stale runs are failed at startup (
store.fail_stale_runs,sweep_store.fail_stale, called frommain.pystartup): a run or sweep still RUNNING / QUEUED when the server starts was cut off by a restart or crash and is marked FAILED, instead of staying "active" for ever. - Deploy note: the busy guard of
pull.shcorrectly refused while the stuck workers existed;./pull.sh --forcewas needed once, then the service restart cleared the lock.
3. How the queue behaves today with 10 or more users
| Situation | Behaviour |
|---|---|
| 10 users start a run together | the first runs, nine wait ("N runs ahead"), first in first out; bulk sweeps share the queue |
| More than 20 waiting | the next request gets 429 "queue is full" (TBS_MAX_QUEUE) |
| A user closes the browser | the run keeps going and keeps its turn |
| CPU protection | one run at a time, workers = CPUs - 2 (4 on the 6-CPU VPS, TBS_MAX_WORKERS), low priority |
| Memory | one run's worth at a time |
| Restart with runs waiting | the in-memory queue is lost; the runs are marked FAILED at the next start and must be run again |
The cost is waiting time: a run takes about 2 s per trading day per worker when the days are not cached; a 6-month run is under a minute with 4 workers, a multi-year uncached run takes many minutes, so the last of ten long runs can wait an hour or more.
4. To do
| # | Item | Why | Notes |
|---|---|---|---|
| 1 | Limit waiting runs per user (for example 2) | one user can queue up to 20 runs and block everybody (there is no per-user limit) | count QUEUED + RUNNING per usrid in start_run / start_sweep, 429 with a clear message |
| 2 | Show an estimated wait in the queued banner | the page only says "N runs ahead" | estimate from the trade-day count of the runs ahead and the measured seconds per day of recent runs |
| 3 | Date-range limit for uncached periods | TBS_MAX_RANGE_DAYS and TBS_MAX_SWEEP_DAYS are 0 (off); a 7-year run is allowed | set a value, or limit only the part the day cache does not hold |
| 4 | Second run slot (for example 2 runs of 2 workers) | short runs wait behind long ones | needs a lock per slot; check CPU on the VPS under load first |
| 5 | Queue survives a restart | waiting runs are lost on every deploy | persist QUEUED rows (they already exist in bt_run) and re-queue them at startup instead of failing them |
| 6 | Watchdog on a running run | a run with no progress for N minutes should be failed and the lock released even if the pool is not the cause | thread in _pump that checks _ACTIVE progress; the pool time limit covers only the pool |
| 7 | Memory cap per run | there is none; many legs over a long range use more memory | measure peak RSS of a 7-year run, then set a limit |
| 8 | Prove the root cause | the fork deadlock is inferred, not captured | if it ever recurs, take a py-spy dump of the stuck process before restarting |
| 9 | Several uvicorn processes | the lock is per process; N processes would each run one backtest | keep one process, or move the lock to the database (an advisory lock) |
| 10 | Cancel of a running run waits for the current trade | cancel is checked between trades | acceptable; worth a "cancelling" state on the banner |
5. Checking the queue yourself (SSH, read only)
cd /home/kss/tradeentry/backend
PYTHONPATH=. venv/bin/python -c "from nseeod import pgs; print(pgs.read_sql(\"SELECT id, status, name, created_at, trades_done, trades_total FROM bt_run WHERE status IN ('RUNNING','QUEUED') ORDER BY id\").to_string())"
ps --ppid $(pgrep -f "uvicorn app.main:app" | head -1) -o pid,etimes,stat,args
uptime
A RUNNING run with trades_done = trades_total for many minutes, no CPU load and old worker processes under uvicorn is the signature of this incident (after the fix it should not happen; a restart clears it).