Skip to main content

TBS run queue - the 3 Nov 2026 stuck queue and capacity to-do list

Written 3 Nov 2026 after an incident on the VPS. Code: backend/app/api/tbs.py (queue), backend/tbs/runner.py and sweep.py (workers), backend/tbs/pool.py (worker pool, new). Related: knowledge-base/tbs-backtester.md (queue and limits), knowledge-base/vps-deploy-busy-guard.md.

1. What happened (3 Nov 2026)​

A user's new run (#518) stayed "Queued: 2 runs ahead of it". Checked over SSH, read only:

ObservationValue
Server load0.00, nothing computing
Runs marked RUNNING in bt_run#499 (started 17:59 IST, 494 of 494 trades done), #517, #518
Processes under the uvicorn processone idle worker, one defunct (dead, never reaped)
Runs cancelled in the same hourabout 18 (a user trying runs and cancelling them)

How a run takes the slot: one lock lets only one backtest execute at a time. A run releases it only when it completely finishes (_execute -> store.finish_run). #499 computed every trade but never got to finish_run, so the lock was never released and every later run waited for it. The cancelled runs left the queue normally; #517 and #518 were the two "ahead".

Most likely cause (not proven; no stack dump was possible): with more than one worker, run_backtest used with ProcessPoolExecutor(...). Leaving that block waits for every worker process to exit. On Linux the pool forked the threaded web server, which copies locks held by other threads (logging, database, DuckDB) into the child; such a child can hang at start or at exit. Windows starts workers fresh, which is why a local run never showed it. The leftover idle and defunct processes fit this.

2. What was fixed (commit 44fdbf6)​

  • Workers are spawned, not forked (tbs/pool.py, make_pool), for backtests and bulk sweeps. Cost: a few seconds of imports at the start of each run.
  • The pool is closed with a time limit (close_pool): pending work is dropped, a worker still alive after 15 s is killed. The lock is always released, whatever a worker does.
  • Stale runs are failed at startup (store.fail_stale_runs, sweep_store.fail_stale, called from main.py startup): a run or sweep still RUNNING / QUEUED when the server starts was cut off by a restart or crash and is marked FAILED, instead of staying "active" for ever.
  • Deploy note: the busy guard of pull.sh correctly refused while the stuck workers existed; ./pull.sh --force was needed once, then the service restart cleared the lock.

3. How the queue behaves today with 10 or more users​

SituationBehaviour
10 users start a run togetherthe first runs, nine wait ("N runs ahead"), first in first out; bulk sweeps share the queue
More than 20 waitingthe next request gets 429 "queue is full" (TBS_MAX_QUEUE)
A user closes the browserthe run keeps going and keeps its turn
CPU protectionone run at a time, workers = CPUs - 2 (4 on the 6-CPU VPS, TBS_MAX_WORKERS), low priority
Memoryone run's worth at a time
Restart with runs waitingthe in-memory queue is lost; the runs are marked FAILED at the next start and must be run again

The cost is waiting time: a run takes about 2 s per trading day per worker when the days are not cached; a 6-month run is under a minute with 4 workers, a multi-year uncached run takes many minutes, so the last of ten long runs can wait an hour or more.

4. To do​

#ItemWhyNotes
1Limit waiting runs per user (for example 2)one user can queue up to 20 runs and block everybody (there is no per-user limit)count QUEUED + RUNNING per usrid in start_run / start_sweep, 429 with a clear message
2Show an estimated wait in the queued bannerthe page only says "N runs ahead"estimate from the trade-day count of the runs ahead and the measured seconds per day of recent runs
3Date-range limit for uncached periodsTBS_MAX_RANGE_DAYS and TBS_MAX_SWEEP_DAYS are 0 (off); a 7-year run is allowedset a value, or limit only the part the day cache does not hold
4Second run slot (for example 2 runs of 2 workers)short runs wait behind long onesneeds a lock per slot; check CPU on the VPS under load first
5Queue survives a restartwaiting runs are lost on every deploypersist QUEUED rows (they already exist in bt_run) and re-queue them at startup instead of failing them
6Watchdog on a running runa run with no progress for N minutes should be failed and the lock released even if the pool is not the causethread in _pump that checks _ACTIVE progress; the pool time limit covers only the pool
7Memory cap per runthere is none; many legs over a long range use more memorymeasure peak RSS of a 7-year run, then set a limit
8Prove the root causethe fork deadlock is inferred, not capturedif it ever recurs, take a py-spy dump of the stuck process before restarting
9Several uvicorn processesthe lock is per process; N processes would each run one backtestkeep one process, or move the lock to the database (an advisory lock)
10Cancel of a running run waits for the current tradecancel is checked between tradesacceptable; worth a "cancelling" state on the banner

5. Checking the queue yourself (SSH, read only)​

cd /home/kss/tradeentry/backend
PYTHONPATH=. venv/bin/python -c "from nseeod import pgs; print(pgs.read_sql(\"SELECT id, status, name, created_at, trades_done, trades_total FROM bt_run WHERE status IN ('RUNNING','QUEUED') ORDER BY id\").to_string())"
ps --ppid $(pgrep -f "uvicorn app.main:app" | head -1) -o pid,etimes,stat,args
uptime

A RUNNING run with trades_done = trades_total for many minutes, no CPU load and old worker processes under uvicorn is the signature of this incident (after the fix it should not happen; a restart clears it).