Skip to main content

Task: VPS Rollout — 3 Oct 2026

Built and verified locally on 2026-10-03. Run everything on the VPS from /home/kss/tradeentry/backend with venv/bin/python, long steps inside tmux.

#PieceNeeds on the VPS
AEOD Chart / TEChart fixes, Corporate Actions universe filter, Stock Universe pagecode only
BTE-FO Index (te_asset / te_price + idx_* membership)F&O history, corporate actions, idx_* tables (Sept rollout steps 6–8), then one build
CMCX F&O history 2003 →already complete on the VPS (checked 2026-10-03); only for a fresh database

No new Python packages, no Alembic migration: TE-FO writes into the existing te_asset / te_price / idx_* tables, and creates the idx_* tables + idx_members_on() itself if they are missing.

0. Before deploying: check the VPS​

Sept rollout state (TE-FO depends on it). The full F&O history must start in 2000, scr_corp_action must exist, and the index tracker must be loaded:

venv/bin/python - <<'EOF' 2>&1 | grep -v "^\[Config\|^\[pgs"
from nseeod import pgs
r = lambda s: pgs.read_sql(s).iloc[0, 0]
print("F&O history :", r("SELECT min(sdate)::text || ' -> ' || max(sdate)::text FROM scr_nsefo_eod"), "(want 2000-06-12 -> today)")
for t in ("scr_corp_action", "idx_membership", "te_asset", "te_price"):
print(f"{t:<16}:", "exists" if r(f"SELECT to_regclass('{t}') IS NOT NULL") else "MISSING")
if r("SELECT to_regclass('idx_membership') IS NOT NULL"):
print("NIFTY 50 today :", r("SELECT count(*) FROM idx_members_on('NIFTY 50', CURRENT_DATE)"), "members (want 50)")
EOF
  • F&O from 2000 + scr_corp_action + idx_membership present → step B can run.
  • Anything missing → finish the Sept rollout steps 6 (corporate actions), 7 (F&O history + lots) and 8 (index tracker) first. Step A deploys fine without them.
  • te_asset / te_price missing → venv/bin/alembic upgrade head.

MCX state (decides step C) - only if scr_mcxfo_eod exists; this command arrives with the deploy in step 1, so run it right after that:

venv/bin/python -m mcxeod.mcx_backfill --status

1. Commit, push, deploy​

On the PC: commit and push main (all of the 2026-10-03 work is in one working tree).

On the VPS, use the repo copy of the deploy script (the /home/kss/pull.sh copy once skipped the frontend rebuild):

cp /home/kss/tradeentry/pull.sh /home/kss/pull.sh && bash /home/kss/pull.sh

It pulls, rebuilds the frontend and docs (both changed), and restarts the tradeentry service. Check the summary shows frontend rebuilt and service restarted.

2. TE-FO Index — te_fo_index.py​

Only after step 0's checks pass. Don't run the Bhav Downloader before this build - its daily hook would otherwise start a first build in the background.

venv/bin/python -m nseeod.te_fo_index build # dry run (~2 min): prints NAV / CAGR / checks
venv/bin/python -m nseeod.te_fo_index build --apply # writes te_asset 'TE-FO', te_price, idx_* 'TE-FO INDEX'
venv/bin/python -m nseeod.te_fo_index members --date 2008-06-30

Expect roughly the PC numbers (small differences are possible where the two F&O databases differ):

CheckPC (2026-10-03)
Range2001-11-29 → last F&O day, 298 expiry cycles
NAV on 2026-09-255,666.91 (CAGR 7.2%, max drawdown −73.4%)
Dry-run checksOHLC consistency: 0 bad rows, 284 scaled, 0 not confirmed, 0 cycles skipped
Written6,169 te_price rows, 47,865 snapshot rows, 679 membership intervals, 207 members now
Members on 2008-06-30227 (includes delisted AIRDECCAN, ADLABSFILM)

If 0 not confirmed / 0 skipped do not hold, stop and compare with the PC before --apply.

If the index tracker is ever rebuilt (index_members build --apply), the TE-FO rows are kept (index_members.EXTERNAL). This protection is new; running an older index_members would wipe them.

3. MCX F&O history — read mcx_backfill --status from step 0​

Checked 2026-10-03: the VPS is already complete - 6,359 trading days, 2003-11-18 → 2026-09-30, 17.0M rows; SILVERM-I 6,298 days from 2004-02-17. Missing only 2026-10-01/02 (next daily MCX download) and two old Saturdays, 2004-07-03 and 2005-01-15 (venv/bin/python -m mcxeod.mcx_backfill --from 2004-07-03 --to 2005-01-15, optional). No archive import on the VPS. The table below stays for a fresh database.

What --status showsDo
Days from 2003-11-10, ~250–300 per year, 0 missing (or only the last day or two)nothing - history is complete
Some missing ranges (weekly-spaced = Saturday sessions; long range = real gap)venv/bin/python -m mcxeod.mcx_backfill --from <first missing date> - downloads only the missing days
Most of 2003–2026 missingarchive import below, then the backfill
scr_mcxfo_eod does not exist (VPS still on the old MCX tables)venv/bin/python -m mcxeod.mcx_infra, venv/bin/python -m mcxeod.mcx_history, then archive import + backfill

Archive import (much faster than downloading 21 years):

# PC (WSL): copy the archive (82 MB)
rsync -av /mnt/d/kss/stock/EODData/MCXBhavf10Nov2003-23Oct2024.zip kss@<vps>:/home/kss/EODData/

# VPS
venv/bin/python -m mcxeod.mcx_archive_import import --src /home/kss/EODData/MCXBhavf10Nov2003-23Oct2024.zip
venv/bin/python -m mcxeod.mcx_backfill --from 2003-11-10 # Saturdays (not in the archive) + Oct 2024 -> today
venv/bin/python -m mcxeod.mcx_backfill --status # want ~0 missing
  • The import is resumable (skips stored days) and runs oldest day first; on the PC it ran at about 1,000 days per 6–7 minutes. Run the backfill after it: change-in-OI comes from the previous stored day.
  • The archive has no Saturday files, but MCX held ~490 Saturday sessions; the backfill fetches them.
  • Verified on the PC: every traded contract identical to a live download; only untraded contracts' theoretical close differs (MCX revised them later).

4. Check in the browser​

PageCheck
EOD Chart · NSE · SPOT · NIFTY (full range), TEChart engineWhole chart fits the pane, latest candles visible, no "zoomed" top slice; switch to RELIANCE / BANKNIFTY - stays fitted
EOD Chart · NSE · FUTTE-FO Index chip next to "Available Stocks" → full history line from Nov 2001
… then click a stock (e.g. ABB)Chart switches at once to ABB FUT on the current expiry (no TE-FO chart under the stock's name)
EOD Chart · MCX · CFUT · SILVERMHistory from 2004 once step 3 is done
Corporate Actions"Universe (any of)" + "As on": TE-FO Index + NIFTY 500 on 01-01-2005 narrows the list and the chips; SME shows only SME stocks trading that day
Stock Universe (sidebar, developer)TE-FO Index today ≈ 207 members; NIFTY 50 = 50; Changes panel lists IN / OUT

5. Daily operation​

  • NSE Bhav Downloader now also extends TE-FO (log tag TE-FO, e.g. TE-FO Index: 1 day(s) from 2026-09-29, ... NAV ...). A failure there is logged only and never fails the bhav sync.
  • After any F&O history backfill, rerun te_fo_index build --apply (the daily update only recomputes from the last expiry).
  • MCX downloads are unchanged; mcx_backfill --status is the quick health check.

Rollback​

  • Code: git revert the commit and run pull.sh again.
  • TE-FO data (derived, rebuildable): DELETE FROM te_price WHERE asset_id = (SELECT id FROM te_asset WHERE code = 'TE-FO'), then the te_asset row, and DELETE FROM idx_constituent/idx_membership/idx_change WHERE idx_id = (SELECT idx_id FROM idx_index WHERE canonical_name = 'TE-FO INDEX').
  • MCX history: additive (new dates only); nothing to roll back.