Task: VPS Rollout — 3 Oct 2026
Built and verified locally on 2026-10-03. Run everything on the VPS from
/home/kss/tradeentry/backend with venv/bin/python, long steps inside tmux.
| # | Piece | Needs on the VPS |
|---|---|---|
| A | EOD Chart / TEChart fixes, Corporate Actions universe filter, Stock Universe page | code only |
| B | TE-FO Index (te_asset / te_price + idx_* membership) | F&O history, corporate actions, idx_* tables (Sept rollout steps 6–8), then one build |
| C | MCX F&O history 2003 → | already complete on the VPS (checked 2026-10-03); only for a fresh database |
No new Python packages, no Alembic migration: TE-FO writes into the existing te_asset / te_price /
idx_* tables, and creates the idx_* tables + idx_members_on() itself if they are missing.
0. Before deploying: check the VPS
Sept rollout state (TE-FO depends on it). The full F&O history must start in 2000,
scr_corp_action must exist, and the index tracker must be loaded:
venv/bin/python - <<'EOF' 2>&1 | grep -v "^\[Config\|^\[pgs"
from nseeod import pgs
r = lambda s: pgs.read_sql(s).iloc[0, 0]
print("F&O history :", r("SELECT min(sdate)::text || ' -> ' || max(sdate)::text FROM scr_nsefo_eod"), "(want 2000-06-12 -> today)")
for t in ("scr_corp_action", "idx_membership", "te_asset", "te_price"):
print(f"{t:<16}:", "exists" if r(f"SELECT to_regclass('{t}') IS NOT NULL") else "MISSING")
if r("SELECT to_regclass('idx_membership') IS NOT NULL"):
print("NIFTY 50 today :", r("SELECT count(*) FROM idx_members_on('NIFTY 50', CURRENT_DATE)"), "members (want 50)")
EOF
- F&O from 2000 +
scr_corp_action+idx_membershippresent → step B can run. - Anything missing → finish the Sept rollout steps 6 (corporate actions), 7 (F&O history + lots) and 8 (index tracker) first. Step A deploys fine without them.
te_asset/te_pricemissing →venv/bin/alembic upgrade head.
MCX state (decides step C) - only if scr_mcxfo_eod exists; this command arrives with the deploy
in step 1, so run it right after that:
venv/bin/python -m mcxeod.mcx_backfill --status
1. Commit, push, deploy
On the PC: commit and push main (all of the 2026-10-03 work is in one working tree).
On the VPS, use the repo copy of the deploy script (the /home/kss/pull.sh copy once skipped the
frontend rebuild):
cp /home/kss/tradeentry/pull.sh /home/kss/pull.sh && bash /home/kss/pull.sh
It pulls, rebuilds the frontend and docs (both changed), and restarts the tradeentry service.
Check the summary shows frontend rebuilt and service restarted.
2. TE-FO Index — te_fo_index.py
Only after step 0's checks pass. Don't run the Bhav Downloader before this build - its daily hook would otherwise start a first build in the background.
venv/bin/python -m nseeod.te_fo_index build # dry run (~2 min): prints NAV / CAGR / checks
venv/bin/python -m nseeod.te_fo_index build --apply # writes te_asset 'TE-FO', te_price, idx_* 'TE-FO INDEX'
venv/bin/python -m nseeod.te_fo_index members --date 2008-06-30
Expect roughly the PC numbers (small differences are possible where the two F&O databases differ):
| Check | PC (2026-10-03) |
|---|---|
| Range | 2001-11-29 → last F&O day, 298 expiry cycles |
| NAV on 2026-09-25 | 5,666.91 (CAGR 7.2%, max drawdown −73.4%) |
| Dry-run checks | OHLC consistency: 0 bad rows, 284 scaled, 0 not confirmed, 0 cycles skipped |
| Written | 6,169 te_price rows, 47,865 snapshot rows, 679 membership intervals, 207 members now |
| Members on 2008-06-30 | 227 (includes delisted AIRDECCAN, ADLABSFILM) |
If 0 not confirmed / 0 skipped do not hold, stop and compare with the PC before --apply.
If the index tracker is ever rebuilt (index_members build --apply), the TE-FO rows are kept
(index_members.EXTERNAL). This protection is new; running an older index_members would wipe them.
3. MCX F&O history — read mcx_backfill --status from step 0
Checked 2026-10-03: the VPS is already complete - 6,359 trading days, 2003-11-18 → 2026-09-30,
17.0M rows; SILVERM-I 6,298 days from 2004-02-17. Missing only 2026-10-01/02 (next daily MCX
download) and two old Saturdays, 2004-07-03 and 2005-01-15
(venv/bin/python -m mcxeod.mcx_backfill --from 2004-07-03 --to 2005-01-15, optional). No archive
import on the VPS. The table below stays for a fresh database.
What --status shows | Do |
|---|---|
| Days from 2003-11-10, ~250–300 per year, 0 missing (or only the last day or two) | nothing - history is complete |
| Some missing ranges (weekly-spaced = Saturday sessions; long range = real gap) | venv/bin/python -m mcxeod.mcx_backfill --from <first missing date> - downloads only the missing days |
| Most of 2003–2026 missing | archive import below, then the backfill |
scr_mcxfo_eod does not exist (VPS still on the old MCX tables) | venv/bin/python -m mcxeod.mcx_infra, venv/bin/python -m mcxeod.mcx_history, then archive import + backfill |
Archive import (much faster than downloading 21 years):
# PC (WSL): copy the archive (82 MB)
rsync -av /mnt/d/kss/stock/EODData/MCXBhavf10Nov2003-23Oct2024.zip kss@<vps>:/home/kss/EODData/
# VPS
venv/bin/python -m mcxeod.mcx_archive_import import --src /home/kss/EODData/MCXBhavf10Nov2003-23Oct2024.zip
venv/bin/python -m mcxeod.mcx_backfill --from 2003-11-10 # Saturdays (not in the archive) + Oct 2024 -> today
venv/bin/python -m mcxeod.mcx_backfill --status # want ~0 missing
- The import is resumable (skips stored days) and runs oldest day first; on the PC it ran at about 1,000 days per 6–7 minutes. Run the backfill after it: change-in-OI comes from the previous stored day.
- The archive has no Saturday files, but MCX held ~490 Saturday sessions; the backfill fetches them.
- Verified on the PC: every traded contract identical to a live download; only untraded contracts' theoretical close differs (MCX revised them later).
4. Check in the browser
| Page | Check |
|---|---|
| EOD Chart · NSE · SPOT · NIFTY (full range), TEChart engine | Whole chart fits the pane, latest candles visible, no "zoomed" top slice; switch to RELIANCE / BANKNIFTY - stays fitted |
| EOD Chart · NSE · FUT | TE-FO Index chip next to "Available Stocks" → full history line from Nov 2001 |
| … then click a stock (e.g. ABB) | Chart switches at once to ABB FUT on the current expiry (no TE-FO chart under the stock's name) |
| EOD Chart · MCX · CFUT · SILVERM | History from 2004 once step 3 is done |
| Corporate Actions | "Universe (any of)" + "As on": TE-FO Index + NIFTY 500 on 01-01-2005 narrows the list and the chips; SME shows only SME stocks trading that day |
| Stock Universe (sidebar, developer) | TE-FO Index today ≈ 207 members; NIFTY 50 = 50; Changes panel lists IN / OUT |
5. Daily operation
- NSE Bhav Downloader now also extends TE-FO (log tag
TE-FO, e.g.TE-FO Index: 1 day(s) from 2026-09-29, ... NAV ...). A failure there is logged only and never fails the bhav sync. - After any F&O history backfill, rerun
te_fo_index build --apply(the daily update only recomputes from the last expiry). - MCX downloads are unchanged;
mcx_backfill --statusis the quick health check.
Rollback
- Code:
git revertthe commit and runpull.shagain. - TE-FO data (derived, rebuildable):
DELETE FROM te_price WHERE asset_id = (SELECT id FROM te_asset WHERE code = 'TE-FO'), then thete_assetrow, andDELETE FROM idx_constituent/idx_membership/idx_change WHERE idx_id = (SELECT idx_id FROM idx_index WHERE canonical_name = 'TE-FO INDEX'). - MCX history: additive (new dates only); nothing to roll back.