Task: NSEEOD Data Rebuild — VPS Rollout (Sept 2026)
Everything below was built and verified locally on 2026-09-26/27. The VPS database is the
reference, so it is rebuilt from the same raw files on the VPS (option A). Copying the local
Parquet / tables is not used, because masid numbers differ between the two databases.
Run everything from /home/kss/tradeentry/backend with venv/bin/python, long steps inside
tmux. Don't run the Bhav Downloader until step 10.
0. Code, packages, settings
cd /home/kss && ./pull.sh
cd /home/kss/tradeentry/backend && venv/bin/pip install -r requirements.txt # adds pypdf, pdfplumber
Add these to backend/.env:
VPS_EQEOD_DATA_PATH=/home/kss/tradeentry/nseeqeoddata
VPS_CORPACT_PATH=/home/kss/EODData/NSE_CA
1. Copy the backups (WSL rsync to /home/kss/EODData/)
Folder (on PC: D:\kss\stock\EODData\) | Size | Needed after the rebuild? |
|---|---|---|
NSE_CM, NSE_MTO | 1.1 GB + 349 MB | no (keep on PC) |
NSE_INDEX | 36 MB | no |
NSE_SME | 53 MB | no |
NSE_FO | several GB | no |
NSE_INDEX_CHANGES | ~450 MB | only to rebuild the tracker |
NSE_CA | 16 MB | yes: the daily corporate-action sync appends to it |
2. Equity history — eq_backfill.py
venv/bin/python -m nseeod.eq_backfill --cm-dir /home/kss/EODData/NSE_CM --mto-dir /home/kss/EODData/NSE_MTO # dry run
venv/bin/python -m nseeod.eq_backfill --cm-dir /home/kss/EODData/NSE_CM --mto-dir /home/kss/EODData/NSE_MTO --apply # ~1 h
3. Index rows — index_backfill.py
venv/bin/python -m nseeod.index_backfill check --dir /home/kss/EODData/NSE_INDEX
venv/bin/python -m nseeod.index_backfill apply --dir /home/kss/EODData/NSE_INDEX --apply
4. SME rows — sme_backfill.py
venv/bin/python -m nseeod.sme_backfill apply --dir /home/kss/EODData/NSE_SME --apply # 29 THEJO rows (2012)
5. Parquet history store — eq_store.py
venv/bin/python -m nseeod.eq_store export
venv/bin/python -m nseeod.eq_store verify # must report 0 mismatched
6. Corporate actions — corp_actions.py
venv/bin/python -m nseeod.corp_actions parse --dir /home/kss/EODData/NSE_CA --apply # runs reconcile too
venv/bin/python -m nseeod.corp_actions accept-high --apply
venv/bin/python -m nseeod.corp_actions ignore-gaps # dry run: check the list first
venv/bin/python -m nseeod.corp_actions ignore-gaps --apply
Review decisions made on the local Corporate Actions page do not carry over. Review on the VPS from now on.
7. F&O history — fo_backfill.py
venv/bin/python -m nseeod.fo_backfill apply --dir /home/kss/EODData/NSE_FO --apply # several hours, tmux
venv/bin/python -m nseeod.fo_backfill lots --apply
8. Index constituent tracker — index_members.py
venv/bin/python -m nseeod.index_members build --dir /home/kss/EODData/NSE_INDEX_CHANGES --apply
The folder carries snapshots.parquet, changes.parquet, changes_b.parquet and name_map_manual.csv from the PC, so no PDF parsing is needed on the VPS.
9. Custom indices — custom_index.py / fo_effect.py
After step 7 (needs the full F&O history) and step 6 (split factors):
venv/bin/alembic upgrade head # creates ci_master / ci_level / ci_holding / ci_event
venv/bin/python -m nseeod.custom_index build --apply # 1x / 2x / 3x, a few minutes
venv/bin/python -m nseeod.custom_index stats
venv/bin/python -m nseeod.fo_effect build --apply # F&O vs non-F&O indices + event study, ~2 min
Expect roughly the PC numbers:
- leverage: 1x CAGR 11.9% (NAV 16,259), 2x 8.2%, 3x ruined 2008-10-23;
- F&O effect: F&O 15.3% vs non-F&O top 750 19.9%; entries +41% before, −3.8% in the first year after.
TE-FO Index (te_fo_index.py, added 2026-10-03), after steps 6–8:
venv/bin/python -m nseeod.te_fo_index build --apply # te_asset 'TE-FO' + te_price + idx_* 'TE-FO INDEX', ~2 min
venv/bin/python -m nseeod.te_fo_index members --date 2008-06-30 # expect 227 members
Expect NAV 5,666.91 on 2026-09-25 (CAGR 7.2%), 298 cycles, 679 membership intervals. If index_members build --apply
runs again later, it leaves the TE-FO rows alone (index_members.EXTERNAL).
10. Frontend + restart
Rebuild the frontend, then restart the tradeentry service. The new pieces are:
- the Corporate Actions page;
- the Raw / Adj / TR chart buttons;
- SME / REIT symbols in the Dashboard list;
- the Custom Indices page (Leverage and F&O effect tabs).
11. Resume daily operation
The Bhav Downloader now also:
- appends to the Parquet store;
- replaces index days;
- keeps OI-only F&O rows and lot sizes;
- syncs corporate actions;
- rebuilds the custom-index series (once the
ci_*tables exist).
Still open after the rollout
- Review the corporate-action queue (also makes the F&O effect study exact before 2006 - it skips unresolved days): about 3,100 rows, mostly
[low]detected jumps from 1998–2001, plus 239 demergers / schemes that need a factor (e.g. RELIANCE / Jio Financial 2023-07-20). -
eq_store trim --keep 90 --apply, only after the VPSverifyis clean. Safe since 2026-09-27: every history reader goes througheq_store(the last two,custom_index.implied_ratesandFoBhav.download_missingfoBhav, were switched then). What still reads the table directly only needs recent rows: latest price, market summary, watchlist, AMI export of a just-loaded day. - Drop the one-time snapshot tables once satisfied:
scr_nseeq_eod_pre_backfill,scr_nseeq_eod_pre_index_backfill,scr_nsefo_eod_pre_backfill. - Constituent tracker: work through
name_review.csv(282 rows, mostly 1999–2003 NIFTY 500) to tighten NIFTY 500 / Next 50 before 2013. - Index history before 2012 (NSE's
ind_close_allarchive starts mid-2012; older rows are NIFTY only). - Old-format F&O lot sizes: ~2% are wrong where a lot changed mid-contract. Fixable from NSE market-activity files.
- F&O and MCX price tables into Parquet (deferred; F&O needs date/month partitions).