Skip to main content

Task: NSEEOD Data Rebuild — VPS Rollout (Sept 2026)

Everything below was built and verified locally on 2026-09-26/27. The VPS database is the reference, so it is rebuilt from the same raw files on the VPS (option A). Copying the local Parquet / tables is not used, because masid numbers differ between the two databases.

Run everything from /home/kss/tradeentry/backend with venv/bin/python, long steps inside tmux. Don't run the Bhav Downloader until step 10.

0. Code, packages, settings​

cd /home/kss && ./pull.sh
cd /home/kss/tradeentry/backend && venv/bin/pip install -r requirements.txt # adds pypdf, pdfplumber

Add these to backend/.env:

VPS_EQEOD_DATA_PATH=/home/kss/tradeentry/nseeqeoddata
VPS_CORPACT_PATH=/home/kss/EODData/NSE_CA

1. Copy the backups (WSL rsync to /home/kss/EODData/)​

Folder (on PC: D:\kss\stock\EODData\)SizeNeeded after the rebuild?
NSE_CM, NSE_MTO1.1 GB + 349 MBno (keep on PC)
NSE_INDEX36 MBno
NSE_SME53 MBno
NSE_FOseveral GBno
NSE_INDEX_CHANGES~450 MBonly to rebuild the tracker
NSE_CA16 MByes: the daily corporate-action sync appends to it

2. Equity history — eq_backfill.py​

venv/bin/python -m nseeod.eq_backfill --cm-dir /home/kss/EODData/NSE_CM --mto-dir /home/kss/EODData/NSE_MTO # dry run
venv/bin/python -m nseeod.eq_backfill --cm-dir /home/kss/EODData/NSE_CM --mto-dir /home/kss/EODData/NSE_MTO --apply # ~1 h

3. Index rows — index_backfill.py​

venv/bin/python -m nseeod.index_backfill check --dir /home/kss/EODData/NSE_INDEX
venv/bin/python -m nseeod.index_backfill apply --dir /home/kss/EODData/NSE_INDEX --apply

4. SME rows — sme_backfill.py​

venv/bin/python -m nseeod.sme_backfill apply --dir /home/kss/EODData/NSE_SME --apply # 29 THEJO rows (2012)

5. Parquet history store — eq_store.py​

venv/bin/python -m nseeod.eq_store export
venv/bin/python -m nseeod.eq_store verify # must report 0 mismatched

6. Corporate actions — corp_actions.py​

venv/bin/python -m nseeod.corp_actions parse --dir /home/kss/EODData/NSE_CA --apply # runs reconcile too
venv/bin/python -m nseeod.corp_actions accept-high --apply
venv/bin/python -m nseeod.corp_actions ignore-gaps # dry run: check the list first
venv/bin/python -m nseeod.corp_actions ignore-gaps --apply

Review decisions made on the local Corporate Actions page do not carry over. Review on the VPS from now on.

7. F&O history — fo_backfill.py​

venv/bin/python -m nseeod.fo_backfill apply --dir /home/kss/EODData/NSE_FO --apply # several hours, tmux
venv/bin/python -m nseeod.fo_backfill lots --apply

8. Index constituent tracker — index_members.py​

venv/bin/python -m nseeod.index_members build --dir /home/kss/EODData/NSE_INDEX_CHANGES --apply

The folder carries snapshots.parquet, changes.parquet, changes_b.parquet and name_map_manual.csv from the PC, so no PDF parsing is needed on the VPS.

9. Custom indices — custom_index.py / fo_effect.py​

After step 7 (needs the full F&O history) and step 6 (split factors):

venv/bin/alembic upgrade head # creates ci_master / ci_level / ci_holding / ci_event
venv/bin/python -m nseeod.custom_index build --apply # 1x / 2x / 3x, a few minutes
venv/bin/python -m nseeod.custom_index stats
venv/bin/python -m nseeod.fo_effect build --apply # F&O vs non-F&O indices + event study, ~2 min

Expect roughly the PC numbers:

  • leverage: 1x CAGR 11.9% (NAV 16,259), 2x 8.2%, 3x ruined 2008-10-23;
  • F&O effect: F&O 15.3% vs non-F&O top 750 19.9%; entries +41% before, −3.8% in the first year after.

TE-FO Index (te_fo_index.py, added 2026-10-03), after steps 6–8:

venv/bin/python -m nseeod.te_fo_index build --apply # te_asset 'TE-FO' + te_price + idx_* 'TE-FO INDEX', ~2 min
venv/bin/python -m nseeod.te_fo_index members --date 2008-06-30 # expect 227 members

Expect NAV 5,666.91 on 2026-09-25 (CAGR 7.2%), 298 cycles, 679 membership intervals. If index_members build --apply runs again later, it leaves the TE-FO rows alone (index_members.EXTERNAL).

10. Frontend + restart​

Rebuild the frontend, then restart the tradeentry service. The new pieces are:

  • the Corporate Actions page;
  • the Raw / Adj / TR chart buttons;
  • SME / REIT symbols in the Dashboard list;
  • the Custom Indices page (Leverage and F&O effect tabs).

11. Resume daily operation​

The Bhav Downloader now also:

  • appends to the Parquet store;
  • replaces index days;
  • keeps OI-only F&O rows and lot sizes;
  • syncs corporate actions;
  • rebuilds the custom-index series (once the ci_* tables exist).

Still open after the rollout​

  • Review the corporate-action queue (also makes the F&O effect study exact before 2006 - it skips unresolved days): about 3,100 rows, mostly [low] detected jumps from 1998–2001, plus 239 demergers / schemes that need a factor (e.g. RELIANCE / Jio Financial 2023-07-20).
  • eq_store trim --keep 90 --apply, only after the VPS verify is clean. Safe since 2026-09-27: every history reader goes through eq_store (the last two, custom_index.implied_rates and FoBhav.download_missingfoBhav, were switched then). What still reads the table directly only needs recent rows: latest price, market summary, watchlist, AMI export of a just-loaded day.
  • Drop the one-time snapshot tables once satisfied: scr_nseeq_eod_pre_backfill, scr_nseeq_eod_pre_index_backfill, scr_nsefo_eod_pre_backfill.
  • Constituent tracker: work through name_review.csv (282 rows, mostly 1999–2003 NIFTY 500) to tighten NIFTY 500 / Next 50 before 2013.
  • Index history before 2012 (NSE's ind_close_all archive starts mid-2012; older rows are NIFTY only).
  • Old-format F&O lot sizes: ~2% are wrong where a lot changed mid-contract. Fixable from NSE market-activity files.
  • F&O and MCX price tables into Parquet (deferred; F&O needs date/month partitions).