Skip to main content

index_history_backfill.py — NSE Index EOD Backfill (pre-2012)

backend/nseeod/index_history_backfill.py fills in index EOD history before NSE's ind_close_all daily archive begins (mid-2012 — see index_backfill.py), one index at a time, straight from each index's actual launch/base date.

Why​

index_backfill.py only repairs 2012→today, because that's as far back as NSE's ind_close_all_DDMMYYYY.csv archive goes. Before 2012, this platform has only ever loaded NIFTY (masid=1, 1991–2011) from a separate one-off source — every other tracked index (Nifty Auto, Nifty Midcap 50, Nifty IT, ...) has no history before its own first ind_close_all appearance, even though NSE Indices Ltd (niftyindices.com) publishes the exact same official daily closing levels per index back to that index's real launch date through its own public "Historical Index Data" report. This tool calls the JSON endpoint behind that report directly.

Source (confirmed by hand, 2026-10-01)​

POST https://www.niftyindices.com/BackPage/getHistoricaldatatabletoString
Content-Type: application/json; charset=utf-8
Referer: https://www.niftyindices.com/reports/historical-data

{"cinfo": "{'name':'<Trading_Index_Name>','startDate':'MM/DD/YYYY','endDate':'MM/DD/YYYY','indexName':'<Trading_Index_Name>'}"}
  • No login/cookie needed — verified working from a cookie-less curl, despite Akamai sitting in front of the rest of niftyindices.com.
  • Max 365 days per call — the site's own front-end JS rejects a wider range; this tool chunks automatically.
  • Response is a JSON array, newest first: {"HistoricalDate": "31 Dec 2020", "OPEN": "13970.00", "HIGH": "...", "LOW": "...", "CLOSE": "13981.75"}. "-" appears for OPEN/HIGH/LOW on many pre-2000 rows where NSE only ever published a single back-computed daily level (no real intraday quote). Filled from CLOSE here, never zeroed — different from utils.parse_index_file's '-' → 0 convention, which is for live ind_close_all CSV rows and isn't appropriate for this back-computed range.
  • The API's name must be NSE's own Trading_Index_Name, from niftyindices.com's own IndexMapping.json (https://liveindexsa.niftyindices.com/assets/json/IndexMapping.json) — it frequently does not match this platform's scr_master.mname (sourced from the ind_close_all BHAV CSV's "Index Name" column). E.g. our "Nifty 10 Yr Benchmark G-Sec" is their "Nifty GS 10Yr".

Scope: typ=2 (equity-category) indices only​

typ=8 (G-Sec/debt) names tested empty against this endpoint even for date ranges this platform already has real data for (tried "Nifty GS Compsite" / Nifty Composite G-Sec Index for 2016 — empty, despite backend/AA/CLAUDE.md rule 1 confirming NSE published it daily from 2015-10-13). niftyindices.com's site appears to serve "Fixed Income" indices through a different, unverified data path. Rather than guess, map only ever considers scr_master rows with typ = 2 AND underly = 0 — revisit typ=8 separately once that path is confirmed.

Never guess a name match​

map only auto-matches a scr_master.mname to an API Trading_Index_Name on exact normalized (uppercase, letters+digits only) equality against either Trading_Index_Name or Index_long_name. Everything else is left with api_name='' and match=none for a human to fill in by hand — never fuzzy-matched (the project's standing rule against guessing scr_master identity — see nseeod/CLAUDE.md rule 1).

Of the 184 typ=2 indices tracked locally (as of 2026-10-01), 161 auto-matched exactly; the other 23 are short/legacy BHAV codes that need a human to look up the right API name, for example:

BANKNIFTY, NIFTY, MIDCPNIFTY, FINNIFTY, NIFTYNXT50, CNX100, DEFTY, JUNIOR,
India VIX, Nifty Free Float Midcap 100, Nifty Free Float Smallcap 100,
Nifty Healthcare Index, Nifty India Internet & E-Commerce,
NIFTY PR 1X Inverse, NIFTY PR 2x Leverage, Nifty Quality 30,
Nifty Tata Group 25% Cap, Nifty TR 1X Inverse, NIFTY TR 1X Inverse,
Nifty TR 2X Leverage, NIFTY TR 2X Leverage, Nifty100 ESG Sector Leaders - Old,
NSE Quality 30

Usage​

Run from backend/:

# 1. Build/refresh the mapping (writes masid,mname,api_name,match)
python -m nseeod.index_history_backfill map --out index_history_mapping.csv

# 2. Edit index_history_mapping.csv by hand: fill api_name for any match=none
# row you've verified on niftyindices.com. Re-run with --in to validate
# your edits (still exact-match only against IndexMapping.json):
python -m nseeod.index_history_backfill map --in index_history_mapping.csv --out index_history_mapping.csv

# 3. Preview, no writes: current DB coverage per index vs how much more the API has
python -m nseeod.index_history_backfill check --map index_history_mapping.csv [--floor 1990-01-01]

# 4. Dry run (default), then write for real
python -m nseeod.index_history_backfill apply --map index_history_mapping.csv [--floor 1990-01-01]
python -m nseeod.index_history_backfill apply --map index_history_mapping.csv --apply
python -m nseeod.index_history_backfill apply --map index_history_mapping.csv --apply --only 987073,987066 # one or a few masids

apply:

  • Only inserts rows strictly before each index's current earliest sdate (via eq_store.get_eod, which reads Parquet + staging together) — it never touches 2012+ rows sourced from ind_close_all.
  • Stops early per index once it sees two consecutive empty 365-day chunks walking backward — that's the index's real launch boundary, so most indices never need to walk all the way to --floor (default 1990-01-01).
  • Before the first write it copies the mapped indices' rows once to scr_nseeq_eod_pre_index_history_backfill (same snapshot-then-write pattern as index_backfill.py and eq_backfill.py).
  • Refreshes Parquet history for just that index (eq_store.append_days(days, masids=[masid])) as soon as it finishes, so a long run's early indices are already queryable before the whole run ends.
  • Sleeps 0.3s between API calls (same courtesy pacing as index_backfill.py's downloader).

Local verification (2026-10-01)​

Ran apply --apply for real on Nifty Auto (masid 987073, floor 2004-01-01): 2,027 new days, 2004-01-01 → 2012-02-17. Confirmed via eq_store.get_eod(987073) that the Parquet history is now continuous (5,641 rows, 2004-01-01 → today) and the snapshot table (scr_nseeq_eod_pre_index_history_backfill, 3,614 rows) is in place for rollback. Not yet run for the full index universe.

VPS rollout — CLI to refill all tracked indices​

Not yet run on the VPS. Once pull.sh has deployed this tool and backend/AA/* (unrelated, same deploy), refill the full index universe in one sitting:

# on the VPS, from /home/kss/tradeentry-VPS/backend (adjust to the actual deploy path)
source venv/bin/activate # or the VPS's actual venv activation path

# 1. Build the mapping fresh from the live IndexMapping.json
python -m nseeod.index_history_backfill map --out index_history_mapping.csv

# 2. Copy the mapping CSV's manually-filled api_name column over from the one
# already reviewed on the local PC (don't re-review 23 rows twice) — e.g.:
# scp index_history_mapping.csv <vps>:/home/kss/tradeentry-VPS/backend/
# then re-validate on the VPS:
python -m nseeod.index_history_backfill map --in index_history_mapping.csv --out index_history_mapping.csv

# 3. Preview before committing (no writes) — sanity-check the day counts look
# like the local run before trusting --apply across ~160 indices:
python -m nseeod.index_history_backfill check --map index_history_mapping.csv --floor 1990-01-01

# 4. Write for real (dry run first is the default - the --apply here is deliberate)
python -m nseeod.index_history_backfill apply --map index_history_mapping.csv --floor 1990-01-01 --apply

Notes for the VPS run specifically:

  • Reuse the same index_history_mapping.csv the local PC ends up with (after step 2 of the local workflow above) rather than re-deciding the 23 manual rows independently — copy it over rather than re-running map blind on the VPS and re-reviewing from scratch.
  • At ~160 mapped indices × up to a few dozen 365-day chunks each (most stop in 1–10 chunks via the two-empty-chunk early exit) and a 0.3s pace between calls, expect roughly 20–40 minutes for the full universe — run it in tmux/screen, not a session that can drop.
  • If it's interrupted partway, it's safe to just re-run apply --apply — _earliest_existing() re-reads the live DB each time, so already-backfilled indices report "nothing to do" and only the unfinished ones continue.
  • After it completes, spot-check with python -m nseeod.eq_store verify (should still report 0 mismatches between Parquet and Postgres for the touched dates) before trimming anything.
  • Rollback if needed: the pre-run snapshot is scr_nseeq_eod_pre_index_history_backfill — restore from it and re-run eq_store.append_days for the affected masids/dates, same as the rollback path documented for index_backfill.py.