index_history_backfill.py — NSE Index EOD Backfill (pre-2012)
backend/nseeod/index_history_backfill.py fills in index EOD history before NSE's
ind_close_all daily archive begins (mid-2012 — see index_backfill.py),
one index at a time, straight from each index's actual launch/base date.
Why
index_backfill.py only repairs 2012→today, because that's as far back as
NSE's ind_close_all_DDMMYYYY.csv archive goes. Before 2012, this platform has only ever loaded
NIFTY (masid=1, 1991–2011) from a separate one-off source — every other tracked index (Nifty
Auto, Nifty Midcap 50, Nifty IT, ...) has no history before its own first ind_close_all
appearance, even though NSE Indices Ltd (niftyindices.com) publishes the exact same official daily
closing levels per index back to that index's real launch date through its own public
"Historical Index Data" report. This tool calls the JSON endpoint behind that report directly.
Source (confirmed by hand, 2026-10-01)
POST https://www.niftyindices.com/BackPage/getHistoricaldatatabletoString
Content-Type: application/json; charset=utf-8
Referer: https://www.niftyindices.com/reports/historical-data
{"cinfo": "{'name':'<Trading_Index_Name>','startDate':'MM/DD/YYYY','endDate':'MM/DD/YYYY','indexName':'<Trading_Index_Name>'}"}
- No login/cookie needed — verified working from a cookie-less
curl, despite Akamai sitting in front of the rest of niftyindices.com. - Max 365 days per call — the site's own front-end JS rejects a wider range; this tool chunks automatically.
- Response is a JSON array, newest first:
{"HistoricalDate": "31 Dec 2020", "OPEN": "13970.00", "HIGH": "...", "LOW": "...", "CLOSE": "13981.75"}."-"appears forOPEN/HIGH/LOWon many pre-2000 rows where NSE only ever published a single back-computed daily level (no real intraday quote). Filled fromCLOSEhere, never zeroed — different fromutils.parse_index_file's'-' → 0convention, which is for liveind_close_allCSV rows and isn't appropriate for this back-computed range. - The API's
namemust be NSE's ownTrading_Index_Name, from niftyindices.com's ownIndexMapping.json(https://liveindexsa.niftyindices.com/assets/json/IndexMapping.json) — it frequently does not match this platform'sscr_master.mname(sourced from theind_close_allBHAV CSV's "Index Name" column). E.g. our"Nifty 10 Yr Benchmark G-Sec"is their"Nifty GS 10Yr".
Scope: typ=2 (equity-category) indices only
typ=8 (G-Sec/debt) names tested empty against this endpoint even for date ranges this platform
already has real data for (tried "Nifty GS Compsite" / Nifty Composite G-Sec Index for 2016 —
empty, despite backend/AA/CLAUDE.md rule 1 confirming NSE published it daily from 2015-10-13).
niftyindices.com's site appears to serve "Fixed Income" indices through a
different, unverified data path. Rather than guess, map only ever considers scr_master rows with
typ = 2 AND underly = 0 — revisit typ=8 separately once that path is confirmed.
Never guess a name match
map only auto-matches a scr_master.mname to an API Trading_Index_Name on exact normalized
(uppercase, letters+digits only) equality against either Trading_Index_Name or Index_long_name.
Everything else is left with api_name='' and match=none for a human to fill in by hand — never
fuzzy-matched (the project's standing rule against guessing scr_master identity — see
nseeod/CLAUDE.md rule 1).
Of the 184 typ=2 indices tracked locally (as of 2026-10-01), 161 auto-matched exactly; the
other 23 are short/legacy BHAV codes that need a human to look up the right API name, for example:
BANKNIFTY, NIFTY, MIDCPNIFTY, FINNIFTY, NIFTYNXT50, CNX100, DEFTY, JUNIOR,
India VIX, Nifty Free Float Midcap 100, Nifty Free Float Smallcap 100,
Nifty Healthcare Index, Nifty India Internet & E-Commerce,
NIFTY PR 1X Inverse, NIFTY PR 2x Leverage, Nifty Quality 30,
Nifty Tata Group 25% Cap, Nifty TR 1X Inverse, NIFTY TR 1X Inverse,
Nifty TR 2X Leverage, NIFTY TR 2X Leverage, Nifty100 ESG Sector Leaders - Old,
NSE Quality 30
Usage
Run from backend/:
# 1. Build/refresh the mapping (writes masid,mname,api_name,match)
python -m nseeod.index_history_backfill map --out index_history_mapping.csv
# 2. Edit index_history_mapping.csv by hand: fill api_name for any match=none
# row you've verified on niftyindices.com. Re-run with --in to validate
# your edits (still exact-match only against IndexMapping.json):
python -m nseeod.index_history_backfill map --in index_history_mapping.csv --out index_history_mapping.csv
# 3. Preview, no writes: current DB coverage per index vs how much more the API has
python -m nseeod.index_history_backfill check --map index_history_mapping.csv [--floor 1990-01-01]
# 4. Dry run (default), then write for real
python -m nseeod.index_history_backfill apply --map index_history_mapping.csv [--floor 1990-01-01]
python -m nseeod.index_history_backfill apply --map index_history_mapping.csv --apply
python -m nseeod.index_history_backfill apply --map index_history_mapping.csv --apply --only 987073,987066 # one or a few masids
apply:
- Only inserts rows strictly before each index's current earliest
sdate(viaeq_store.get_eod, which reads Parquet + staging together) — it never touches 2012+ rows sourced fromind_close_all. - Stops early per index once it sees two consecutive empty 365-day chunks walking backward —
that's the index's real launch boundary, so most indices never need to walk all the way to
--floor(default1990-01-01). - Before the first write it copies the mapped indices' rows once to
scr_nseeq_eod_pre_index_history_backfill(same snapshot-then-write pattern asindex_backfill.pyandeq_backfill.py). - Refreshes Parquet history for just that index (
eq_store.append_days(days, masids=[masid])) as soon as it finishes, so a long run's early indices are already queryable before the whole run ends. - Sleeps 0.3s between API calls (same courtesy pacing as
index_backfill.py's downloader).
Local verification (2026-10-01)
Ran apply --apply for real on Nifty Auto (masid 987073, floor 2004-01-01): 2,027 new days,
2004-01-01 → 2012-02-17. Confirmed via eq_store.get_eod(987073) that the Parquet history is now
continuous (5,641 rows, 2004-01-01 → today) and the snapshot table
(scr_nseeq_eod_pre_index_history_backfill, 3,614 rows) is in place for rollback. Not yet run for
the full index universe.
VPS rollout — CLI to refill all tracked indices
Not yet run on the VPS. Once pull.sh has deployed this tool and backend/AA/* (unrelated, same
deploy), refill the full index universe in one sitting:
# on the VPS, from /home/kss/tradeentry-VPS/backend (adjust to the actual deploy path)
source venv/bin/activate # or the VPS's actual venv activation path
# 1. Build the mapping fresh from the live IndexMapping.json
python -m nseeod.index_history_backfill map --out index_history_mapping.csv
# 2. Copy the mapping CSV's manually-filled api_name column over from the one
# already reviewed on the local PC (don't re-review 23 rows twice) — e.g.:
# scp index_history_mapping.csv <vps>:/home/kss/tradeentry-VPS/backend/
# then re-validate on the VPS:
python -m nseeod.index_history_backfill map --in index_history_mapping.csv --out index_history_mapping.csv
# 3. Preview before committing (no writes) — sanity-check the day counts look
# like the local run before trusting --apply across ~160 indices:
python -m nseeod.index_history_backfill check --map index_history_mapping.csv --floor 1990-01-01
# 4. Write for real (dry run first is the default - the --apply here is deliberate)
python -m nseeod.index_history_backfill apply --map index_history_mapping.csv --floor 1990-01-01 --apply
Notes for the VPS run specifically:
- Reuse the same
index_history_mapping.csvthe local PC ends up with (after step 2 of the local workflow above) rather than re-deciding the 23 manual rows independently — copy it over rather than re-runningmapblind on the VPS and re-reviewing from scratch. - At ~160 mapped indices × up to a few dozen 365-day chunks each (most stop in 1–10 chunks via the
two-empty-chunk early exit) and a 0.3s pace between calls, expect roughly 20–40 minutes for
the full universe — run it in
tmux/screen, not a session that can drop. - If it's interrupted partway, it's safe to just re-run
apply --apply—_earliest_existing()re-reads the live DB each time, so already-backfilled indices report "nothing to do" and only the unfinished ones continue. - After it completes, spot-check with
python -m nseeod.eq_store verify(should still report 0 mismatches between Parquet and Postgres for the touched dates) before trimming anything. - Rollback if needed: the pre-run snapshot is
scr_nseeq_eod_pre_index_history_backfill— restore from it and re-runeq_store.append_daysfor the affected masids/dates, same as the rollback path documented forindex_backfill.py.