Skip to main content

index_backfill.py — NSE Index EOD Repair

backend/nseeod/index_backfill.py reloads index rows in scr_nseeq_eod (scr_master.typ 2, and 8 for G-Sec indices) from NSE's daily ind_close_all_DDMMYYYY.csv files.

The problem (found 2026-09-27)​

Many 2012–2026 days had only 1–3 indices, or none:

YearDays with no index rowMedian indices per dayExpected
2012–20183–17 a yearas expected30–70
20202532~80
2023–20242 eachas expected~100

In 2020 NIFTY itself had only 132 of 252 days; for example, only two rows between 20 March and 3 April. The VPS had exactly the same counts, so the fault was in the loading, not in a sync.

Cause: utils.updateindexbhav only inserted. scr_nseeq_eod has a primary key on (masid, sdate) and the insert is one batch. If any index row for that day already existed (NIFTY from another source, or an earlier partial run), the whole day's file was rejected. Only the old rows survived, typically NIFTY alone or NIFTY / Nifty 100 / Nifty Next 50.

Fix: the loader is split into utils.parse_index_file() and utils.write_index_day(). write_index_day deletes that day's rows for exactly the indices in the file, then inserts them. The daily Bhav Downloader and this tool both use it, so a re-run now replaces a day instead of failing.

Source​

URLCoverage
nsearchives.nseindia.com/content/indices/ind_close_all_DDMMYYYY.csv (niftyindices.com Daily_Snapshot as fallback)From mid-2012 (15-06-2012 exists, 02-01-2012 doesn't). 30 indices in 2012, ~80 in 2020, ~160 in 2026

Index rows before 2012 (only NIFTY, 1991–2011) came from another source and are not touched.

Backup​

D:\kss\stock\EODData\NSE_INDEX\{year}\ind_close_all_DDMMYYYY.csv, plus manifest.csv (date, file, rows, SHA-1) and not_published.txt.

A file is kept only when it is a real CSV with an Index Name column and every row's date equals the file's date. The days to fetch are the equity trading days (eq_store.available_days).

Usage​

Run from backend/:

python -m nseeod.index_backfill download --dir D:/kss/stock/EODData/NSE_INDEX [--from 2012-01-01 --to ...]
python -m nseeod.index_backfill check --dir D:/kss/stock/EODData/NSE_INDEX # per year: short days, file vs DB rows
python -m nseeod.index_backfill apply --dir D:/kss/stock/EODData/NSE_INDEX # dry run: days that are short
python -m nseeod.index_backfill apply --dir D:/kss/stock/EODData/NSE_INDEX --apply [--all]

apply:

  • By default it reloads only days where the DB has fewer index rows than the file; --all reloads every day.
  • Before the first write it copies all index rows once to scr_nseeq_eod_pre_index_backfill.
  • It uses the same parser as the daily loader: the BHAV name-change map applies, and new index names are added to scr_master as typ 2.
  • At the end it refreshes the Parquet history for the index symbols only, using eq_store.append_days(days, masids=...).

Local run (2026-09-27)​

Download: 3,613 files (mid-2012 → 2026-09-25). NSE never published 40 dates, mostly January–June 2012, before the archive starts, plus a few single days.

Check: 2,064 of 3,613 days were short, in every year, not only 2020:

  • 134 days had no index row at all;
  • 1,169 were short by 10 or more indices;
  • 592 were short by exactly one (a different index each time, e.g. Nifty Midcap 50 in 2012).

Apply: all 2,064 days reloaded, 0 failures, in 3.8 minutes. Index rows on those days went from 96,401 to 177,373.

  • 6 genuinely new 2026 indices were added to scr_master (Nifty AI Catalysts, Nifty Next 100, …). Local index data had stopped on 2026-07-13.
  • 166 index Parquet files were refreshed; eq_store verify still matches on 8,744 of 8,744 dates.
Year20122016201920202022202420252026
Median indices / day, after30607275 (was 32)92111132140
  • NIFTY 2020: 252 of 252 days (was 132).
  • Days with no index row are now only 4, all dates NSE never published.
  • Two days still count one row "short" (2013-02-08, 2014-09-15): two names in NSE's file map to one index through the name-change table, so that's expected.

Test: March 2020 (local)​

21 trading days (10 March 2020 was Holi) went from 388 index rows to 1,701: 74 typ-2 indices a day, plus the G-Sec ones. NIFTY is complete; 09-03-2020 closes at 10,451.45.