Skip to main content

fo_backfill.py — F&O EOD Quality Check and History Rebuild

backend/nseeod/fo_backfill.py rebuilds scr_nsefo_eod day by day from local backups of NSE's F&O bhav copies, from the first F&O day, 2000-06-12. It uses the same loader as the daily Bhav Downloader (utils.updatefobhav).

Quality check (2026-09-27)​

CheckFindingFix
CoverageHistory started 2009-08-31. 6 ordinary weekdays had no F&O rows: 2020-12-04, 2022-02-25, 2022-08-04, 2022-10-28, 2023-01-13, 2023-03-29. The other missing / short days are Saturday, Sunday and Muhurat special sessions.Rebuild 2000-06-12 → today
Lot sizelotsize was 0 on every rowExact NewBrdLotQty from the new format (2024-07-08 →); older days from the OI gcd (below)
Untraded contractsThe loader dropped every contract with 0 volume, including those with open interest, which left gaps in option-chain OIKeep rows with OPEN_INT > 0 (about +4,000 a day; decided 2026-09-27)
Open/high/low = 0NSE reports 0 for untraded contracts; 8,849 traded rows also had itStored as a flat bar at the close; svolume = 0 still marks untraded rows
Close outside high–low20,707 rowsNSE's own settlement-style closing price; not a load error, so kept
Duplicates54,000 duplicate (masid, date) rows, from 2018 onAdjusted strikes (ITC 430 and 430.5 CE) had been stored under one name by older loads. The current naming keeps them apart, and the rebuild splits them.
Continuous seriesNIFTY-I / -II / -III stopped in Aug 2025getThursdays assumed the last Thursday; NSE moved monthly expiry to Tuesday in Sep 2025. They are now ranked from each day's actual futures expiries (nearest = -I), which works under any expiry rule. Checked: on expiry day 30-09-2025 NIFTY-I is still the Sep contract, and it rolls to Oct on 01-10-2025.
Re-runsThe table has no primary keyFutures are inserted first with a delete of the whole date, so a re-run replaces the day (documented in match_scr_masters_and_fo_update)

Lot sizes for old-format days​

The old bhav copy has no lot size, but open interest and change in OI are in shares, and so always whole lots. apply reads every file and keeps, per underlying + expiry (all contracts of one underlying and expiry share a lot), the gcd of all those values. lots then writes it:

  • contracts through scr_master (underly, edate);
  • continuous series from the real future they match that day (same underlying, date, close and OI).

The gcd is read from the files, not the table: scr_nsefo_eod.oi is float32, which is not exact above 16.7 million.

Cross-check: NSE's daily F&O market-activity zips (archives/fo/mkt/foDDMMYYYY.zip, from about 2011, with gaps) carry traded quantity and number of contracts, so lot = quantity ÷ contracts, exactly (NIFTY 15-07-2019: 6,509,325 ÷ 86,791 = 75).

Backups​

D:\kss\stock\EODData\NSE_FO\{year}\:

  • foDDMONYYYYbhav.csv.zip (old format, to 2024-07-05);
  • BhavCopy_NSE_FO_0_0_0_YYYYMMDD_F_0000.csv.zip (UDiFF, from 2024-07-08);
  • manifest.csv and not_published.txt.

Built mostly from your existing RAR backups (user request), and downloaded only where they had nothing:

SourceF&O files taken
postgres/rAUG302009 to JUL 18 2014 .rar790
postgres/FROM19JUL2014to 26 FEB 2019 .rar1,095
postgres/FROM27 FEB 2019 TO MAY 26 2023.rar680
tradeentryscrbhavcopies.rar455 (UDiFF 2024-07 → 2026)
NSE archivethe rest: 2000-06 → 2009-08, 2023-05 → 2024-07, 2026 after the rar, 14 rar files whose inner date didn't match their name

NSE_Futures_and_Options_Bhav_EOD_Data_Files_from_2001_to_2023.zip is password-protected and was not used.

Usage​

Run from backend/:

python -m nseeod.fo_backfill import --dir D:/kss/stock/EODData/NSE_FO --rar "<backup.rar>" [--rar ...]
python -m nseeod.fo_backfill download --dir D:/kss/stock/EODData/NSE_FO [--from --to] # only missing days
python -m nseeod.fo_backfill apply --dir D:/kss/stock/EODData/NSE_FO [--from --to --limit] [--apply]
python -m nseeod.fo_backfill lots [--apply]

apply:

  • Takes a one-time snapshot scr_nsefo_eod_pre_backfill before the first write.
  • Is resumable (LOG_PATH/fo_backfill_done.txt) and keeps the lot evidence in LOG_PATH/fo_lot_gcd.json.
  • Logs failed days to fo_backfill_failed.txt.

Don't run the F&O Bhav Downloader while it runs.

Local run (2026-09-27): results​

  • All 6,521 days loaded, 2000-06-12 → 2026-09-25, 0 failures. 47.4 million rows.
    • OI-only contracts: about 1 million rows a year recently.
    • No high < low anywhere.
  • Coverage: 0 trading days without F&O rows, apart from 17 early-2000 days and 2 special sessions that NSE never published. The 6 formerly missing weekdays now have 7,800–20,500 rows each.
  • Continuous series: NIFTY-I / -II / -III present every day, including after the Sep 2025 Tuesday switch.
  • Adjusted strikes separated: ITC 430 CE and 430.5 CE (Jul 2023) are distinct again. 0 duplicate (masid, date) rows.

Old-file faults fixed on the way. utils.read_old_fo_csv is the shared reader for the loader and the lot collector:

FaultFiles
Header row repeated inside the data2002–2003
The whole day repeated (header + rows twice), which gave exact duplicates28 days, 2002–2003
Two records glued on one line (missing newline, 30 fields)some 2002 files
Column OPTIONTYPE instead of OPTION_TYP878 files from mid-2003
Day without a leading zero in the date ("1-MAR-2002")older files (download check)

Speed: new contracts are now added to the in-memory master cache instead of reloading 1.6M+ rows, so a day takes about 2 s instead of 5.5 s. The daily loader benefits too.

Lot sizes​

lots --apply filled 36.2 million contract rows and the continuous series.

  • Accuracy vs NSE's exact traded quantity ÷ contracts (market-activity files, 11 dates 2011–2024): 4,780 of 4,873 futures = 98.1% exact.
  • All errors are contracts whose lot changed mid-life:
    • a bonus or split (INFY's Dec 2014 bonus: Feb-2015 contract stored as 125, true 250);
    • NSE's periodic lot revisions (the 2024 revision had the most).
  • Why: the gcd over a contract's whole life mixes the old and the new lot, and returns a divisor. A per-day gcd alone is worse, because far months have too few values on one day.
  • Possible improvement (not done): use NSE's daily market-activity files (archives/fo/mkt/foDDMMYYYY.zip, patchy before 2013, ~1 GB) for exact per-day lots.

Tests before the full run​

  • June 2000: the first F&O days load (NIFTY futures only, as index options began June 2001). NIFTY-I/-II/-III = Jul/Aug/Sep 2000 contracts.
  • 4–10 July 2024, across the format switch: 0 failures, no duplicates. OI-only rows kept (3,500–4,300 a day). Lot 25 from 2024-07-08. The implied lot from turnover is 25 on both sides, so volume (contracts) and turnover (lakh) mean the same in both formats.
  • 29-09 → 03-10-2025: the continuous series roll correctly on the Tuesday expiry.