Skip to main content

bse_backfill.py — Historical Backfill

Resumable historical download for BSE equity + SENSEX/BANKEX derivatives + index spot. Mirrors mcxeod/mcx_backfill.py's design (live-download-driven, not a from-local-backup rebuild like nseeod/eq_backfill.py — BSE has no local backup archive the way NSE's EODData folder does).

cd backend
python -m bseeod.bse_backfill # resume -> today
python -m bseeod.bse_backfill --to 2021-12-31 # resume -> that date
python -m bseeod.bse_backfill --from 2020-01-01 --to 2020-12-31 # refill a gap
python -m bseeod.bse_backfill --limit 25 --pause 1.0
python -m bseeod.bse_backfill --status # read-only coverage report

Equity vs. Index Rows in One Table​

scr_bseeq_eod holds both equity (typ=30 masids) and SENSEX/BANKEX index-spot (typ=31 masids) rows — same table, same convention scr_nseeq_eod uses for NSE indices (see bse_index.py). Every date-coverage check in this module therefore joins against scr_bse_master's typ, never queries the price table alone:

_EQUITY_JOIN = "JOIN scr_bse_master m ON m.masid = e.masid AND m.typ = 30"
Confirmed live bug, now fixed

A plain "any row for this date" check wrongly treats a date with only the 2 SENSEX/BANKEX index rows as fully backfilled. Confirmed live 2026-10-11, right after a full equity wipe-and-rebackfill (see bse_equity.py): --status reported 369 dates as "already filled" purely from their leftover index rows, with equity data for all of them actually missing — pending_dates() and last_backfilled_date() were querying scr_bseeq_eod unscoped. Fixed to join against the equity typ everywhere; status()'s printed summary is labelled "(equity only — excludes SENSEX/BANKEX index rows)" as a result.

Equity drives resumability; derivatives (scr_bsefo_eod) and index are attempted alongside each equity day but tracked separately and never gate progress — a zero-trade SENSEX/BANKEX day, or a day before BANKEX existed, or a date before the index-summary file's 2025-04-15 availability cutoff, is a legitimately empty result, not a failure.


run_backfill​

Retries with exponential backoff per date (pause * 2^attempt); a date that exhausts retries is left unrecorded so the next run retries it automatically, never silently marked done.

The per-day Parquet-rebuild performance trap​

Confirmed live bug, now fixed — this is why progress looked stalled

An earlier version called bse_store.append_days([day]) inside the loop, once per date. append_days() calls rebuild_catalog(), a full DuckDB scan over every stock's Parquet file. With ~1,650 outstanding days, that's ~1,650 full catalog rebuilds over thousands of files — the loop wasn't frozen, it was quietly stuck doing expensive, redundant work between each print. Fixed: the per-day append was removed entirely, and bse_store.export_all() — which reads Postgres once in batches and rewrites every file plus the catalog in one pass — runs exactly once, after the whole backfill loop finishes. Confirmed: a 12-day trial that previously felt stalled now completes (loop + full catalog rebuild) in under 30 seconds.

if ok:
print("[bse_backfill] rebuilding Parquet store from Postgres (one pass)...")
bse_store.export_all()

Progress prints every 10 dates (not 25) with flush=True on every print() call in this module — Windows terminals can buffer stdout, so an unflushed print can sit invisible for a while even when the loop itself is moving.


pending_dates(start, end)​

BSE trading days in range with no equity row yet (see the _EQUITY_JOIN note above). Used both to build the work list for a run and, via status(), to report outstanding gaps as collapsed date ranges.

status(end=None)​

Read-only coverage report — overall equity range/rows, equity days per year, derivatives overall range/rows, and outstanding gaps (collapsed into ranges when consecutive missing dates are within 7 days of each other, so a run of ordinary weekend/holiday gaps doesn't fragment the output into hundreds of one-day "ranges"). Prints a ready-to-paste --from <date> command for the first gap found.


  • bse_downloader.py — BseProcessor, whose Equity/Fo/Index updaters this calls directly (not through run_update(), for finer-grained retry control per updater)
  • bse_store.py — export_all(), called once at the end
  • bse_equity.py — scrip_code matching — the bug whose fix required the full equity wipe this module's gap-fill command was then used to re-run