bse_backfill.py — Historical Backfill
Resumable historical download for BSE equity + SENSEX/BANKEX derivatives +
index spot. Mirrors mcxeod/mcx_backfill.py's design (live-download-driven,
not a from-local-backup rebuild like nseeod/eq_backfill.py — BSE has no
local backup archive the way NSE's EODData folder does).
cd backend
python -m bseeod.bse_backfill # resume -> today
python -m bseeod.bse_backfill --to 2021-12-31 # resume -> that date
python -m bseeod.bse_backfill --from 2020-01-01 --to 2020-12-31 # refill a gap
python -m bseeod.bse_backfill --limit 25 --pause 1.0
python -m bseeod.bse_backfill --status # read-only coverage report
Equity vs. Index Rows in One Table
scr_bseeq_eod holds both equity (typ=30 masids) and SENSEX/BANKEX
index-spot (typ=31 masids) rows — same table, same convention
scr_nseeq_eod uses for NSE indices (see
bse_index.py). Every date-coverage check in this module
therefore joins against scr_bse_master's typ, never queries the price
table alone:
_EQUITY_JOIN = "JOIN scr_bse_master m ON m.masid = e.masid AND m.typ = 30"
A plain "any row for this date" check wrongly treats a date with only the
2 SENSEX/BANKEX index rows as fully backfilled. Confirmed live 2026-10-11,
right after a full equity wipe-and-rebackfill (see
bse_equity.py): --status
reported 369 dates as "already filled" purely from their leftover index
rows, with equity data for all of them actually missing — pending_dates()
and last_backfilled_date() were querying scr_bseeq_eod unscoped. Fixed
to join against the equity typ everywhere; status()'s printed summary
is labelled "(equity only — excludes SENSEX/BANKEX index rows)" as a
result.
Equity drives resumability; derivatives (scr_bsefo_eod) and index are
attempted alongside each equity day but tracked separately and never gate
progress — a zero-trade SENSEX/BANKEX day, or a day before BANKEX existed,
or a date before the index-summary file's
2025-04-15 availability cutoff,
is a legitimately empty result, not a failure.
run_backfill
Retries with exponential backoff per date (pause * 2^attempt); a date
that exhausts retries is left unrecorded so the next run retries it
automatically, never silently marked done.
The per-day Parquet-rebuild performance trap
An earlier version called bse_store.append_days([day]) inside the loop,
once per date. append_days() calls rebuild_catalog(), a full DuckDB
scan over every stock's Parquet file. With ~1,650 outstanding days,
that's ~1,650 full catalog rebuilds over thousands of files — the loop
wasn't frozen, it was quietly stuck doing expensive, redundant work between
each print. Fixed: the per-day append was removed entirely, and
bse_store.export_all() — which reads Postgres once in batches and
rewrites every file plus the catalog in one pass — runs exactly once,
after the whole backfill loop finishes. Confirmed: a 12-day trial that
previously felt stalled now completes (loop + full catalog rebuild) in
under 30 seconds.
if ok:
print("[bse_backfill] rebuilding Parquet store from Postgres (one pass)...")
bse_store.export_all()
Progress prints every 10 dates (not 25) with flush=True on every
print() call in this module — Windows terminals can buffer stdout, so an
unflushed print can sit invisible for a while even when the loop itself is
moving.
pending_dates(start, end)
BSE trading days in range with no equity row yet (see the
_EQUITY_JOIN note above). Used both to build the work list for a run and,
via status(), to report outstanding gaps as collapsed date ranges.
status(end=None)
Read-only coverage report — overall equity range/rows, equity days per
year, derivatives overall range/rows, and outstanding gaps (collapsed into
ranges when consecutive missing dates are within 7 days of each other, so a
run of ordinary weekend/holiday gaps doesn't fragment the output into
hundreds of one-day "ranges"). Prints a ready-to-paste --from <date>
command for the first gap found.
Related
- bse_downloader.py —
BseProcessor, whoseEquity/Fo/Indexupdaters this calls directly (not throughrun_update(), for finer-grained retry control per updater) - bse_store.py —
export_all(), called once at the end - bse_equity.py — scrip_code matching — the bug whose fix required the full equity wipe this module's gap-fill command was then used to re-run