bse_downloader.py — Orchestrator
BseProcessor orchestrates BSE bhavcopy downloads and database updates.
Mirrors nseeod/downloadbhav.py's BhavProcessor: same
AddLog/AddHistory/UpdateState/stats.json status-reporting contract
(so the same frontend polling pattern — root CLAUDE.md rule 6 — will work
unchanged once a BSE tab exists), same rollback-baseline-before-writes
design.
Own stats.json under Config.BSE_CSV_PATH, so a BSE sync can never
clobber NSE's or MCX's status file — the same isolation nseeod and
mcxeod's downloaders already have.
DEFAULT_START_DATE
DEFAULT_START_DATE = datetime.date(2020, 1, 1)
Phase-1 scope per the approved plan. Unlike NSE's updater (whose tables are
never empty in practice, so it defaults to "7 days ago" when it can't find
a prior date), BSE's tables genuinely start empty on a fresh setup, so
MIN_DATE is this literal constant, not a relative fallback.
run_update()
Rollback baseline captured before any writes
last_safe_date = lastupdateDate() — the earlier of MAX(sdate) across
scr_bseeq_eod (equity-only, see below) and scr_bsefo_eod, minus one day
— is captured before Equity.update()/Fo.update()/Index.update()
run. If anything raises, deleteadateabove(last_safe_date) removes every
row newer than that baseline from both tables, restoring a clean state.
Same discipline as nseeod/downloadbhav.py's BhavProcessor.run_update().
_append_parquet() — called once, not per updater
Parquet history is appended for every date actually committed this run —
queried directly from the DB (sdate > last_safe_date), the same "AMI
dates" pattern nseeod/downloadbhav.py uses — after Equity, Fo and
Index have all had their turn, never from inside any individual updater.
def _append_parquet(self, dates):
"""Must run AFTER Equity, Fo and Index have all written that date's
rows into scr_bseeq_eod - calling this before Index runs silently
drops that date's SENSEX/BANKEX rows from Parquet."""
This was a real bug, confirmed live 2026-10-11: an earlier version called
the Parquet append from inside bse_equity.py's update_a_day(), which
runs before bse_index.py writes that date's SENSEX/BANKEX rows (the
orchestrator runs Equity, then Fo, then Index). bse_store.verify() caught
it directly — one date showed 4,504 Postgres rows but only 4,502 Parquet
rows, the 2 missing being exactly that date's SENSEX+BANKEX index rows.
Index/CorpActions failures never fail the sync
Both self.Index.update() and self._sync_corp_actions() are wrapped in
their own try/except, logging but swallowing any exception — a hiccup in
either never blocks equity/derivatives, which are the load-bearing data.
run_update_for_date(target_date)
Force-updates one specific date — calls Equity.update_a_day,
Fo.update_a_day, Index.update_a_day (best-effort), then
_append_parquet([target_date]) and _sync_corp_actions(), with the same
rollback-on-exception pattern (deleteadate(target_date) on failure).
This is what bse_backfill.py calls per-day during a
historical catch-up, and what a single-date CLI invocation uses directly.
get_database_status()
is_up_to_date = bool(eq_date and eq_date >= latest)
Deliberately equity-only — unlike NSE's status check (which requires
both scr_nseeq_eod and scr_nsefo_eod current), BSE's derivatives and
index are secondary data riding alongside the equity sync, not required for
"up to date" status.
Utility Functions
lastupdateDate(), deleteadate(mdate), deleteadateabove(mdate) — same
shape as their nseeod/downloadbhav.py counterparts, scoped to
{EQUITY_TABLE, FO_TABLE} (scr_bseeq_eod, scr_bsefo_eod).
deleteadate* touch the whole scr_bseeq_eod table, including index rowsA rollback deletes by sdate across the entire table, not scoped to
equity masids — appropriate here, since a rollback is meant to fully
undo everything written for that date (equity and index), not just the
equity half.
Related
- bse_equity.py, bse_derivatives.py, bse_index.py — the three updaters this orchestrates
- bse_store.py —
_append_parquet's target - bse_corp_actions.py —
_sync_corp_actions's target - bse_backfill.py — the resumable historical driver