Skip to main content

bse_downloader.py — Orchestrator

BseProcessor orchestrates BSE bhavcopy downloads and database updates. Mirrors nseeod/downloadbhav.py's BhavProcessor: same AddLog/AddHistory/UpdateState/stats.json status-reporting contract (so the same frontend polling pattern — root CLAUDE.md rule 6 — will work unchanged once a BSE tab exists), same rollback-baseline-before-writes design.

Own stats.json under Config.BSE_CSV_PATH, so a BSE sync can never clobber NSE's or MCX's status file — the same isolation nseeod and mcxeod's downloaders already have.


DEFAULT_START_DATE​

DEFAULT_START_DATE = datetime.date(2020, 1, 1)

Phase-1 scope per the approved plan. Unlike NSE's updater (whose tables are never empty in practice, so it defaults to "7 days ago" when it can't find a prior date), BSE's tables genuinely start empty on a fresh setup, so MIN_DATE is this literal constant, not a relative fallback.


run_update()​

Rollback baseline captured before any writes​

last_safe_date = lastupdateDate() — the earlier of MAX(sdate) across scr_bseeq_eod (equity-only, see below) and scr_bsefo_eod, minus one day — is captured before Equity.update()/Fo.update()/Index.update() run. If anything raises, deleteadateabove(last_safe_date) removes every row newer than that baseline from both tables, restoring a clean state. Same discipline as nseeod/downloadbhav.py's BhavProcessor.run_update().

_append_parquet() — called once, not per updater​

Parquet history is appended for every date actually committed this run — queried directly from the DB (sdate > last_safe_date), the same "AMI dates" pattern nseeod/downloadbhav.py uses — after Equity, Fo and Index have all had their turn, never from inside any individual updater.

def _append_parquet(self, dates):
"""Must run AFTER Equity, Fo and Index have all written that date's
rows into scr_bseeq_eod - calling this before Index runs silently
drops that date's SENSEX/BANKEX rows from Parquet."""

This was a real bug, confirmed live 2026-10-11: an earlier version called the Parquet append from inside bse_equity.py's update_a_day(), which runs before bse_index.py writes that date's SENSEX/BANKEX rows (the orchestrator runs Equity, then Fo, then Index). bse_store.verify() caught it directly — one date showed 4,504 Postgres rows but only 4,502 Parquet rows, the 2 missing being exactly that date's SENSEX+BANKEX index rows.

Index/CorpActions failures never fail the sync​

Both self.Index.update() and self._sync_corp_actions() are wrapped in their own try/except, logging but swallowing any exception — a hiccup in either never blocks equity/derivatives, which are the load-bearing data.


run_update_for_date(target_date)​

Force-updates one specific date — calls Equity.update_a_day, Fo.update_a_day, Index.update_a_day (best-effort), then _append_parquet([target_date]) and _sync_corp_actions(), with the same rollback-on-exception pattern (deleteadate(target_date) on failure). This is what bse_backfill.py calls per-day during a historical catch-up, and what a single-date CLI invocation uses directly.


get_database_status()​

is_up_to_date = bool(eq_date and eq_date >= latest)

Deliberately equity-only — unlike NSE's status check (which requires both scr_nseeq_eod and scr_nsefo_eod current), BSE's derivatives and index are secondary data riding alongside the equity sync, not required for "up to date" status.


Utility Functions​

lastupdateDate(), deleteadate(mdate), deleteadateabove(mdate) — same shape as their nseeod/downloadbhav.py counterparts, scoped to {EQUITY_TABLE, FO_TABLE} (scr_bseeq_eod, scr_bsefo_eod).

deleteadate* touch the whole scr_bseeq_eod table, including index rows

A rollback deletes by sdate across the entire table, not scoped to equity masids — appropriate here, since a rollback is meant to fully undo everything written for that date (equity and index), not just the equity half.