BSEEOD Package — Overview
backend/bseeod/ handles the full lifecycle of BSE (Bombay Stock Exchange)
end-of-day data: equity cash-market prices, SENSEX/BANKEX index-spot
values, SENSEX/BANKEX futures and options, and BSE's own corporate-action
feed — plus a per-stock Parquet history store mirroring nseeod/eq_store.py.
It deliberately mirrors nseeod's architecture (orchestrator pattern,
stats.json progress file, rollback-baseline discipline, resumable
backfill) and follows the precedent mcxeod set for a second exchange: a
fully separate schema. BSE instruments are never written to
scr_master — that table is NSE-only — and never to scr_mcx_master
either.
Phase 1 scope (approved 2026-10-11): equity + SENSEX/BANKEX derivatives +
index spot, maintained from 2020-01-01 onward (deep pre-2020 history is
an explicit phase-2 follow-up). Survivorship-bias-free index-membership
tracking (a BSE equivalent of nseeod/index_members.py) is not built
yet — see Not Yet Built below.
Package File Map
| File | Role |
|---|---|
bse_infra.py | Infrastructure bootstrap — idempotent DDL for scr_bse_master / scr_bseeq_eod / scr_bsefo_eod, their indexes, and the modi_date trigger |
bse_equity.py | Equity cash-market ingestion — BseEodEqUpdater, both bhavcopy eras, the debt-instrument filter |
bse_derivatives.py | SENSEX/BANKEX futures & options ingestion — BseEodFoUpdater |
bse_index.py | SENSEX/BANKEX index-spot ingestion — BseEodIndexUpdater |
bse_downloader.py | Orchestrator — BseProcessor, same AddLog/stats.json/rollback contract as nseeod/downloadbhav.py's BhavProcessor |
bse_history.py | Trading calendar, BSEWorkingDays(), holiday-calendar seeding from NSE's (exch=3) |
bse_backfill.py | Resumable historical download (2020-01-01 → today) |
bse_store.py | Per-stock Parquet history store, mirrors nseeod/eq_store.py |
bse_corp_actions.py | BSE's own corporate-action feed — download, parse, scr_bse_corp_action |
High-Level Architecture
Data Flow — Daily Sync
Why the Parquet append order matters: scr_bseeq_eod holds both equity
and SENSEX/BANKEX index rows (same table, same convention eq_store.py
uses for NSE). Appending to Parquet right after the equity step — before
BseEodIndexUpdater has written that date's index rows — silently drops
the index rows from the Parquet copy for that date. Confirmed live
2026-10-11: one early version did exactly this and left Parquet two rows
short of Postgres for the affected date. BseProcessor._append_parquet()
and bse_backfill.py's per-day loop both call the Parquet append only
after every writer for that date has run.
Two Bhavcopy Eras
Confirmed live against bseindia.com/markets/marketinfo/bhavcopy
(2026-10-11, BSE's own discontinuation notice, Notice No 20240610-33):
| Era | Equity URL | Derivatives URL |
|---|---|---|
| UDiFF (≥ 2024-07-08) | .../download/BhavCopy/Equity/BhavCopy_BSE_CM_0_0_0_{YYYYMMDD}_F_0000.CSV — plain CSV, same column vocabulary as NSE's UDiFF format | .../download/BhavCopy/Derivative/BhavCopy_BSE_FO_0_0_0_{YYYYMMDD}_F_0000.CSV |
| Old format (< 2024-07-08) | .../download/BhavCopy/Equity/EQ{DDMMYY}_CSV.ZIP + a separate EQ_ISINCODE_{DDMMYY}.zip for ISIN | .../download/BhavCopy/Derivative/bhavcopy{DD-MM-YY}.zip — a completely different legacy schema, no lot-size column |
See bse_equity.py and bse_derivatives.py for the full column mapping of each.
Index-Summary File — a Hard Data Gap
SENSEX/BANKEX index-spot OHLC comes from
https://www.bseindia.com/bsedata/Index_Bhavcopy/INDEXSummary_{DDMMYYYY}.csv.
Confirmed live by binary-searching HTTP 200/404 across 2024–2025 (cross-checked
against actual weekdays and the seeded BSE holiday calendar to rule out false
negatives from weekends/holidays): this file is only available from
2025-04-15 onward. Every date before 2025-04-14 returns 404, with a sharp
cutover to 200 starting exactly 2025-04-15. This is a separate, later cutover
than the equity/derivatives UDiFF switch, and BSE exposes no older-format
fallback for it.
SENSEX/BANKEX index-spot history before 2025-04-15 is not obtainable from
this source at all — bse_index.py treats dates before INDEX_DATA_START
(2025-04-15) as a known no-op, not a failure, so it never pollutes a
backfill's failure counts. Equity and derivatives are unaffected — the
project's 2020-01-01 start applies to those as normal.
Key Database Tables
scr_bse_master — BSE Security Master
| Column | Type | Description |
|---|---|---|
masid | SERIAL PK | Auto-increment ID |
mname | VARCHAR(100) | Display name (company name or contract name) — not unique, see below |
scrip_code | VARCHAR(20) UNIQUE | BSE's stable numeric identifier — the real join key for equity rows |
typ | SMALLINT | 30=EQ, 31=IDX (SENSEX/BANKEX spot), 32=FUTIDX, 33=OPTIDX |
underly | INTEGER | FK → masid of the index root (futures/options only) |
edate | DATE | Expiry (derivatives only) |
strike / opttyp | REAL / CHAR(2) | Options only |
monthindex | SMALLINT | 0=dated, 1/2/3=continuous -I/-II/-III (index futures only) |
isin | VARCHAR(12) | Kept current from each day's bhavcopy, same pattern as scr_master.isin |
tag1 | VARCHAR(100) | BSE's SctySrs/SC_GROUP code (A, B, F, IF, ...) — reference only |
mname is not a unique identifier on BSEConfirmed live 2026-10-11: BSE's ~12-char truncated company name is
neither unique nor stable the way NSE's SYMBOL is. Two unrelated
companies can truncate to the same name (SHELTER at scrip_code 526839
is a completely different company from SHELTER at 543963 — different
ISINs), and the same company's name string can differ between the
old-format and UDiFF eras (ABB LTD. vs ABB, same scrip_code 500002).
Matching by mname instead of scrip_code first merged unrelated
companies' histories together and, separately, fragmented single
companies' histories across duplicate masids — 43% of BSE equity
histories (2,842 of 6,667 stocks) were affected before this was caught
and fixed. Every matching function in this package resolves equity rows by
scrip_code, never by mname. See
bse_equity.py — match_scr_bse_masters_and_update
for the full writeup.
scr_bseeq_eod — Equity + Index-Spot Prices
masid, sdate composite PK. sopen/shigh/slow/sclose/svolume/svalue/ prevclose/slast/delqty/avgprice. Holds both equity stocks (typ=30
masids) and SENSEX/BANKEX index-spot rows (typ=31 masids) — same table,
same convention scr_nseeq_eod uses for NSE indices. Any date-coverage
query must join against scr_bse_master's typ to tell the two apart —
see the warning in bse_backfill.py.
scr_bsefo_eod — SENSEX/BANKEX Derivatives
masid, sdate, sopen/shigh/slow/sclose/svolume/oi/coi/lotsize. No enforced
PK (same "delete the date, then insert" rule as scr_nsefo_eod). lotsize
is 0 for old-format (pre-UDiFF) rows — that era's file has no lot-size
column at all, unlike NSE's old format (which at least has OI to
gcd-derive it from). A BSE lot-size-by-date reference table is a documented
follow-up, not solved here.
scr_bse_corp_action
Mirrors scr_corp_action's shape, keyed by scrip_code instead of
symbol/isin (BSE's feed has neither). See
bse_corp_actions.py.
Calendar — shared with nseeod, discriminated by exch
holidays / specialdays gain exch = 3 for BSE, seeded once from NSE's
exch = 1 rows (India's BSE/NSE trading calendars are effectively
identical) via bse_history.seed_bse_holidays_from_nse().
Running Standalone
cd backend
python -m bseeod.bse_infra # bootstrap/verify schema
python -m bseeod.bse_history --apply # seed holidays (exch=3) from NSE's
python -c "from bseeod.bse_downloader import BseProcessor; BseProcessor().run_update()"
python -m bseeod.bse_backfill --from 2020-01-01 # resumable historical catch-up
python -m bseeod.bse_backfill --status # read-only coverage report
python -m bseeod.bse_store export # full Parquet rebuild from Postgres
python -m bseeod.bse_store verify # Parquet vs Postgres, every date
python -m bseeod.bse_corp_actions download --dir <path> # corp-action backup
python -m bseeod.bse_corp_actions parse --dir <path> --apply
API
Not yet wired to a FastAPI router (app/api/bse_routes.py is a planned
follow-up, mirroring app/api/bhav_downloader.py's surface). Today,
BseProcessor is driven from Python directly or via bse_backfill.py's CLI.
Frontend
Not yet built — a BSE tab/section in BhavDownloader.tsx is a planned
follow-up, reusing the existing polling pattern (root CLAUDE.md rule 6).
Not Yet Built
- Survivorship-bias-free index membership — a BSE equivalent of
nseeod/index_members.py'sidx_*tables andidx_members_on()function, for BSE indices (SENSEX, BSE100, BSE500, ...).index_members.py's schema is already index-name-keyed, not exchange-keyed, so BSE indices could in principle be added as more rows in the sameidx_index/idx_constituent/idx_membershiptables and reuseidx_members_on()unchanged — what's missing is BSE-specific scraping (constituent weightage files, index-change announcements), which is in a different format from NSE's PDFs/press releases and hasn't been built. reconcile()— the price-gap verification pass for corporate actions (nseeod/corp_actions.py'sreconcile()), which needsprice_windows()/price_moves()helpers not yet ported tobse_store.py.- BSE lot-size-by-date reference table for old-format (pre-UDiFF) derivative days, which carry no lot-size column at all.
- Corporate-action price adjustment —
bse_store.get_eod(adjust=...)raisesNotImplementedErrorfor anything but'none'; wiring it up toscr_bse_corp_actionis a direct port ofeq_store.get_eod's_apply_adjust, blocked only onreconcile()existing first (unreconciledAUTOfactors are usable but unverified). - FastAPI routes + frontend tab (see above).
Related Docs
- bse_infra.py — Infrastructure Bootstrap
- bse_equity.py — Equity Ingestion
- bse_derivatives.py — SENSEX/BANKEX Derivatives
- bse_index.py — Index Spot
- bse_downloader.py — Orchestrator
- bse_history.py — Trading Calendar
- bse_backfill.py — Historical Backfill
- bse_store.py — Parquet History Store
- bse_corp_actions.py — Corporate Actions