Skip to main content

BSEEOD Package — Overview

backend/bseeod/ handles the full lifecycle of BSE (Bombay Stock Exchange) end-of-day data: equity cash-market prices, SENSEX/BANKEX index-spot values, SENSEX/BANKEX futures and options, and BSE's own corporate-action feed — plus a per-stock Parquet history store mirroring nseeod/eq_store.py.

It deliberately mirrors nseeod's architecture (orchestrator pattern, stats.json progress file, rollback-baseline discipline, resumable backfill) and follows the precedent mcxeod set for a second exchange: a fully separate schema. BSE instruments are never written to scr_master — that table is NSE-only — and never to scr_mcx_master either.

Phase 1 scope (approved 2026-10-11): equity + SENSEX/BANKEX derivatives + index spot, maintained from 2020-01-01 onward (deep pre-2020 history is an explicit phase-2 follow-up). Survivorship-bias-free index-membership tracking (a BSE equivalent of nseeod/index_members.py) is not built yet — see Not Yet Built below.


Package File Map​

FileRole
bse_infra.pyInfrastructure bootstrap — idempotent DDL for scr_bse_master / scr_bseeq_eod / scr_bsefo_eod, their indexes, and the modi_date trigger
bse_equity.pyEquity cash-market ingestion — BseEodEqUpdater, both bhavcopy eras, the debt-instrument filter
bse_derivatives.pySENSEX/BANKEX futures & options ingestion — BseEodFoUpdater
bse_index.pySENSEX/BANKEX index-spot ingestion — BseEodIndexUpdater
bse_downloader.pyOrchestrator — BseProcessor, same AddLog/stats.json/rollback contract as nseeod/downloadbhav.py's BhavProcessor
bse_history.pyTrading calendar, BSEWorkingDays(), holiday-calendar seeding from NSE's (exch=3)
bse_backfill.pyResumable historical download (2020-01-01 → today)
bse_store.pyPer-stock Parquet history store, mirrors nseeod/eq_store.py
bse_corp_actions.pyBSE's own corporate-action feed — download, parse, scr_bse_corp_action

High-Level Architecture​


Data Flow — Daily Sync​

Why the Parquet append order matters: scr_bseeq_eod holds both equity and SENSEX/BANKEX index rows (same table, same convention eq_store.py uses for NSE). Appending to Parquet right after the equity step — before BseEodIndexUpdater has written that date's index rows — silently drops the index rows from the Parquet copy for that date. Confirmed live 2026-10-11: one early version did exactly this and left Parquet two rows short of Postgres for the affected date. BseProcessor._append_parquet() and bse_backfill.py's per-day loop both call the Parquet append only after every writer for that date has run.


Two Bhavcopy Eras​

Confirmed live against bseindia.com/markets/marketinfo/bhavcopy (2026-10-11, BSE's own discontinuation notice, Notice No 20240610-33):

EraEquity URLDerivatives URL
UDiFF (≥ 2024-07-08).../download/BhavCopy/Equity/BhavCopy_BSE_CM_0_0_0_{YYYYMMDD}_F_0000.CSV — plain CSV, same column vocabulary as NSE's UDiFF format.../download/BhavCopy/Derivative/BhavCopy_BSE_FO_0_0_0_{YYYYMMDD}_F_0000.CSV
Old format (< 2024-07-08).../download/BhavCopy/Equity/EQ{DDMMYY}_CSV.ZIP + a separate EQ_ISINCODE_{DDMMYY}.zip for ISIN.../download/BhavCopy/Derivative/bhavcopy{DD-MM-YY}.zip — a completely different legacy schema, no lot-size column

See bse_equity.py and bse_derivatives.py for the full column mapping of each.


Index-Summary File — a Hard Data Gap​

SENSEX/BANKEX index-spot OHLC comes from https://www.bseindia.com/bsedata/Index_Bhavcopy/INDEXSummary_{DDMMYYYY}.csv. Confirmed live by binary-searching HTTP 200/404 across 2024–2025 (cross-checked against actual weekdays and the seeded BSE holiday calendar to rule out false negatives from weekends/holidays): this file is only available from 2025-04-15 onward. Every date before 2025-04-14 returns 404, with a sharp cutover to 200 starting exactly 2025-04-15. This is a separate, later cutover than the equity/derivatives UDiFF switch, and BSE exposes no older-format fallback for it.

SENSEX/BANKEX index-spot history before 2025-04-15 is not obtainable from this source at all — bse_index.py treats dates before INDEX_DATA_START (2025-04-15) as a known no-op, not a failure, so it never pollutes a backfill's failure counts. Equity and derivatives are unaffected — the project's 2020-01-01 start applies to those as normal.


Key Database Tables​

scr_bse_master — BSE Security Master​

ColumnTypeDescription
masidSERIAL PKAuto-increment ID
mnameVARCHAR(100)Display name (company name or contract name) — not unique, see below
scrip_codeVARCHAR(20) UNIQUEBSE's stable numeric identifier — the real join key for equity rows
typSMALLINT30=EQ, 31=IDX (SENSEX/BANKEX spot), 32=FUTIDX, 33=OPTIDX
underlyINTEGERFK → masid of the index root (futures/options only)
edateDATEExpiry (derivatives only)
strike / opttypREAL / CHAR(2)Options only
monthindexSMALLINT0=dated, 1/2/3=continuous -I/-II/-III (index futures only)
isinVARCHAR(12)Kept current from each day's bhavcopy, same pattern as scr_master.isin
tag1VARCHAR(100)BSE's SctySrs/SC_GROUP code (A, B, F, IF, ...) — reference only
mname is not a unique identifier on BSE

Confirmed live 2026-10-11: BSE's ~12-char truncated company name is neither unique nor stable the way NSE's SYMBOL is. Two unrelated companies can truncate to the same name (SHELTER at scrip_code 526839 is a completely different company from SHELTER at 543963 — different ISINs), and the same company's name string can differ between the old-format and UDiFF eras (ABB LTD. vs ABB, same scrip_code 500002). Matching by mname instead of scrip_code first merged unrelated companies' histories together and, separately, fragmented single companies' histories across duplicate masids — 43% of BSE equity histories (2,842 of 6,667 stocks) were affected before this was caught and fixed. Every matching function in this package resolves equity rows by scrip_code, never by mname. See bse_equity.py — match_scr_bse_masters_and_update for the full writeup.

scr_bseeq_eod — Equity + Index-Spot Prices​

masid, sdate composite PK. sopen/shigh/slow/sclose/svolume/svalue/ prevclose/slast/delqty/avgprice. Holds both equity stocks (typ=30 masids) and SENSEX/BANKEX index-spot rows (typ=31 masids) — same table, same convention scr_nseeq_eod uses for NSE indices. Any date-coverage query must join against scr_bse_master's typ to tell the two apart — see the warning in bse_backfill.py.

scr_bsefo_eod — SENSEX/BANKEX Derivatives​

masid, sdate, sopen/shigh/slow/sclose/svolume/oi/coi/lotsize. No enforced PK (same "delete the date, then insert" rule as scr_nsefo_eod). lotsize is 0 for old-format (pre-UDiFF) rows — that era's file has no lot-size column at all, unlike NSE's old format (which at least has OI to gcd-derive it from). A BSE lot-size-by-date reference table is a documented follow-up, not solved here.

scr_bse_corp_action​

Mirrors scr_corp_action's shape, keyed by scrip_code instead of symbol/isin (BSE's feed has neither). See bse_corp_actions.py.

Calendar — shared with nseeod, discriminated by exch​

holidays / specialdays gain exch = 3 for BSE, seeded once from NSE's exch = 1 rows (India's BSE/NSE trading calendars are effectively identical) via bse_history.seed_bse_holidays_from_nse().


Running Standalone​

cd backend
python -m bseeod.bse_infra # bootstrap/verify schema
python -m bseeod.bse_history --apply # seed holidays (exch=3) from NSE's
python -c "from bseeod.bse_downloader import BseProcessor; BseProcessor().run_update()"
python -m bseeod.bse_backfill --from 2020-01-01 # resumable historical catch-up
python -m bseeod.bse_backfill --status # read-only coverage report
python -m bseeod.bse_store export # full Parquet rebuild from Postgres
python -m bseeod.bse_store verify # Parquet vs Postgres, every date
python -m bseeod.bse_corp_actions download --dir <path> # corp-action backup
python -m bseeod.bse_corp_actions parse --dir <path> --apply

API​

Not yet wired to a FastAPI router (app/api/bse_routes.py is a planned follow-up, mirroring app/api/bhav_downloader.py's surface). Today, BseProcessor is driven from Python directly or via bse_backfill.py's CLI.

Frontend​

Not yet built — a BSE tab/section in BhavDownloader.tsx is a planned follow-up, reusing the existing polling pattern (root CLAUDE.md rule 6).

Not Yet Built​

  • Survivorship-bias-free index membership — a BSE equivalent of nseeod/index_members.py's idx_* tables and idx_members_on() function, for BSE indices (SENSEX, BSE100, BSE500, ...). index_members.py's schema is already index-name-keyed, not exchange-keyed, so BSE indices could in principle be added as more rows in the same idx_index / idx_constituent / idx_membership tables and reuse idx_members_on() unchanged — what's missing is BSE-specific scraping (constituent weightage files, index-change announcements), which is in a different format from NSE's PDFs/press releases and hasn't been built.
  • reconcile() — the price-gap verification pass for corporate actions (nseeod/corp_actions.py's reconcile()), which needs price_windows()/price_moves() helpers not yet ported to bse_store.py.
  • BSE lot-size-by-date reference table for old-format (pre-UDiFF) derivative days, which carry no lot-size column at all.
  • Corporate-action price adjustment — bse_store.get_eod(adjust=...) raises NotImplementedError for anything but 'none'; wiring it up to scr_bse_corp_action is a direct port of eq_store.get_eod's _apply_adjust, blocked only on reconcile() existing first (unreconciled AUTO factors are usable but unverified).
  • FastAPI routes + frontend tab (see above).