Skip to main content

bse_equity.py — Equity Ingestion

Downloads and parses BSE's equity bhavcopy, resolves each row against scr_bse_master, and replaces that day's rows in scr_bseeq_eod. Mirrors nseeod/EquityBhav.py + the equity-parsing half of nseeod/utils.py, simplified: BSE's equity bhavcopy is one self-contained file, unlike NSE's cm+MTO+index split, so there's no separate index/delivery-volume downloader here (SENSEX/BANKEX index spot is bse_index.py).


Two Eras​

Confirmed live 2026-10-11 against bseindia.com/markets/marketinfo/bhavcopy (BSE's own discontinuation notice, Notice No 20240610-33):

UDiFF (≥ 2024-07-08)​

https://www.bseindia.com/download/BhavCopy/Equity/BhavCopy_BSE_CM_0_0_0_{YYYYMMDD}_F_0000.CSV

Same column vocabulary as NSE's UDiFF format: TradDt, BizDt, Sgmt, Src, FinInstrmTp, FinInstrmId, ISIN, TckrSymb, SctySrs, ..., OpnPric, HghPric, LwPric, ClsPric, ..., TtlTradgVol, TtlTrfVal, ..., PrvsClsgPric, LastPric. FinInstrmId is BSE's numeric scrip code. Filtered to Sgmt == 'CM' (cash market) before the debt-instrument filter.

Old format (< 2024-07-08, covers the project's 2020-01-01 start)​

https://www.bseindia.com/download/BhavCopy/Equity/EQ{DDMMYY}_CSV.ZIP → EQ{DDMMYY}.CSV: SC_CODE, SC_NAME, SC_GROUP, SC_TYPE, OPEN, HIGH, LOW, CLOSE, LAST, PREVCLOSE, NO_TRADES, NO_OF_SHRS, NET_TURNOV, TDCLOINDI. No ISIN — that needs the separate https://www.bseindia.com/download/BhavCopy/Equity/EQ_ISINCODE_{DDMMYY}.zip (EQ_ISINCODE_{DDMMYY}.CSV: same columns + ISIN_CODE, TRADING_DATE), merged in by SC_CODE — the same "second file, merge by key" pattern as NSE's MTO delivery-volume merge (nseeod/utils.py's merge_delivery). Filtered to SC_TYPE == 'Q' (regular quotation-priced) before the debt-instrument filter.


A Missing-Newline Bug in the Old-Format File​

Confirmed live on 2021-12-29 (EQ291221.CSV, line 1773): BSE's old-format equity file occasionally drops the newline between two records, gluing them onto one line and dropping the first record's trailing empty TDCLOINDI field in the process:

531358,CHOICE INT. ,X ,Q,152.50,...,2504395.00,531359,SHRIRAM ASSE,XT,Q,98.20,...,19638.00,

27 comma-fields instead of 2×14 — not a clean multiple of the 14-column header, so NSE's read_old_fo_csv chunking trick (built for a different glued-record fault in 2002–2003 NSE F&O files) doesn't directly apply.

_read_old_equity_csv() repairs this by field shape, not a line-number allowlist, so it self-heals if the fault recurs on other dates: whenever a NET_TURNOV-shaped field (\d+\.\d{2}) is immediately followed by what looks like a fresh SC_CODE rather than the expected (often blank) TDCLOINDI, a blank TDCLOINDI is reinserted to re-align the record boundary.

The companion ISIN file for the same date parses cleanly with plain pd.read_csv — the corruption is specific to the main EQ*.CSV file.


Debt Instruments Are Mixed Into the "Equity" File​

BSE's equity bhavcopy is not equity-only — it carries every SctySrs/SC_GROUP traded on the cash segment. Confirmed live 2026-10-11 against a full day's file (2024-08-01):

GroupWhat it actually is
G100% government securities / Sovereign Gold Bonds, ISIN prefix IN0 (e.g. IN0020110048 "915GOI2024", IN0020160027 an SGB tranche)
FA mix: ETFs (ISIN prefix INF, e.g. INF204KB10R1 "07AGG", a Nippon India ETF) and corporate NCDs/bonds (ISIN prefix INE but a non-equity instrument-type code — 04/07/08/09 — at isin[7:9], e.g. INE134E07588 "760PFCL35", a PFC bond) under one SctySrs

Project decision 2026-10-11: remove only debt (government securities and corporate bonds/NCDs); keep plain equity, ETFs, REITs/InvITs (group IF), rights entitlements (group R), and partly-paid/DVR shares (ISIN prefix IN9..., e.g. IN9155A01020 "TATAMTRDVR") — none of those are debt.

_equity_only() implements this: drop group G outright, and within group F drop rows whose ISIN starts INE (keep INF). Every other group is left alone. A row with no ISIN at all (old-format date whose EQ_ISINCODE file didn't merge) only gets the group-G check — group F can't be split into ETF/NCD without the ISIN, so it's left in rather than guessed.

Without this filter, debt rows load straight into scr_bseeq_eod and scr_bse_master as if they were stocks — every row has valid OHLCV and inserts cleanly, so it's caught only by a human noticing oddly-named Parquet files (SGBAPR28I, 760PFCL35, ...) in a diff, never by an automated check.


match_scr_bse_masters_and_update​

Resolves each row's scrip_code to scr_bse_master.masid (inserting new rows as needed), then replaces that day's scr_bseeq_eod rows.

Why scrip_code, not mname​

This function originally matched by mname (company name text), mirroring how NSE's match_scr_masters_and_update matches by mname — reasonable for NSE, where SYMBOL is a reliable unique identifier, but wrong for BSE. Confirmed live 2026-10-11, both directions:

  1. Two unrelated companies can truncate to the same ~12-char name. "SHELTER" at scrip_code 526839 is a completely different company from "SHELTER" at 543963 — different ISINs (INE413C01013 vs INE013V01011). Matching by mname would silently merge their price histories into one masid.
  2. The same company's name string varies across bhav eras. ABB appears as "ABB LTD." in the old-format era and "ABB" in UDiFF, both scrip_code 500002. Matching by mname split what should be one stock's history across two masids.

The scale was severe: 2,842 of 6,667 distinct BSE stocks (43%) had their price history fragmented across duplicate masids by the time this was caught (after a full 2020→today backfill). The fix — matching by scrip_code, BSE's actual stable identifier, which never changes and never collides — required a schema migration (026cf2bcd55f, moving the UNIQUE constraint off mname), updating bse_store.py's Parquet file naming to key on scrip_code too (filenames could still collide on disk even after the DB-side fix), and a full wipe-and-rebackfill of equity data (derivatives and index data were untouched — not affected by this bug).

mname and isin are kept current on every sync via pgs.bulk_update_by_key — cheap no-ops when unchanged, and the mechanism by which a company's display name naturally updates in place (as its SC_NAME text drifts across bhav eras) instead of ever needing a second masid for it.

Dual-listing pairs resolve to separate masids — correctly​

A side effect of matching by scrip_code: BSE's dual-listing pairs (same company, same ISIN, two scrip codes — e.g. AMBUJA CEME at both 100425 and 500425, one with real trading volume and one with a token volume, looking like an alternate-segment listing for the same stock) now resolve to two separate masids, which is the accurate representation — they really are two distinct tradeable listings, even though they share a company. This also eliminates the UniqueViolation crash the old mname-based matching hit when both rows of such a pair landed in the same day's insert batch (confirmed live: duplicate key value violates unique constraint "pk_bseeq_eod").


BseEodEqUpdater​

Daily sync driver — mirrors nseeod.EquityBhav.NseEodEqUpdater, simplified (no separate index/delivery-volume sub-updater). update() walks BSEWorkingDays() from the day after MAX(sdate) (or MIN_DATE on an empty table) up to the cutoff trading time; update_a_day(mdate) is the single-date entry point used by both BseProcessor.run_update_for_date and bse_backfill.py's per-day loop.

Deliberately does not append to Parquet itself — see the Overview's note on append ordering.