bse_equity.py — Equity Ingestion
Downloads and parses BSE's equity bhavcopy, resolves each row against
scr_bse_master, and replaces that day's rows in scr_bseeq_eod. Mirrors
nseeod/EquityBhav.py + the equity-parsing half of nseeod/utils.py,
simplified: BSE's equity bhavcopy is one self-contained file, unlike NSE's
cm+MTO+index split, so there's no separate index/delivery-volume
downloader here (SENSEX/BANKEX index spot is bse_index.py).
Two Eras
Confirmed live 2026-10-11 against bseindia.com/markets/marketinfo/bhavcopy
(BSE's own discontinuation notice, Notice No 20240610-33):
UDiFF (≥ 2024-07-08)
https://www.bseindia.com/download/BhavCopy/Equity/BhavCopy_BSE_CM_0_0_0_{YYYYMMDD}_F_0000.CSV
Same column vocabulary as NSE's UDiFF format: TradDt, BizDt, Sgmt, Src, FinInstrmTp, FinInstrmId, ISIN, TckrSymb, SctySrs, ..., OpnPric, HghPric, LwPric, ClsPric, ..., TtlTradgVol, TtlTrfVal, ..., PrvsClsgPric, LastPric.
FinInstrmId is BSE's numeric scrip code. Filtered to Sgmt == 'CM'
(cash market) before the debt-instrument filter.
Old format (< 2024-07-08, covers the project's 2020-01-01 start)
https://www.bseindia.com/download/BhavCopy/Equity/EQ{DDMMYY}_CSV.ZIP →
EQ{DDMMYY}.CSV: SC_CODE, SC_NAME, SC_GROUP, SC_TYPE, OPEN, HIGH, LOW, CLOSE, LAST, PREVCLOSE, NO_TRADES, NO_OF_SHRS, NET_TURNOV, TDCLOINDI.
No ISIN — that needs the separate
https://www.bseindia.com/download/BhavCopy/Equity/EQ_ISINCODE_{DDMMYY}.zip
(EQ_ISINCODE_{DDMMYY}.CSV: same columns + ISIN_CODE, TRADING_DATE),
merged in by SC_CODE — the same "second file, merge by key" pattern as
NSE's MTO delivery-volume merge (nseeod/utils.py's merge_delivery).
Filtered to SC_TYPE == 'Q' (regular quotation-priced) before the
debt-instrument filter.
A Missing-Newline Bug in the Old-Format File
Confirmed live on 2021-12-29 (EQ291221.CSV, line 1773): BSE's old-format
equity file occasionally drops the newline between two records, gluing
them onto one line and dropping the first record's trailing empty
TDCLOINDI field in the process:
531358,CHOICE INT. ,X ,Q,152.50,...,2504395.00,531359,SHRIRAM ASSE,XT,Q,98.20,...,19638.00,
27 comma-fields instead of 2×14 — not a clean multiple of the 14-column
header, so NSE's read_old_fo_csv chunking trick (built for a different
glued-record fault in 2002–2003 NSE F&O files) doesn't directly apply.
_read_old_equity_csv() repairs this by field shape, not a line-number
allowlist, so it self-heals if the fault recurs on other dates: whenever a
NET_TURNOV-shaped field (\d+\.\d{2}) is immediately followed by what
looks like a fresh SC_CODE rather than the expected (often blank)
TDCLOINDI, a blank TDCLOINDI is reinserted to re-align the record
boundary.
The companion ISIN file for the same date parses cleanly with plain
pd.read_csv — the corruption is specific to the main EQ*.CSV file.
Debt Instruments Are Mixed Into the "Equity" File
BSE's equity bhavcopy is not equity-only — it carries every
SctySrs/SC_GROUP traded on the cash segment. Confirmed live 2026-10-11
against a full day's file (2024-08-01):
| Group | What it actually is |
|---|---|
G | 100% government securities / Sovereign Gold Bonds, ISIN prefix IN0 (e.g. IN0020110048 "915GOI2024", IN0020160027 an SGB tranche) |
F | A mix: ETFs (ISIN prefix INF, e.g. INF204KB10R1 "07AGG", a Nippon India ETF) and corporate NCDs/bonds (ISIN prefix INE but a non-equity instrument-type code — 04/07/08/09 — at isin[7:9], e.g. INE134E07588 "760PFCL35", a PFC bond) under one SctySrs |
Project decision 2026-10-11: remove only debt (government securities
and corporate bonds/NCDs); keep plain equity, ETFs, REITs/InvITs (group
IF), rights entitlements (group R), and partly-paid/DVR shares (ISIN
prefix IN9..., e.g. IN9155A01020 "TATAMTRDVR") — none of those are
debt.
_equity_only() implements this: drop group G outright, and within
group F drop rows whose ISIN starts INE (keep INF). Every other group
is left alone. A row with no ISIN at all (old-format date whose
EQ_ISINCODE file didn't merge) only gets the group-G check — group F
can't be split into ETF/NCD without the ISIN, so it's left in rather than
guessed.
Without this filter, debt rows load straight into scr_bseeq_eod and
scr_bse_master as if they were stocks — every row has valid OHLCV and
inserts cleanly, so it's caught only by a human noticing oddly-named
Parquet files (SGBAPR28I, 760PFCL35, ...) in a diff, never by an
automated check.
match_scr_bse_masters_and_update
Resolves each row's scrip_code to scr_bse_master.masid (inserting new
rows as needed), then replaces that day's scr_bseeq_eod rows.
Why scrip_code, not mname
This function originally matched by mname (company name text), mirroring
how NSE's match_scr_masters_and_update matches by mname — reasonable
for NSE, where SYMBOL is a reliable unique identifier, but wrong for
BSE. Confirmed live 2026-10-11, both directions:
- Two unrelated companies can truncate to the same ~12-char name.
"SHELTER"atscrip_code 526839is a completely different company from"SHELTER"at543963— different ISINs (INE413C01013vsINE013V01011). Matching bymnamewould silently merge their price histories into onemasid. - The same company's name string varies across bhav eras. ABB
appears as
"ABB LTD."in the old-format era and"ABB"in UDiFF, bothscrip_code 500002. Matching bymnamesplit what should be one stock's history across twomasids.
The scale was severe: 2,842 of 6,667 distinct BSE stocks (43%) had
their price history fragmented across duplicate masids by the time this
was caught (after a full 2020→today backfill). The fix — matching by
scrip_code, BSE's actual stable identifier, which never changes and
never collides — required a schema migration
(026cf2bcd55f, moving the
UNIQUE constraint off mname), updating
bse_store.py's Parquet file naming to key
on scrip_code too (filenames could still collide on disk even after the
DB-side fix), and a full wipe-and-rebackfill of equity data (derivatives
and index data were untouched — not affected by this bug).
mname and isin are kept current on every sync via
pgs.bulk_update_by_key — cheap no-ops when unchanged, and the mechanism
by which a company's display name naturally updates in place (as its
SC_NAME text drifts across bhav eras) instead of ever needing a second
masid for it.
Dual-listing pairs resolve to separate masids — correctly
A side effect of matching by scrip_code: BSE's dual-listing pairs (same
company, same ISIN, two scrip codes — e.g. AMBUJA CEME at both 100425
and 500425, one with real trading volume and one with a token volume,
looking like an alternate-segment listing for the same stock) now resolve
to two separate masids, which is the accurate representation — they
really are two distinct tradeable listings, even though they share a
company. This also eliminates the UniqueViolation crash the old
mname-based matching hit when both rows of such a pair landed in the
same day's insert batch (confirmed live: duplicate key value violates unique constraint "pk_bseeq_eod").
BseEodEqUpdater
Daily sync driver — mirrors nseeod.EquityBhav.NseEodEqUpdater, simplified
(no separate index/delivery-volume sub-updater). update() walks
BSEWorkingDays() from the day after MAX(sdate) (or MIN_DATE on an
empty table) up to the cutoff trading time; update_a_day(mdate) is the
single-date entry point used by both BseProcessor.run_update_for_date
and bse_backfill.py's per-day loop.
Deliberately does not append to Parquet itself — see the Overview's note on append ordering.
Related
- Overview —
scr_bse_masteris not unique onmname - bse_store.py —
symbol_path - bse_infra.py — the schema this writes into