sme_backfill.py — NSE SME (Emerge) EOD History
backend/nseeod/sme_backfill.py adds NSE Emerge (SME, series SM / ST) EOD rows that the main CM
bhav copy doesn't carry. The source is NSE's separate SME bhav copy.
Findings (2026-09-27)
SME looked like it started in 2025, but the data was mostly there. Two reasons:
-
The Dashboard never listed SME symbols. The SPOT symbol list in
chart_service.get_stocksfiltered ontyp IN (1, 2). It now includes 10 (SME) and 11 (REIT / InvIT). -
SME companies that grew into the mainboard are typ 1. A read-only scan of the CM bhav backups since 2012 found 815 symbols that ever traded as SM / ST:
Kind typ 10 typ 1 Always SME 652 0 SME, now mainboard (same symbol) 4 159 Tagging by current segment is correct: their SME-era rows are stored under the same symbol. The 4 migrated symbols still at typ 10 are left alone (
nseeod/CLAUDE.mdrule 2: never reclassify silently).
The prices were also nearly complete. The CM bhav copy carries SME series from 2012-11-02, and where both files cover a day they match (checked 15-07-2019: 71 of 71). NSE Emerge began in September 2012, so only Sep–Oct 2012 was missing: 29 rows of one symbol, THEJO (the first Emerge listing, 18-09-2012, later mainboard).
Source
nsearchives.nseindia.com/archives/sme/bhavcopy/smeDDMMYY.csv:
- columns
MARKET, SERIES, SYMBOL, SECURITY, PREV_CL_PR, OPEN_PRICE, HIGH_PRICE, LOW_PRICE, CLOSE_PRICE, NET_TRDVAL, NET_TRDQTY, …(2012 files start withTrade Date); - available from Sep 2012 to late 2025. It is not published for days without SME trades, and NSE stopped the separate file in late 2025, when the CM bhav copy already had full SME coverage.
Backup
D:\kss\stock\EODData\NSE_SME\{year}\smeDDMMYY.csv (3,159 files), plus manifest.csv and
not_published.txt (308 days).
Usage
Run from backend/:
python -m nseeod.sme_backfill download --dir D:/kss/stock/EODData/NSE_SME [--from 2012-09-01 --to ...]
python -m nseeod.sme_backfill check --dir D:/kss/stock/EODData/NSE_SME # SME rows in files vs in DB, per year
python -m nseeod.sme_backfill apply --dir D:/kss/stock/EODData/NSE_SME [--apply]
apply only inserts (symbol, date) rows the DB doesn't have. It never changes an existing row.
- It uses the same pieces as the CM ingest: rows with no trades are dropped, then
one_row_per_symbolandrepair_ohlcare applied. - Masters go through
match_scr_masters_and_update(globalmname); new symbols get typ 10. - At the end it refreshes the Parquet history for the touched symbols only.
Local run (2026-09-27)
| Years | SME rows in files | Missing in DB |
|---|---|---|
| 2012 | 74 | 29 (THEJO, Sep–Oct) → added |
| 2013–2025 | ~285,000 | 0 |
eq_store verify matches on the touched dates.
Split / bonus for SME come from NSE's SME corporate-action feed (from 2013, see corp_actions.py):
- 19 of 19 SME splits and 133 of 152 SME bonuses match the price gap (the rest are too small to check or have no prices);
- 1 bonus (OSIAHYPER 2022-06-20) and 5 detected SME jumps are on the review page.