Skip to main content

sme_backfill.py — NSE SME (Emerge) EOD History

backend/nseeod/sme_backfill.py adds NSE Emerge (SME, series SM / ST) EOD rows that the main CM bhav copy doesn't carry. The source is NSE's separate SME bhav copy.

Findings (2026-09-27)​

SME looked like it started in 2025, but the data was mostly there. Two reasons:

  1. The Dashboard never listed SME symbols. The SPOT symbol list in chart_service.get_stocks filtered on typ IN (1, 2). It now includes 10 (SME) and 11 (REIT / InvIT).

  2. SME companies that grew into the mainboard are typ 1. A read-only scan of the CM bhav backups since 2012 found 815 symbols that ever traded as SM / ST:

    Kindtyp 10typ 1
    Always SME6520
    SME, now mainboard (same symbol)4159

    Tagging by current segment is correct: their SME-era rows are stored under the same symbol. The 4 migrated symbols still at typ 10 are left alone (nseeod/CLAUDE.md rule 2: never reclassify silently).

The prices were also nearly complete. The CM bhav copy carries SME series from 2012-11-02, and where both files cover a day they match (checked 15-07-2019: 71 of 71). NSE Emerge began in September 2012, so only Sep–Oct 2012 was missing: 29 rows of one symbol, THEJO (the first Emerge listing, 18-09-2012, later mainboard).

Source​

nsearchives.nseindia.com/archives/sme/bhavcopy/smeDDMMYY.csv:

  • columns MARKET, SERIES, SYMBOL, SECURITY, PREV_CL_PR, OPEN_PRICE, HIGH_PRICE, LOW_PRICE, CLOSE_PRICE, NET_TRDVAL, NET_TRDQTY, … (2012 files start with Trade Date);
  • available from Sep 2012 to late 2025. It is not published for days without SME trades, and NSE stopped the separate file in late 2025, when the CM bhav copy already had full SME coverage.

Backup​

D:\kss\stock\EODData\NSE_SME\{year}\smeDDMMYY.csv (3,159 files), plus manifest.csv and not_published.txt (308 days).

Usage​

Run from backend/:

python -m nseeod.sme_backfill download --dir D:/kss/stock/EODData/NSE_SME [--from 2012-09-01 --to ...]
python -m nseeod.sme_backfill check --dir D:/kss/stock/EODData/NSE_SME # SME rows in files vs in DB, per year
python -m nseeod.sme_backfill apply --dir D:/kss/stock/EODData/NSE_SME [--apply]

apply only inserts (symbol, date) rows the DB doesn't have. It never changes an existing row.

  • It uses the same pieces as the CM ingest: rows with no trades are dropped, then one_row_per_symbol and repair_ohlc are applied.
  • Masters go through match_scr_masters_and_update (global mname); new symbols get typ 10.
  • At the end it refreshes the Parquet history for the touched symbols only.

Local run (2026-09-27)​

YearsSME rows in filesMissing in DB
20127429 (THEJO, Sep–Oct) → added
2013–2025~285,0000

eq_store verify matches on the touched dates.

Split / bonus for SME come from NSE's SME corporate-action feed (from 2013, see corp_actions.py):

  • 19 of 19 SME splits and 133 of 152 SME bonuses match the price gap (the rest are too small to check or have no prices);
  • 1 bonus (OSIAHYPER 2022-06-20) and 5 detected SME jumps are on the review page.