Skip to main content

index_members.py — Index Constituent Tracker

backend/nseeod/index_members.py answers "which stocks were in this index on that date?", including stocks that later left, went bust or were delisted. It implements the plan in Tasks → Index Constituent Tracker; see also Survivorship Bias.

Design: hybrid (decided 2026-09-27)​

The original plan used only NSE's "Constituents of index" snapshot PDFs. They have gaps, so they are combined with NSE Indices' change announcements:

SourceGivesCoverage
Monthly "Indices – Market Capitalisation & Weightage" zips (niftyindices.com → Reports → Monthly)Month-end snapshot: symbol, name, industry, close, index mcap, weightage %Jun 2013 →. NIFTY 50 / Bank / Next 50 (from Oct 2015) to today; NIFTY 100 / 200 / 500 / Midcap 150 / Smallcap 250 only to Mar 2022 (later zips carry just 5–7 indices)
Index-change press releases (niftyindices.com/media, ind_prsDDMMYYYY.pdf, 1,505 since 1998)Exact effective date of every inclusion / exclusionSymbols listed from about 2004–2011; older releases list company names only
Today's constituent lists (nsearchives…/ind_<index>list.csv)The starting pointToday

Build: start from today's lists and walk backwards through the changes. An inclusion means the stock was not a member before; an exclusion means it was.

  • Symbols are renamed back across NSE symbol changes (scr_corp_action).
  • At every monthly snapshot, the walk is compared with NSE's list and re-anchored on it, so an error can never spread past one month.
  • Demerger placeholders (DUMMYREL, DUMMYVEDL1 …) are dropped; they are not tradable.

Snapshots are read with pdfplumber (table columns; with pypdf the mcap and weightage run together). Press releases are read with pypdf, as text.

Tables (PostgreSQL)​

The schema is from the tasks page. The first four tables and the view are as planned; idx_change, idx_membership and idx_members_on() were added for the hybrid. DDL is owned by index_members.ensure_tables().

TableContent
idx_indexCanonical names (NIFTY 50 …) and size
idx_aliasCNX / S&P CNX → NIFTY names (rebrand ended 2015-11)
idx_constituentMonthly snapshots with weightage, mcap, industry and close: 227,185 rows, 4,021 index-months, every index in the zips (not only the 8 tracked)
idx_symbol_aliasNSE symbol changes (from scr_corp_action SYMBOL_CHANGE)
idx_changeParsed announcements: effective date, IN/OUT, symbol, company, release file (11,699 rows for the tracked indices)
idx_membershipExact intervals [from_date, to_date) per index and symbol (+ masid), with how each end is known: change announcement, snapshot anchor, symbol change, current list, coverage start
view v_index_constituentSnapshots, plus today's symbol for each (as planned)
function idx_members_on(index, date)Members on any date
SELECT * FROM idx_members_on('NIFTY 50', '2014-06-30'); -- 50 rows, incl. CAIRN, SSLT, IDFC ...
SELECT symbol, weightage_pct FROM v_index_constituent
WHERE index_name = 'NIFTY 500' AND as_of_date = '2018-01-31'; -- weights where a snapshot exists

Usage​

Run from backend/:

python -m nseeod.index_members download --dir D:/kss/stock/EODData/NSE_INDEX_CHANGES # releases + today's lists
python -m nseeod.index_members download-weightage --dir ... # 159 monthly zips, Jun 2013 ->
python -m nseeod.index_members parse --dir ... # -> snapshots.parquet, changes.parquet (~20 min)
python -m nseeod.index_members build --dir ... [--apply] # intervals + checks + idx_* tables
python -m nseeod.index_members members --index "NIFTY 50" --date 2015-06-30

build is a full reload of the derived tables and writes build_checks.csv: per snapshot, the walked list vs NSE's list.

Backup folder D:\kss\stock\EODData\NSE_INDEX_CHANGES\:

  • prs_list.csv, plus pdf/{year}/ with 955 equity releases and extracted .txt;
  • weightage/{year}/indices_data*.zip (159, 350 MB);
  • current/{date}/;
  • the two caches and unparsed_releases.csv.

Results (local, 2026-09-27) and known limits​

  • Snapshot checks: 809 of 857 exact (94%). The other 48 are small:

    • rename timing (PHILIPCARB → PCBL, BURGERKING → RBA);
    • stocks moved between Smallcap 250 and 500 in the same month;
    • one Aug–Sep 2021 rebalance whose PDF niftyindices no longer serves (6 releases are unavailable);
    • early CNX-era months.

    Re-anchoring limits each case to "switch day within that month approximate".

  • Member counts (checked every 6 months, 2013–2026) are exact for NIFTY 50 / 100 / 200 / 500 / Bank. 2016–2019 shows 51 / 101 / 201 / 501, which is real: Tata Motors and Tata Motors DVR were both constituents.

  • Weak spots:

    • NIFTY 200 / 500 / Midcap 150 / Smallcap 250 in Apr 2022 – 2024 are off by 1–3 names (no snapshots to re-anchor on);
    • NIFTY Next 50 before Oct 2015 (46–48 members).
  • Survivorship check: YESBANK is in NIFTY 50 from 2015-03-27 to 2020-03-27, and CAIRN, IDEA and ZEEL have their real exit dates.

Stage B: before the snapshots (1999 → 2013, done 2026-09-27)​

Before June 2013 there are no snapshots, and the announcements mostly list company names without symbols. Stage B (python -m nseeod.index_members parse-b --dir …, then build):

  1. Reads every pre-2013-06-28 release with pdfplumber, in page order.

    • The old nseindia.com menu printed into pre-2006 releases ("----- 1 Jet Airways (India) Ltd.") is stripped.
    • Doubled-character 2011 PDFs ("IInnddiiaabbuullllss") are repaired.
    • The index is read from headings and from "excluded from S&P CNX Nifty" lines.
  2. Matches names to symbols from every name source:

    • NSE corporate-action records (company + symbol since 1995);
    • the name-change, symbol-change and delisted files;
    • EQUITY_L.csv;
    • the snapshots and the 2011+ releases that carry both.

    A candidate must have traded within 15 days of the effective date (bhav history). Matching compares the company part only (up to "Ltd."), ignores filler words and plurals, and needs at least 60% word overlap. Ties go to the symbol trading on the exact day, then to the closest full name.

  3. Resolves ambiguity with membership: walking back past an inclusion, the stock must be a member, so the walk picks the strong name candidate that is. Example: "Bajaj Auto Ltd." in 2010 is BAJAJ-AUTO, not BAJAJHLDNG (which carried the Bajaj Auto name before the 2008 demerger).

  4. Manual map name_map_manual.csv (company,symbol) overrides everything. It fixes names the sources only know by their later name: P&G, ZEETELE, INDRAYON, VSNL, TATATEA, BHARTI. Review list: name_review.csv has unmatched, fuzzy and tie-broken rows (282). Add fixes to the manual map, then rerun parse-b and build.

Stage B events are used before 2013-06-28, Stage A (symbol releases) after.

Accuracy by index: idx_index.history_from and exact_from. exact_from is the earliest date after which every checked member count is within ±1 of the index size.

IndexHistory fromExact (±1) fromBefore 2013
NIFTY 501999-05-261999-09-08exact every year
NIFTY 1002006-06-262007-12-17within ±2
NIFTY NEXT 501999-02-032014-05-05within ±4
NIFTY 2002011-11-222024-04-16within ±2 (the ±3 drift in 2022–24 comes from missing snapshots)
NIFTY 5001999-02-032020-06-30±15 for 2003–2012; about −30 for 1999–2002 (weak)
NIFTY BANK2006-03-02— (no fixed size)10 change rows
NIFTY Midcap 150 / Smallcap 2502013-06-282025launched 2016; no earlier history

Example: idx_members_on('NIFTY 50', '2008-01-31') gives 50 symbols as they traded then, including BAJAJAUTO, HEROHONDA, INFOSYSTCH, REL, RPL, STER, SATYAMCOMP, VSNL, CAIRN, SUZLON and UNITECH.

To improve NIFTY 500 / Next 50: work through name_review.csv (mostly 1999–2003 NIFTY 500 small caps delisted long ago). The per-date drift in build_checks.csv shows which announcement dates to look at first.

Stock Universe page​

Sidebar → Stock Universe (/stock-universe, DEVELOPER, added 2026-10-03). A read-only verification page for the universes the app filters on.

  • Universe (one or more of SME, TE-FO Index and every index with membership rows) and As on (default today).
  • One row per stock, with a "since" column per chosen universe: membership start, plus the end date for a stock that later left.
  • First / last traded date from the EOD store. Last traded shows red when it is more than 45 days before the as-on date.
  • Any / All: stocks in any of the chosen universes, or in every one of them. Symbol search and CSV export.
  • Count per universe, with a warning when the as-on date is before that index's history_from.
  • Changes (IN / OUT) panel: one universe's events between two dates (idx_change; SME = first / last EOD row).
UniverseSource
NSE indices, TE-FO INDEXidx_members_on(name, date), survivorship-free
SMEscr_master.typ = 10 stocks trading on the date: first EOD row ≤ date, last row within 45 days (none before Sep 2012)

Shared logic: backend/nseeod/universe.py, also used by the Corporate Actions universe filter. API (app/api/stock_universe.py): GET /api/stock-universe/universes, /members?universe=A,B&as_of=, /changes?universe=A&from_date=&to_date=.

What it showed on first use (2026-10-03): NIFTY 500 has 510 members on 2005-01-01, the known pre-2020 drift described above. TE-FO Index had 52 members that day, NIFTY 50 has 50 today, and 573 SME stocks are trading today.