index_members.py — Index Constituent Tracker
backend/nseeod/index_members.py answers "which stocks were in this index on that date?",
including stocks that later left, went bust or were delisted. It implements the plan in
Tasks → Index Constituent Tracker; see also
Survivorship Bias.
Design: hybrid (decided 2026-09-27)
The original plan used only NSE's "Constituents of index" snapshot PDFs. They have gaps, so they are combined with NSE Indices' change announcements:
| Source | Gives | Coverage |
|---|---|---|
| Monthly "Indices – Market Capitalisation & Weightage" zips (niftyindices.com → Reports → Monthly) | Month-end snapshot: symbol, name, industry, close, index mcap, weightage % | Jun 2013 →. NIFTY 50 / Bank / Next 50 (from Oct 2015) to today; NIFTY 100 / 200 / 500 / Midcap 150 / Smallcap 250 only to Mar 2022 (later zips carry just 5–7 indices) |
Index-change press releases (niftyindices.com/media, ind_prsDDMMYYYY.pdf, 1,505 since 1998) | Exact effective date of every inclusion / exclusion | Symbols listed from about 2004–2011; older releases list company names only |
Today's constituent lists (nsearchives…/ind_<index>list.csv) | The starting point | Today |
Build: start from today's lists and walk backwards through the changes. An inclusion means the stock was not a member before; an exclusion means it was.
- Symbols are renamed back across NSE symbol changes (
scr_corp_action). - At every monthly snapshot, the walk is compared with NSE's list and re-anchored on it, so an error can never spread past one month.
- Demerger placeholders (
DUMMYREL,DUMMYVEDL1…) are dropped; they are not tradable.
Snapshots are read with pdfplumber (table columns; with pypdf the mcap and weightage run
together). Press releases are read with pypdf, as text.
Tables (PostgreSQL)
The schema is from the tasks page. The first four tables and the view are as planned; idx_change,
idx_membership and idx_members_on() were added for the hybrid. DDL is owned by
index_members.ensure_tables().
| Table | Content |
|---|---|
idx_index | Canonical names (NIFTY 50 …) and size |
idx_alias | CNX / S&P CNX → NIFTY names (rebrand ended 2015-11) |
idx_constituent | Monthly snapshots with weightage, mcap, industry and close: 227,185 rows, 4,021 index-months, every index in the zips (not only the 8 tracked) |
idx_symbol_alias | NSE symbol changes (from scr_corp_action SYMBOL_CHANGE) |
idx_change | Parsed announcements: effective date, IN/OUT, symbol, company, release file (11,699 rows for the tracked indices) |
idx_membership | Exact intervals [from_date, to_date) per index and symbol (+ masid), with how each end is known: change announcement, snapshot anchor, symbol change, current list, coverage start |
view v_index_constituent | Snapshots, plus today's symbol for each (as planned) |
function idx_members_on(index, date) | Members on any date |
SELECT * FROM idx_members_on('NIFTY 50', '2014-06-30'); -- 50 rows, incl. CAIRN, SSLT, IDFC ...
SELECT symbol, weightage_pct FROM v_index_constituent
WHERE index_name = 'NIFTY 500' AND as_of_date = '2018-01-31'; -- weights where a snapshot exists
Usage
Run from backend/:
python -m nseeod.index_members download --dir D:/kss/stock/EODData/NSE_INDEX_CHANGES # releases + today's lists
python -m nseeod.index_members download-weightage --dir ... # 159 monthly zips, Jun 2013 ->
python -m nseeod.index_members parse --dir ... # -> snapshots.parquet, changes.parquet (~20 min)
python -m nseeod.index_members build --dir ... [--apply] # intervals + checks + idx_* tables
python -m nseeod.index_members members --index "NIFTY 50" --date 2015-06-30
build is a full reload of the derived tables and writes build_checks.csv: per snapshot, the
walked list vs NSE's list.
Backup folder D:\kss\stock\EODData\NSE_INDEX_CHANGES\:
prs_list.csv, pluspdf/{year}/with 955 equity releases and extracted.txt;weightage/{year}/indices_data*.zip(159, 350 MB);current/{date}/;- the two caches and
unparsed_releases.csv.
Results (local, 2026-09-27) and known limits
-
Snapshot checks: 809 of 857 exact (94%). The other 48 are small:
- rename timing (PHILIPCARB → PCBL, BURGERKING → RBA);
- stocks moved between Smallcap 250 and 500 in the same month;
- one Aug–Sep 2021 rebalance whose PDF niftyindices no longer serves (6 releases are unavailable);
- early CNX-era months.
Re-anchoring limits each case to "switch day within that month approximate".
-
Member counts (checked every 6 months, 2013–2026) are exact for NIFTY 50 / 100 / 200 / 500 / Bank. 2016–2019 shows 51 / 101 / 201 / 501, which is real: Tata Motors and Tata Motors DVR were both constituents.
-
Weak spots:
- NIFTY 200 / 500 / Midcap 150 / Smallcap 250 in Apr 2022 – 2024 are off by 1–3 names (no snapshots to re-anchor on);
- NIFTY Next 50 before Oct 2015 (46–48 members).
-
Survivorship check: YESBANK is in NIFTY 50 from 2015-03-27 to 2020-03-27, and CAIRN, IDEA and ZEEL have their real exit dates.
Stage B: before the snapshots (1999 → 2013, done 2026-09-27)
Before June 2013 there are no snapshots, and the announcements mostly list company names without
symbols. Stage B (python -m nseeod.index_members parse-b --dir …, then build):
-
Reads every pre-2013-06-28 release with
pdfplumber, in page order.- The old nseindia.com menu printed into pre-2006 releases ("----- 1 Jet Airways (India) Ltd.") is stripped.
- Doubled-character 2011 PDFs ("IInnddiiaabbuullllss") are repaired.
- The index is read from headings and from "excluded from S&P CNX Nifty" lines.
-
Matches names to symbols from every name source:
- NSE corporate-action records (company + symbol since 1995);
- the name-change, symbol-change and delisted files;
EQUITY_L.csv;- the snapshots and the 2011+ releases that carry both.
A candidate must have traded within 15 days of the effective date (bhav history). Matching compares the company part only (up to "Ltd."), ignores filler words and plurals, and needs at least 60% word overlap. Ties go to the symbol trading on the exact day, then to the closest full name.
-
Resolves ambiguity with membership: walking back past an inclusion, the stock must be a member, so the walk picks the strong name candidate that is. Example: "Bajaj Auto Ltd." in 2010 is BAJAJ-AUTO, not BAJAJHLDNG (which carried the Bajaj Auto name before the 2008 demerger).
-
Manual map
name_map_manual.csv(company,symbol) overrides everything. It fixes names the sources only know by their later name: P&G, ZEETELE, INDRAYON, VSNL, TATATEA, BHARTI. Review list:name_review.csvhas unmatched, fuzzy and tie-broken rows (282). Add fixes to the manual map, then rerunparse-bandbuild.
Stage B events are used before 2013-06-28, Stage A (symbol releases) after.
Accuracy by index: idx_index.history_from and exact_from. exact_from is the earliest date after
which every checked member count is within ±1 of the index size.
| Index | History from | Exact (±1) from | Before 2013 |
|---|---|---|---|
| NIFTY 50 | 1999-05-26 | 1999-09-08 | exact every year |
| NIFTY 100 | 2006-06-26 | 2007-12-17 | within ±2 |
| NIFTY NEXT 50 | 1999-02-03 | 2014-05-05 | within ±4 |
| NIFTY 200 | 2011-11-22 | 2024-04-16 | within ±2 (the ±3 drift in 2022–24 comes from missing snapshots) |
| NIFTY 500 | 1999-02-03 | 2020-06-30 | ±15 for 2003–2012; about −30 for 1999–2002 (weak) |
| NIFTY BANK | 2006-03-02 | — (no fixed size) | 10 change rows |
| NIFTY Midcap 150 / Smallcap 250 | 2013-06-28 | 2025 | launched 2016; no earlier history |
Example: idx_members_on('NIFTY 50', '2008-01-31') gives 50 symbols as they traded then, including
BAJAJAUTO, HEROHONDA, INFOSYSTCH, REL, RPL, STER, SATYAMCOMP, VSNL, CAIRN, SUZLON and UNITECH.
To improve NIFTY 500 / Next 50: work through name_review.csv (mostly 1999–2003 NIFTY 500 small caps
delisted long ago). The per-date drift in build_checks.csv shows which announcement dates to look at first.
Stock Universe page
Sidebar → Stock Universe (/stock-universe, DEVELOPER, added 2026-10-03). A read-only verification page
for the universes the app filters on.
- Universe (one or more of SME, TE-FO Index and every index with membership rows) and As on (default today).
- One row per stock, with a "since" column per chosen universe: membership start, plus the end date for a stock that later left.
- First / last traded date from the EOD store. Last traded shows red when it is more than 45 days before the as-on date.
- Any / All: stocks in any of the chosen universes, or in every one of them. Symbol search and CSV export.
- Count per universe, with a warning when the as-on date is before that index's
history_from. - Changes (IN / OUT) panel: one universe's events between two dates (
idx_change; SME = first / last EOD row).
| Universe | Source |
|---|---|
NSE indices, TE-FO INDEX | idx_members_on(name, date), survivorship-free |
SME | scr_master.typ = 10 stocks trading on the date: first EOD row ≤ date, last row within 45 days (none before Sep 2012) |
Shared logic: backend/nseeod/universe.py, also used by the Corporate Actions universe filter. API
(app/api/stock_universe.py): GET /api/stock-universe/universes, /members?universe=A,B&as_of=,
/changes?universe=A&from_date=&to_date=.
What it showed on first use (2026-10-03): NIFTY 500 has 510 members on 2005-01-01, the known pre-2020 drift described above. TE-FO Index had 52 members that day, NIFTY 50 has 50 today, and 573 SME stocks are trading today.