Skip to main content

Historical Margin Pipeline

Package backend/margin/ (full run instructions in its README.md). Margins land in margin_daily inside {NSEFO_DATA_PATH}/_catalog/margin.db and {MCXFO_DATA_PATH}/_catalog/margin.db.

What each source is (and is not)​

  • Arihant PDFs (NSE) - per-underlying, futures-style margin. No per-strike or portfolio SPAN. The old "MTM" column is the exposure margin; from 2024 it also contains the additional/ELM add-on in the same number.
  • NSE SPAN files + marginism (NSE) - https://nsearchives.nseindia.com/archives/nsccl/span/nsccl.YYYYMMDD.s.zip is public and was reachable back to 2017 (probed 2026-10-20). Computed long/short per lot for configured underlyings.
  • Fyers (MCX) - POST /api/v3/multiorder/margin, one lot long and short. Needs the daily Fyers login.
  • Estimated (MCX, optional) - volatility-based approximation, labelled source='estimated'.

Findings from the first local run​

  • Arihant URLs change by era and the site answers HTTP 200 with an HTML page for missing days, so a PDF is accepted only if it starts with %PDF and the printed date equals the requested date.
  • marginism SPAN% agrees with Arihant to within 5% of total margin on 65 of 70 overlapping rows; the 5 differences are MIDCPNIFTY in May 2023 where Arihant shows 0% exposure (set it in exposure_rate_history).

SPAN cache (fast exact margin for TBS runs)​

The daily NSE SPAN file is a ~60 MB XML and takes 3-4 s to read, so the exact margin of a 6 month TBS run (one file per trading day) used to take minutes. Every contract in the file already carries its 16 scenario risk array, so the margin of a set of legs is plain arithmetic. margin/span_cache.py therefore keeps the parsed index commodities (NIFTY, BANKNIFTY, FINNIFTY, MIDCPNIFTY, NIFTYNXT50) of each day as a gzip pickle (<NSEFO_DATA_PATH>/_raw_margin/spancache/<yyyy>/<yyyymmdd>.pkl.gz, about 1 MB, loads in about 50 ms). The figures are identical to parsing the file (same objects). The raw zip stays the source of truth; an unreadable cache file is rebuilt from it.

  • The nightly margin job writes the cache while it parses the day (nse_span.compute_day).
  • A TBS margin run builds the caches of days it lacks in parallel first (span_cache.warm), so the first run over an old period pays the parse once and every later run takes seconds.
  • Backfill the archive once: cd backend && python -m margin.span_cache build --from 2023-05-01 --workers 4 (status shows how many days are cached).

Known gaps​

No Arihant file found in probes before 2023 or between mid-2025 and Oct 2026 (patterns may differ - gaps lists them); MCX Fyers data starts the day the job first runs with a valid token; lot sizes before the Arihant/F&O-store history need lots --import-lots.