Historical Margin Pipeline
Package backend/margin/ (full run instructions in its README.md). Margins land in margin_daily inside
{NSEFO_DATA_PATH}/_catalog/margin.db and {MCXFO_DATA_PATH}/_catalog/margin.db.
What each source is (and is not)
- Arihant PDFs (NSE) - per-underlying, futures-style margin. No per-strike or portfolio SPAN. The old "MTM" column is the exposure margin; from 2024 it also contains the additional/ELM add-on in the same number.
- NSE SPAN files + marginism (NSE) -
https://nsearchives.nseindia.com/archives/nsccl/span/nsccl.YYYYMMDD.s.zipis public and was reachable back to 2017 (probed 2026-10-20). Computed long/short per lot for configured underlyings. - Fyers (MCX) -
POST /api/v3/multiorder/margin, one lot long and short. Needs the daily Fyers login. - Estimated (MCX, optional) - volatility-based approximation, labelled
source='estimated'.
Findings from the first local run
- Arihant URLs change by era and the site answers HTTP 200 with an HTML page for missing days, so a PDF is accepted only
if it starts with
%PDFand the printed date equals the requested date. - marginism SPAN% agrees with Arihant to within 5% of total margin on 65 of 70 overlapping rows; the 5 differences are
MIDCPNIFTY in May 2023 where Arihant shows 0% exposure (set it in
exposure_rate_history).
SPAN cache (fast exact margin for TBS runs)
The daily NSE SPAN file is a ~60 MB XML and takes 3-4 s to read, so the exact margin of a 6 month TBS run (one file per trading day) used to take minutes. Every contract in the file already carries its 16 scenario risk array, so the margin of a set of legs is plain arithmetic. margin/span_cache.py therefore keeps the parsed index commodities (NIFTY, BANKNIFTY, FINNIFTY, MIDCPNIFTY, NIFTYNXT50) of each day as a gzip pickle (<NSEFO_DATA_PATH>/_raw_margin/spancache/<yyyy>/<yyyymmdd>.pkl.gz, about 1 MB, loads in about 50 ms). The figures are identical to parsing the file (same objects). The raw zip stays the source of truth; an unreadable cache file is rebuilt from it.
- The nightly margin job writes the cache while it parses the day (
nse_span.compute_day). - A TBS margin run builds the caches of days it lacks in parallel first (
span_cache.warm), so the first run over an old period pays the parse once and every later run takes seconds. - Backfill the archive once:
cd backend && python -m margin.span_cache build --from 2023-05-01 --workers 4(statusshows how many days are cached).
Known gaps
No Arihant file found in probes before 2023 or between mid-2025 and Oct 2026 (patterns may differ - gaps lists them);
MCX Fyers data starts the day the job first runs with a valid token; lot sizes before the Arihant/F&O-store history need
lots --import-lots.