MCXEOD Package — Overview
backend/mcxeod/ is a custom-built Python package that handles the full
lifecycle of MCX (Multi Commodity Exchange) end-of-day data: downloading
daily bhavcopies from MCX India via a TLS-impersonating session, resolving
instruments against a dedicated MCX security master, building continuous
front-month series, and persisting everything to PostgreSQL.
It deliberately mirrors nseeod's architecture — same orchestrator
pattern, same stats.json progress file, same rollback-baseline discipline —
but is a fully separate package with its own tables. MCX instruments are
never written to scr_master; that table is NSE-only.
Package File Map
| File | Role |
|---|---|
mcx_infra.py | Infrastructure bootstrap — idempotent DDL for scr_mcx_master / scr_mcxfo_eod, their indexes, and the modi_date trigger. Single source of truth; mcx_downloader.py delegates here rather than duplicating the DDL |
mcx_downloader.py | Orchestrator — McxDownloader (network) + McxProcessor (DB/business logic) |
mcx_backfill.py | Resumable historical download (2003-11-18 → today) |
mcx_history.py | Trading calendar + holiday builder, MCXWorkingDays() |
mcx_specs.py | Static contract facts only: family and pricequote lookup tables |
app/api/mcx_routes.py | REST endpoints (/api/mcx-downloader/*) |
CLAUDE.md | Package-specific rules — read alongside backend/nseeod/CLAUDE.md, most of which still applies |
High-Level Architecture
Data Flow — Daily Sync
Download Modes
| Mode | Method | Trigger |
|---|---|---|
| Auto sync (all missing dates) | McxProcessor.run_update() | POST /api/mcx-downloader/download (no params) |
| Single date | McxProcessor.run_update_for_date(date) | POST /api/mcx-downloader/download?date=YYYY-MM-DD |
| Date range | McxProcessor.run_range_download(start, end) | POST /api/mcx-downloader/download?from_date=...&to_date=... |
| Historical backfill | mcx_backfill.run_backfill() | CLI only — python -m mcxeod.mcx_backfill |
Routes must call run_update / run_update_for_date / run_range_download —
never McxProcessor.download_date() directly, which has no rollback
baseline captured.
Contract typ Codes
Deliberately distinct numbering from the NSE codes used in scr_master:
typ | Meaning | Bhavcopy InstrumentName |
|---|---|---|
| 20 | Tradable symbol / underlying (GOLD, GOLDM, ALUMINI, MCXBULLDEX) | — |
| 21 | Commodity future | FUTCOM |
| 22 | Option on commodity future | OPTFUT |
| 23 | Index future | FUTIDX |
| 24 | Index option | OPTIDX |
Continuous front-month series are ordinary scr_mcx_master rows —
SYMBOL-I / -II / -III carrying monthindex 1/2/3 and a NULL edate,
exactly how nseeod stores NIFTY-I in scr_master. There is no separate
"continuous" table (an earlier scr_mcx_continuous design was removed —
every consumer had to UNION two tables for no benefit).
GOLD-I typ=21 monthindex=1 edate=NULL underly=<GOLD masid>
GOLD-II typ=21 monthindex=2 edate=NULL
GOLD05AUG26 typ=21 monthindex=0 edate=2026-08-05
Each tradable symbol (GOLD, GOLDM, ALUMINI) gets its own typ=20 row
and is its own underlying — mirroring NSE, where a contract's underly
points at its own spot instrument, never at a shared "family" row. Family
grouping (GOLD covering GOLD/GOLDM/GOLDPETAL) lives only in the
analytics-only family column (see mcx_specs.py).
Key Database Tables
Postgres-only — no Parquet. MCX EOD is ~15k rows/day, squarely Postgres
territory (Parquet exists in nseieod because 1-minute bars are millions of
rows/day, a different problem entirely). Both tables are created
idempotently by McxProcessor._ensure_tables_exist() on first write.
scr_mcx_master — MCX Security Master
| Column | Type | Description |
|---|---|---|
masid | SERIAL PK | Auto-increment ID |
mname | VARCHAR(100) UNIQUE | Contract name, NSE-style: GOLDM28AUG26, ZINC24JAN25285CE, GOLD-I |
typ | SMALLINT | 20=symbol, 21=FUTCOM, 22=OPTFUT, 23=FUTIDX, 24=OPTIDX |
underly | INTEGER | FK → masid of the tradable symbol (typ=20) row |
edate | DATE | Expiry date (NULL for symbols and continuous rows) |
strike | REAL | Strike price (options only) |
opttyp | CHAR(2) | CE / PE (options only) |
monthindex | SMALLINT | 0=dated contract, 1/2/3=continuous -I/-II/-III |
family | VARCHAR(20) | Analytics grouping (e.g. GOLD for GOLD/GOLDM/GOLDPETAL) |
unit | VARCHAR(10) | KGS / GRMS / BBL / mmBtu / MWH / MT / BALES / UNIT — read off the bhavcopy, not hardcoded |
pricequote | REAL | Units the quoted price refers to (default 1) — see mcx_specs.py |
crea_date / modi_date | TIMESTAMP | modi_date maintained by a Postgres trigger, never by application code |
scr_mcxfo_eod — MCX EOD Prices
| Column | Type | Description |
|---|---|---|
masid | INTEGER | FK → scr_mcx_master |
sdate | DATE | Trading date |
sopen / shigh / slow / sclose | REAL | OHLC |
svolume | REAL | Contracts traded, in lots |
svaluel | REAL | Turnover value, in lakhs |
oi | REAL | Open interest |
coi | REAL | Change in OI — derived, MCX does not publish it (see below) |
avgprice | REAL | Average traded price, per quoted unit |
lotsize | REAL | Contract lot size — read off the bhavcopy per day, not stored in the master |
f5min / f15min / l5min / l15min | REAL | Reserved for the future mcxieod intraday module |
PRIMARY KEY (masid, sdate). A day's rows are replaced with DELETE +
INSERT (McxProcessor._write_eod), never upserted row-by-row.
coi is derived, not published. After inserting a day's rows, _write_eod
runs an UPDATE ... FROM against the prior session's OI for the same masid.
It is therefore NULL for the very first date in the table, and meaningless
across a futures roll for continuous rows (the underlying contract changed).
avgprice is per quoted unit, not per lot:
avgprice = (svaluel * 100000 * pricequote) / (svolume * lotsize)
svolume is in lots, so dividing value by it directly yields value-per-lot,
not a price — an earlier version did exactly that and produced a 100x-inflated
figure that looked plausible enough to survive review.
Calendar tables — shared with nseeod, discriminated by exch
| Table | MCX rows | Notes |
|---|---|---|
holidays | exch = 2 | Weekdays MCX did not trade. Gained morning_open / evening_open (nullable) — MCX holidays are per-session; most close only the morning session while the evening still trades. NULL for pre-current-year history (an EOD turnover feed can't say which session traded). Only both-sessions-closed is a true non-trading day. |
specialdays | exch = 2 | Weekends MCX did trade — 490 Saturday sessions (2004–2025) + 5 Sunday/Muhurat sessions. |
NSE rows use exch = 1 in the same two tables; deletes and holiday
rebuilds are always exch-scoped so an MCX refresh can never touch NSE data
and vice versa.
Running Standalone
# Always run as a module from backend/ directory
cd backend
python -m mcxeod.mcx_infra # bootstrap/verify schema (run FIRST on a fresh DB)
python -m mcxeod.mcx_downloader # daily sync (run_update)
python -m mcxeod.mcx_history # rebuild trading calendar + holidays
python -m mcxeod.mcx_backfill # resume -> today
python -m mcxeod.mcx_backfill --to 2015-12-31 # resume -> that date
python -m mcxeod.mcx_backfill --from 2004-01-01 --to 2004-12-31 # refill a gap
curl_cffi is a hard dependencyMcxDownloader and McxHistoryClient use requests.Session(impersonate="chrome131")
to get past MCX's TLS fingerprinting — plain requests is blocked. It is
pinned in backend/requirements.txt, in the unmarked core section so the Linux
VPS installs it too — never under a sys_platform marker.