Skip to main content

mcx_downloader.py — McxDownloader & McxProcessor

The core of the package. All REST API calls and the backfill script ultimately land on McxProcessor.

Class: McxDownloader​

Handles network requests to MCX India using TLS impersonation.

get_xsrf_token() → str | None​

Fetches the bhavcopy page (https://www.mcxindia.com/market-data/bhavcopy) to retrieve the anti-CSRF token embedded in the page HTML, along with session cookies. Falls back to "en" on failure.

fetch_bhavcopy_raw(target_date: date) → list[dict] | None​

GET GetDateWiseBhavCopy?InstrumentName=ALL&fromDate=DD/MM/YYYY with the XSRF token as __RequestVerificationToken. Returns the raw Data array from the JSON response — one dict per instrument row across every commodity — or None on HTTP error / empty response (holiday or future date).

Class: McxProcessor​

Orchestrates bhavcopy download, master resolution, EOD storage, and AMI export. Mirrors nseeod.downloadbhav.BhavProcessor in shape and naming.

Constructor​

proc = McxProcessor()
  • Loads existing mcx_stats.json from Config.MCX_CSV_PATH (if present)
  • _tables_ready = False — DDL runs once per process, not once per date (a 6,300-date backfill would otherwise re-run CREATE TABLE / CREATE TRIGGER per date). The DDL itself lives in mcx_infra.ensure_infrastructure() — _ensure_tables_exist() is a thin wrapper adding this cache flag, not a second copy of the schema.

Orchestration methods​

run_update()​

Auto sync — fills every missing date up to the latest MCX trading date.

start defaults to mcx_date + 1 day, or latest - 30 days when the table is empty (a bounded first fill rather than reaching back to 2003 — full history is mcx_backfill's job).

run_update_for_date(target_date: date) → bool​

Force-updates a single date. Captures last_safe_date first, calls download_date(target_date), rolls back on exception.

run_range_download(start_date, end_date)​

Downloads every weekday in the range via download_date(), pacing requests with RANGE_DOWNLOAD_PAUSE (0.6s) between each. Used by the from_date/to_date REST endpoint for "sync recent dates" — it filters weekends with weekday() < 5, which is safe only because it targets recent dates. Do not reuse for historical work — MCX ran 490 Saturday sessions and 5 Sunday sessions historically; use MCXWorkingDays() instead for anything spanning older dates.

Also the loop run_update() calls internally — so the pause protects every routine auto-sync, not just a manual date-range request.

MCX's WAF blocks unpaused request bursts

Observed 2026-08-19: a batch with no pause between dates got one request through, then 403 Access Denied on every request after it — including, minutes later, a previously-working date tested completely separately. The block is IP/session-level, not per-date. fetch_bhavcopy_raw()'s generic exception handling logs a block identically to a genuine holiday ("No MCX data available... (holiday or future date)"), so that message alone is not proof of a real non-trading day — see mcxeod/CLAUDE.md rule 17.

Critical rule: last_safe_date (lastmcxupdateDate()) is captured before any DB writes in all three entry points above, so a mid-run crash can be rolled back cleanly.

download_date(target_date: date) → bool​

Downloads and fully processes one MCX trading date — the unit of work every other method builds on.

Has no rollback baseline of its own — always called through run_update / run_update_for_date / run_range_download / mcx_backfill, never directly from a route.

Routes never call download_date directly

It has no rollback baseline. REST endpoints call run_update / run_update_for_date / run_range_download, which all capture last_safe_date first.

Transform methods​

process_bhavcopy_data(raw_items, target_date) → DataFrame​

Turns the raw MCX JSON rows into the standardised frame every later step consumes. Key steps:

  1. Rename MCX's JSON keys to internal column names (Symbol → symbol, ExpiryDate → expiry_raw, Volume → svolume, etc.)
  2. Parse expiry_raw ("31AUG2026") to a real date, plus an NSE-style DDMMMYY token used to build mname
  3. Build opttyp as a plain object-dtype column of "CE" / "PE" / None — not Series.where(cond, None), which leaves a pandas.NA behind that psycopg2 stringifies to 'nan', overflowing CHAR(2) on insert
  4. Split "225.810 KGS" into qty_k = 225.810 (thousands) and unit = "KGS"
  5. Derive lotsize = (qty_k * 1000) / svolume per row, only for traded rows — MCX revises lot sizes between contract cycles, so a hardcoded table would go stale silently
  6. Look up family / pricequote from mcx_specs.py
  7. Derive avgprice (see formula in Overview)
  8. Classify typ via _classify() and build mname via _build_mname()
  9. Drop rows whose expiry didn't parse — they can't be keyed reliably

_classify(row) → int (static)​

FUTCOM / OPTFUT / FUTIDX / OPTIDX → typ, falling back to the option flag when InstrumentName doesn't disambiguate. Uses mcx_specs.is_index() to route index symbols to 23/24 instead of 21/22.

_build_mname(row) → str​

NSE-style contract name: SYMBOL + DDMMMYY for futures (GOLDM28AUG26), plus + STRIKE + CE/PE for options (ZINC24JAN25285CE).

build_continuous(df) → DataFrame​

For every symbol, sorts its futures by expiry and takes the nearest three as SYMBOL-I / -II / -III with monthindex 1/2/3. These carry edate=None — the row is permanent, only the contract behind it rolls over time. Returns an empty frame if there are no futures rows.

Master resolution​

resolve_masids(con, df) → DataFrame​

Ensures every symbol, dated contract, and continuous series has a scr_mcx_master row, then attaches the resolved masid back onto df.

Lookups use pgs.read_sql(..., params={...}) with named :param binds (the query goes through SQLAlchemy text()) — psycopg2-style %(name)s placeholders raise a syntax error here. Raw cursor.execute() calls elsewhere in this file (_write_eod, deleteadateabove) still use %s; the two styles coexist deliberately.

_insert(con, frame, table)​

Wraps pgs.insertdataframe and raises RuntimeError if it returns False. insertdataframe swallows its own exception, prints Insert Error:, rolls back, and returns False — an unchecked call would report success ("Added 15,862 new MCX contracts") while writing nothing. Always call self._insert(), never pgs.insertdataframe directly.

_prepare_for_insert(frame) → DataFrame (static)​

Pins every non-numeric column to object dtype holding real None before insert. pandas 3.x gives text columns the new str dtype whose missing value is pd.NA; psycopg2 has no adapter for it and stringifies it instead.

EOD write​

_write_eod(con, df, target_date) → int​

Replace-then-derive pattern:

coi is derived because MCX's bhavcopy has no CHG_IN_OI equivalent — NSE's does. The UPDATE joins each row to the max sdate strictly before target_date, so coi is NULL for the very first row of a masid.

AMI export​

generate_mcx_ami_format(target_date) → bool​

Writes Rates/MCX/YYYYMMDD.txt for AmiBroker. Continuous futures only — WHERE m.monthindex > 0 AND m.typ IN (21, 23). Excludes:

ExcludedRows on 2026-08-04Why
Options (typ 22/24)15,620~95% of a day, not chartable series
Dated futures (monthindex = 0)152expiry-stamped tickers die monthly — AmiBroker would fill up with thousands of short dead symbols
Kept: continuous90one rolling chart per commodity

Everything stays queryable in scr_mcxfo_eod; this filter only governs what AmiBroker receives.

Status / stats​

get_database_status() → dict​

{
    "latest_trading_date": date,   # weekday-based estimate, see below
    "mcx_date": date | None,       # MAX(sdate) from scr_mcxfo_eod
    "is_up_to_date": bool,
}

get_latest_trading_date() → date​

Most recent MCX weekday whose bhavcopy should already be published, then walks back over weekends.

No same-day case — unlike NSE

MCX trades until ~23:45, but the bhavcopy for that session does not appear until roughly 08:00 the following morning (confirmed 2026-08-19) — not same-night. nseeod publishes same evening after 17:00; MCX does not follow that pattern, so get_latest_trading_date() never returns today, only yesterday or earlier:

now = datetime.datetime.now()
candidate = now.date() - datetime.timedelta(days=1)   # start at "yesterday"
if now.hour < 8:
    candidate -= datetime.timedelta(days=1)             # not published yet

This isn't cosmetic — get_database_status() / run_update() depend on it. Requesting a date whose bhavcopy genuinely doesn't exist yet looks identical to a holiday or a WAF block once it reaches fetch_bhavcopy_raw()'s generic "no data" handling (see the warning above). An overly-optimistic cutover here silently manufactures that exact false negative.

Weekday-only, unlike the calendar-driven backfill

get_latest_trading_date() and run_range_download() both use a plain weekday() < 5 check — fine for "what's the most recent business day right now", wrong for historical ranges. Use MCXWorkingDays() for anything that needs to account for MCX's Saturday/Sunday sessions.

add_log(message, source="MCX")​

Appends to stats["Logs"], keeps the last 100, prints [McxProcessor][{source}] {message}. ASCII-only — Windows cp1252 print() crashes on non-ASCII characters (backend rule 2).

add_history(date_label, status, message)​

Inserts at the front of stats["history"], keeps the last 50. Status values: SUCCESS, FAILED, SKIPPED, UP_TO_DATE.

Standalone Utility Functions​

lastmcxupdateDate() → date | None​

MAX(sdate) from scr_mcxfo_eod — the rollback baseline captured before any write in run_update / run_update_for_date / run_range_download.

deleteadate(mdate)​

Deletes every scr_mcxfo_eod row for exactly mdate.

deleteadateabove(mdate)​

Deletes every scr_mcxfo_eod row with sdate > mdate — crash-recovery rollback, called by McxProcessor._rollback().

mcx_stats.json Structure​

Written to {MCX_CSV_PATH}/mcx_stats.json after every state change. Shape mirrors nseeod's stats.json (status, progress, Logs, history).

Frontend polling needs the hasSeenRunning guard

mcx_stats.json still holds the previous run's terminal status when a new sync starts, so a terminal status is only trustworthy once RUNNING has actually been observed (backend rule 6). Call the frontend's startMcxPolling(false) on a fresh trigger — passing true defeats the guard; true means "the server already confirmed a sync is in flight" (mount / tab-switch only).