Skip to main content

mcx_spot_downloader.py — MCX Spot Prices

backend/mcxeod/mcx_spot_downloader.py downloads MCX spot prices (the cash-market reference price of each commodity) and stores one OHLC row per symbol per trading day in scr_mcxsp_eod. It is the spot counterpart of the F&O bhavcopy pipeline in mcx_downloader.py.

Who uses the data: Range Analysis prices MCX from spot, because NeoWave needs spot data (see Range Analysis — Overview).

Table scr_mcxsp_eod​

Created by mcx_infra.ensure_infrastructure(), and therefore by the spot downloader itself (see below).

ColumnTypeMeaning
masidINTEGERThe symbol's typ = 20 row in scr_mcx_master
sdateDATETrading day
sopen / shigh / slow / scloseREALDay OHLC built from all of that day's spot ticks, across every location
  • Primary key: (masid, sdate).
  • No other columns: there is no volume, OI or location. The location is deliberately dropped (see the module docstring).

How it runs​

TriggerWhat happens
MCX Bhav Downloader (UI, POST /api/mcx-downloader/download)Every mode — auto sync (run_update), single date (run_update_for_date) and date range (run_range_download) — runs McxProcessor._sync_spot() after the F&O download and before the final SUCCESS/FAILED status. It is logged in the same Logs/History panel with the tag MCX-SPOT.
CLIpython -m mcxeod.mcx_spot_downloader (options below)

Per symbol, for every typ = 20 row in scr_mcx_master:

  1. Resume point. Start from the day after that symbol's own latest sdate, or from 2005-01-01 if it has no rows yet.
  2. Fetch. One archive request (GET /GetSpotMarketArchive, Location=ALL) usually returns the whole range.
  3. Save the raw response as MCX_CSV_PATH/SpotArchive_<SYMBOL>_<from>_<to>.csv, as an audit trail.
  4. Aggregate the ticks into day OHLC, and reject any day that isn't an MCX trading day according to MCXWorkingDays() (holidays and special days).
  5. Write. Past dates use ON CONFLICT DO NOTHING, so settled history is never overwritten. Today's date uses ON CONFLICT DO UPDATE.
  6. Roll back on failure, per symbol only: rows newer than that symbol's pre-run date are deleted, and the run carries on with the next symbol.

Pacing: 0.6 s between symbols, because MCX's WAF blocks unpaused bursts (mcxeod/CLAUDE.md rule 17).

Failures never affect the F&O sync. A spot failure is logged as a separate (spot) FAILED history entry.

CLI​

cd backend
python -m mcxeod.mcx_spot_downloader # resume -> today, every symbol
python -m mcxeod.mcx_spot_downloader --symbol GOLD # one symbol
python -m mcxeod.mcx_spot_downloader --from 2005-01-01 --to 2005-12-31 # refill a range
python -m mcxeod.mcx_spot_downloader --pause 1.0 # slower pacing

The run prints one line per symbol, then a summary:

GOLD ticks=43 days=15 written=15 invalid=0
Done. 32/32 symbol(s) processed successfully, 405 day-rows written, 0 invalid rows skipped.

written=0 with ticks=0 is normal for symbols that MCX publishes no spot archive for, such as CASTORSEED, RUBBER and the MCXBULLDEX / MCXMETLDEX indices.

The September 2026 fix​

Spot sync was silently broken from about 2026-09-05 to 2026-09-26

Two separate problems (mcxeod/CLAUDE.md rule 19).

1. MCX stopped answering "bare" archive requests​

MCX began answering a plain GET /GetSpotMarketArchive with an HTML "404" page at HTTP 200. The code tried to read it as JSON, failed, logged a warning, and wrote nothing. Every machine stopped at 2026-09-04.

The request parameters had not changed. The fix makes the request look like the spot page's own jQuery call:

BeforeAfter
Page visit firstnoGET /market-data/spot-market-price once per session
Referer / Originnonethe spot page / https://www.mcxindia.com
Acceptdefaultapplication/json, text/javascript, */*; q=0.01
Browser fingerprintchrome110chrome131 (same as McxDownloader)

A non-JSON reply is now logged as "archive returned non-JSON (blocked or endpoint changed)" rather than being treated as "no data".

If this breaks again, fetch /assets/customjs/SpotMarketPrice.js from mcxindia.com and compare its GetSpotArchive() call with McxSpotDownloader.fetch_archive().

2. The VPS never had the table​

McxSpotProcessor.run() read from scr_mcxsp_eod without creating it. Only the F&O write path called ensure_infrastructure(). On the VPS, every spot sync therefore failed with relation "scr_mcxsp_eod" does not exist, and that failure was swallowed as a spot-only history entry.

run() now calls ensure_infrastructure(con) first, so the table is created where it's missing. On an empty table, the first run is the full backfill: every symbol starts from 2005-01-01.

3. The date-range option never synced spot​

Until 2026-09-26, run_range_download() (the UI's date-range option) did not call _sync_spot(). Only auto sync and single date did, and auto sync did it after setting SUCCESS. The UI stops polling on a terminal status, so the MCX-SPOT lines could be missed. Spot now runs inside run_range_download() before the terminal status, and auto sync reuses that path.

Reading the log: "Spot sync complete: 4/32 symbol(s)" counts only the symbols that needed fetching; current symbols are skipped. The symbols MCX has no spot archive for are re-checked on every run (one request each).

One-time backfill on a new server​

cd /home/kss/tradeentry/backend
venv/bin/python -m mcxeod.mcx_spot_downloader

This was done on the VPS on 2026-09-26: 173,520 rows for 54 symbols, 2005-01-29 to 2026-09-25. From then on, the Bhav Downloader keeps the table current.

Troubleshooting​

SymptomCheck
History shows (spot) FAILEDThe backend log has the exception (mcx_spot_sync_failed).
Every symbol shows ticks=0Look for "archive returned non-JSON" in the log, which means MCX changed or blocked the call. Compare with the page's JS (above).
One symbol always ticks=0That symbol probably has no spot archive at MCX, which is expected for some agri symbols and indices.
Rows stop at a dateRun the CLI for that symbol and read its output line.