mcx_spot_downloader.py — MCX Spot Prices
backend/mcxeod/mcx_spot_downloader.py downloads MCX spot prices (the cash-market
reference price of each commodity) and stores one OHLC row per symbol per trading day in
scr_mcxsp_eod. It is the spot counterpart of the F&O bhavcopy pipeline in
mcx_downloader.py.
Who uses the data: Range Analysis prices MCX from spot, because NeoWave needs spot data (see Range Analysis — Overview).
Table scr_mcxsp_eod
Created by mcx_infra.ensure_infrastructure(), and therefore by the spot downloader
itself (see below).
| Column | Type | Meaning |
|---|---|---|
masid | INTEGER | The symbol's typ = 20 row in scr_mcx_master |
sdate | DATE | Trading day |
sopen / shigh / slow / sclose | REAL | Day OHLC built from all of that day's spot ticks, across every location |
- Primary key:
(masid, sdate). - No other columns: there is no volume, OI or location. The location is deliberately dropped (see the module docstring).
How it runs
| Trigger | What happens |
|---|---|
MCX Bhav Downloader (UI, POST /api/mcx-downloader/download) | Every mode — auto sync (run_update), single date (run_update_for_date) and date range (run_range_download) — runs McxProcessor._sync_spot() after the F&O download and before the final SUCCESS/FAILED status. It is logged in the same Logs/History panel with the tag MCX-SPOT. |
| CLI | python -m mcxeod.mcx_spot_downloader (options below) |
Per symbol, for every typ = 20 row in scr_mcx_master:
- Resume point. Start from the day after that symbol's own latest
sdate, or from 2005-01-01 if it has no rows yet. - Fetch. One archive request (
GET /GetSpotMarketArchive,Location=ALL) usually returns the whole range. - Save the raw response as
MCX_CSV_PATH/SpotArchive_<SYMBOL>_<from>_<to>.csv, as an audit trail. - Aggregate the ticks into day OHLC, and reject any day that isn't an MCX trading day according to
MCXWorkingDays()(holidays and special days). - Write. Past dates use
ON CONFLICT DO NOTHING, so settled history is never overwritten. Today's date usesON CONFLICT DO UPDATE. - Roll back on failure, per symbol only: rows newer than that symbol's pre-run date are deleted, and the run carries on with the next symbol.
Pacing: 0.6 s between symbols, because MCX's WAF blocks unpaused bursts (mcxeod/CLAUDE.md rule 17).
Failures never affect the F&O sync. A spot failure is logged as a separate
(spot) FAILED history entry.
CLI
cd backend
python -m mcxeod.mcx_spot_downloader # resume -> today, every symbol
python -m mcxeod.mcx_spot_downloader --symbol GOLD # one symbol
python -m mcxeod.mcx_spot_downloader --from 2005-01-01 --to 2005-12-31 # refill a range
python -m mcxeod.mcx_spot_downloader --pause 1.0 # slower pacing
The run prints one line per symbol, then a summary:
GOLD ticks=43 days=15 written=15 invalid=0
Done. 32/32 symbol(s) processed successfully, 405 day-rows written, 0 invalid rows skipped.
written=0 with ticks=0 is normal for symbols that MCX publishes no spot archive for,
such as CASTORSEED, RUBBER and the MCXBULLDEX / MCXMETLDEX indices.
The September 2026 fix
Two separate problems (mcxeod/CLAUDE.md rule 19).
1. MCX stopped answering "bare" archive requests
MCX began answering a plain GET /GetSpotMarketArchive with an HTML "404" page at HTTP 200.
The code tried to read it as JSON, failed, logged a warning, and wrote nothing. Every machine
stopped at 2026-09-04.
The request parameters had not changed. The fix makes the request look like the spot page's own jQuery call:
| Before | After | |
|---|---|---|
| Page visit first | no | GET /market-data/spot-market-price once per session |
Referer / Origin | none | the spot page / https://www.mcxindia.com |
Accept | default | application/json, text/javascript, */*; q=0.01 |
| Browser fingerprint | chrome110 | chrome131 (same as McxDownloader) |
A non-JSON reply is now logged as "archive returned non-JSON (blocked or endpoint changed)" rather than being treated as "no data".
If this breaks again, fetch /assets/customjs/SpotMarketPrice.js from mcxindia.com and
compare its GetSpotArchive() call with McxSpotDownloader.fetch_archive().
2. The VPS never had the table
McxSpotProcessor.run() read from scr_mcxsp_eod without creating it. Only the F&O write
path called ensure_infrastructure(). On the VPS, every spot sync therefore failed with
relation "scr_mcxsp_eod" does not exist, and that failure was swallowed as a spot-only history
entry.
run() now calls ensure_infrastructure(con) first, so the table is created where it's missing.
On an empty table, the first run is the full backfill: every symbol starts from 2005-01-01.
3. The date-range option never synced spot
Until 2026-09-26, run_range_download() (the UI's date-range option) did not call
_sync_spot(). Only auto sync and single date did, and auto sync did it after setting
SUCCESS. The UI stops polling on a terminal status, so the MCX-SPOT lines could be missed.
Spot now runs inside run_range_download() before the terminal status, and auto sync reuses
that path.
Reading the log: "Spot sync complete: 4/32 symbol(s)" counts only the symbols that needed fetching; current symbols are skipped. The symbols MCX has no spot archive for are re-checked on every run (one request each).
One-time backfill on a new server
cd /home/kss/tradeentry/backend
venv/bin/python -m mcxeod.mcx_spot_downloader
This was done on the VPS on 2026-09-26: 173,520 rows for 54 symbols, 2005-01-29 to 2026-09-25. From then on, the Bhav Downloader keeps the table current.
Troubleshooting
| Symptom | Check |
|---|---|
History shows (spot) FAILED | The backend log has the exception (mcx_spot_sync_failed). |
Every symbol shows ticks=0 | Look for "archive returned non-JSON" in the log, which means MCX changed or blocked the call. Compare with the page's JS (above). |
One symbol always ticks=0 | That symbol probably has no spot archive at MCX, which is expected for some agri symbols and indices. |
| Rows stop at a date | Run the CLI for that symbol and read its output line. |