MCX IEOD (1-minute data)
IEOD is the project's name for intraday (1-minute bar) data. This page covers the MCX commodity-derivatives
store: where the data comes from, how it is stored, how to load it, and how it is checked. The code lives in
backend/mcxieod/; the daily (EOD) MCX pipeline is the separate backend/mcxeod/ package.
One-minute MCX options data exists only from July 2025 (July 2025 to September 2026). There is no option history before that. The 2020-2025 yearly archives contain a few stray option files (about 1,100 files in four months: 2021-05, 2021-11, 2023-01 and 2024-08); they are not continuous history and are deliberately not imported.
What is available
| Instrument | 1-minute data | Source | Status |
|---|---|---|---|
| Futures, month 1 / 2 / 3 (rolling series) | 2020-01 to 2025-04 | yearly zips 2020.zip .. 2025.zip | Loaded on the VPS (25.9M bars), quality check passed |
| Futures, 2025-05 to 2025-12 | May to December 2025 | 2025Missing/<MON>2025.zip (daily RARs, supplied by the vendor on 2026-10-16) | Loaded and checked on the local PC (172 of 172 trading days); VPS run pending |
| Futures, 2026-01 to 2026-09 | 2026 | 2026/MCX_IEOD_Data-YYYYMM.rar | Loaded on the VPS (5.9M bars incl. 2025-07-07), quality check passed |
| Options, 2025-07 to 2025-12 | July 2025 onward | 2025Options/MCXOption<MON>2025.zip (daily files) | Loaded on the VPS, quality check passed |
| Options, 2026-01 to 2026-09 | 2026 | 2026/MCX_IEOD_Option-YYYYMM.rar | Loaded on the VPS, quality check passed |
| Index futures (MCXBULLDEX, MCXMETLDEX, MCXENRGDEX) | where present in the master | archives | Imported if the master has the row |
| Spot index files (MCXAGRI, MCXMETAL ...) | present in archives | - | Skipped (not futures) |
Source data
Everything sits under csvs/EODIEODMCX/ (MCXIEOD_SOURCE_PATH). The backfill discovers the files by name, so
new archives dropped into the folder are picked up without code changes.
| Path | Holds | Notes |
|---|---|---|
2020.zip .. 2025.zip | One RAR per month, futures only (2020-01 to 2025-04) | RAR names carry junk (MCX IEOD Data - hy55_6# - 202004.rar); the month is the trailing 6 digits |
2026/MCX_IEOD_Data-YYYYMM.rar | Futures, one RAR per month, 2026-01 to 2026-09 | |
2026/MCX_IEOD_Data-202606.zip | June 2026 futures again, as daily RARs | Merged with the monthly RAR, duplicates dropped |
2026/MCX_IEOD_Option-YYYYMM.rar | Options, one RAR per month, 2026-01 to 2026-09 | 2,000 to 3,900 contracts per month |
2025Missing/<MON>2025.zip | Futures, daily RARs (MCX_Future_IEOD - dd-mm-2025.rar), 2025-05 to 2025-12 | Fills the gap in the yearly archives. JUNE2025.zip was first delivered with June 2026 data inside (see below) |
2025Options/MCXOption<MON>2025.zip | Options, daily RARs (MCX_Option_IEOD - dd-mm-2025.rar), 2025-07 to 2025-12 | Also one stray futures day, 2025-07-07 |
None of the RARs has a password.
Futures files
A futures file is a rolling series, not a contract: GOLD-1 is the near month, GOLD-2 the next, GOLD-3
the far month. The series rolls to the next contract at expiry.
<ticker>,<date>,<time>,<open>,<high>,<low>,<close>,<volume>,<o/i>
GOLD-1,01/01/2020,09:00:00,39108.00,39113.00,39061.00,39061.00,22,15138
Option files
An option file is one contract. The name is the symbol, the expiry as YYMMDD, the strike and CE or PE:
COPPER2609231200PE.csv -> COPPER, expiry 2026-09-23, strike 1200, put
COPPER250822860PE.csv -> COPPER, expiry 2025-08-22, strike 860, put
The contract is matched to its scr_mcx_master row (typ 22 commodity option, 24 index option) by underlying,
expiry date, strike and CE/PE, never by name. The master spells these names differently
(COPPER24AUG261230CE), and matching by value avoids any formatting mismatch. A contract that has no master row is
counted and skipped; the backfill never creates master rows.
- Date is
mm/dd/YYYY, time is IST and is the bar open (MCX opens at 09:00). - Some 2023 files have a trailing comma on every row.
- The vendor omits minutes with no trade. Thin series and far-from-the-money options are sparse by nature.
What is skipped and why
| Skipped | Reason |
|---|---|
| Stray option files in the 2020-2025 yearly zips | Not continuous history (see the warning above) |
Spot index files (MCXAGRI.csv) | Not futures |
Roman-suffix files (ALUMINI-I.csv) | Not clean copies of the numeric files (ALUMINIUM-I 2020 differs from ALUMINIUM-1); numeric files win |
Futures months 4 and 5 (CARDAMOM-4, GOLD-4) | The master only carries -I, -II, -III (--max-month changes this) |
Futures symbols not in scr_mcx_master | Currency pairs, 718GS2033 and similar. Reported at the end of the run, never auto-registered |
Option contracts not in scr_mcx_master | Counted per underlying at the end of the run |
Mislabelled vendor files
The first JUNE2025.zip held 22 daily files named ...-06-2026 whose bars were dated June 2026, a copy of 2026 data and not June 2025. It was replaced. The backfill now checks every series: if most of its bars fall outside the archive month, the series is not written, the unit is not marked done, and the run reports it as a failure. The other 2025Missing zips were verified file by file (every file's bar dates match its name). A zip whose name has a typo (JUNE025.zip) still loads, because the month then comes from the dates in the member names.
Duplicate rows in the source
Some vendor files repeat whole blocks. In April 2024 ALUMINI-1 holds every bar twice (about 379k duplicate
rows that month; 896k over the whole 2020-2025 futures load). Exact repeats are dropped and counted. A repeat of
the same minute with different values is counted as a conflict and the last one wins. The VPS futures load had no
conflicts.
Parquet store
Root: Config.MCXIEOD_DATA_PATH (LOCAL_MCXIEOD_DATA_PATH / VPS_MCXIEOD_DATA_PATH in backend/.env; the
folder is ignored by git). The layout and schema match the BSE 1-minute store.
{root}/G/GOLD/2020/10_JANUARY futures: <root_masid>_<MONTHNAME>, no file extension
{root}/options/G/GOLD/2026/10_SEPTEMBER options: same name, under options/
{root}/_catalog/mcx_ieod_backfill.json finished archive units (resume state)
{root}/_catalog/tmp/ temporary extraction folder (emptied after each unit)
| Column | Meaning |
|---|---|
masid | Futures: scr_mcx_master continuous row, SYMBOL-1 -> SYMBOL-I, -2 -> -II, -3 -> -III (typ 21 commodity, 23 index). Options: the contract's own master row (typ 22 / 24) |
sdate | Bar open (IST) converted to UTC, plus 59 seconds (the :59 convention used by the NSE and BSE stores) |
sopen, shigh, slow, sclose | Prices as quoted by the exchange (GOLD per 10 g; option prices are premiums) |
svolume, oi | Volume and open interest from the vendor |
filled | 0 for all imported bars (reserved for a future gap fill) |
The file name uses the root masid: the typ-20 symbol (GOLD = 10). All three futures series of a symbol share
one file. All of a symbol's option contracts for a month share one options file, separated by masid. MCX
instruments are never written to the NSE scr_master.
Refilling missing days from Fyers
Ten trading days (2021-11-19, 2022-07-07, 2022-07-15, 2022-10-25, 2023-09-28, 2023-11-27, 2024-05-01, 2024-05-20,
2024-06-17, 2024-07-11) have no bars in the vendor files. backend/mcxieod/fyers_fill.py refills whole days from the
Fyers expired-contract API (/history/fno/expired/historical-data, which does serve expired MCX futures with OI):
- The dated futures that traded that day come from the EOD store and are ranked per symbol by expiry (nearest =
-I, next =-II, third =-III). - Each contract is fetched as
MCX:<SYMBOL><YY><MON>FUT(for exampleMCX:CRUDEOIL22NOVFUT). - Every contract's minute high and low are compared with its EOD row (0.5% tolerance). Only contracts that pass are
written (
--apply). Volume ratio is reported for information. - Fyers returns every minute of the session; bars with zero volume are dropped so the series matches the vendor's traded-minutes-only data.
cd backend # needs a valid FYERS_ACCESS_TOKEN (expires daily): python -m scratch.fyers_download_indiavix --login
python -m mcxieod.fyers_fill plan # contracts that would be fetched, no API calls
python -m mcxieod.fyers_fill run # fetch + cross-check, writes nothing
python -m mcxieod.fyers_fill run --apply # also write the contracts that pass
Run on 2026-10-16 on the VPS: 696 contracts, 489 returned data, all 489 matched EOD high and low (100%), volume ratio median 1.00; 146,647 bars written. The remaining 207 contracts returned no data: mostly third-month contracts and commodities Fyers does not archive (KAPAS, NICKEL, RUBBER, MCXMETLDEX, STEELREBAR). Month-1 of the ten liquid symbols was covered 99 of 100 times (ALUMINIUM on 2023-11-27 returned nothing), month-2 100 of 100. Five of the ten days (2021-11-19, 2023-11-27, 2024-05-01, 2024-05-20, 2024-06-17) turned out to be evening-only sessions (bars from 17:00 only), which explains why the vendor skipped them. After the refill the futures store has data on all 1,739 MCX trading days from 2020-01 to 2026-09.
Running the backfill
Run everything from backend/, never from inside the package folder. On the VPS use the virtualenv's Python
(venv/bin/python); there is no bare python.
cd backend
python -m mcxieod.backfill plan # list archive units, done / pending, gaps. No writes
python -m mcxieod.backfill run --kind opt --from 2026-09 --to 2026-09 --symbols ZINC --root <scratch dir> # sample
python -m mcxieod.backfill run # everything pending: futures and options
python -m mcxieod.backfill run --kind fut # futures only (or --kind opt)
python -m mcxieod.backfill verify # read the store back
| Option | Meaning |
|---|---|
--kind all/fut/opt | Futures, options or both (default both) |
--src DIR | Folder holding the archives. Default MCXIEOD_SOURCE_PATH from .env (csvs/EODIEODMCX), or env MCX_EODIEOD_SOURCE |
--root DIR | Parquet root. Default MCXIEOD_DATA_PATH. Use a scratch folder for samples |
--from, --to | Month range YYYY-MM (default 2020-01 to 2026-12) |
--symbols | Comma list, for a sample run. A run with --symbols never marks a unit done |
--max-month | Highest futures SYMBOL-N to import (default 3) |
--force | Redo units already marked done |
--unrar PATH | RAR extractor (UnRAR, 7z or bsdtar). Also env MCX_UNRAR |
--tmp DIR | Extraction folder. Default {root}/_catalog/tmp, on the data disk and not the system /tmp |
The unit of work is one archive month per kind (state keys YYYYMM for futures, OYYYYMM for options). It is
extracted to the temp folder, parsed, written and deleted. Finished units are skipped on a re-run, and
rewriting a unit is idempotent (merge on masid + sdate). The run prints one line per unit and a summary at
the end, never one line per file. Options are buffered per symbol and written once per symbol-month (flushed
every 3 million rows to bound memory).
plan also reports months with no archive at all and months built from daily files. The 2025 option months and
June 2026 futures are daily files, so a missing day inside those months shows up as a gap in the quality check
(calendar section), not as an error in the backfill.
On the VPS
The 2025-07 to 2026-09 load (89 units) finished on 2026-10-16 in 1,210 seconds (20 minutes), faster than the 35 minutes estimated from archive size (about 3 seconds per MB for the daily-file months, less for the monthly RARs).
./pull.sh, then make sure an extractor is installed:apt install unrar(7zip or bsdtar also work).- Copy the new archives into
/home/kss/tradeentry/csvs/EODIEODMCX: the2026/folder and the2025Options/folder (VPS_MCXIEOD_SOURCE_PATH). Keep the file names unchanged. venv/bin/python -m mcxieod.backfill plan: the futures 2020-01 to 2025-04 units show as done; the 2025-07 to 2026-09 futures and options units show as pending.nohup venv/bin/python -m mcxieod.backfill run > ~/mcxieod_backfill.log 2>&1 &, then watch the log.venv/bin/python -m mcxieod.quality check.
The VPS has 16 GB of memory and plenty of disk. Futures 2020-2025 took 221 seconds; options 2025-07 to 2026-09 took most of the 20 minutes.
On Windows the extractor is found automatically at C:\Program Files\WinRAR\UnRAR.exe.
Quality check
cd backend
python -m mcxieod.quality check [--kind all/fut/opt] [--symbols GOLD,CRUDEOIL] [--from 2020-01] [--to 2026-09] [--root DIR]
NSE bars have a fixed count per day (375, or 335 before 2010-01-04). MCX does not: sessions run from 09:00 to 23:30 or 23:55 IST depending on US daylight saving, agricultural contracts end at 17:00, and there are weekend and Muhurat sessions. Completeness is therefore judged against each series-day's own first-to-last-bar window, and against the MCX trading calendar.
| Check | Applies to | Type | What it reports |
|---|---|---|---|
Duplicate (masid, sdate) | both | HARD | Must be 0 |
| Bars before 09:00 IST | both | HARD | Must be 0 |
Seconds other than :59 | both | HARD | Must be 0 |
| High below low, open/close outside the high-low range | both | HARD | Must be 0 |
| Price at or below zero, negative volume or OI | both | HARD | Must be 0 |
| Bars after the contract's expiry date | options | HARD | Must be 0; a hit means the contract was mapped to the wrong master row |
| Per-underlying summary | options | Report | Bars and contracts per symbol, days with data |
| Liquid month-1 series coverage | futures | Report | Median coverage, days under 80%, longest in-day gap, late open (first bar after 09:30), early close (last bar before 23:25) |
| Calendar | futures | Report | MCX trading days with no data, data on non-trading days (loaded months only) |
| EOD cross-check | both | HARD / info | Minute high and low against mcx_fo_store within 0.5%, volume ratio. Runs only where the EOD store has the day |
Coverage is bars / minutes between the first and last bar of the day. The gap statistics are meaningful only
for the liquid month-1 series: GOLD, GOLDM, SILVER, SILVERM, CRUDEOIL, NATURALGAS, COPPER, ZINC, LEAD and
ALUMINIUM.
Results so far
Futures 2020-01 to 2025-04, VPS (25.9M bars, 93 series): all integrity checks passed. The EOD cross-check on 68,329 series-days matched minute highs and lows within 0.5% for 99.9%, and minute volume summed to the EOD volume at a median ratio of 1.00. CRUDEOIL, NATURALGAS, SILVERM, GOLDM and SILVER month-1 series had 96-100% median coverage; ZINC, LEAD and ALUMINIUM are sparse (27-74%) because the vendor omits no-trade minutes. Ten trading days had no data in any liquid series (see below).
Futures 2025-07 and 2026-01 to 2026-09, VPS (5.88M bars, 77 series): all integrity checks passed. The EOD cross-check on 12,896 series-days matched highs and lows for 99.9%, volume ratio 1.00. The 2026 calendar check is perfect: 194 trading days, 0 without data. The 18 days listed as missing are 2025-07-08 to 2025-07-31, because only 2025-07-07 exists for that month (see the futures gap below).
Options 2025-07 to 2026-09, VPS (68.5M bars, 22,475 contracts, 323 trading days; loaded 65.5M bars from 34,265 contract files): CRUDEOIL 14.5M bars, CRUDEOILM 12.2M, GOLDM 9.9M, NATURALGAS 9.1M, SILVERM 8.5M, NATGASMINI 6.2M. No duplicates, no bars after expiry, no malformed prices. The EOD cross-check on 281,079 contract-days matched highs 99.9% and lows 99.8%, volume ratio 1.01. One integrity check failed: a single bar at 08:47 IST on 2025-12-31 (one lot in a SILVER option, before the 09:00 open). It is a vendor print and was left in place.
2026-09 sample, local PC (futures GOLD, COPPER, ZINC and options for the same symbols): futures 74,611 bars and options 416,671 bars over 236 contracts. All integrity checks passed, no option bar falls after its expiry date. The EOD cross-check matched on 192 futures series-days (highs and lows 100%) and on 2,161 option contract-days (highs and lows 100%, volume ratio 1.01). This confirms the option name-to-master mapping.
The 2025 option months could not be tested locally because the local master has no 2025 contracts. The VPS master does (about 200,000 option contracts from 2017, including 98 ZINC contracts expiring in July and August 2025).
Known limits and next steps
- Futures for 2025-05 to 2025-12 were missing from the original archives and were supplied later in
2025Missing/. After loading them,planno longer reportsNO ARCHIVEfor any month from 2020-01 to 2026-09. - Reloading changed units. Each finished unit stores a fingerprint of its source files. If files are added to a
month later (for example 2025-07 gained the
2025Missingdaily files), the nextrunredoes that month automatically. Units recorded before fingerprints existed are treated as done; redo those with--forceand a--from/--torange. - Some trading days have no bars in the vendor files. Ten days from 2021-11-19 to 2024-07-11 are marked as
trading days by the MCX calendar but have no data in any liquid series. For 2024-05-01 and 2024-05-20 the cause is
evening-only sessions (morning closed); the other eight are probably the same (not individually confirmed). The
quality check lists them under
no-data days. - 118 option contracts are not in the VPS master and were skipped, all ZINC strikes (for example
ZINC260122297PE; 75 of them in 2026, the rest in 2025). That is 0.3% of the files. The master is built from the daily bhavcopy; the cause for these strikes is not confirmed. They can be added if the master gains the rows, by re-running with--forcefor the affected months. - Daily-file months (the six 2025 option months, June 2026 futures) are only as complete as the daily files supplied. Check the quality check's calendar section for missing days after loading.
- No gap fill (the NSE
filldata()equivalent). Most MCX gaps are minutes with no trade, so filling is a per-series decision. - Run the full backfill and the quality check, including the EOD cross-check, on the VPS.
Related
- Cost of Carry reads MCX futures from the EOD store.
- Developer rules for the package:
backend/mcxieod/CLAUDE.md. EOD pipeline rules:backend/mcxeod/CLAUDE.md.