mcx_backfill.py — Historical Backfill

Downloads bhavcopies for every known MCX trading day from MCX_START (2003-11-18) onward. Calendar-driven, not weekday-driven — MCX ran 490 Saturday sessions and 5 Sunday sessions between 2004 and 2026, which a Mon–Fri walk would silently discard.

cd backend
python -m mcxeod.mcx_backfill                                      # resume -> today
python -m mcxeod.mcx_backfill --to 2015-12-31                      # resume -> that date
python -m mcxeod.mcx_backfill --from 2004-01-01 --to 2004-12-31    # refill a gap
python -m mcxeod.mcx_backfill --limit 25 --pause 0.8                # trial run

---

Resumability

--from defaults to the day after the newest date in scr_mcxfo_eod, so a plain re-run just carries on. Passing --from explicitly overrides this — that's how you go back and refill an old gap.

pending_dates(start, end) → list[date]

Within the resolved range, the work list is calendar trading days minus dates already present (MCXWorkingDays(start, end) from mcx_history.py, set-differenced against SELECT DISTINCT sdate FROM scr_mcxfo_eod). This means a date that failed mid-run leaves a gap that the next plain run will pick back up — the resume point being MAX(sdate) does not itself skip failures, but if a later date succeeded, the run summary's failed_dates is how you target it explicitly:

python -m mcxeod.mcx_backfill --from 2020-03-17 --to 2020-03-17

---

run_backfill(start, end, pause, limit, max_retries) → dict

Returns:

{"attempted": int, "ok": int, "empty": int, "failed": int, "failed_dates": [date, ...]}

• ok — date downloaded and written successfully
• empty — MCX responded but returned nothing, even though the calendar says this should be a trading day. Logged as mcx_backfill_calendar_disagreement rather than silently treated as a holiday — a request failure is never recorded as a holiday; only mcx_history.py decides what a holiday is.
• failed — all max_retries attempts raised an exception. The date is left unrecorded so the next run retries it automatically (via pending_dates's set-difference).

Retry backoff is exponential: wait = pause * (2 ** attempt).

---

CLI Arguments

Flag	Default	Meaning
--to YYYY-MM-DD	today	Fill up to this date
--from YYYY-MM-DD	day after MAX(sdate), or MCX_START if empty	Override the resume point
--pause	0.6	Seconds between requests
--limit	none	Stop after N dates (trial runs)

At the default pause, a ~6,300-date full backfill takes roughly 2 hours.

---

Related

• mcx_history.py — supplies MCXWorkingDays() and MCX_START
• mcx_downloader.py — McxProcessor.download_date() does the actual per-date work