BSE (SENSEX / BANKEX) 1-minute IEOD - EODIEOD (default) + Fyers
Source policy (decided 2026-10-16): EODIEOD is the DEFAULT source for BSE 1-minute IEOD. EODIEOD (bought vendor data,
csvs/EODIEODBSE, monthlyBSE_Option_MMYY.rar) is the reference; Fyers is the secondary source:
EODIEOD (default) Fyers (secondary) Role Reference history, Sept 2023 onward; the current month is uploaded by the user and loaded with eodieod_bse loadGap filler (contracts / minutes the vendor lacks), days not yet delivered by the vendor, nightly live update ( fyers_ieod daily), cross-checkWhy Real OI on every row incl. 2023, volume closest to the exchange EOD, closing session and Muhurat complete, 62+ contracts Fyers never listed Free; has futures; API available any day; token must be re-issued daily Coverage SENSEX / BANKEX options only (stock options and SENSEX50 files are skipped; no futures) SENSEX / BANKEX options and futures, Aug 2023 onward Current store state: Fyers backfill first, then the vendor loaded in
fillmode (adds only minutes the store lacks, never overwrites). On the overlapping minutes the two sources agree on close 99.6 %, OI 99.7 %, volume 95 %; the differences are minute-boundary trade assignment. Making EODIEOD win on overlapping minutes (eodieod_bse load --mode replace --force, about 20 minutes on the VPS) is the logical next step of this policy and is decided separately - until then Fyers rows win on overlap. Order of precedence for new data: EODIEOD file for the month if delivered -> otherwise Fyers (daily/fetch) -> the vendor load later fills or (after the replace decision) supersedes it.
Current month (daily archives): the vendor delivers the running month as one small RAR per trading day,
BSE_Option_IEOD - DD-MM-YYYY.rar(same per-contract CSV layout as the monthly files), in a Google Drive folder. On the VPS a second read-only rclone remotegdrive_vendor(the EODIEOD Gmail; the backup remotegdriveis untouched) copies them tocsvs/EODIEODBSE/daily/:rclone copy "gdrive_vendor,root_folder_id=<folder id>:" /home/kss/tradeentry/csvs/EODIEODBSE/daily --include "*.rar", thenpython -m bseieod.eodieod_bse load --src /home/kss/tradeentry/csvs/EODIEODBSE/daily(archive labelsDYYYYMMDD; finished days are skipped,--forcereloads). Verified 2026-10-16 on 1-16 Oct: 11 files, 11 MB, 917,074 rows, 257 stock-option files skipped.
Status: Phases A-D done, 2026-10-16. Dry-run results: backend/scratch/fyers_expired_probe_bse.py (modes: default / deep / depth), outputs in backend/tmp/fyers_expired_*.txt.
0. Progress
-
Phase A done (2026-10-14, local PC): package
backend/bseieod/built; Parquet basedata/bseieod/fyers(Config.BSEIEOD_FYERS_PATH, envLOCAL_/VPS_BSEIEOD_DATA_PATH+/fyers); manifestdata/bseieod/fyers/_catalog/catalog.db.- Slice: SENSEX Sept-2025 sample (60 + 60 + 1 future) and BANKEX Sept-2025 (40): 0 errors, ~385k bars, 1 API call per contract.
- Checks passing: no duplicates, all bars inside 09:15-15:29,
:59seconds, OHLC sane; EOD cross-check: high 100 %, low 100 %, volume median ratio 1.00; re-writing a contract leaves the store unchanged (idempotent). - Findings: the EOD
closeinscr_bsefo_eodis a late-session weighted price / settlement on expiry day (last-30-min VWAP matches 57 %, last bar 36 %), and EOD OI is a post-close figure (last-bar OI within 5 % on ~65 %) - both reported as INFO, not failures. Illiquid options have sparse minutes (BANKEX median 327 bars/day) and are stored raw. Far-OTM strikes returnno_data(statusempty). - Commands (from
backend/):python -m bseieod.fyers_ieod discover|fetch|verify|status --und SENSEX BANKEX --from YYYY-MM-DD --to YYYY-MM-DD [--sample N] [--limit N] [--dry](dates are contract expiry dates).
-
Phase B done (2026-10-14, local PC): full Sept-2025 expiry month, both underlyings, 1,583 contracts in ~12 min, 1,583 API calls, 0 errors.
SENSEX BANKEX contracts with data / empty 1,115 / 354 238 / 157 1-min bars 3.25 M 0.84 M median bars per contract-day 374 312 contract-days with full 375 bars 48 % 8 % EOD cross-check high / low 100 % / 100 % 100 % / 100 % EOD cross-check volume (median ratio, within 1 %) 1.00, 65 % 1.01, 53 % All integrity checks pass (no duplicates, nothing outside 09:15-15:29,
:59seconds, OHLC sane). Open points: minute-bar volume equals EOD volume exactly on only part of the contract-days (cause not confirmed; high/low match everywhere, so no missing bars suspected); BANKEX is thin (illiquid strikes, stored raw).statuscannot run while a fetch is running (DuckDB single-writer lock on the manifest). -
Phase D built + tested locally (2026-10-15): nightly
dailycommand for LIVE contracts, volume-filtered:- Flow:
options-chain-v3(one call per expiry, max 50 strikes each side of ATM; SENSEX nearest 5 expiries, BANKEX 3) -> keep strikes withvolume > 0+ the nearest 3 monthly futures -> ordinary/data/historywithoi_flag=1for that session (Fyers returns every minute twice; the store de-duplicates; bars after 15:29 are dropped). - Result for session 2026-10-14: 446 contracts (415 SENSEX, 31 BANKEX), 454 API calls, 3 minutes, 416k bars, all integrity checks OK. Idempotent (rerun skips contracts already pulled for the session). Run after 15:45 IST:
python -m bseieod.fyers_ieod daily [--day YYYY-MM-DD] [--expiries N] [--dry]. Do not run while afetchis running (manifest lock). - Not yet done: EOD cross-check for these days (needs the day's BSE bhav in
scr_bsefo_eod), scheduling on the VPS, automatic Fyers login.
- Flow:
-
Phase C done on the VPS (2026-10-15/16): Fyers backfill of all 79,271 contracts (0 errors, 109.8 M bars), then the EODIEOD vendor refill:
python -m bseieod.eodieod_bse loadover 37 monthly RARs (Sept 2023 - Sept 2026, SENSEX/BANKEX options; 51.6 M rows read, 990,639 new, 50.6 M overlapping and agreeing on close 99.6 %, OI 99.7 %, volume 95 %). Fill mode never overwrites existing rows.- Store after refill: SENSEX 90.36 M rows / 38,167 contracts, BANKEX 20.42 M / 12,210. Integrity clean; EOD cross-check high/low 99.6 % / 99.1 % (SENSEX), 99.7 % / 99.9 % (BANKEX); volume within 1 % of EOD 85 % / 83 % (was 57 % / 68 %).
- Lessons: session window is 09:15-16:05 and special days only widen it (BSE closing session 15:30-15:39 from ~Jul 2026, 16:00 prints on expiry days are in the exchange volume, Muhurat);
specialdayshas a wrong time for 2025-10-21; vendor archives also hold stock options and SENSEX50 files (skipped) and one corrupt CSV; vendor has no futures. - Still open: Fyers re-fetch (
requeue --from 2026-10-01) for the closing-session bars of October 2026, schedulingdailyon the VPS, daily Fyers login.
-
Extrapolation for Phase C: ~168 SENSEX + ~100 BANKEX expiries at ~1 call per contract = roughly 70-80k calls, ~8-9 h at 150 calls/min, within one token day.
-
Next: Phase C (VPS full backfill, newest expiry first).
1. What the dry run proved
| Item | Result |
|---|---|
| API | GET https://api-t1.fyers.in/data/history/fno/expired/{expiry-dates, underlying-symbols, historical-data}, header Authorization: APP_ID:ACCESS_TOKEN |
| Candle shape | [epoch, o, h, l, c, volume, oi] - 7 columns, real OI (Breeze gives OI = 0) |
| SENSEX | expiries listed from 2023-07-28; 1-min futures data from 2023-08-25; ATM options verified 2024-25 |
| BANKEX | futures 1-min from 2023-10-13 (many partial days 2023-24); ATM options verified 2025-26 |
Empty (no_data) | July-2023 contracts, far OTM / not-yet-listed options |
| Limits (old, community posts - superseded by 1b) | expiry-dates: max 366 d/request; minute candles: max 100 d/request; 10 req/s, 200/min, 100,000/day (> 3 minute-cap breaches in a day blocks the account until midnight) |
| Token | expires daily; manual login (fyers_download_indiavix.py --login) |
Fallback if Fyers has holes: ICICI Breeze (BFO / BSESEN / BANKEX, only from ~2024-06, OI = 0) - scratch/breeze_probe_bse.py.
1b. Fyers API limits - official, Standard plan (read 2026-10-16)
Source: FYERS support - API rate limits for Standard and Prime plans. Our account is on the Standard plan. This replaces the numbers in section 1 / 2 below, which came from community posts and from the pre-revision limits (10/s, 200/min, 100,000/day for all API types).
| Standard (ours) | Prime | |
|---|---|---|
| Data APIs (History, Expired F&O, Option Chain, Quotes, Depth, Screeners, Reports) | 5 /s, 50 /min, 5,000 /day | 10 /s, 500 /min, 500,000 /day |
| Transaction / user-info APIs | 10 /s, 200 /min, 100,000 /day | 10 /s, 500 /min, 500,000 /day |
| History coverage | daily: last 5 years; intraday: last 1 year | daily: all (20+ y); intraday: all (5+ y) |
| Expired F&O | daily: 2 years from 2024; intraday: 10 requests/day | 7+ years, Prime limits |
| 1-second data | no | yes |
| Data socket / TBT socket symbols | 100 / 1 | 5,000 / 15 |
Not stated by Fyers: socket connections per plan, the HTTP response / blocking time on a breach, an effective date.
Consequences for this project
- The backfill of 15-16 Oct 2026 (~79,000 expired-F&O calls, full history to Aug 2023, 8-12 h at 150 calls/min) ran without any limit being applied - the revised Standard limits were evidently not enforced on the account then. Under Standard it could not be repeated: expired F&O intraday is 10 requests/day. The data already stored stays valid; only new bulk pulls are affected.
- Nightly
fyers_ieod daily= ~450 ordinary History calls on live contracts + 2 spot calls: fits 5,000/day, but at 50/min it takes ~10 minutes (pace at least 1.3 s per call, not the current 0.40 s). fyers_ieod spot: intraday coverage is only the last 1 year on Standard - the 2017 floor in the code will simply return no bars before ~Oct 2025. Longer spot history needs another source (or Prime).- Option-chain collection for BSE shares the same 5,000/day: 4 indices x 3 expiries x 125 snapshots = ~1,500/day, plus
daily~450 - fine, but leaves little for ad-hoc pulls. bseieod/fyers_client.pyis plan-aware since 2026-10-16 (FYERS_PLANenv, defaultstandard): Standard = at most 4 calls/s, 45/min, 4,500/day (published 5 / 50 / 5,000), the day's count is persisted per IST day in_catalog/fyers_usage.jsonand shared by every run, and the 11th expired-F&O candle request of a day raisesBudgetStop; Prime = 9/s, 450/min, 450,000/day, no expired cap.--budget 0(the default) = what the plan still allows today. Tests:tests/test_fyers_client_limits.py(virtual clock).- This is why the EODIEOD vendor files are the primary 1-minute source and Fyers is only the gap-filler / futures / spot source.
2. Workload (why the VPS does the big run) - historical; Standard limits in 1b make such a run impossible today
- A contract's whole life (listing to expiry) is normally under 100 days, so one
historical-datacall per contract returns every day of its life. - Rough size: SENSEX ~170 expiries (monthly + weekly) x ~300 contracts, BANKEX ~100 expiries x ~200 -> ~70-80k candle calls + ~300
underlying-symbolscalls. - At the self-imposed 150 calls/min (below the 200/min cap): ~8-9 hours. Under the 100k/day cap, so it fits in one token day, and the manifest makes it resumable across days.
- Output: roughly 150 million rows / year for options, stored as monthly Parquet (a few GB per year). Local PC only needs a small slice.
3. Design
New package backend/bseieod/ (does not touch the NSE nseieod code in phase 1; same Parquet schema so readers can be unified later).
| File | Purpose |
|---|---|
fyers_client.py | HTTP client: auth header, shared rate limiter (token bucket 150/min, 8/s), retry/backoff on 429/5xx, hard stop on repeated 429, 401 -> stop with "re-login" message. Reads Config.FYERS_*. |
bse_universe.py | Expiry discovery (yearly chunks of at most 366 days) and contract list per expiry -> parses each contract into (underlying, expiry, strike, CE/PE or FUT) using the expiry we asked for + trailing strike+CE/PE regex (never trusts the weekly-code in the symbol). |
bse_master.py | Resolve/insert masids in scr_bse_master (typ 31 spot, 32 futures, 33 options; names via bseeod.bse_derivatives.build_mnames). Contracts that bhav never had (2023 to mid-2024) are inserted. Report-only --dry mode. |
bse_ieod_store.py | Parquet writer: {BSEIEOD_DATA_PATH}/{S}/{SENSEX}/{year}/{underly}_{MONTH}, columns masid, sdate, sopen, shigh, slow, sclose, svolume, oi, filled; dedupe on (masid, sdate); idempotent merge. |
bse_catalog.py | DuckDB _catalog/catalog.db under BSEIEOD_DATA_PATH with table fyers_contracts(symbol, underlying, expiry, kind, strike, status, bars, first_day, last_day, fetched_at, error) - the resume manifest. |
fyers_ieod.py | CLI (python -m bseieod.fyers_ieod ..., run from backend/): discover, fetch, spot, verify, report. |
bse_quality.py | Quality checks (section 5). |
Transform per bar: epoch = bar open UTC -> IST -> +59 s (matches GDFL / NSE store); drop bars outside 09:15-15:29 (special sessions via specialdays); drop duplicate timestamps; filled = 0.
Rules from CLAUDE.md to follow: ASCII-only log text, aggregate skip/warning lists once at the end (never per day / per file), config only through app.config.Config, run from backend/ with python -m.
Config / env (new)
BSEIEOD_DATA_PATH (env LOCAL_BSEIEOD_DATA_PATH default data/bseieod; VPS_BSEIEOD_DATA_PATH default bseieoddata), added next to the existing MCXIEOD_* block in app/config.py; add a row to the env table in CLAUDE.md. ICICI_* keys stay (Breeze fallback only).
4. Phases
Phase A - Local PC test (small slice, no VPS)
config.pypath + package skeleton + catalog.fyers_ieod discover --underlying SENSEX --from 2025-09-01 --to 2025-09-30-> manifest rows only (about 6 expiries);--dryprints counts, no candles, no writes.fyers_ieod fetch --underlying SENSEX --from 2025-09-01 --to 2025-09-30 --limit 50-> 50 contracts into Parquet; then the same for BANKEX and for 1 futures + 1 weekly expiry.fyers_ieod verifyon that slice (section 5) - must pass before moving on. Target: ~1-2k calls, a few minutes.- Idempotency test: rerun
fetch-> zero new bars, zero duplicate rows; kill mid-run and rerun -> resumes. - Spot:
fyers_ieod spot --underlying SENSEX --from 2025-01-01(normal/history,BSE:SENSEX-INDEX/BSE:BANKEX-INDEX, 90-day chunks) - and probe how far back spot 1-min goes.
Phase B - Local month, then sign-off
- Run one full calendar month for both underlyings (about 5-7k calls, ~40 min). Review the quality report with you. Adjust rules (fill policy, tolerances). Commit.
Phase C - VPS full backfill (the rest of the days)
git pushfrom local,./pull.shon the VPS (new package + config, no frontend change); setVPS_BSEIEOD_DATA_PATHinbackend/.env; check disk space.- Token: run the Fyers login on the VPS (paste the redirect URL; works over SSH), then start the run in
tmux/nohup:python -m bseieod.fyers_ieod discover --underlying SENSEX,BANKEX --from 2023-07-01 --to <yesterday>python -m bseieod.fyers_ieod fetch --underlying SENSEX,BANKEX --workers 1(single worker; the limiter is the bottleneck, not CPU).
- Newest expiries first (recent data matters most), then back to 2023. Stops cleanly on 401 / daily budget (default 90k); rerun next day after re-login - manifest skips finished contracts.
verify+reportover the full range; list of failed/empty contracts is aggregated once at the end.- Local PC does not need the full history; if wanted later, rsync the Parquet store (see
backup_data_gdrive.shfor the existing approach).
Phase D - Daily update (after backfill)
- Daily job after 15:30 on the VPS:
discoverfor new expiries,fetchfor contracts still live (the expired API only returns expired contracts, so a contract becomes fetchable the day after expiry; for the live ones use the normal/historywith the live symbol, or wait one day). Needs a fresh Fyers token each morning -> decide on TOTP auto-login (open question 2). - Hook status into the existing sync/Dashboard later (out of scope here).
Phase E - API / UI (later)
exchangeparam onieod_spotfut.py/ieod_options.pyor new/api/bse-ieod; BSE option inIeodSpotFut.jsx; docs page underdocs/docs/developer/packages/BSEEOD/.
5. Quality checks (the "nseieod-equivalent")
| Check | Rule |
|---|---|
| Bars per day | 375 expected (09:15-15:29) per liquid script; special days from specialdays (Muhurat etc.) |
| Liquid set | SENSEX + BANKEX spot and front weekly/monthly futures: gaps detected, filled by the existing FillingMissingTimes rules (flags -1..-6), filled column. |
| Options | Stored raw (sparse): illiquid strikes legitimately have minutes with no trade; no synthetic bars written. Report per contract: bars, first/last day, % of minutes covered. |
| Out-of-session | bars outside 09:15-15:29 removed; counted in the report |
| OHLC sanity | low is at most open/close, which are at most high; no negative price/volume/OI; no zero-price bars |
| Timestamps | unique per (masid, sdate); all end in :59 (store convention) |
| Cross-check vs EOD | per contract-day, minute-aggregate (O/first, H/max, L/min, C/last, volume/sum, OI/last) vs scr_bsefo_eod where it exists (2024-07-08+ UDiFF, older zip data); report days differing beyond tolerance (price 0.5%, volume exact or flag) |
| Coverage | every (expiry, contract) in the manifest has a status: ok, empty (no_data), error; counts by status in the final report |
| Futures vs spot | front future close within a sane basis of spot at the same minute |
6. Verification (how we know it works)
- Local slice: counts per contract = manifest bars; rerun is a no-op; DuckDB query of Parquet matches manifest totals.
- 20 random contract-days compared by hand with the Fyers web chart / bhav.
verifyreport clean (or every exception explained) before the VPS run is started.- VPS: spot-check the same 20 contract-days after backfill; disk usage and run time recorded back into this page.
9. BSE tab on the IEOD Spot-Fut page + 1-minute spot (2026-10-16)
The page's exchange switch now has BSE (NSE stays as it was; MCX is still a disabled stub). Everything BSE goes through its own router so the NSE endpoints are untouched.
| Piece | File |
|---|---|
| Reader (scr_bse_master + the Parquet store, IST conversion, whitelisted ticker) | backend/bseieod/reader.py |
API /api/ieod-bse: tickers, expiries?ticker&mode=FUT|OPT, strikes, range, download (CSV), preview | backend/app/api/ieod_bse.py |
| Page: BSE button, Spot / Future / Options, expiry + strike + CE/PE dropdowns (NSE keeps its free-text M1/ATM) | frontend/src/pages/IeodSpotFut.jsx, services/api.js (ieodBse) |
| 1-minute spot download | python -m bseieod.fyers_ieod spot --und SENSEX BANKEX (needs a fresh Fyers token); daily also pulls the session's spot |
| Tests | tests/test_bse_reader.py, tests/test_bse_spot_cli.py |
Spot download, first run on the VPS (2026-10-16). python -m bseieod.fyers_ieod spot pulled SENSEX 679,824 and BANKEX 678,085 one-minute bars, 28 May 2019 to 15 Oct 2026 (74 calls per underlying pair, a rerun makes 0 calls). Fyers served 2019 onward although the Standard plan documents only the last year of intraday history - another limit that was not enforced (see 1b). Quality (fyers_ieod verify now also prints a spot section): 99.0 % (SENSEX) / 98.3 % (BANKEX) of the days have exactly 375 bars, the first bar is 091559, no duplicates; against the index rows of scr_bseeq_eod the day high and low match on every common day (the EOD close is BSE's official value, so the last-bar close matches only 70-78 %, informational). Findings: (1) on 2020-12-01, 2021-01-01 and 2021-08-05 the feed delivers high and low swapped (1,105 bars) - candles_to_df now puts them right and the stored bars were repaired; (2) index volume exists only from 2025, OI never; (3) the 381 bars after 15:39 are the Diwali Muhurat evening sessions (2019-2024); (4) 28-29 May 2019 and 14 Aug 2019 are short days.
Behaviour: CSV columns are the NSE ones (Ticker, Date YYYYMMDD, Time HHMMSS, Open..OI); Time is bar-open + 59 s in IST, so the 09:15 bar is 091559. Spot = index bars (volume/OI 0) stored as yearly {S}/{UND}/{year}/{root_masid}_EQ; Future = one dated contract (dropdown lists only contracts that have bars; no stitched continuous series); Options = a listed expiry + strike (or ATM = the strike nearest the index close of the From Date, from scr_bseeq_eod) + CE/PE. Options come from the EODIEOD vendor (and Fyers for gaps), futures and spot from Fyers.
Checked against the vendor's own CSV for SENSEX25SEP2582200CE (22-25 Sep 2025): 1,500 of 1,500 minutes present, close 100 %, OI 100 %, volume 99.7 %, high 99.9 %.
8. Nightly auto-sync (21:00 IST) - built 2026-10-16
backend/bseieod/auto_sync.py, started as a background task from app/main.py (same pattern as the live OI collector). Off by default except on the VPS (BSE_AUTO_SYNC, BSE_AUTO_SYNC_TIME, see CLAUDE.md).
| Step | What happens |
|---|---|
| 1. Guard | cross-process lock file + stats.json status (RUNNING and fresh) -> skip; a lock/RUNNING older than 6 h is treated as dead |
| 2. BSE EOD | BseProcessor.run_update(finalize=False): equity, F&O, index spot, Parquet append, corporate actions, index changes. The page keeps showing RUNNING until the IEOD part is done |
| 3. IEOD | needs a Drive folder link saved for the current month (BSE tab > Daily Auto-Sync). rclone lsjson / rclone copy --include "*.rar" through the read-only remote gdrive_vendor -> csvs/EODIEODBSE/daily/<YYYY-MM>/ -> eodieod_bse.load_archives (fill mode, finished days skipped) |
| 4. Finish | status SUCCESS / FAILED; last-run info saved in scr_config (BseAutoSync) |
Rules (decided with the user):
- No link for the current month (new month, link not pasted yet): the EOD part still runs, the IEOD part is skipped with a red log line, history
MISSING_LINK, the run ends FAILED and the BSE tab shows a red banner. - A saved link whose folder has no file dated in that month: error after the 5th, warning before (the vendor may be late).
- The vendor file for the latest trading day not uploaded yet: warning only; the next run catches up.
- Futures 1-min is not in the vendor feed: one note line per run, nothing downloaded.
- Logs: every step writes through
BseProcessor.AddLoginto the BSEstats.json, so the page's Live Log Stream and Ingestion records show the whole run (sourcesBseAuto,DriveSync,BseIeod).
API (/api/bse-downloader): GET /auto-sync, POST /auto-sync/check, PUT /auto-sync/link, DELETE /auto-sync/link/{month}, PUT /auto-sync/enabled, POST /auto-sync/run. POST /download now also answers 409 while a BSE sync is running.
VPS checklist: rclone remote gdrive_vendor configured for the service user (or RCLONE_CONFIG pointing at its rclone.conf); the vendor folder shared with that Google account; the backend restarted (./pull.sh) so the scheduler starts. Manual run: python -m bseieod.auto_sync run [--dry] [--skip-eod] [--local-dir DIR]. Tests: backend/tests/test_bse_auto_sync.py.
10. Two stores: eodieod (vendor) and fyers (2026-10-16)
Until now everything - Fyers backfill, nightly Fyers update, vendor refill, spot - went into one folder, data/bseieod/fyers, so the vendor data could not be told apart from Fyers data (vendor refill ran in fill mode, so a Fyers bar already there won). The store is now split, same layout in both:
| Store | Folder | Holds | Written by |
|---|---|---|---|
| EODIEOD (default source) | data/bseieod/eodieod | vendor options (SENSEX / BANKEX) | eodieod_bse.py load, nightly auto-sync |
| Fyers | data/bseieod/fyers | spot (Fyers only), dated futures (vendor has none), option history and gap fill | fyers_ieod.py fetch / daily / spot |
Each store has its own _catalog/catalog.db (the archive manifest eodieod_archives is in the eodieod one). The reader (reader.py, used by the BSE tab) reads both and the vendor bar wins on a duplicate minute; Fyers fills what the vendor lacks. fyers_ieod verify checks both stores (--store eodieod|fyers|both). Config: Config.BSEIEOD_VENDOR_PATH.
VPS rollout. The old vendor rows stay inside fyers/ (harmless, vendor wins in the reader). Reload the vendor archives into the new store:
cd /home/kss/tradeentry/backend && source venv/bin/activate
python -m bseieod.eodieod_bse scan # all archives show 'pending' (new catalog)
python -m bseieod.eodieod_bse load --dry # parse only
python -m bseieod.eodieod_bse load # full reload into data/bseieod/eodieod (run in tmux)
python -m bseieod.fyers_ieod verify --store eodieod
10b. Missing SENSEX futures + index EOD gap (2026-10-16)
- Futures: for the SENSEX expiry 2026-09-24 Fyers lists the date but returns an empty futures list, so
discovernever queued it (BANKEX had its contract). The expired-candle call does answer for the monthly symbol (BSE:SENSEX26SEPFUT, 6,929 bars from 2026-08-28), sobse_universe.build_universenow builds the symbol from the date whenever a listed futures expiry has none (reported assynth_fut). Rundiscoverthenfetchfor SENSEX futures. - Index EOD gap:
scr_bseeq_eodSENSEX had BSE's own rows from 2025-04-15 and the old CSV up to 2022-06-17.fyers_ieod eodpulls Fyers DAILY candles (5 years on Standard) and inserts only the missing days (existing rows are compared and reported, never changed).
7. Risks / open questions
- Illiquid-option minutes: keep sparse (recommended) or fill-forward? Fill-forward makes OI Analysis simpler but multiplies rows.
- Daily token on the VPS: keep the manual paste (fine for the one-off backfill) or add TOTP auto-login for the daily job? (VPS has no Chrome; would need the Fyers REST login flow with TOTP, not Selenium.)
- Fyers API is not in the public docs for all fields: parameter names were taken from community SDK notes and confirmed by the dry run; re-check if Fyers changes them.
- BANKEX 2023-24 partial days: Fyers has holes there; Breeze (from 2024-06) can patch some days; earlier gaps may remain.
- Before Aug/Oct 2023: not available from Fyers (or Breeze). Needs a paid vendor (ask GDFL whether they carry BFO).
- Account block risk (the 150/min figure predates the Standard-plan limits in 1b; the client now enforces 45/min): strict limiter, stop-on-429 logic is mandatory before the first VPS run.