nseIEOD.py — Price Retrieval & Ingestion

helpers.py — Common Market-Time Functions

Created at backend/nseieod/helpers.py. Contains shared constants and functions used across the nseieod package.

MarketOpeningTime(d) -> datetime.time

Returns the NSE market opening time for a given date.

Date range	Opening time	Bars / day (1-min)
Before 2010-01-04	09:55	335
From 2010-01-04	09:15	375

from nseieod.helpers import MarketOpeningTime

MarketOpeningTime("2009-12-31")   # datetime.time(9, 55)
MarketOpeningTime("2010-01-04")   # datetime.time(9, 15)
MarketOpeningTime("2024-06-01")   # datetime.time(9, 15)

MarketClosingTime(d=None) -> datetime.time

Returns datetime.time(15, 30) — currently constant for all dates. The d parameter is reserved for future use.

from nseieod.helpers import MarketClosingTime

MarketClosingTime()               # datetime.time(15, 30)
MarketClosingTime("2024-06-01")   # datetime.time(15, 30)

normalize_fromdate(s) / normalize_todate(s)

Auto-append market open/close time when the caller passes a date-only string. Used internally by Con_Price_Getter.__init__. Special-day times are applied automatically.

Input	Output	Reason
"2011-01-01"	"2011-01-01 09:15:00"	normal post-2010 open
"2009-12-31"	"2009-12-31 09:55:00"	pre-2010 open
"2009-10-17"	"2009-10-17 18:15:00"	Muhurat (special day)
"2011-01-01 09:30:00"	"2011-01-01 09:30:00"	already has time, unchanged
"2011-02-01" (todate)	"2011-02-01 15:30:00"	normal close
"2009-10-17" (todate)	"2009-10-17 19:15:00"	Muhurat close (special day)

Special-day awareness — Fix 1.2

Both functions now query the Postgres specialdays table via liquid.get_special_day_info() before falling back to normal session rules.

Day type	MarketOpeningTime	MarketClosingTime
Special (Muhurat, Budget, …)	starttime from specialdays DB row	endtime from specialdays DB row
Normal before 2010-01-04	09:55	15:30
Normal from 2010-01-04	09:15	15:30

The DB lookup is cached for the process lifetime — no extra round-trip per call.

---

Con_Price_Getter — Date + Time Support (Enhance 1.1)

Con_Price_Getter now accepts both date-only and datetime strings for fromdate / todate.

Signature (unchanged externally)

Con_Price_Getter(
    script,
    fromdate = "",          # "2011-01-01"  or  "2011-01-01 09:30:00"
    todate   = "",          # "2011-02-01"  or  "2011-02-01 14:00:00"
    opttype  = "SPOT",
    expiry   = "M1",
    strike   = "ATM",
    ...
)

Auto-normalization rules

When fromdate / todate have no time component, they are expanded:

Parameter	Supplied	Resolved to
fromdate	"2011-01-01"	"2011-01-01 09:15:00" (market open)
fromdate	"2009-12-31"	"2009-12-31 09:55:00" (pre-2010 open)
todate	"2011-02-01"	"2011-02-01 15:30:00" (market close)
fromdate	"2011-01-01 09:30:00"	"2011-01-01 09:30:00" (kept as-is)

Examples

# Both date-only — auto-fills open/close
cp = Con_Price_Getter("NIFTY", fromdate="2011-01-01", todate="2011-02-01")
# Effective range: 2011-01-01 09:15:00 IST  to  2011-02-01 15:30:00 IST

# Mixed — explicit from-time, date-only to
cp = Con_Price_Getter("NIFTY", fromdate="2011-01-01 09:30:00", todate="2011-02-01")
# Effective range: 2011-01-01 09:30:00 IST  to  2011-02-01 15:30:00 IST

# Both explicit
cp = Con_Price_Getter("NIFTY", fromdate="2011-01-01 09:30:00", todate="2011-02-01 14:00:00")
# Effective range: 2011-01-01 09:30:00 IST  to  2011-02-01 14:00:00 IST

IsDate(s) and default expiry / opt_under (hardening)

IsDate(s) decides whether expiry is a literal calendar date (edate='...' lookup) or a shorthand code (M1, W2, Y1, ...). pd.to_datetime alone is too lenient for this: pd.to_datetime("") returns NaT (no exception), and pd.to_datetime("M1") parses as 0001-01-01 00:00:01. Both used to make IsDate return True, producing invalid SQL like edate='' or edate='M1'.

IsDate(s) now first requires s to match ^\d{4}-\d{2}-\d{2}([ T]\d{2}:\d{2}(:\d{2})?)?$ (an ISO date, optionally with HH:MM[:SS]) before attempting pd.to_datetime. Anything else — including "", "M1", "W2" — returns False.

Con_Price_Getter.__init__ also now fills in safe defaults when callers pass blank strings for non-SPOT lookups:

Condition	Default applied
opttype != "SPOT" and expiry is blank	expiry = "M1" (front month, matches the constructor's documented default)
opttype in ("CE", "PE") and opt_under is blank	opt_under = "SPOT"

# Before: raised psycopg2.errors.InvalidDatetimeFormat (edate='' / edate='M1')
Con_Price_Getter("NIFTY", opttype="FUT", expiry="")

# After: expiry defaults to "M1", resolves normally
Con_Price_Getter("NIFTY", opttype="FUT", expiry="")

---

"W{n}" expiries — continuous rolling weekly series

__getexpiry(masid, typ, cn) resolves shorthand expiry codes to a real edate, but only handles typ == "M" / "Y" (grouping scr_master edates by month/year). "W{n}" is handled separately and before __getexpiry is reached, because a single edate doesn't make sense for a weekly request spanning [fromdate, todate] — most ranges cross several weekly expiries.

In getprices(), the CE/PE branch intercepts "W{n}" up front:

if not IsDate(self.expiry) and self.expiry[0].upper() == "W":
    cn = int(self.expiry[1:])
    result = self.__get_weekly_series_prices(underl, mprice, cn)
    if result is None:
        return f"No {self.expiry} expiry available for {self.script} from {self.fromdate}!!!"
    return result

__weekly_edates(masid, start_date, end_date) -> list[Timestamp]

Returns the distinct scr_master edates for masid in [start_date, end_date] whose gap to the previous or next edate is <= 8 days — i.e. edates that are part of an actual weekly series. This excludes underlyings/periods with monthly expiries only (e.g. NIFTY before weekly options existed, ~28-35 day gaps).

__get_weekly_series_prices(underl, mprice, cn) -> DataFrame | str | None

Builds the continuous "W{cn}" series across [fromdate, todate]:

• Looks ahead up to todate + 60 + cn*7 days to gather the full list of weekly edates (__weekly_edates).
• Walks the weeks in order: for each weekly edate e (segment boundary), the active contract is the cn-th edate counting from the current position (weekly[i + cn - 1]). The segment runs from the previous boundary (or fromdate for the first segment) through min(e, todate).
• For each segment, resolves the strike (ATM or fixed) against that contract's edate, looks up its masid, and fetches its bars via get_prices_by_masid.
• Rolls to the next contract starting the day after each expiry — mirroring how monthindex gives a continuous near-month future.
• Concatenates all segments, sorted by sdate.

Return values:

Return	Meaning
None	No weekly series exists at all for this underlying/period (e.g. NIFTY in 2016 — only monthly options). getprices() then returns "No {expiry} expiry available for {script} from {fromdate}!!!".
"Empty Data!!!!"	A weekly series exists but no price data was found for any segment.
DataFrame	Concatenated multi-contract bars, sorted by sdate. Expiry boundaries appear as price discontinuities (the new contract's price series resets relative to the expiring one).

# NIFTY had no weekly options in 2016 -- "No data" message
Con_Price_Getter("NIFTY", fromdate="2016-06-01", todate="2016-06-30",
                 opttype="CE", expiry="W1", strike="ATM").getprices()
# -> "No W1 expiry available for NIFTY from 2016-06-01 09:15:00!!!"

# BANKNIFTY had weekly options in 2016 -- "W1" rolls through all 4
# weekly contracts traded in June 2016, not just one
Con_Price_Getter("BANKNIFTY", fromdate="2016-06-01", todate="2016-06-30",
                 opttype="PE", expiry="W1", strike=17000).getprices()
# -> DataFrame spanning contracts expiring 2016-06-02, -09, -16, -23, -30

---

get_prices_by_masid — UTC timestamp filtering

get_prices_by_masid(masid, from_date, to_date) applies the same open/close normalization. The DuckDB query now uses full UTC timestamps (not date-only) so session boundaries are respected:

# Parquet sdate is stored as raw UTC.
# 2011-01-01 09:15:00 IST  ==  2011-01-01 03:45:00 UTC
# DuckDB filter: sdate >= '2011-01-01 03:45:00' AND sdate <= '...'

This ensures bars from 09:15–09:29 on the first day are not accidentally excluded.

---

_write_parquet_bars(df, preserve_filled=False) — now handles spot too

Previously this only wrote F&O bars (one file per {root}_{MONTH_NAME}). It now branches on whether each row is spot or F&O:

# df columns: masid, underly, sdate (naive IST, bar-end :59),
#             sopen, shigh, slow, sclose, svolume, oi[, filled]
# underly must equal masid itself for spot/index rows (the DB's underly=0
# convention means "is its own root" -- callers resolve that before calling).

# Spot (masid == underly): {data_path}/{Letter}/{Name}/{year}/{masid}_EQ
# F&O  (masid != underly): {data_path}/{Letter}/{Name}/{year}/{underly}_{MONTH_NAME}
# deduped on (masid, sdate), keep='last' so new rows win on overlap.

preserve_filled=True keeps each row's incoming filled value as-is instead of forcing it to 0 — used by insert_clean_csv below for pre-cleaned vendor backfills that already carry meaningful fill-flag values (see nseieod/CLAUDE.md rule 7). update_fo_file (fresh GDFL ingestion) still calls this with the default preserve_filled=False, unchanged.

insert_clean_csv(csv_path, masid, fromdate=None, todate=None)

Inserts a pre-cleaned 1-minute OHLCV CSV straight into Parquet for a single, already-known masid — no Ticker parsing, no scr_nsefo staging. Built for nseieod/insert_liquid_csv.py (below), which backfills vendor data the GDFL pipeline never covered: NIFTY/BANKNIFTY spot coverage currently stops at end-2016, and NIFTY-I/BANKNIFTY-I (near-month continuous future) have no Parquet files at all yet.

Expected CSV columns: Ticker, Date/Time, Open, High, Low, Close, Volume, Open Interest, Lotsize, Change, filled — Ticker/Lotsize/Change are dropped, the caller resolves masid itself (typically from the filename, not the Ticker column), and filled is preserved as-is.

ieod = IEod(fromdate_as_date, todate_as_date, source_dir, datapath=Config.IEOD_DATA_PATH)
ieod.insert_clean_csv("csvs/Liquid/cleaned/NIFTY.csv", masid=1,
                       fromdate="2016-01-01", todate="2023-02-06")

fromdate/todate here are plain 'YYYY-MM-DD' strings used only to filter which rows get written — separate from IEod.__init__'s own fromdate/ todate, which must be real datetime.date objects.

---

update_index_file() / update_eq_file() — implemented (were stubs)

Both now resolve each GDFL root-level (spot) ticker to a scr_master masid and write through _write_parquet_bars(), instead of update_index_file() just dumping unique ticker names to tickers.csv and update_eq_file() being a bare pass. They share one implementation, _update_root_file():

# update_index_file: GFDLCM_INDICES_{ddmmyyyy}.csv, strips ".NSE_IDX"
# update_eq_file:     GFDLCM_STOCK_{ddmmyyyy}.csv,   strips ".NSE"
# (longer suffix stripped first -- ".NSE" is a substring of ".NSE_IDX")

Masid resolution (_resolve_index_masid / _resolve_eq_masid, backed by a shared _root_master_cache of every scr_master row with underly=0):

• EQ/stock tickers (e.g. RELIANCE.NSE) match scr_master.mname exactly — no aliasing needed.
• Index tickers (e.g. NIFTY 50.NSE_IDX) often don't match literally — most of GDFL's ~70 index labels match after normalizing (remove spaces, uppercase: "NIFTY IT" → Nifty IT), but the four tradeable indices have their own short F&O-underlying name, handled via _INDEX_NAME_ALIASES: NIFTY 50→NIFTY, NIFTY BANK→BANKNIFTY, NIFTY FIN SERVICE→FINNIFTY, NIFTY MID SELECT→MIDCPNIFTY. Anything that still doesn't resolve is skipped and logged (not auto-inserted into scr_master) — verified against a real GFDLCM_INDICES_*.csv sample: 29 of 70 tickers resolved, 41 skipped (mostly sectoral/strategy indices with no existing scr_master row, e.g. NIFTY SMLCAP 50, the GS bond indices, the 1X/2X lev-inv series).

Both write with masid == underly (spot convention) and the default preserve_filled=False, same as update_fo_file — these are fresh real exchange bars, not a vendor backfill.

pack() / update_folder() — new segment parameter

ieod.pack(year, segment="NFO")    # default -- futures & options (unchanged)
ieod.pack(year, segment="INDEX")  # GFDLCM_INDICES_{ddmmyyyy}.csv -> update_index_file()
ieod.pack(year, segment="EQ")     # GFDLCM_STOCK_{ddmmyyyy}.csv   -> update_eq_file()

Both methods resolve the handler/prefix pair via a shared _segment_handler() lookup; an unknown segment raises ValueError immediately. Run pack() once per segment to import all three for the same period — pack_gdfl_range() still only drives segment="NFO" automatically (index_prefix/eq_prefix don't vary across GDFL_VERSIONS the way fo_prefix does, so there's nothing for its version-splitting logic to resolve for them); it now prints a pointer to call pack(year, segment="INDEX"/"EQ") separately instead of the old "not implemented yet" message.

GDFL_VERSIONS — coverage now ends 2026-05-31

The v3 entry's "to" field, previously open-ended (None, meaning "current date"), is now capped at 2026-05-31. From 2026-06-01 onward, 1-minute data comes from a different vendor ("EODIEOD" data), not GDFL CSVs — see nseieod/CLAUDE.md rule 9 for the full explanation. This means pack_gdfl_range() and any GDFL-file presence check built on this table (e.g. scratch/move_gdfl_copied.py's missing-file report) now correctly stop expecting GDFL CSVs past 2026-05-31 instead of treating them as missing.

IEod.__init__ — new version=4 (EODIEOD) profile + overridable prefixes

def __init__(self, fromdate, todate, path, version=3, fo_prefix=None, datapath="",
            index_prefix=None, eq_prefix=None):

Two changes, both backward compatible (every existing caller passes fo_prefix/version/datapath as keywords, never relies on positional order beyond fromdate, todate, path):

1. fo_prefix default is now None (was the literal "GFDLNFO_CONTRACT_"). When omitted, it resolves based on version — "GFDLNFO_CONTRACT_" for any version other than 4, unchanged from before.

2. index_prefix/eq_prefix are now constructor parameters, not hardcoded inside __init__. Previously the only way to change them was to set ieod.index_prefix = "..." on the instance after construction (a pattern nseieod/eodieod/ingest.py used to have to do).

3. version=4 is a new, EODIEOD-specific profile (see nseieod/CLAUDE.md rule 12). Ticker parsing is unaffected — generatemnamever2's if version == 2: ... else: # version 3 ... already treats any version other than 2 as v3-style (DDMMMYY direct), which is exactly what nseieod/eodieod/convert.py's converted tickers use. Only the default file-name prefixes change, to three new module-level constants:

   Constant	Value
   EODIEOD_FO_PREFIX	EODIEODNFO_CONTRACT_
   EODIEOD_INDEX_PREFIX	EODIEODCM_INDICES_
   EODIEOD_EQ_PREFIX	EODIEODCM_STOCK_

   So IEod(f, t, folder, version=4, datapath=Config.IEOD_DATA_PATH) looks for EODIEODCM_INDICES_ddmmyyyy.csv etc instead of the GFDLCM_INDICES_ GDFL name — the converted file's data origin (EODIEOD vs GDFL) is visible in its filename even after packing. nseieod/eodieod/convert.py imports these same three constants for its _Emitter output, so the writer and reader can never drift apart.

version=4 is not added to GDFL_VERSIONS and is never passed to pack_gdfl_range() — that automation is a date-driven scan for real, vendor-delivered GDFL CSVs, which doesn't apply to EODIEOD's manually-invoked convert/ingest flow (nseieod/eodieod/run.py).

---

insert_liquid_csv.py — Liquid CSV Backfill Script

Reads every CSV in csvs/Liquid/cleaned/ and inserts it via IEod.insert_clean_csv(). The filename itself (minus .csv) is the scr_master.mname to resolve — _resolve_masid() tries an exact match first, then _ → - (source files use NIFTY_I.csv for the scr_master mname NIFTY-I). Each CSV's own Ticker column is ignored entirely.

cd backend
python -m nseieod.insert_liquid_csv --from 2016-01-01 --to 2023-02-06

from nseieod.insert_liquid_csv import insert_liquid_csvs
insert_liquid_csvs("2016-01-01", "2023-02-06")

---

main.py — Verification Entry Point

Orchestrates verification of the entire Parquet store by walking the data/ directory tree and calling ScriptDetails.verify() for each script.

---

Call Flow

---

Functions

process_a_script(mpath, mname, upto, from_begin=False)

Verifies one script's Parquet data:

Step	Action
1	ScriptDetails.open(mname, mpath) — load or create audit record
2	sd.initiate() — load working-days calendar and parent refs
3	sd.ieod.verify(upto, from_begin) — scan Parquet files
4	Print sd.ieod.spot_missing_times summary to console
5	sd.save() — upsert updated JSON state to scriptdetails

Setting from_begin=True clears all prior audit state and re-verifies from scratch (full re-check).

# Verify NIFTY up to today, resuming from last checked date
process_a_script(
    mpath      = r"D:\data\N\NIFTY",
    mname      = "NIFTY",
    upto       = "2025-05-27",
    from_begin = False
)

---

process_a_letter_folder(mpath, up_to, from_begin=False)

Iterates every script subfolder inside a single letter folder (e.g. data/N/) and calls process_a_script for each.

# Verify all scripts starting with "N"
process_a_letter_folder(
    mpath      = r"D:\data\N",
    up_to      = "2025-05-27",
    from_begin = False
)

---

verify_scripts(mDatapath, upto, all=False)

Top-level runner. Scans the entire Parquet store.

Parameter	Type	Description
mDatapath	str	Root data path (e.g. D:\data)
upto	str	Verify up to this date YYYY-MM-DD
all	bool	True = full re-check; False = resume from last checkpoint

from main import verify_scripts

# Incremental verification (resume from last saved checkpoint)
verify_scripts(r"D:\data", upto="2025-05-27", all=False)

# Full re-verification from scratch (slow — use for data repairs)
verify_scripts(r"D:\data", upto="2025-05-27", all=True)

---

Entry Point (bottom of file)

The file currently has a hardcoded call at the bottom:

verify_scripts(mDatapath, "2007-12-31", True)

caution

This runs automatically when the file is imported or executed directly. Adjust upto and all before running to avoid unintended full re-verification of the entire dataset.

---

Typical Usage Scenarios

1. Daily Incremental Check (most common)

Run after each day's data is packed:

import datetime
verify_scripts(DATA_PATH, upto=datetime.date.today().isoformat(), all=False)

2. Verify a New Script Added to the Store

process_a_script(
    mpath      = r"D:\data\M\MIDCPNIFTY",
    mname      = "MIDCPNIFTY",
    upto       = "2025-05-27",
    from_begin = True    # First-time check — no prior state
)

3. Re-verify After a Data Repair

# Re-verify only the affected script
process_a_script(
    mpath      = r"D:\data\B\BANKNIFTY",
    mname      = "BANKNIFTY",
    upto       = "2025-05-27",
    from_begin = True
)

---

Console Output

During verification, process_a_script prints:

---------------------------NIFTY-------------------------------------------
[list of spot_missing_times DataFrames, if any]
--------------------Process NIFTY  Done !!----------------------------------

Detailed missing-date and missing-time info is persisted to PostgreSQL via ScriptDetails.save() — use the scriptdetails table for reporting rather than relying on console output in production.