nseIEOD.py — Ingestion, Packing & Price Retrieval

This module contains two classes and three helper functions that form the core of the NSEIEOD pipeline.

---

Helper Functions

IsDate(s) → bool

def IsDate(s):
    try:
        pd.to_datetime(s)
        return True
    except (ValueError, TypeError):
        return False

Returns True if s can be parsed as a date by pandas.to_datetime. Used in generatemnamever2 to distinguish real expiry date strings (e.g. "29MAY25") from shorthand codes like "M1" or "W2".

---

get_last_thursday(year, month) → datetime

Returns the last Thursday of the given month — the standard NSE monthly expiry day. Used when a GDFL ticker encodes expiry as a month-year code that must be resolved to an actual calendar date.

# Example
get_last_thursday(2025, 5)  # → datetime(2025, 5, 29)

---

save_gdfl_index_config(typ, notes)

Reads a CSV file named {typ}.csv from the current directory and inserts its rows as a JSON blob into the scr_config table. Used for saving GDFL index-mapping configuration once.

---

Class: IEod

Handles two jobs — ingestion (CSV → Parquet, via update_fo_file/_write_parquet_bars, with scr_master lookups/inserts only) and packing (PostgreSQL staging → Parquet, via pack/pack_a_script).

Constructor

IEod(
    fromdate,                           # Start date (date or 'YYYY-MM-DD')
    todate,                             # End date
    path,                               # Source folder with GDFL CSV files
    version=3,                          # GDFL format version (1, 2, or 3)
    fo_prefix="GFDLNFO_CONTRACT_",      # CSV filename prefix
    datapath=""                         # Root path of the Parquet store
)

---

Ingestion Methods

workingdays()

Populates self.dates — a DataFrame of NSE working days from fromdate through 5 years ahead, using utils.NSEWorkingDays.
Called automatically in __init__.

---

get_monthly_expiry(mdate) → datetime

Given any date in a month, returns the actual monthly expiry date (last Thursday, walking back day-by-day if that Thursday is a holiday).

---

generatemname(row) → row

Dispatcher: routes to generatemnamever2 for version ≥ 2, otherwise uses version-1 logic (NIFTY / BANKNIFTY only).

Parses the raw GDFL ticker string into structured fields:

Output field	Example	Description
mname	NIFTY29MAY2525000CE	Canonical instrument name
edate	2025-05-29	Expiry date YYYY-MM-DD
strike	25000	Strike price string
opttyp	CE	Option type (CE, PE, or "" for futures)
underly	1	Underlying masid

---

generatemnamever2(row) → row

Version 2/3 ticker parser. Uses two regex patterns:

Pattern	Matches
fpattern	Futures: ...DDMMMYYF UT (v3) or ...YYMMM FUT (v2)
wpattern	Options/weekly: DDMMMYY

---

changetype(row) → row

Assigns the typ code based on whether the instrument has a strike price and whether it has a weekly or monthly expiry:

Has Strike	Expiry	typ
No	Weekly	4
No	Monthly	3
Yes	Weekly	6
Yes	Monthly	5

---

update_fo_file(conn)

Processes one trading day's F&O CSV file. Writes bars directly to Parquet — no scr_nsefo staging table is used. scr_master is still read (and written to, for brand-new instruments) to resolve each ticker's masid/underly.

---

_write_parquet_bars(df)

Writes 1-minute F&O bars straight into the Parquet store under self.data_path (no scr_nsefo staging involved). Input df must have columns masid, underly, sdate (naive IST, GDFL bar-end :59), sopen, shigh, slow, sclose, svolume, oi.

• Converts sdate from naive IST to UTC.
• Looks up each underly root's mname from scr_master (read-only).
• Groups bars by (underly, IST year, IST month) and writes/merges each group into {data_path}/{Letter}/{Name}/{year}/{root}_{MONTH_NAME}, matching the naming convention used by pack_a_script.
• Merges with any existing file, dedupes on (masid, sdate), sorts by sdate. If the existing file has a filled column, new rows get filled=0.

---

_archive_raw(mfile)

Called at the start of update_fo_file / update_index_file, before a source CSV is processed. It copies the file into a raw/ subfolder under IEOD_DATA_PATH (data_path), preserving its folder structure relative to folder_path. The raw/ tree thus records exactly which source files have been processed.

Source location	Archived to
folder_path/2019/GFDLNFO_CONTRACT_05062019.csv	data_path/raw/2019/GFDLNFO_CONTRACT_05062019.csv
folder_path/GFDLNFO_CONTRACT_05062019.csv (flat)	data_path/raw/2019/... (nested by currentdate.year)

Behaviour notes:

• Idempotent — if the destination already exists it is skipped (the file was processed before).
• Non-blocking — any copy error is logged ([raw] could not archive ...) and processing continues.
• No-op when data_path is empty.

---

update_index_file(conn)

Reads a GFDLCM_INDICES_{DDMMYYYY}.csv file and prints ticker diagnostics. Full insert logic is not yet implemented — prints ticker list to console only.

---

update_eq_file(conn)

Stub — pass. Equity intraday ingestion not yet implemented.

---

update_folder()

The main ingestion entry point. Iterates every NSE working day between fromdate and todate, sets self.currentdate, and calls update_fo_file.

# Usage
mu = IEod(date(2024, 1, 1), date(2024, 3, 31), source_path, version=3)
mu.update_folder()

---

Packing Methods

pack(year: int, fromdate=None, todate=None)

Imports one calendar year of GDFL FO (options) CSVs directly into Parquet — no scr_nsefo / scr_nseeq staging table involved. For every NSE working day in [fromdate, todate] (default: the full calendar year), looks for {folder_path}/{fo_prefix}{ddmmyyyy}.csv and — if present — calls update_fo_file(), the same per-day method update_folder() uses. No 1-minute bar is ever written to Postgres; scr_master is still queried/updated (new contracts need a masid), but that's a registry lookup, not bar storage.

Returns {"imported": int, "missing_dates": [date, ...]} and prints both the summary and every missing date.

ieod = IEod(date(2017, 1, 1), date(2017, 12, 31), r"D:\...\csvs\2017",
            fo_prefix="GFDLNFO_NIFTY_BNK_OPT_", version=1, datapath=DATA)
ieod.pack(2017)

---

pack_gdfl_range(year=None, fromdate=None, todate=None, datapath=None)

Module-level, version-aware wrapper around pack(). Callers don't need to know which GDFL file-naming version/prefix applies to a given date — it resolves that from the GDFL_VERSIONS table (see nseieod/CLAUDE.md rule 9), splits the requested range at every version boundary AND every calendar-year boundary (CSVs live under {Config.CSV_PATH}/{year}/), builds a fresh IEod per chunk with the right fo_prefix/version, and calls .pack() on each.

from nseieod.nseIEOD import pack_gdfl_range

pack_gdfl_range(year=2017)
pack_gdfl_range(fromdate=date(2020, 1, 1), todate=date(2021, 6, 30))  # spans v2 -> v3

Only FO (options/futures) is wired up end-to-end today — update_index_file() and update_eq_file() are still stubs, so a version-3 segment (which also has Index/EQ GDFL files) prints a note rather than silently skipping them.

---

Class: Con_Price_Getter

Retrieves historical 1-minute price bars from the Parquet store for a given instrument and configuration.

Constructor

Con_Price_Getter(
    script,             # Underlying name: "NIFTY", "BANKNIFTY", etc.
    datapath,           # Root Parquet store path
    fromdate="",        # Start date 'YYYY-MM-DD'  (default: '2008-01-01')
    todate="",          # End date   'YYYY-MM-DD'  (default: today)
    opttype="SPOT",     # Instrument: "SPOT", "FUT", "CE", "PE"
    expiry="M1",        # Expiry: date string | "M1"/"M2" | "W1"/"W2" | "Y1"
    strike="ATM",       # Strike: "ATM" or a specific price
    current_date="",    # Reference date for ATM/strike resolution
    current_time="",    # Reference time HH:MM:SS (uses IEOD price if set)
    opt_under="SPOT"    # Underlying for ATM calc: "SPOT" or "FUT"
)

---

getprices() → DataFrame | str

Main entry point. Resolves masid based on the configuration then reads the Parquet files.

---

get_contract_prices(masid, underl) → DataFrame | str

Iterates the month calendar, reads each Parquet file via __get_contract_month, concatenates, and trims to exact date range.

Passes a shared eq_cache dict into each __get_contract_month call so the yearly _EQ file is read from disk only once per year, not once per month.

---

get_prices_by_masid(masid, from_date, to_date) → DataFrame

Direct price fetch by masid — no script name, opttype, or expiry resolution needed. Suitable for internal use (e.g. liquid.py, notebooks, batch jobs).

Feature	Detail
Input	masid (int) + date range
_EQ cache	Reads yearly file once per year (eq_cache)
Legacy fallback	Falls back to {root}_{MONTH} if _EQ missing
Month names	Always UPPER() → matches actual file names (1_JANUARY)
Date parsing	Accepts YYYY-MM-DD, DD-MM-YYYY, DD-MM-YY, DD/MM/YYYY

# Fetch NIFTY spot bars directly by masid
getter = Con_Price_Getter("NIFTY", Config.IEOD_DATA_PATH)
df = getter.get_prices_by_masid(masid=1,
                                from_date="2024-01-01",
                                to_date="2024-03-31")

---

__get_contract_month(underl, masid, m, y, mpat, eq_cache=None) → DataFrame

Reads one Parquet file for a given month, converts sdate to IST, and filters to the requested masid. Month name is normalised to UPPER() before building the file path.

File resolution order for spot instruments:

1. {year}/{underl}_EQ (yearly file) — used from shared eq_cache if available
2. {year}/{underl}_{MONTH} (legacy monthly fallback) — if _EQ absent

For F&O instruments: {year}/{underl}_{MONTH} only.

Returns an empty DataFrame if no file is found.

---

__get_eod_price(id) → float | None

Fetches the EOD close price from PostgreSQL for current_date:

• opt_under="SPOT" → queries scr_nseeq_eod
• opt_under="FUT" → resolves monthindex=1 contract, queries scr_nsefo_eod

---

__get_ieod_price(id) → float | None

Fetches the intraday close price at current_date current_time from the Parquet store. Used for ATM resolution when current_time is set.

---

__getexpiry(masid, typ, cn) → date

Resolves an expiry shorthand to an actual calendar date by scanning scr_master expiry dates:

Code	Meaning
M1	1st upcoming monthly expiry from fromdate
M2	2nd upcoming monthly expiry
W1	1st upcoming weekly expiry
W2	2nd upcoming weekly expiry
Y1	1st upcoming yearly expiry

---

__get_strike(id, expi) → DataFrame

Fetches all distinct strikes for a given underlying and expiry from scr_master, adding a gap column (diff between consecutive strikes). Used to compute ATM via the most common strike interval.

---

__get_matching_or_next(df, val) → float

Returns val from the strikes list if it exists; otherwise returns the next higher available strike. Handles boundary conditions (val below minimum or above maximum available strike).

---

Usage Examples

from nseieod.nseIEOD import IEod, Con_Price_Getter, pack_gdfl_range
import datetime
from app.config import Config

# Ingest one month (datapath still required by IEod -- writes Parquet there)
mu = IEod(datetime.date(2024, 1, 1), datetime.date(2024, 1, 31),
          r"D:\GDFL\csvs", version=3, datapath=Config.IEOD_DATA_PATH)
mu.update_folder()

# Bulk-import a whole year of GDFL CSVs -- version/prefix auto-resolved per date
pack_gdfl_range(year=2017)

# Get NIFTY spot prices for Jan 2024
# (Con_Price_Getter no longer takes datapath -- reads Config.IEOD_DATA_PATH internally)
cp = Con_Price_Getter("NIFTY",
                      fromdate="2024-01-01", todate="2024-01-31",
                      opttype="SPOT")
df = cp.getprices()

# Get front-month BANKNIFTY future
cp = Con_Price_Getter("BANKNIFTY",
                      fromdate="2024-01-01", todate="2024-01-31",
                      opttype="FUT", expiry="M1")
df = cp.getprices()

# Get ATM NIFTY call option (using EOD price as reference)
cp = Con_Price_Getter("NIFTY",
                      fromdate="2024-01-01", todate="2024-01-25",
                      opttype="CE", expiry="M1",
                      current_date="2024-01-01")
df = cp.getprices()

# Direct masid-based fetch (no script name / opttype needed)
cp = Con_Price_Getter("NIFTY")
df = cp.get_prices_by_masid(masid=1,
                            from_date="2024-01-01",
                            to_date="2024-03-31")