NSEIEOD — Intraday EOD Data Pipeline
The nseieod package ingests, packs, verifies, and retrieves NSE F&O intraday
(1-minute bar) data sourced from GDFL CSV files.
Package Purpose
| Stage | What it does |
|---|---|
| Ingest | Reads raw GDFL F&O CSV files, parses tickers, resolves masid from scr_master, and inserts 1-minute bars into PostgreSQL staging tables |
| Pack | Exports staging data year-by-year from PostgreSQL into a month-partitioned Parquet store, then clears the staging rows |
| Verify (QC) | Scans the Parquet store in-memory for missing trading days and missing 1-minute bars and reports them (fix_data.check). Coverage tracking lives in DuckDB catalog.db (liquid_scripts) |
| Fill | Fills the gaps found by the QC with synthetic / pattern-based bars (fix_data.filldata, using ut_Classes) |
| Retrieve | Reads price series directly from Parquet for a given instrument (spot / future / option), expiry, and strike |
IEOD Data Pipeline
Data history & sources
| Period | Coverage | Source |
|---|---|---|
| 2007 – 2016 (May) | Only 4 scripts — NIFTY spot, NIFTY future, BANKNIFTY spot, BANKNIFTY future | Collected from various sources, already packed into Parquet |
| 2016 (Jun) onwards | Full GDFL feed (see breakup below) | GDFL daily IEOD CSVs |
| Later addition | India VIX index IEOD | — |
In the DuckDB liquid_scripts table we store only the spot masid for each
liquid root — but each row still tracks spot_start_date, cfut_start_date, and
fut_start_date separately.
| Term | Meaning |
|---|---|
| Spot | Underlying index/equity (underly = 0 in scr_master) |
| CFuture | Continuous future — monthindex > 0 in scr_master (rolling near-month) |
| Future | A real future contract with an expiry date (edate) |
Two update stages
| Stage | Status | Description |
|---|---|---|
| Production Stage | current | Back-fill: extract, pack to raw/, QC, then fill into the live store |
| Regular Stage | future | Day-to-day incremental updates once back-fill is complete |
Data from 2007 → 2016 May is already updated in the Parquet store. Everything from 2016 Jun → today flows through the Production Stage below.
Production Stage flow
Stage 1 — Extract
Daily GDFL IEOD data is extracted into CSV_PATH, organised under a {year} subfolder.
Stage 2 — Pack to raw/
The CSV files are ingested and written to Parquet (the format described below).
All of these Parquet files are written under the raw/ subfolder of
IEOD_DATA_PATH — these are the raw files (unverified, unfilled).
Stage 3 — Quality Check (period-aware)
Scan the raw/ subfolder for any missing trading days or missing 1-minute bars.
Because GDFL coverage changed over time, the QC must apply this file breakup when
deciding which files are expected (and therefore flagged as missing):
| Period | Expected GDFL content |
|---|---|
| Jun 2016 → May 2020 | NIFTY & BANKNIFTY options IEOD only (NFO) |
| Jun 2020 → Mar 2022 | NFO — all stocks + Index options, no EQ |
| Mar 2022 → onwards | All EQ IEOD + all NFO contracts + Index IEOD files |
QC reads from the
raw/subfolder inIEOD_DATA_PATH, not the live root.
Stage 4 — Fill
Once a range is checked, run fix_data filldata to fill the gaps. The fill
writes into the root folder of IEOD_DATA_PATH (the live store), promoting the
verified/filled data out of raw/.
Architecture Overview
File Structure
backend/nseieod/
├── nseIEOD.py # IEod (ingest + pack) + Con_Price_Getter (read)
├── helpers.py # NSE calendar + market-time (single source of truth)
├── fix_data.py # Quality check + gap fill (check / filldata)
├── catalog.py # DuckDB catalog manager (liquid_scripts table)
├── ut_Classes.py # Data repair: Resu_Holidays, Restore_Missingdates, FillingMissingTimes
├── getprices.py # CLI to export liquid-script prices to CSV
├── gdfl/ # GDFL ZIP extraction + per-period file checks
└── script_data.json # Script metadata cache
Parquet Data Store Layout
All packed data lives under a single root folder (configured via mDatapath):
data/
└── {FIRST_LETTER}/
└── {SCRIPT_NAME}/
└── {YEAR}/
└── {masid}_{MONTH_NAME} ← extensionless Parquet file
Examples:
data/N/NIFTY/2021/1_JANUARY
data/N/NIFTY/2021/1_FEBRUARY
data/B/BANKNIFTY/2022/2_MARCH
Each file contains all contracts (spot + futures + options) for that script
in that month. Rows are filtered by masid at read time.
Parquet Schema
| Column | Type | Description |
|---|---|---|
masid | int | Script master ID (FK → scr_master) |
sdate | datetime (UTC) | Bar timestamp |
sopen | float | Open price |
shigh | float | High price |
slow | float | Low price |
sclose | float | Close price |
svolume | int | Volume |
oi | int | Open Interest (0 for equity/spot) |
GDFL Format Versions
GDFL changed their file format over time. IEod handles all three versions:
| Version | Period | Characteristics |
|---|---|---|
| 1 | Before May 2020 | NIFTY & BANKNIFTY only; prefix GFDLNFO_NIFTY_BNK_OPT_; monthly expiry as YYMMM |
| 2 | May 2020 – Jan 2021 | Multi-script; monthly expiry encoded as YYMMM |
| 3 | Jan 2021 onwards | Multi-script; weekly/monthly expiry as DDMMMYY (default) |
Dependencies
pandas # DataFrames, Parquet I/O
pyarrow # Parquet engine
python-dateutil # relativedelta for month arithmetic
nseeod # Internal package (utils, pgs)
The nseeod internal package provides:
utils—NSEWorkingDays,get_single_value_from_sql,clean_floatpgs— PostgreSQL pool:conn,read_sql,insertdataframe,updatetable_from_df,update_or_delete_sql
Data repair helpers (Resu_Holidays, Restore_Missingdates, FillingMissingTimes) are now
native to this package in nseieod/ut_Classes.py — no longer imported from nseeod.
Database Tables
| Table | Purpose |
|---|---|
scr_master | Master instrument registry (masid, mname, underly, typ, edate, strike, opttyp, monthindex) |
scr_nsefo | Staging: raw intraday F&O 1-min bars (cleared after packing) |
scr_nseeq | Staging: raw intraday equity/index 1-min bars (cleared after packing) |
scr_nsefo_eod | Daily EOD F&O prices (used for ATM resolution + whole-day restore) |
scr_nseeq_eod | Daily EOD equity/index prices (used for ATM resolution + whole-day restore) |
scr_config | Global config & stats (mname, cobj, typ, sdate, notes) |
holidays / specialdays | NSE holiday calendar + special-session windows (cached by helpers.py) |
Coverage / quality tracking is not in PostgreSQL — it lives in DuckDB
catalog.db (liquid_scripts). See Quality Checking.
Instrument Type Codes (scr_master.typ)
typ | Instrument | Expiry |
|---|---|---|
| 0 | Underlying index/equity (spot) | — |
| 3 | Future | Monthly |
| 4 | Future | Weekly |
| 5 | Option (CE or PE) | Monthly |
| 6 | Option (CE or PE) | Weekly |
Quick-Start Workflow
Run from backend/ with the venv active. Paths come from Config (.env) — no
hard-coded paths needed.
import datetime
from app.config import Config
from nseieod.nseIEOD import IEod, Con_Price_Getter
# 1. Ingest CSVs into PostgreSQL staging, then pack to Parquet
mu = IEod(datetime.date(2024, 1, 1), datetime.date(2024, 3, 31),
Config.CSV_PATH, fo_prefix="GFDLNFO_CONTRACT_", version=3,
datapath=Config.IEOD_DATA_PATH)
mu.update_folder() # CSV -> staging (also archives source to raw/)
mu.pack(2024) # staging -> Parquet
# 2. Quality check + fill (in-memory; writes a report to CSV_PATH)
from nseieod.fix_data import check, filldata
check('2024-01-01', '2024-03-31', masid=1)
filldata('2024-01-01', '2024-03-31', masid=1, apply=True)
# 3. Retrieve prices (datapath comes from Config internally)
cp = Con_Price_Getter("NIFTY", fromdate="2024-01-01", todate="2024-03-31",
opttype="FUT", expiry="M1")
df = cp.getprices()
print(df.head())
CLI equivalents:
python -m nseieod.fix_data check --from 2024-01-01 --to 2024-03-31 --masid 1
python -m nseieod.fix_data filldata --from 2024-01-01 --to 2024-03-31 --masid 1 --apply
python -m nseieod.catalog # inspect liquid_scripts coverage table