Skip to main content

Liquid Scripts — Quality Checking & Gap Filling

The fix_data.py module (backend/nseieod/fix_data.py) maintains complete 1-minute bar coverage for the 4 most important scripts in the system.

Everything is computed in-memory directly from the Parquet store — there are no DuckDB tracking tables. Both entry points print the same missing-period report:

FunctionPurpose
check(from_date, to_date, masid=None)Scan the range and print the missing-period report (no writes)
filldata(from_date, to_date, masid=None, apply=False)Print the same report; with apply=True, fill the gaps into Parquet

Why "Liquid Scripts"?​

These 4 instruments trade every day with high volume and underpin almost every strategy and scanner:

masidNameTypeParquet location
1NIFTYSpot indexN/NIFTY/{year}/1_EQ
2BANKNIFTYSpot indexB/BANKNIFTY/{year}/2_EQ
3NIFTY-INear-month futureN/NIFTY/{year}/1_{MONTH}
4BANKNIFTY-INear-month futureB/BANKNIFTY/{year}/2_{MONTH}
masid=3 and 4 are always the near-month

scr_master holds a rolling near-month masid — masid=3 is always "current NIFTY near-month", masid=4 "current BANKNIFTY near-month". All historical futures data is stored under these masids — no per-date contract resolution needed.


NSE Trading Hours History​

NSE changed its market-open time on January 4, 2010:

PeriodOpenCloseExpected bars/day
Before 2010-01-0409:5515:29335
From 2010-01-0409:1515:29375

Special trading days (Muhurat, Budget Day, special sessions) use the window from the Postgres specialdays table instead of the normal session. Session times come from helpers.MarketOpeningTime / MarketClosingTime (special-day aware).

GDFL timestamp convention

GDFL stores bar-end timestamps at HH:MM:59 UTC. The module floors all timestamps to :00 before comparing, so 09:15:59 and 09:15:00 are the same minute.


How It Works​

For each NSE trading day in range:
read the day's bars from Parquet (read_day)
compare against expected minutes (check_day -> missing_times)
group consecutive missing minutes (_find_clusters)
-> each gap becomes one report row

No state is persisted. check only reads; filldata --apply additionally appends synthetic bars back to the Parquet file (the single source of truth).

Start-date guard​

Before scanning, check / filldata look up each script's start date in the DuckDB liquid_scripts table and never scan before it — so dates with no data yet are not falsely flagged as missing. The column used depends on the script kind:

Kindscr_master ruleStart-date column
Spotunderly = 0spot_start_date
CFuture (continuous future)underly > 0, monthindex > 0cfut_start_date
Future (dated contract)underly > 0, monthindex = 0fut_start_date

If the relevant start date is NULL, the full requested range is scanned. When a clamp happens the run prints, e.g.:

[masid 1] start_date 2016-06-01 -> skipping dates before it (requested from 2007-01-01)

Gap Fill Rules (FillingMissingTimes)​

filldata delegates per-day filling to FillingMissingTimes in ut_Classes.py. Every filled bar gets svolume = 0 and a negative filled flag identifying how it was produced:

Gap sizeStrategyfilled flag
1 barCarry the adjacent bar. If it is the session-opening bar, fill from the next bar instead of the previous one.-1
2–5 barsLinear interpolation on close → synthetic OHLC-2
6–20 barsScale the previous trading day's pattern to fit the gap-3
>20 barsSame previous-day pattern scaling-4
Vendor-replacement threshold

Clusters with filled < -3 (i.e. -4 and below) are low-confidence reconstructions and should be replaced with another vendor's data when available.

Whole missing days​

The report also flags fully missing trading days (zero bars). These are grouped into consecutive runs and cannot be bar-filled — they need vendor / EOD restore (Restore_Missingdates):

Situationfilled flag
Single missing day-5
Run of >1 consecutive missing days-6

They appear as a single row spanning the whole session, e.g.:

1 2007-01-23 09:55:00 15:29:00 335 -6;WHOLE DAY MISSING - needs vendor/EOD restore

filldata --apply does not fill these (FillingMissingTimes needs adjacent bars); they are reported only.


The Report​

Both check and filldata print the same table — one row per consecutive missing-bar gap:

Missing periods: 24 gap(s) across 20 day(s), 1740 bars total
(19 intraday gap(s), 5 whole-missing day(s))

masid date from to bars note
----------------------------------------------------------------------
1 2007-01-02 09:55:00 09:55:00 1 -1;Single-bar carry fill
1 2007-01-02 13:20:00 13:29:00 10 -3;6-20 bar previous-day pattern
1 2007-01-22 14:00:00 14:24:00 25 -4;>20 bar previous-day pattern (replace with another vendor)
1 2007-01-23 09:55:00 15:29:00 335 -6;WHOLE DAY MISSING - needs vendor/EOD restore
1 2007-01-24 09:55:00 15:29:00 335 -6;WHOLE DAY MISSING - needs vendor/EOD restore
1 2007-01-29 14:00:00 14:04:00 5 -2;2-5 bar linear interpolation
1 2007-01-31 09:55:00 09:56:00 2 -2;2-5 bar linear interpolation

The summary's second line splits the count into intraday gaps vs whole-missing days. Special trading days get an extra [details start-end] suffix in the note column.


Public API​

check(from_date, to_date, masid=None)​

from nseieod.fix_data import check

check() # all 4 scripts, 2007-2015
check('2010-01-01', '2010-12-31', masid=1)

Returns the clusters as a DataFrame (liquid_masid, trade_date, start_time, end_time, bar_count).

filldata(from_date, to_date, masid=None, apply=False)​

from nseieod.fix_data import filldata

filldata('2007-01-01', '2007-01-31', masid=1) # dry run (report only)
filldata('2007-01-01', '2007-01-31', masid=1, apply=True) # report + write fills
ParameterDefaultDescription
from_date / to_date2007-01-01 / 2015-12-31Date range YYYY-MM-DD
masidNoneOne script (1-4) or all 4
applyFalseFalse = report only; True = write fills to Parquet

Command Line​

Run from backend/:

# Check (report only)
python -m nseieod.fix_data check --from 2007-01-01 --to 2007-01-31 --masid 1

# Fill — dry run (report only), then apply
python -m nseieod.fix_data filldata --from 2007-01-01 --to 2007-01-31 --masid 1
python -m nseieod.fix_data filldata --from 2007-01-01 --to 2007-01-31 --masid 1 --apply
ArgumentDefaultDescription
--from2007-01-01Start date
--to2015-12-31End date
--masidall (1–4)Restrict to one liquid script
--applyoff(filldata only) write fills to Parquet

Importable building blocks​

from nseieod.fix_data import (
check, filldata,
check_day, fill_day, read_day,
get_expected_times,
LIQUID_SCRIPTS,
)

get_expected_bars, get_nse_working_days, get_special_day_info and the NSE calendar constants live in nseieod/helpers.py.


Known Limitations & Edge Cases​

SituationBehaviour
Gap at session open with no usable neighbourunfillable — left as-is
Whole day missing (no bars)Reported as -5/-6; not bar-filled — needs Restore_Missingdates / vendor data
Day with > expected barsTreated as complete (extra = optional 15:30 closing-auction bar)
Special trading dayExpected window comes from the Postgres specialdays table

Daily Maintenance (after market close)​

from datetime import date
from nseieod.fix_data import check, filldata

TODAY = date.today().isoformat()
check(TODAY, TODAY) # review the report
filldata(TODAY, TODAY, apply=True) # fill any intraday gaps