Liquid Scripts — Quality Checking & Gap Filling

The fix_data.py module (backend/nseieod/fix_data.py) maintains complete 1-minute bar coverage for the 4 most important scripts in the system.

Everything is computed in-memory directly from the Parquet store — there are no DuckDB tracking tables. Both entry points print the same missing-period report:

Function	Purpose
`check(from_date, to_date, masid=None)`	Scan the range and print the missing-period report (no writes)
`filldata(from_date, to_date, masid=None, apply=False)`	Print the same report; with `apply=True`, fill the gaps into Parquet

Why "Liquid Scripts"?

These 4 instruments trade every day with high volume and underpin almost every strategy and scanner:

masid	Name	Type	Parquet location
1	NIFTY	Spot index	`N/NIFTY/{year}/1_EQ`
2	BANKNIFTY	Spot index	`B/BANKNIFTY/{year}/2_EQ`
3	NIFTY-I	Near-month future	`N/NIFTY/{year}/1_{MONTH}`
4	BANKNIFTY-I	Near-month future	`B/BANKNIFTY/{year}/2_{MONTH}`

masid=3 and 4 are always the near-month

scr_master holds a rolling near-month masid — masid=3 is always "current NIFTY near-month", masid=4 "current BANKNIFTY near-month". All historical futures data is stored under these masids — no per-date contract resolution needed.

NSE Trading Hours History

NSE changed its market-open time on January 4, 2010:

Period	Open	Close	Expected bars/day
Before 2010-01-04	09:55	15:29	335
From 2010-01-04	09:15	15:29	375

Special trading days (Muhurat, Budget Day, special sessions) use the window from the Postgres specialdays table instead of the normal session. Session times come from helpers.MarketOpeningTime / MarketClosingTime (special-day aware).

GDFL timestamp convention

GDFL stores bar-end timestamps at HH:MM:59 UTC. The module floors all timestamps to :00 before comparing, so 09:15:59 and 09:15:00 are the same minute.

How It Works

For each NSE trading day in range:
  read the day's bars from Parquet  (read_day)
  compare against expected minutes  (check_day -> missing_times)
  group consecutive missing minutes (_find_clusters)
  -> each gap becomes one report row

No state is persisted. check only reads; filldata --apply additionally appends synthetic bars back to the Parquet file (the single source of truth).

Start-date guard

Before scanning, check / filldata look up each script's start date in the DuckDB liquid_scripts table and never scan before it — so dates with no data yet are not falsely flagged as missing. The column used depends on the script kind:

Kind	`scr_master` rule	Start-date column
Spot	`underly = 0`	`spot_start_date`
CFuture (continuous future)	`underly > 0`, `monthindex > 0`	`cfut_start_date`
Future (dated contract)	`underly > 0`, `monthindex = 0`	`fut_start_date`

If the relevant start date is NULL, the full requested range is scanned. When a clamp happens the run prints, e.g.:

  [masid 1] start_date 2016-06-01 -> skipping dates before it (requested from 2007-01-01)

Gap Fill Rules (`FillingMissingTimes`)

filldata delegates per-day filling to FillingMissingTimes in ut_Classes.py. Every filled bar gets svolume = 0 and a negative filled flag identifying how it was produced:

Gap size	Strategy	`filled` flag
1 bar	Carry the adjacent bar. If it is the session-opening bar, fill from the next bar instead of the previous one.	`-1`
2–5 bars	Linear interpolation on close → synthetic OHLC	`-2`
6–20 bars	Scale the previous trading day's pattern to fit the gap	`-3`
>20 bars	Same previous-day pattern scaling	`-4`

Vendor-replacement threshold

Clusters with filled < -3 (i.e. -4 and below) are low-confidence reconstructions and should be replaced with another vendor's data when available.

Whole missing days

The report also flags fully missing trading days (zero bars). These are grouped into consecutive runs and cannot be bar-filled — they need vendor / EOD restore (Restore_Missingdates):

Situation	`filled` flag
Single missing day	`-5`
Run of >1 consecutive missing days	`-6`

They appear as a single row spanning the whole session, e.g.:

    1  2007-01-23  09:55:00  15:29:00    335  -6;WHOLE DAY MISSING - needs vendor/EOD restore

filldata --apply does not fill these (FillingMissingTimes needs adjacent bars); they are reported only.

The Report

Both check and filldata print the same table — one row per consecutive missing-bar gap:

  Missing periods: 24 gap(s) across 20 day(s), 1740 bars total
  (19 intraday gap(s), 5 whole-missing day(s))

  masid        date      from        to   bars  note
  ----------------------------------------------------------------------
      1  2007-01-02  09:55:00  09:55:00      1  -1;Single-bar carry fill
      1  2007-01-02  13:20:00  13:29:00     10  -3;6-20 bar previous-day pattern
      1  2007-01-22  14:00:00  14:24:00     25  -4;>20 bar previous-day pattern (replace with another vendor)
      1  2007-01-23  09:55:00  15:29:00    335  -6;WHOLE DAY MISSING - needs vendor/EOD restore
      1  2007-01-24  09:55:00  15:29:00    335  -6;WHOLE DAY MISSING - needs vendor/EOD restore
      1  2007-01-29  14:00:00  14:04:00      5  -2;2-5 bar linear interpolation
      1  2007-01-31  09:55:00  09:56:00      2  -2;2-5 bar linear interpolation

The summary's second line splits the count into intraday gaps vs whole-missing days. Special trading days get an extra [details start-end] suffix in the note column.

Public API

`check(from_date, to_date, masid=None)`

from nseieod.fix_data import check

check()                                   # all 4 scripts, 2007-2015
check('2010-01-01', '2010-12-31', masid=1)

Returns the clusters as a DataFrame (liquid_masid, trade_date, start_time, end_time, bar_count).

`filldata(from_date, to_date, masid=None, apply=False)`

from nseieod.fix_data import filldata

filldata('2007-01-01', '2007-01-31', masid=1)              # dry run (report only)
filldata('2007-01-01', '2007-01-31', masid=1, apply=True)  # report + write fills

Parameter	Default	Description
`from_date` / `to_date`	`2007-01-01` / `2015-12-31`	Date range `YYYY-MM-DD`
`masid`	`None`	One script (1-4) or all 4
`apply`	`False`	`False` = report only; `True` = write fills to Parquet

Command Line

Run from backend/:

# Check (report only)
python -m nseieod.fix_data check --from 2007-01-01 --to 2007-01-31 --masid 1

# Fill — dry run (report only), then apply
python -m nseieod.fix_data filldata --from 2007-01-01 --to 2007-01-31 --masid 1
python -m nseieod.fix_data filldata --from 2007-01-01 --to 2007-01-31 --masid 1 --apply

Argument	Default	Description
`--from`	`2007-01-01`	Start date
`--to`	`2015-12-31`	End date
`--masid`	all (1–4)	Restrict to one liquid script
`--apply`	off	(`filldata` only) write fills to Parquet

Importable building blocks

from nseieod.fix_data import (
    check, filldata,
    check_day, fill_day, read_day,
    get_expected_times,
    LIQUID_SCRIPTS,
)

get_expected_bars, get_nse_working_days, get_special_day_info and the NSE calendar constants live in nseieod/helpers.py.

Known Limitations & Edge Cases

Situation	Behaviour
Gap at session open with no usable neighbour	`unfillable` — left as-is
Whole day missing (no bars)	Reported as `-5`/`-6`; not bar-filled — needs `Restore_Missingdates` / vendor data
Day with > expected bars	Treated as `complete` (extra = optional 15:30 closing-auction bar)
Special trading day	Expected window comes from the Postgres `specialdays` table

Daily Maintenance (after market close)

from datetime import date
from nseieod.fix_data import check, filldata

TODAY = date.today().isoformat()
check(TODAY, TODAY)                    # review the report
filldata(TODAY, TODAY, apply=True)     # fill any intraday gaps

Why "Liquid Scripts"?​

NSE Trading Hours History​

How It Works​

Start-date guard​

Gap Fill Rules (FillingMissingTimes)​

Whole missing days​

The Report​

Public API​

check(from_date, to_date, masid=None)​

filldata(from_date, to_date, masid=None, apply=False)​

Command Line​

Importable building blocks​

Known Limitations & Edge Cases​

Daily Maintenance (after market close)​