`fix_data.py` — IEOD Gap Filler

Location: backend/nseieod/fix_data.py

Scans the 4 liquid scripts for missing 1-minute bars and fills them using FillingMissingTimes. Everything runs in-memory directly from the Parquet store — there are no DuckDB tracking tables.

Two subcommands, both printing the same missing-period report:

check — scan and print the report (never writes)
filldata — print the report; with --apply, write filled bars back to Parquet

See Quality Checking for the full walkthrough.

Fill Rules (FillingMissingTimes)

Each filled bar is tagged with a negative filled flag identifying how it was produced, and svolume is always set to 0.

Gap Size	Method	`filled` flag
1 bar	Carry the adjacent bar. If the missing bar is the session opening bar (`MarketOpeningTime`), fill from the next bar instead of the previous one.	`-1`
2–5 bars	Linear interpolation on close between prev & next, then synthetic OHLC built from the interpolated close path	`-2`
6–20 bars	Scale the previous trading day's intraday pattern to fit the gap	`-3`
>20 bars	Same previous-day pattern scaling	`-4`

Vendor-replacement threshold

Any cluster with filled < -3 (i.e. -4, -5, -6) is a low-confidence reconstruction and should be replaced with data from another vendor when available.

Session open/close times come from helpers.MarketOpeningTime / MarketClosingTime (special-day aware), and the previous trading day is resolved via helpers.get_nse_working_days (skips weekends & holidays).

Whole-Day Restore (Restore_Missingdates)

Fully missing trading days are flagged in the check / filldata report (grouped into consecutive runs) but are not bar-filled — they are restored separately by Restore_Missingdates in ut_Classes.py:

Source	`filled` flag
Real bars from a secondary vendor file	`0` (real data)
EOD-pattern reconstruction, single missing day	`-5`
EOD-pattern reconstruction, >1 consecutive missing days	`-6`

In the report a whole-missing day shows as one full-session row:

    1  2007-01-23  09:55:00  15:29:00    335  -6;WHOLE DAY MISSING - needs vendor/EOD restore

EOD reconstruction reads the day's Open/High/Low/Close from scr_nseeq_eod / scr_nsefo_eod and warps the previous day's intraday shape onto those anchors (Open → first extreme → second extreme → Close). All reconstructed bars use svolume = 0.

Usage

cd backend
venv\Scripts\activate

# Check — report only, never writes
python -m nseieod.fix_data check --from 2007-01-01 --to 2007-01-31 --masid 1

# Fill — dry run (report only)
python -m nseieod.fix_data filldata --from 2007-01-01 --to 2007-01-31 --masid 1

# Fill — actually write to Parquet
python -m nseieod.fix_data filldata --from 2007-01-01 --to 2007-01-31 --masid 1 --apply

CLI Arguments

Argument	Default	Description
`--from`	`2007-01-01`	Start date
`--to`	`2015-12-31`	End date
`--masid`	all (1–4)	Restrict to one liquid script
`--apply`	off	(`filldata` only) write fills to Parquet; omit for dry-run

Report Output (example)

  Missing periods: 24 gap(s) across 20 day(s), 1740 bars total
  (19 intraday gap(s), 5 whole-missing day(s))

  masid        date      from        to   bars  note
  ----------------------------------------------------------------------
      1  2007-01-02  09:55:00  09:55:00      1  -1;Single-bar carry fill
      1  2007-01-02  13:20:00  13:29:00     10  -3;6-20 bar previous-day pattern
      1  2007-01-22  14:00:00  14:24:00     25  -4;>20 bar previous-day pattern (replace with another vendor)
      1  2007-01-23  09:55:00  15:29:00    335  -6;WHOLE DAY MISSING - needs vendor/EOD restore
      1  2007-01-29  14:00:00  14:04:00      5  -2;2-5 bar linear interpolation

With filldata --apply, a one-line confirmation follows:

  Filled 65 bar(s) across 15 day(s).

What Happens on `--apply`

Re-checks the day with check_day (exact minute comparison)
Generates synthetic bars via FillingMissingTimes
Timestamps set to HH:MM:59 UTC (GDFL bar-end convention)
Appends filled rows with their filled flag (-1 … -4, see Fill Rules)
De-duplicates and sorts by sdate
Writes the combined DataFrame back to the same Parquet file (single source of truth)

Filled Bar Flags

Filled bars carry a negative filled flag in the Parquet schema so they can be identified, audited, or excluded from analysis:

`filled`	Meaning
`0`	Real bar (original GDFL data, or secondary-vendor restore)
`-1`	Single-bar carry fill
`-2`	2–5 bar linear interpolation
`-3`	6–20 bar previous-day pattern
`-4`	>20 bar previous-day pattern (replace with another vendor)
`-5`	Whole single day rebuilt from EOD OHLC pattern
`-6`	Whole multi-day run rebuilt from EOD OHLC pattern

df = pd.read_parquet(path)
real_bars   = df[df["filled"] == 0]
filled_bars = df[df["filled"] < 0]
low_quality = df[df["filled"] < -3]   # candidates for vendor replacement

Recommended Workflow

# 1. Check first — review the report
python -m nseieod.fix_data check    --from 2007-01-01 --to 2015-12-31

# 2. Dry-run the fill (same report)
python -m nseieod.fix_data filldata --from 2007-01-01 --to 2015-12-31

# 3. Apply
python -m nseieod.fix_data filldata --from 2007-01-01 --to 2015-12-31 --apply

Quality Checking — full guide to check / filldata
ut_Classes.py — FillingMissingTimes (bar gaps) and Restore_Missingdates (whole days)

Fill Rules (FillingMissingTimes)​

Whole-Day Restore (Restore_Missingdates)​

Usage​

CLI Arguments​

Report Output (example)​

What Happens on --apply​

Filled Bar Flags​

Recommended Workflow​

Related​