Skip to main content

utils.py — Shared Helpers

The largest file in the package. Contains NSE calendar utilities, file downloaders, bhav parsers, and DB write helpers used by all other modules.

Function Map​

Calendar Functions​

NseHolidays() → list[str]​

Returns all NSE holidays from the holidays table as YYYY-MM-DD strings.

NSEWorkingDays(fromdate, todate) → DataFrame​

Builds a pandas date range excluding NSE holidays, then appends Muhurat trading days. Returns DataFrame with sdate as both column and index.

wd = NSEWorkingDays(date(2026, 5, 1), date(2026, 5, 31))
# returns DataFrame with all NSE working days in May 2026

get_next_nseworkingdate(fromdate) → date​

Returns the next NSE working day after fromdate using a 20-day window.

lastupdateDate() → date | None​

Returns min(MAX(eq_sdate), MAX(fo_sdate)) - 1 day. Used to establish the safe rollback point before any DB writes.

HolidayExpiries(fdate, tdate) → list[date]​

Returns list of Thursdays that fall on NSE holidays (these are expiry adjustment dates). F&O expiry is moved backward when the last Thursday is a holiday.

getThursdays(mDate, mExpi) → DataFrame​

Returns a DataFrame of the next 16 weekly Thursdays with Roman numeral suffixes (-I, -II, -III) identifying monthly expiry order. Used to build adjusted futures mname.

to_roman(num) → str​

Converts integer to Roman numeral string (1→I, 2→II, 3→III, etc.).

File Download Functions​

getnsebhavzip(url, mtargetfolder, parent) → bool​

Downloads and extracts a ZIP file from NSE archives. 3 retries with 1s sleep. Defined twice in the file (bug — see junk analysis). Both versions are functionally identical.

getnsefile(url, target, parent) → bool​

Downloads a plain CSV/text file using the niftyindices.com session headers. Validates response is not an HTML error page before writing to disk. 3 retries.

requestnsefile(url, target, parent) → bool​

Simpler version of getnsefile without HTML validation. Used for delivery volume .DAT files.

download_eqbhavcopy(mDate, mpath, parent) → bool​

Downloads old-format equity bhav ZIP. Handles the edge case where NSE extracts the ZIP into a folder with the same name as the target CSV.

download_delivary_volume_file(mDate, mpath, parent) → bool​

Downloads MTO_DDMMYYYY.DAT (delivery volume data).

getnseFObhavcopy(mDate, mpath, parent) → bool​

Downloads old-format F&O bhav ZIP.

get_nseindex_info(mDate, mpath, parent) → bool​

Downloads ind_close_all_DDMMYYYY.csv from niftyindices.com.

Filename Builder Functions​

getnsefilaname(mDate, prefix, wzip=False) → str​

Builds old-format NSE filename. Examples:

  • getnsefilaname(date(2026,5,22), 'cm') → cm22MAY2026bhav.csv
  • getnsefilaname(date(2026,5,22), 'cm', True) → cm22MAY2026bhav.csv.zip

getnse_new_filaname(mDate, prefix, wzip=False) → str​

Builds new-format NSE filename (post-April 2025). Examples:

  • getnse_new_filaname(date(2026,5,22), 'cm') → BhavCopy_NSE_CM_0_0_0_20260522_F_0000.csv

Bhav Parser Functions​

process_and_update_EqBhav(mDate, conn, bhavfile, mDelPath, parent)​

Parses old-format equity CSV. Filters SERIES ∈ ['EQ', 'BE']. Renames columns to match DB schema. Merges delivery data. Calls match_scr_masters_and_update.

new_process_and_update_EqBhav(mDate, conn, bhavfile, mDelPath, parent)​

Parses new-format equity CSV (TradDt, TckrSymb, etc.). Same logic as above.

getfodf(mDate, mPath) → (DataFrame, str)​

Reads old-format F&O bhav CSV, returns DataFrame + source file path.

new_getfodf(mDate, mPath) → (DataFrame, str)​

Reads new-format F&O bhav CSV. Maps new column names to old internal names. Maps new instrument codes (STO→OPTSTK, STF→FUTSTK, IDO→OPTIDX, IDF→FUTIDX).

updatefobhav(mDate, mPath, conn, exps, parent)​

Full F&O processing pipeline:

updateindexbhav(mdate, mpath, conn, parent)​

Processes index snapshot CSV:

  1. Reads Indexinfo.csv for name mapping (Old→New index names)
  2. Merges with scr_master to resolve masid
  3. Inserts new index symbols into scr_master if needed
  4. Writes OHLCV to scr_nseeq_eod via updatetable_from_df
  5. Writes PE/PB/Div to scr_indexpepb

DB Helper Functions​

match_scr_masters_and_update(conn, table, df, typ, mdate, parent)​

Resolves symbol names to masid via scr_master. Inserts new symbols if needed. Writes data to table.

match_scr_masters_and_fo_update(conn, table, df, mdate, parent, Opt=False)​

Same as above but for F&O. Handles options separately (Opt=True) because options have additional columns (strike, opttyp).

match_scr_masters_and_underlying(conn, df, parent)​

Resolves F&O underlying symbols (NIFTY, BANKNIFTY etc.) from scr_master where typ<=2. Used before processing F&O to link each contract to its underlying security.

get_indexinfo_path(mpath) → str​

Finds Indexinfo.csv with 4 fallback locations:

  1. nseeod/Indexinfo.csv (canonical — committed to git)
  2. backend/Indexinfo.csv
  3. Parent of mpath
  4. mpath itself

Task System Functions​

restore_tasks(typ) → dict | None​

Reads task JSON from tasks table for the given typ.

save_task(typ, notes, pending, conn)​

Upserts task JSON into tasks table (INSERT or UPDATE based on typ).

Duplicate definition

save_task() is defined twice in utils.py (lines 164 and 401). Both are identical. The second definition silently overwrites the first. See Junk Analysis.

add_nse_site_task(masterKey, issuedate, issuedetails, update_date)​

Tracks rare NSE site anomalies (e.g., ZIP extracting into a folder). Stored in tasks table with typ=4.

Miscellaneous​

get_single_value_from_sql(mSql) → any | False​

Wrapper around pgs.getsqlValue that returns the first column of the first row.

clean_float(val) → int | float​

Returns int(val) if value is a whole number, otherwise returns float. Not used in main flow.

SaveHolidays(path)​

Exports holidays and specialdays tables to CSV files in path.

updateholidays(mpath)​

Reads Holidays.csv and Muhurat.csv from mpath and reloads the DB tables.