Skip to main content

downloadbhav.py — BhavProcessor

The orchestrator for the entire bhav download pipeline. All REST API calls ultimately land here via FastAPI BackgroundTasks.

Class: BhavProcessor​

Constructor​

bp = BhavProcessor()
  • Loads existing stats.json from CSV_PATH (if present)
  • Initialises NseEodEqUpdater and NseEodFoUpdater
  • Sets testingMode = True (process only 1 day per run — set to False for full sync)

Methods​

run_update()​

Full sync — downloads all missing dates since last DB record.

Critical rule: last_safe_date is captured before any DB writes so rollback is safe.

run_update_for_date(target_date: date)​

Force-downloads and processes a single specific date.

  1. Downloads delivery volume for target_date
  2. Runs Equity.update_a_day(target_date)
  3. Runs Equity.indexupdater.update(target_date)
  4. Runs Fo.update_a_day(target_date, holidays)
  5. Calls _generate_ami(target_date)
  6. On failure: calls deleteadate(target_date) to roll back

get_database_status() → dict​

{
    "latest_trading_date": date,   # Last NSE working day
    "eq_date": date | None,        # MAX(sdate) from scr_nseeq_eod
    "fo_date": date | None,        # MAX(sdate) from scr_nsefo_eod
    "is_up_to_date": bool          # Both eq and fo >= latest
}

get_latest_trading_date() → date​

Returns the most recent NSE working day. If current time < 17:00, uses yesterday's date (bhav copies published after market close).

AddLog(message, source)​

Appends to stats["Logs"] list and writes stats.json. Keeps last 100 logs. Prints to console with format [BhavDownloader][{source}] {message}.

ASCII-only messages

On Windows (cp1252), Unicode characters (→, …, ✓) in log messages crash print(). Always use ASCII: -> not →, ... not ….

AddHistory(date_label, status, message)​

Appends to stats["history"]. Status values: SUCCESS, FAILED, Started, Running, UP_TO_DATE. Any status not in that allowed set sets has_errors = True. Keeps last 50 records.

UpdateState(key, value)​

Convenience wrapper: self.stats[key] = value then savetojson().

updatecount()​

Increments current, recalculates progress percentage (capped at 99 until done), saves stats.json.

Standalone Utility Functions​

deleteadate(mdate)​

deleteadate(datetime.date(2026, 5, 22))

Deletes all rows for exactly mdate from all 5 core tables: scr_missing_bhav, scr_missing_eod, scr_nseeq_eod, scr_nsefo_eod, scr_indexpepb.

Used for: rollback of a single failed date.

deleteadateabove(mdate)​

deleteadateabove(datetime.date(2026, 5, 20))

Deletes all rows where sdate > mdate from all 5 tables.

Used for: crash-recovery rollback to restore DB to a clean pre-crash state.

updateHolidays()​

Calls ut.updateholidays(Config.CSV_PATH) — reads Holidays.csv and Muhurat.csv from CSV_PATH and reloads the holidays and specialdays tables.

GetHolidays()​

Calls ut.SaveHolidays(Config.CSV_PATH) — exports current holiday data to CSV files.

stats.json Structure​

Written to {CSV_PATH}/stats.json after every state change:

{
    "status": "RUNNING | SUCCESS | FAILED | READY",
    "progress": 0,
    "date": "2026-05-22",
    "last_log": "Last log message",
    "LastEqBhav": "2026-05-22",
    "LastFOBhav": "2026-05-22",
    "EqStat": "Up to Date",
    "FOStat": "Up to Date",
    "Logs": [
        {
            "message": "...",
            "source": "NseEodEqUpdater",
            "date": "2026-05-22",
            "timestamp": "05:30:00 PM"
        }
    ],
    "history": [
        {
            "date": "2026-05-22",
            "timestamp": "2026-05-22 05:30:00 PM",
            "status": "SUCCESS",
            "message": "Equity, delivery, and indices synced successfully."
        }
    ]
}

The frontend polls GET /api/bhav-downloader/status every 1500 ms which reads this file.