PC ↔ VPS Sync Process

Two pieces of state live outside git because they are too large / binary to track:

What	Size	Local path	VPS path
nseieoddata (Parquet store)	~30 GB	D:\kss\stock\tradeentry-VPS\nseieoddata	/home/kss/tradeentry/nseieoddata
tedb (PostgreSQL database)	several GB	Local Postgres 17 instance	VPS Postgres (Docker, postgres:16)

They need different tools because they're different kinds of state: flat Parquet files vs. a live database engine.

State	Tool	Why
nseieoddata	rsync over SSH	Delta-transfer — only sends changed bytes, safe to re-run anytime
tedb	pg_dump (plain SQL) → transfer → psql restore	Postgres-native, safe migration method; raw file-level copy of a live DB risks corruption

The two syncs are NOT interchangeable order — DB always goes first

The local PC is the single source of truth for this workflow: everything ends up mirrored from the PC, and orphan.py is never run on the VPS. That only works safely if the database is synced before the Parquet store is mirrored with --delete. See Part 4 for the full reasoning — the short version: --delete decides what to remove by comparing against the local Parquet folder only, with no awareness of what the VPS's database currently references. If the VPS's database still has rows the PC doesn't (which is normal any time the two have drifted), deleting their Parquet data first breaks the live VPS and risks silently destroying VPS-only historical data that was never re-added locally. Swap the database first, and the same --delete comparison becomes safe and correct.

---

Part 1 — One-Time Environment Setup

Install WSL (Windows side)

rsync isn't available natively on Windows, so we use WSL (Windows Subsystem for Linux) to get a real Linux rsync binary that can read the Windows drive directly via /mnt/d/....

wsl --install

Restart the PC, then launch Ubuntu (wsl from PowerShell) and create a Linux username/password on first run.

Install rsync and an SSH client inside Ubuntu:

sudo apt update
sudo apt install -y rsync openssh-client tmux

Disk footprint

WSL2 + Ubuntu uses roughly 1-2 GB (a sparse, dynamically-growing virtual disk). It does not copy nseieoddata into the Linux filesystem — rsync reads/writes the Windows files directly through the /mnt/d/... mount.

SSH key access from WSL

The Windows OpenSSH key (C:\Users\<user>\.ssh\id_ed25519) needs to be copied into WSL's home directory with stricter permissions — Linux SSH refuses keys that are "too open" (a common gotcha when keys are read directly off the Windows-mounted drive).

mkdir -p ~/.ssh
cp /mnt/c/Users/<user>/.ssh/id_ed25519 ~/.ssh/
cp /mnt/c/Users/<user>/.ssh/id_ed25519.pub ~/.ssh/
cp /mnt/c/Users/<user>/.ssh/config ~/.ssh/
chmod 600 ~/.ssh/id_ed25519
chmod 644 ~/.ssh/id_ed25519.pub ~/.ssh/config

Test the connection:

ssh <vps-ip> "echo connected ok"

On first connect you'll be asked to verify the host key fingerprint — type the full word yes (not just y).

Long transfers — use tmux

Both the database dump and the full Parquet sync can take a while over the network. Wrap either one in tmux so it survives a dropped connection or a closed terminal window:

tmux new -s sync_job
# ... run the long command inside ...
# detach (leave it running):     Ctrl+b  then  d
# reattach later:                 tmux attach -t sync_job
# list sessions:                  tmux ls

Don't nest tmux sessions

If tmux attach says sessions should be nested with care, unset $TMUX to force, you're already inside that session — you don't need to attach, you're looking at it. Check with echo $TMUX and tmux display-message -p '#S'.

Closing the WSL window can kill the session

Closing the WSL terminal window sometimes tears down the whole WSL instance (not just that shell), which takes any tmux sessions with it — tmux attach will report no sessions. If that happens, nothing is corrupted (rsync/pg_dump are safe to re-run from scratch); just start a fresh tmux new -s sync_job and re-run the command.

---

Part 2 — Syncing tedb (PostgreSQL) — always do this FIRST

Postgres data files can't be safely rsync'd while the server is running (risk of reading a half-written page). The supported way to move a Postgres database is pg_dump + restore.

Plain SQL format, not custom format

Local Postgres is 17.x; the VPS runs Postgres in Docker as postgres:16. A custom-format dump (-F c) made by pg_dump 17 will not pg_restore on the VPS — the archive header version doesn't match. Plain SQL format (-F p) sidesteps this entirely, so that's what backup_tedb.sh uses. Don't switch back to -F c for VPS transfers.

1. Dump + ship, via backup_tedb.sh

Run from WSL, from the repo root:

cd /mnt/d/kss/stock/tradeentry-VPS
./backup_tedb.sh

This script (see backup_tedb.sh at the repo root):

• Dumps local tedb with pg_dump -F p, gzips it, and timestamps it under D:\kss\postgrebackup
• Keeps a rolling window of the last 5 local backups
• Rsyncs the whole backup folder (no --delete — the VPS keeps full history even as local rotates) to /home/kss/backups/ on the VPS

It reads DB connection details from backend/.env, so there's nothing to hand-edit.

2. Back up the VPS's current database first

Before touching anything on the VPS, take a safety-net dump of its current tedb — this is what lets you undo the swap if something looks wrong afterward:

ssh kss@<vps-ip>
pg_dump -h 127.0.0.1 -U postgres -d tedb -F p | gzip > /home/kss/backups/tedb_vps_before_swap_$(date +%Y%m%d_%H%M%S).sql.gz

(-h 127.0.0.1 is required — the VPS's Postgres runs in Docker and only exposes a TCP port, not the default Unix socket psql looks for.)

3. Restore into a fresh database, verify, then swap names

Restoring directly on top of the live tedb is a one-way door. Restore into a new database first, spot-check it, and only then swap it in — if anything looks wrong, tedb was never touched:

# still on the VPS
psql -h 127.0.0.1 -U postgres -c "CREATE DATABASE tedb_new;"
gunzip -c /home/kss/backups/tedb_<TIMESTAMP>.sql.gz | psql -h 127.0.0.1 -U postgres -d tedb_new

# sanity check row counts before swapping anything in
psql -h 127.0.0.1 -U postgres -d tedb_new -c "
  SELECT 'scr_master' AS tbl, COUNT(*) FROM scr_master
  UNION ALL SELECT 'scr_nseeq_eod', COUNT(*) FROM scr_nseeq_eod
  UNION ALL SELECT 'scr_nsefo_eod', COUNT(*) FROM scr_nsefo_eod
  UNION ALL SELECT 'holidays', COUNT(*) FROM holidays;
"

ALTER DATABASE ... RENAME refuses to run while anything is connected to that database — stop the backend service first:

sudo systemctl stop tradeentry

psql -h 127.0.0.1 -U postgres -c "ALTER DATABASE tedb RENAME TO tedb_old;"
psql -h 127.0.0.1 -U postgres -c "ALTER DATABASE tedb_new RENAME TO tedb;"

4. Deploy code and restart the service

bash ~/pull.sh

pull.sh (repo root) does git pull, detects that backend files changed, and restarts the tradeentry systemd service for you — which also covers starting the service back up after the stop above. See pull.sh's own header comment for what it does step by step; it's a separate tool from this doc's scope (code deploy vs. data sync) but is always the next step after a DB swap.

Once you've confirmed everything on the VPS looks healthy for a few days, tedb_old (and the pre-swap safety dump) can be dropped/deleted to reclaim space — not before.

---

Part 3 — Syncing nseieoddata (Parquet Store) — always do this SECOND

Only run this after Part 2's database swap has completed. See Part 4 for why.

rsync's delta-transfer algorithm only sends bytes that actually changed. Historical Parquet files never change once written, so after the first sync, subsequent syncs are fast regardless of total size.

Always dry-run first

rsync -avzn -i --delete \
  /mnt/d/kss/stock/tradeentry-VPS/nseieoddata/ \
  kss@<vps-ip>:/home/kss/tradeentry/nseieoddata/ > /tmp/dryrun.log

grep -c '^>f'  /tmp/dryrun.log   # files that would actually transfer data
grep -c '^cd'  /tmp/dryrun.log   # brand-new directories
grep '^deleting' /tmp/dryrun.log # everything that would be REMOVED from the VPS -- read this list

-n (dry-run) and -i (itemize-changes) together tell you exactly what would happen before anything is sent. Read the deleting lines every time, not just the counts — that's the destructive part.

Real sync

rsync -avz --delete --partial --progress --stats \
  -e "ssh -o ServerAliveInterval=15 -o ServerAliveCountMax=3" \
  /mnt/d/kss/stock/tradeentry-VPS/nseieoddata/ \
  kss@<vps-ip>:/home/kss/tradeentry/nseieoddata/

Flag	Meaning
-a	Archive mode — preserves timestamps/permissions, needed for delta detection
-v	Verbose
-z	Compress during transfer (big win on slow links)
--delete	Mirrors deletions — removes files on the VPS that no longer exist locally
--partial	Keeps a partially-transferred file if the connection drops mid-file, so the next run resumes it instead of starting that file over from 0%
--progress	Per-file progress
--stats	Summary block at the end (files transferred, bytes sent, etc.)
-e "ssh -o ServerAliveInterval=15 -o ServerAliveCountMax=3"	Makes SSH ping the connection every 15s and give up after 3 missed pings (~45s) instead of hanging silently forever on a dead connection

A "running" transfer can be silently dead

A real incident: a sync sat for 13 hours looking alive in ps aux (normal-looking sleep state, low CPU — expected for a network wait) while actually transferring zero bytes the whole time, because the network connection dropped and plain SSH never noticed. The -e "ssh -o ServerAlive..." flags above fix this going forward. If a sync ever feels stuck, verify it's actually moving data before waiting it out:

cat /proc/<rsync-pid>/io | grep -E 'rchar|wchar'
sleep 5
cat /proc/<rsync-pid>/io | grep -E 'rchar|wchar'

If the numbers don't change, it's dead — kill <pid> (find it with ps aux | grep rsync, there will be two: the local rsync and an ssh ... rsync --server helper, kill both) and restart the same command. --partial means you don't lose everything already transferred.

For the full plain-language breakdown of every flag (including the dry-run ones above) and how to read rsync's output, see rsync Explained.

Trailing slashes matter

Both paths end in /. That syncs the contents of nseieoddata into the remote folder, not a nested nseieoddata/nseieoddata/ copy.

_catalog/catalog.db is included by design

This sync intentionally does not exclude _catalog/catalog.db — the PC is meant to be the single source of truth for liquid_scripts/name_changes too. One side effect: catalog.db's parquet_file_index table (an internal cache used by orphan.py) stores Windows-style file paths, which won't match anything on the VPS's Linux filesystem. This is harmless and self-correcting — since orphan.py is never run on the VPS in this workflow, it simply never gets used there. If that ever changes, expect one wasted full re-index the first time orphan.py runs on the VPS after a sync.

Run this same command anytime you add/remove data locally (new bhav downloads, GDFL/EODIEOD packs, liquid-script fills, etc.) — just always run Part 2 first if the database has also changed.

---

Part 4 — Why Order Matters

This is the one rule that matters most in this whole doc: sync the database before you --delete-sync the Parquet store.

rsync --delete decides what to remove purely by comparing "does this file exist in my local source folder" — it has no idea what the VPS's database currently references. That comparison is only correct once the VPS's database is guaranteed to match the local one (i.e., right after Part 2's swap).

If you run the --delete Parquet sync before the database swap:

1. It can break the live VPS right now. If the VPS's current database still has a row for some masid the PC doesn't (which happens naturally as the two drift between syncs), that row is still "live" as far as the VPS's running backend is concerned. Deleting its Parquet data out from under it means anything querying that masid on the VPS starts returning empty/missing data immediately — a real functional regression, not a cleanup.
2. It can permanently destroy VPS-only data. The PC and VPS databases can each accumulate data the other doesn't have (a real example hit during this process: the VPS had 2016 India VIX bars the PC's copy was missing entirely). If a masid on the VPS holds a date range that was never captured locally, deleting it removes the only copy that ever existed — there's nothing to restore it from.

Swapping the database first (Part 2) means that by the time --delete runs, "not present in the local Parquet folder" and "no longer has a row in the VPS's database" are the same true statement — so the deletion is unambiguous and safe.

---

Part 5 — Verifying the Sync

Files (nseieoddata)

A clean dry-run with 0 under ^>f (real file transfers), 0 under ^cd (new directories), and no deleting lines confirms the two copies are byte-identical.

Database (tedb)

Don't trust pg_stat_user_tables right after a restore

n_live_tup is a statistics-collector estimate that only updates after ANALYZE or autovacuum runs. Right after a restore it can show 0 rows on a table that's actually full — this looks like data loss but isn't.

Use real counts instead (same query as Part 2 step 3, run again post-swap):

SELECT 'scr_master' AS tbl, COUNT(*) FROM scr_master
UNION ALL SELECT 'scr_nseeq_eod', COUNT(*) FROM scr_nseeq_eod
UNION ALL SELECT 'scr_nsefo_eod', COUNT(*) FROM scr_nsefo_eod
UNION ALL SELECT 'holidays', COUNT(*) FROM holidays;

Run the same query on PC and VPS and compare — matching counts confirm the databases are in sync.

---

Part 6 — Routine Incremental Sync (day-to-day)

Parts 2–3 are the full-mirror procedure — needed for the first migration or after major divergence. Day to day, use the incremental tools instead; they move only what changed, in minutes not hours.

The multi-node rule: VPS is the hub

Three nodes exist: Home PC (dev + sometimes ingests), VPS (production, sometimes ingests), Office PC (read-only mirror for programmers). The rule that keeps them from diverging:

• VPS is always the single current copy. Everything else pulls FROM it.
• Home PC pushes up to the VPS right after doing new work (backfills, ingestion done locally) — and never treats its own copy as current once the VPS has moved past it.
• Office PC only pulls from the VPS (nightly cron or on demand). It never ingests, never pushes.
• Never ingest the same trading day on two nodes independently. Whichever node runs a day's EODIEOD/bhav ingestion, that's the only place it happens; the sync afterward brings the others up to date. Check the EODIEOD Upload page's "Next pending working day" (or pipeline.get_checkpoint()) before starting ingestion anywhere.

Parquet — plain rsync, direction depends on where ingestion ran

# Home PC ingested -> push up to VPS (no --delete needed for routine adds)
rsync -avz --partial --progress --stats \
  -e "ssh -o ServerAliveInterval=15 -o ServerAliveCountMax=3" \
  /mnt/d/kss/stock/tradeentry-VPS/nseieoddata/ \
  kss@<vps-ip>:/home/kss/tradeentry/nseieoddata/

# VPS ingested -> pull down to Home PC (or Office PC; just swap the local path)
rsync -avz --partial --progress --stats \
  -e "ssh -o ServerAliveInterval=15 -o ServerAliveCountMax=3" \
  kss@<vps-ip>:/home/kss/tradeentry/nseieoddata/ \
  /mnt/d/kss/stock/tradeentry-VPS/nseieoddata/

Postgres — incremental_pg_sync.sh (repo root)

No full dump/restore needed for routine catch-up:

./incremental_pg_sync.sh push   # Home PC -> VPS   (after local ingestion)
./incremental_pg_sync.sh pull   # VPS -> Home PC   (after VPS ingestion)

How it decides what to move:

• scr_master — uses the crea_date/modi_date audit columns (added 2026-07-13, see nseeod/CLAUDE.md rule 4): pulls only rows whose modi_date is newer than the destination's newest, upserts by masid.
• scr_nseeq_eod / scr_nsefo_eod — no modi_date on these 30M+-row tables. Instead: rewind a safety window (default 30 days, SAFETY_WINDOW_DAYS=45 ./incremental_pg_sync.sh push to override) behind the destination's newest sdate, delete that range on the destination, reload it fresh from the source — inside ONE transaction, so a dropped connection rolls back cleanly instead of leaving the range deleted-but-empty. The window also re-captures corrections to recent dates (the downloadbhav.py rollback-and-redownload pattern can rewrite an already-synced date).

Never run push and pull at the same time, and never run either while an ingestion is actively writing on either side.

---

Part 7 — Deploying Code: ONE Command

ssh kss@<vps-ip>
bash ~/tradeentry/pull.sh

That's the whole deploy. pull.sh (repo root) automates everything that used to be manual — each step runs ONLY when the relevant files actually changed:

Step	Trigger	What it runs
Git pull	always	git pull origin main
Self-update	pull.sh itself changed	re-executes the NEW pull.sh before doing anything else
Backend deps	any requirements*.txt changed	uv pip install -r backend/requirements.txt into backend/venv (pip fallback if uv missing)
Frontend	frontend/** changed	npm install + npm run build
Docs	docs/** changed	npm install + npm run build
Service restart	backend OR frontend changed	sudo systemctl restart tradeentry (after the builds, so it serves fresh output)

Why this exists — the 2026-07-14 loguru incident

A web-page EODIEOD upload appeared to "hang" on the VPS with zero output. Hours of debugging later, the actual cause was ModuleNotFoundError: No module named 'loguru' — crashing the background task instantly, before it could write any status. The dependency gap had TWO layers, which is why it survived a deploy:

1. pull.sh never installed Python packages at all — it only restarted the service.
2. The service startup (uv run app.py --no-build) DOES install backend/requirements.txt on every restart — but that file never listed loguru (or fyers-apiv3, redis, python-multipart, alembic), and was UTF-16-corrupted on top of it. So the install step ran and installed nothing new, every time.

The same class of gap had already hit twice the same night: the frontend served a stale build after a deploy (fixed by the frontend step above), and unrar was missing as a system package (apt-installed manually — see the deployment guide's system-packages step; pull.sh deliberately does not install apt packages).

Rules to keep it working

• backend/requirements.txt is THE server manifest. The root requirements.txt is the full local-dev environment (includes Windows-only/Jupyter packages that would fail on Linux) — the VPS never installs it. When backend code gains a new import, add it to backend/requirements.txt in the same commit.
• Keep both requirements files UTF-8. PowerShell's pip freeze > requirements.txt writes UTF-16, which silently breaks text tooling (a UTF-16 file greps as binary garbage). From PowerShell use pip freeze | Out-File requirements.txt -Encoding utf8, or run the freeze from WSL/cmd instead.
• New system-level tools (apt packages like unrar) still need a manual install once per VPS — and a note in the deployment guide's system-packages step.

---

Troubleshooting

Symptom	Cause	Fix
Web-triggered pipeline "hangs" at 0%, status stays READY, Run History empty	A background task crashed on import (ModuleNotFoundError, e.g. the 2026-07-14 loguru incident) BEFORE it could write any status	Run the same pipeline once via CLI on the VPS (venv/bin/python -m nseieod.eodieod.run pipeline --date ...) — the traceback prints directly; then fix backend/requirements.txt and redeploy (Part 7)
Pipeline stalls at the extract step, or "No RAR extraction tool found"	unrar is a system package — pull.sh installs Python/npm deps only, never apt packages	sudo apt install unrar (once per VPS); verify with which unrar
grep finds nothing in requirements.txt even though the package is listed / file prints as spaced-out letters	File saved as UTF-16 (PowerShell's pip freeze > file default) — text tools see binary garbage	Convert once: iconv -f UTF-16LE -t UTF-8 (or re-freeze with pip freeze | Out-File -Encoding utf8); check with file requirements.txt
./script.sh fails with $'\r': command not found in WSL	File has Windows CRLF line endings (saved by a Windows editor)	sed -i 's/\r$//' <file> — recheck with file <file> (should not say "with CRLF")
Uploads fail with generic 413 in the browser	nginx client_max_body_size (default 1MB) rejects it before FastAPI ever sees it	See the deployment guide's 413 entry — client_max_body_size 350M; in the /api/ block
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed	VPS Postgres runs in Docker — no local Unix socket	Add -h 127.0.0.1 to every psql/pg_dump call on the VPS
ALTER DATABASE ... RENAME hangs or errors "database is being accessed by other users"	The tradeentry service still holds a connection	sudo systemctl stop tradeentry first, restart it after (or let pull.sh do it)
Start-Service fails: "Cannot open ... service on computer '.'"	PowerShell isn't running as Administrator	Right-click PowerShell → "Run as Administrator" (a normal launch, even from a pinned icon, is not enough), or use services.msc → right-click → Start
Postgres won't start: FATAL: could not reattach to shared memory ... error code 1455	Windows page file too small / system low on free RAM at that moment	Close memory-heavy apps and retry; if it persists, increase the page file size or reboot to clear fragmented virtual memory
tmux attach says "no sessions" after closing the WSL window	Closing the window can tear down the whole WSL instance, not just that shell	Not harmful — tmux new -s sync_job and re-run the command; rsync/pg_dump are safe to resume/redo
tmux new -s <name> immediately shows [exited]	Something in WSL's shell startup (.bashrc etc.) is erroring out	Skip tmux for that run and execute the command directly in the plain WSL shell, keeping the window open until it finishes
Ctrl+C doesn't seem to stop a running transfer	Rare — usually it does; if a long-running Python-side script seems unresponsive to it, let it finish rather than force-killing mid-write	N/A — specific to custom scripts, not rsync itself
pg_restore: error: ... database "tedb" does not exist	Forgot to CREATE DATABASE before restoring	Run CREATE DATABASE tedb_new; first (see Part 2)
WARNING: database "postgres" has a collation version mismatch	OS collation library updated (e.g. Windows Update)	ALTER DATABASE <name> REFRESH COLLATION VERSION; on template1 and postgres
SSH host-key prompt rejects input	Typed y instead of the full word	Type yes and press Enter
n_live_tup shows 0 after restore	Stale autovacuum statistics	Use SELECT COUNT(*) instead, or run ANALYZE;
Private key "UNPROTECTED PRIVATE KEY FILE" error in WSL	Key copied from /mnt/c/... keeps loose Windows permissions	Copy the key into ~/.ssh/ inside WSL and chmod 600 it