Skip to main content

rsync Explained

This is a plain-language companion to the PC ↔ VPS Sync Process doc — every flag used there, explained on its own, plus how to read what rsync prints while it runs.

The one idea that matters​

rsync copies files from one place to another, but it's smart about it: if a file already exists on the destination and hasn't changed, it skips it entirely. If a file changed slightly, it only sends the changed bytes, not the whole file again. That's why the first sync of a large folder is slow, and every sync after that is fast — as long as most files haven't changed.

It is safe to re-run the exact same command as many times as you want. If it gets interrupted (closed terminal, dropped connection, Ctrl+C), just run it again — it picks up from wherever it left off, nothing gets corrupted by re-running it.

The flags, one at a time​

-a — archive mode​

Shorthand for "copy everything about this file that matters": contents, timestamps, permissions, and recurse into subfolders. Without -a, rsync might copy file contents but forget timestamps — which breaks its own ability to detect "did this file change?" on the next run. Always use -a.

-v — verbose​

Prints the name of every file as it's handled. Without this, rsync runs silently and just shows a final summary.

-z — compress during transfer​

Squeezes the data smaller while it's traveling over the network, then unpacks it on the other end. Helps most on slow connections; costs a little CPU time on both ends. Worth it for our case (many files over the internet, not a local network).

-n — dry run​

Simulates the whole operation and changes nothing. Prints exactly what it would do — same output format as a real run — but no bytes are actually sent, nothing is deleted. Always run with -n first before any command that includes --delete.

-i — itemize changes​

Instead of just a filename, prints an 11-character code in front of every file explaining why it's being touched. The codes look cryptic but break down like this:

>f+++++++++   a brand-new file — being sent in full
>f..t......   an existing file — only its timestamp changed, no data resent
>f.st......   an existing file — size AND timestamp differ, data is being resent
.d..t......   a directory — only needs a timestamp touch-up, nothing inside changed
cd+++++++++   a brand-new directory
*deleting     this file/folder is being REMOVED (only appears with --delete)

Rule of thumb: the first character tells you the type (> = file being transferred, . = nothing meaningful moving, c = created, * = special like deleting). Everything after that is why.

--delete — mirror deletions​

Without this flag, rsync only adds and updates — it never removes anything from the destination, even if you deleted it locally. With --delete, the destination becomes an exact mirror: anything present on the destination but not in your source folder gets removed.

This is the one destructive flag here

Every other flag in this list is safe to run blind. This one isn't — it deletes real files on the other end. Always pair it with -n first and actually read the *deleting lines before running for real. See the Sync Process doc for a real example of why running this at the wrong time (before the database was synced) would have been dangerous, independent of the file list itself being correct.

--partial — keep incomplete transfers​

Normally, if a big file's transfer gets interrupted partway (dropped connection, closed terminal), rsync throws away what it had and starts that file over from 0% next time. --partial keeps the partial data instead, so the next run resumes from where it stopped rather than re-sending the whole file.

-e "ssh -o ServerAliveInterval=15 -o ServerAliveCountMax=3" — detect a dead connection​

This isn't an rsync flag itself — -e tells rsync what command to use for the network connection, here a customized ssh call. Plain SSH, left to its defaults, can sit forever on a connection that silently died (network dropped, VPS hiccuped) without ever noticing — it just waits, looking perfectly "alive" in ps aux, transferring nothing. ServerAliveInterval=15 makes SSH ping the other end every 15 seconds; ServerAliveCountMax=3 gives up after 3 missed replies (~45 seconds) instead of hanging indefinitely. Real incident this fixes: a sync sat for 13 hours, looked completely normal in ps aux, and had transferred zero bytes the entire time.

If you didn't have this flag on and suspect a sync is actually dead, not just slow, check whether it's really moving data before waiting it out:

cat /proc/<rsync-pid>/io | grep -E 'rchar|wchar'
sleep 5
cat /proc/<rsync-pid>/io | grep -E 'rchar|wchar'

Unchanged numbers between the two checks means it's dead — find both processes (ps aux | grep rsync, there will be two: the local rsync and an ssh ... rsync --server helper) and kill both, then re-run the same command.

--progress — per-file progress bar​

Shows a live progress line for whichever file is currently transferring (bytes done, percentage, speed, ETA). Useful for eyeballing "is this actually moving or stuck" — see the --stats section below for how to tell for certain.

--stats — summary block at the end​

Prints a final report once everything is done:

Number of files: 44,811 (reg: 27,750, dir: 17,061)
Number of created files: 26,404 (reg: 12,610, dir: 13,794)
Number of deleted files: 46 (reg: 21, dir: 25)
Number of regular files transferred: 23,655
Total file size: 30,229,665,007 bytes
Total transferred file size: 27,002,823,031 bytes

What each line means:

  • Number of files — total files+folders rsync looked at (in both source and destination combined)
  • Number of created files — brand new on the destination (didn't exist there before)
  • Number of deleted files — only appears with --delete; how many were removed
  • Number of regular files transferred — how many actually had data sent (new + changed; unchanged files are not counted here)
  • Total file size vs Total transferred file size — the first is everything in the source; the second is only what was actually sent over the network this run

--exclude 'PATTERN'​

Skips anything matching PATTERN entirely — never reads it, never transfers it, never considers deleting it. Used when there's a file you deliberately don't want mirrored (we chose not to use this for _catalog/catalog.db in the end — see the Sync Process doc for why).

Why the same-looking sync can take very different amounts of time​

If a sync feels slow even though "most files are already there," it's usually one of these, not a bug:

  1. --delete needs the full picture first. To know what to delete, rsync has to list every file on both sides before it can compare anything — that listing step scales with total file count, not with how much actually changed. Tens of thousands of files means a slow listing phase even when almost nothing needs to move.
  2. A file's timestamp changed, even if most of its content didn't. If something rewrote a file locally (even to remove a handful of rows), rsync sees a newer timestamp and re-sends that whole file.
  3. Upload speed, not download speed, governs this. Most home/office connections upload much slower than they download — this is usually the real bottleneck for a big first sync.
  4. No --partial on an earlier run — an interrupted file restarts from scratch instead of resuming.

Quick reference: commands used in this project​

# Dry run — always do this before anything with --delete
rsync -avzn -i --delete SOURCE/ DEST/

# Real sync, safe to re-run anytime, resumable if interrupted
rsync -avz --delete --partial --progress --stats SOURCE/ DEST/

# Plain additive sync (no deletions) — for routine "just add what's new"
rsync -avz --partial --progress --stats SOURCE/ DEST/

One more thing that trips people up: both paths need a trailing / on the source folder. SOURCE/ copies the contents of that folder into DEST/; SOURCE (no slash) copies the folder itself into DEST/, creating an unwanted DEST/SOURCE/ nesting.