Application-Consistent Proxmox Backups

Your backup job says OK every night. The snapshot verifies, the datastore looks healthy, and then one day you restore the VM that runs your billing database and MySQL comes up complaining about a half-written table. The backup was never broken. It was crash-consistent, and nobody told the database a snapshot was about to happen.

Three Levels of Consistency

The words get used loosely, so pin them down first. Every Proxmox backup lands in one of three categories, and which one you get depends on the backup mode and what is installed inside the guest.

Crash-consistent means the disk image is a valid point-in-time copy of the block device, and nothing more. Any write that the guest had in flight, in a page cache, or buffered in application memory is simply absent. Restoring is equivalent to booting a machine that lost power at that instant. Filesystems replay their journals, databases run recovery, and most of the time it works.

Filesystem-consistent means the guest was told to flush and quiesce its filesystems before the snapshot. Dirty pages are on disk, the journal is clean, and the filesystem mounts without replay. This is what fs-freeze buys you.

Application-consistent means the applications themselves reached a defined state before the snapshot: transactions committed or rolled back, caches flushed, no partially written multi-file state. Nothing in Proxmox VE gives you this automatically. You have to arrange it.

Stop mode is the only option that is application-consistent with zero configuration, because a cleanly shut down guest has nothing left in memory. It is also the option nobody wants for a production VM. Everything else in this post is about getting close to that guarantee without the downtime.

What the QEMU Guest Agent Actually Does

When agent is enabled on a VM and the agent is running inside it, PVE issues two QMP commands around the snapshot: guest-fsfreeze-freeze before, guest-fsfreeze-thaw after. Inside the guest, qemu-ga calls the kernel's FIFREEZE ioctl on every mounted filesystem that supports it. The kernel flushes dirty pages, completes in-flight metadata updates, and then blocks all new writes until the thaw.

The freeze window is short, usually well under a second, because PVE only needs it held long enough to establish the snapshot. During that window writes to the frozen filesystems block. Reads continue.

Enable it on the VM and install the agent in the guest:

# On the PVE node
qm set 110 --agent enabled=1,freeze-fs-on-backup=1

# Inside a Debian or Ubuntu guest
apt-get install -y qemu-guest-agent
systemctl enable --now qemu-guest-agent

freeze-fs-on-backup=1 is the default, but it is worth setting explicitly so nobody wonders later. The reason it exists at all is that a small number of workloads behave badly under a freeze, and turning it off is the escape hatch.

On Windows the same freeze command means something more useful. The QEMU guest agent MSI from the virtio-win ISO installs a VSS provider, and a freeze request drives the Volume Shadow Copy Service. VSS-aware writers, including SQL Server and Exchange, are notified and quiesce themselves properly. Windows guests get real application consistency from the freeze alone, provided the VSS component was actually selected during install and the writers are healthy. Check with vssadmin list writers and look for writers in a stable state with no errors. The Proxmox backup client for Windows guide covers the agent-in-guest alternative for physical Windows machines.

Where Filesystem Freeze Stops Being Enough

A frozen filesystem is a clean starting point for crash recovery, not a substitute for it. Understanding which of your workloads is fine and which is not saves a lot of pointless hook writing.

InnoDB is fine. With innodb_flush_log_at_trx_commit=1, every committed transaction is in the redo log on disk before the client gets its acknowledgement. A frozen-filesystem snapshot captures a valid redo log, and MySQL replays it on next start. You lose nothing that was committed.

PostgreSQL is fine, under one condition: the whole cluster has to be in the snapshot. The data directory, pg_wal, and every tablespace must live on disks that were captured while the same freeze was held. PVE freezes all the guest's filesystems and snapshots all its disks together, so a normal multi-disk VM is fine. A VM with an NFS-mounted tablespace is not, because that mount is not part of the snapshot.

MyISAM is not fine. Non-transactional engines have no recovery log. A snapshot taken mid-write leaves a table that needs REPAIR TABLE, and sometimes that is not enough. Same story for older application state files written without atomic rename.

In-memory state is not fine. Redis with only periodic RDB saves loses everything since the last save. Elasticsearch and other Lucene-based stores need a flush to have their translog in a useful position. Message queues holding unacknowledged work in memory lose it.

Multi-VM applications are not fine and cannot be fixed at this layer. If your app server and database are separate VMs, freezing each one independently gives you two snapshots taken at different instants. There is no cross-VM atomic freeze in Proxmox VE. Either accept the skew, or get consistency at the application layer by making the app tolerate a database that is slightly behind.

Hooks Inside the Guest: fsfreeze-hook

qemu-ga can run a script on freeze and thaw. The hook is called with freeze before the filesystems are frozen and thaw after they are released, which makes it the right place for anything the application needs to do at that moment.

The hook is not enabled by default on Debian and Ubuntu. The agent needs the -F flag:

# /etc/default/qemu-guest-agent
DAEMON_ARGS="-F/etc/qemu/fsfreeze-hook"

systemctl restart qemu-guest-agent

# The default hook dispatches to every executable in this directory
install -d -m 0755 /etc/qemu/fsfreeze-hook.d

The stock /etc/qemu/fsfreeze-hook script runs every executable file in fsfreeze-hook.d, passing the same freeze or thaw argument through. Drop one file per application:

#!/bin/sh
# Called with "freeze" before the snapshot, "thaw" after.
set -e

case "$1" in
    freeze)
        # Force an RDB write so the on-disk dump matches memory
        /usr/bin/redis-cli --no-auth-warning SAVE
        ;;
    thaw)
        : # nothing to undo
        ;;
esac

Keep these fast. The freeze hook runs while PVE waits, and a hook that takes 30 seconds adds 30 seconds to every backup of that guest before the freeze even starts. Cheap flushes belong here. Logical dumps do not.

There is one thing the guest hook is genuinely bad at, and it catches people out: session-scoped locks. FLUSH TABLES WITH READ LOCK in MySQL only holds while the connection that issued it stays open, so a hook that runs mysql -e 'FLUSH TABLES WITH READ LOCK' releases the lock the instant the client exits. Working around that means holding a background session across the freeze and thaw, which is fragile in exactly the situation where you least want fragility. If you need a read lock held across the snapshot, do it from the host side instead.

Hooks on the Host: vzdump Scripts

PVE calls an external script at defined points in a backup job. This is where anything slow or coordinated belongs, because it runs before the freeze and outside its time budget.

The script is called with three arguments: phase, mode, and VMID. The phases that matter here are job-start and job-end for once-per-job work, and backup-start, backup-end, and backup-abort for per-guest work. PVE ships a documented example at /usr/share/pve-docs/examples/vzdump-hook-script.pl.

#!/bin/bash
# Called as: <script> <phase> <mode> <vmid>
set -euo pipefail

phase="${1:-}"
vmid="${3:-}"

# Guests that need a logical dump before the snapshot
case "$vmid" in
    110|112) needs_dump=1 ;;
    *)       needs_dump=0 ;;
esac

case "$phase" in
    backup-start)
        [ "$needs_dump" = 1 ] || exit 0
        # Runs inside the guest via the agent, before fs-freeze
        qm guest exec "$vmid" --timeout 900 -- \
            /usr/local/sbin/pre-backup-dump.sh
        ;;
    backup-end|backup-abort)
        [ "$needs_dump" = 1 ] || exit 0
        qm guest exec "$vmid" --timeout 60 -- \
            /bin/rm -rf /var/backups/predump || true
        ;;
esac

exit 0

Register it globally in /etc/vzdump.conf with script: /usr/local/bin/vzdump-app-hook.sh, or per job in the backup job's advanced options. The file has to be executable, and it runs on the PVE node rather than on the Proxmox Backup Server.

The dump script itself lives in the guest and is ordinary:

#!/bin/bash
set -euo pipefail

out=/var/backups/predump
mkdir -p "$out"

# PostgreSQL: consistent dump of every database, no locks held
pg_dumpall --clean --file="$out/pgdump.sql.tmp"
mv "$out/pgdump.sql.tmp" "$out/pgdump.sql"

# MySQL: --single-transaction for InnoDB, no read lock needed
mysqldump --single-transaction --routines --events \
    --all-databases > "$out/mysql.sql.tmp"
mv "$out/mysql.sql.tmp" "$out/mysql.sql"

sync

Now the snapshot contains both a crash-consistent copy of the live data directory and a logical dump that was taken transactionally. Restore the VM, and if the live database recovers cleanly you ignore the dump. If it does not, you have a guaranteed-good copy sitting in /var/backups/predump. The cost is disk space in the guest and dump time on every run, which is why you scope it to the VMIDs that need it.

For guests you back up with proxmox-backup-client directly rather than through PVE, there is no hook mechanism at all. Wrap the dump and the backup call in one script and drive it from a systemd timer. Our guide to backing up any Linux server to PBS covers that wrapper pattern.

Verify the Freeze Actually Happened

This is the step people skip. Everything above can be configured correctly and still not run, and the backup will report success either way. Read the task log:

Two lines, freeze then thaw. If they are missing, either the agent is disabled on the VM or freeze-fs-on-backup is off. If you see a timeout instead, the agent flag is set but nothing is answering inside the guest:

Note what follows the error. The backup continued and finished successfully. PVE treats a failed freeze as a warning, not a failure, which is defensible behaviour and also the reason a broken setup can go unnoticed for a year. If application consistency matters to you, alert on that string. Our post on backup monitoring and alerting covers wiring log conditions into notifications.

The only real proof is a restore. Bring the VM up in an isolated network, start the database, and read its startup log. InnoDB will tell you plainly whether it performed crash recovery and whether it succeeded. Fold that check into the restore drills you are already running.

Common Mistakes

• Assuming snapshot mode quiesces anything. It does not. Without the agent installed and running, every snapshot-mode backup is crash-consistent.
• Setting agent enabled=1 and stopping there. The flag is a promise the guest has to keep. Verify with qm guest cmd <vmid> ping.
• Putting a mysqldump in the fsfreeze hook. It runs inside the freeze path and stalls the backup. Slow work belongs in a host-side backup-start hook.
• Expecting FLUSH TABLES WITH READ LOCK from a hook to hold. The lock dies with the session that took it.
• Forgetting Windows VSS was optional at install time. The base guest agent installs without the VSS provider. Confirm with vssadmin list writers.
• Treating multi-VM skew as solvable here. Two guests frozen separately are two different points in time. Fix that in the application, not in the backup job.
• Never testing. A consistency configuration that has never survived a restore is a hypothesis.

Wrapping Up

Proxmox Backup Server stores whatever the hypervisor hands it, and by default that is a crash-consistent image. Enabling the QEMU guest agent moves you up to filesystem consistency for a cost measured in milliseconds, and on Windows it gets you real VSS quiescing at the same time. Beyond that, application consistency is something you build: short flush hooks inside the guest for in-memory state, host-side hooks running logical dumps before the freeze for databases that deserve a belt-and-braces copy. Then check the task log for the freeze lines, and prove the whole thing with a restore before you need one.