External snapshot terminology¶

External snapshots on a disk level consist of layered chains of disk images. An image in the chain can have a backing image placed below. Any chunk in the current image which was not written explicitly is transparent and if read the data from the backing image is passed through. An image placed on top of the current image is called overlay.

The bottommost backing image at the end of the chain is also usually described as base image.

The topmost overlay is the image which is being written to by the VM and is also described as the active layer or image.

Handling of bitmaps during snapshots¶

Creating an external snapshot involves adding a overlay on top of the previously active image. Libvirt requires that all block-dirty-bitmaps which correspond to the checkpoint must be created in the new overlay before any write from the guest reaches the overlay to continue tracking which blocks are dirtied.

Since there are no new bitmaps created by qemu or qemu-img by default when creating an overlay, we need to re-create the appropriate bitmaps (see below) in the new overlay based on the previously active bitmaps in the active image. The new bitmaps are created with the same granularity.

After taking a snapshot of the vda disk from the example above placed into vda-2.qcow2 the following topology will be created:

+----------------+
| disk: vda      |
+-------+--------+
        |
+-------v--------+    +----------------+
| vda-2.qcow2    |    | vda-1.qcow2    |
|                |    |                |
| bitmaps: chk-a +----> bitmaps: chk-a |
|          chk-b |    |          chk-b |
|                |    |                |
+----------------+    +----------------+

Manipulating bitmaps in shell¶

NOTE: Any of the examples expect that the full image chain isn't used by any running VM at the time.

qemu-img info command reports information about dirty bitmaps in an image:

$ qemu-img info -f qcow2 vda-1.qcow2
 image: vda-1.qcow2
 file format: qcow2
 virtual size: 100 MiB (104857600 bytes)
 disk size: 220 KiB
 cluster_size: 65536
 Format specific information:
     compat: 1.1
     compression type: zlib
     lazy refcounts: false
     bitmaps:
         [0]:
             flags:
                 [0]: in-use
                 [1]: auto
             name: chk-a
             granularity: 65536
         [1]:
             flags:
                 [0]: auto
             name: chk-b
             granularity: 65536
     refcount bits: 16
     corrupt: false

The flags have following meanings:

auto - recording

The bitmap is automatically activated when the image is opened for writing and thus it's actively recording writes.

in-use - inconsistent

The bitmap was not properly saved when the qemu process was shut down last time thus didn't consistently record all the changed sectors.

It's recommended to use --output=json parameter to work with a machine readable output rather than trying to process the human readable output by scripts. For processing JSON in shell the jq tool can be used.

The qemu-img bitmap command allows modification of block-dirty-bitmaps of an offline image. It supports the following operations relevant to this document (see man page for full list of operations):

--add NAME

Creates a new bitmap named NAME. Optionally -g can be used to specify granularity.

--remove NAME

Deletes bitmap NAME.

--merge SRCBITMAP -b SRCFILE -F SRCFILEFMT DSTBITMAP

Merges bitmap SRCBITMAP from SRCFILE into DSTBITMAP.

Checking bitmap health¶

QEMU optimizes disk writes by only updating the bitmaps in certain cases. This also can cause problems in cases when e.g. QEMU crashes.

For a chain of corresponding bitmaps in a backing chain images to be considered valid and eligible for use for an incremental backup with virDomainBackupBegin the bitmaps intended to be used must conform to the following rules:

1. active/topmost image must contain the bitmap

2. if a bitmap with the same name is contained in one of the backing images it must be a contiguous subchain starting from the topmost image which contains the bitmaps (no gaps)

3. all of the above bitmaps must be marked as recording

4. all of the above bitmaps must not be inconsistent

(See also the qemuBlockBitmapChainIsValid helper method in src/qemu/qemu_block.c)

Creating external snapshots manually¶

To create the same topology outside of libvirt (e.g when doing snapshots offline) the following pseudo-algorithm ensures that the new image after snapshot will work with backups. OVERLAY corresponds to the new overlay image, ACTIVE corresponds to the topmost image of the active chain prior to the snapshot.

create image OVERLAY on top of ACTIVE

for each BITMAP in ACTIVE:
    let GRANULARITY = granularity of BITMAP in ACTIVE

    if BITMAP isn't RECORDING or is INCONSISTENT:
        continue

    create RECORDING bitmap named BITMAP in OVERLAY with GRANULARITY

Committing external snapshots manually¶

block commit refers to an operation where data from a subchain of the backing chain is merged down into the backing image of the subchain removing all images in the subchain.

COMMIT_TOP refers to the top of the subchain to merge into COMMIT_BASE (which stays in the new chain).

It's strongly advised to use virDomainBlockCommit API in libvirt directly if possible. Inactive VMs can be started with VIR_DOMAIN_START_PAUSED flag (virsh start --paused) to prevent OS from running.

Otherwise the following pseudo-algorithm can be used:

Note: A valid bitmap chain is a set of images containing bitmaps which conform to the rules about valid bitmaps mentioned above.

commit data from COMMIT_TOP to COMMIT_BASE

let BITMAPS = valid bitmap chains in COMMIT_TOP

for each BITMAP in BITMAPS
    let GRANULARITY = granularity of BITMAP in ACTIVE

    if BITMAP is not present in COMMIT_BASE:
        create RECORDING bitmap named BITMAP in COMMIT_BASE with GRANULARITY

    for each IMAGE between COMMIT_TOP(inclusive) and COMMIT_BASE(exclusive):
        if BITMAP is not present in IMAGE:
            break

        merge BITMAP in IMAGE into BITMAP in COMMIT_BASE