Skip to content

linstor: incremental snapshots on secondary storage - #13746

Merged
rp- merged 7 commits into
apache:4.22from
LINBIT:linstor-4.22-incremental-snapshots
Oct 21, 2026
Merged

rp- merged 7 commits into
apache:4.22from
LINBIT:linstor-4.22-incremental-snapshots

Conversation

@rp-

@rp- rp- commented Jul 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

In short, this PR adds incremental snapshot support for Linstor backuped (secondary storage) volume snapshots.
It uses the same configuration keys as the file based KVM incremental snapshots settings.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Tested on a Linstor cluster.

@rp-
rp- requested review from JoaoJandre and sureshanaparti July 30, 2026 09:03
@rp- rp- self-assigned this Jul 30, 2026
@rp- rp- changed the title linstor: incremental snapshots on secondary storage via qemu-img reba… linstor: incremental snapshots on secondary storage Jul 30, 2026
@codecov

codecov Bot commented Jul 30, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 43.57143% with 79 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.81%. Comparing base (0339f31) to head (af37556).
⚠️ Report is 105 commits behind head on 4.22.

Files with missing lines Patch % Lines
...e/wrapper/LinstorBackupSnapshotCommandWrapper.java 0.00% 37 Missing ⚠️
...tore/driver/LinstorPrimaryDataStoreDriverImpl.java 51.06% 18 Missing and 5 partials ⚠️
...storage/datastore/db/SnapshotDataStoreDaoImpl.java 42.85% 7 Missing and 1 partial ⚠️
...cloudstack/storage/datastore/util/LinstorUtil.java 78.57% 1 Missing and 5 partials ⚠️
...torage/datastore/ObjectInDataStoreManagerImpl.java 0.00% 2 Missing ⚠️
...om/cloud/storage/snapshot/SnapshotManagerImpl.java 75.00% 1 Missing and 1 partial ⚠️
...java/org/apache/cloudstack/utils/qemu/QemuImg.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               4.22   #13746      +/-   ##
============================================
+ Coverage     17.69%   17.81%   +0.11%     
- Complexity    15833    16030     +197     
============================================
  Files          5925     5928       +3     
  Lines        533534   534402     +868     
  Branches      65273    65403     +130     
============================================
+ Hits          94421    95183     +762     
- Misses       428434   428438       +4     
- Partials      10679    10781     +102     
Flag Coverage Δ
uitests 3.69% <ø> (-0.01%) ⬇️
unittests 18.89% <43.57%> (+0.12%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…se diff

Linstor volume snapshots were always backed up to NFS secondary storage as
full qcow2 copies, so every snapshot consumed full-volume space. The existing
KVM incremental snapshot feature (kvm.incremental.snapshot) is limited to
file-based primary storage because it relies on libvirt checkpoints backed by
qcow2-persistent dirty bitmaps, which do not exist for raw DRBD devices.

Add incremental snapshots for Linstor through a content diff instead: the
agent creates an empty qcow2 overlay backed by the raw snapshot device (with
an explicit virtual size, clipping the trailing DRBD metadata), then
safe-rebases it onto the previous snapshot qcow2 on secondary storage. qemu
writes only the clusters in which the two backing files differ, yielding a
delta qcow2 chained to its parent by a relative backing name (portable across
mount points). No bitmaps, no libvirt version gates, no LINSTOR snapshot
retained on primary, and it works for running and stopped VMs alike. Because
the diff is content-based, any ready previous snapshot is a correct diff base,
which makes fallback-to-full trivial.

The feature is gated by the existing cluster-scoped kvm.incremental.snapshot
setting plus the pool being Linstor; chain length is capped by
snapshot.delta.max and the end_of_chain rotation. Encrypted volumes stay
full-only (a rebase would need the LUKS secret for delta and backing file).

- SnapshotDataStoreDaoImpl.findParent: parent selection for Linstor volumes
  (non-encrypted) uses the plain search instead of requiring
  kvm_checkpoint_path, so chains can start; ObjectInDataStoreManagerImpl then
  links parent_snapshot_id on the image store ref as usual.
- SnapshotObject.getCorrectIncrementalParent: fall back to the plain parent
  instead of throwing when no ref carries a checkpoint path. The TO
  constructor still early-returns for such parents, keeping parents[] null so
  the file-based agent paths are unaffected, while getParent() correctly
  drives convert-before-use and chain-aware deletion.
- SnapshotManagerImpl: generalize the three checkpoint-keyed predicates
  (extract URL flattening, endLastChainIfNeeded, copySnapshotChainToZone)
  through isIncrementalChainRef(), which also recognizes parent-linked KVM
  chains; XenServer per-member chain copy semantics are preserved.
- Linstor driver: copySnapshot passes fullSnapshot=false and the parent
  install path when the dest ref has a ready, same-sized parent on the same
  store; the parent link is cleared whenever a full copy was made (encrypted,
  parent missing/unready/resized or agent-side fallback) so the DB chain
  always matches what is on disk.
- LinstorBackupSnapshotCommandWrapper: new incremental branch (qemu-img
  create -F raw -b <snapdev> + safe rebase onto the parent + rebase -u to the
  relative backing name); falls back to the full convert path if the parent
  file is missing. The answer TO reports parentSnapshotPath so
  SnapshotObject.processEvent keeps the parent link only for real deltas.

Restore, volume/template-from-snapshot, extract and cross-zone copy all reuse
the existing machinery: qemu-img follows the relative backing chain on the
same mount, and chain members are flattened via ConvertSnapshotCommand before
standalone use. Delete ordering is handled by the generic parent-link logic
(children hide their parents, leaf-first physical deletion).
@rp-
rp- force-pushed the linstor-4.22-incremental-snapshots branch from 0a4822c to 8643df8 Compare July 30, 2026 09:27
@DaanHoogland
DaanHoogland requested a review from Copilot July 30, 2026 12:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds support for incremental (content-diff) snapshot backups to secondary storage for Linstor volumes, reusing the existing KVM incremental snapshot configuration semantics.

Changes:

  • Introduces content-based incremental chaining for Linstor snapshot backups via parent snapshot install paths.
  • Updates snapshot parent selection logic to avoid applying checkpoint-based (cross-role) handling to content-based chains.
  • Adds/extends unit tests covering incremental-chain detection, parent-path resolution, and chain bookkeeping adjustments.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java Centralizes “is part of incremental chain” detection for KVM (checkpoint-based and content-based parent-link chains).
server/src/test/java/com/cloud/storage/snapshot/SnapshotManagerImplTest.java Adds tests for incremental-chain detection behavior across hypervisors and ref states.
plugins/storage/volume/linstor/src/main/java/org/apache/cloudstack/storage/datastore/driver/LinstorPrimaryDataStoreDriverImpl.java Implements incremental-parent resolution on secondary storage and clears chain parent when a full copy is taken.
plugins/storage/volume/linstor/src/test/java/org/apache/cloudstack/storage/datastore/driver/LinstorPrimaryDataStoreDriverImplTest.java Adds tests for incremental parent path selection and parent-link clearing behavior.
plugins/storage/volume/linstor/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LinstorBackupSnapshotCommandWrapper.java Adds agent-side incremental qcow2 creation using safe rebase onto the parent qcow2 backing file.
plugins/storage/volume/linstor/src/main/java/com/cloud/api/storage/LinstorBackupSnapshotCommand.java Adds a new command option for passing the parent snapshot install path to the agent.
engine/storage/src/main/java/org/apache/cloudstack/storage/datastore/ObjectInDataStoreManagerImpl.java Skips checkpoint-based parent handling for volumes using content-based chains.
engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/SnapshotObject.java Makes incremental-parent resolution tolerate chains without checkpoints (fallback to plain parent).
engine/storage/snapshot/src/test/java/org/apache/cloudstack/storage/snapshot/SnapshotObjectTest.java New tests for incremental-parent selection logic with/without checkpoint-bearing refs.
engine/schema/src/main/java/org/apache/cloudstack/storage/datastore/db/SnapshotDataStoreDao.java Adds usesContentBasedChain(volumeId) API to distinguish chain types.
engine/schema/src/main/java/org/apache/cloudstack/storage/datastore/db/SnapshotDataStoreDaoImpl.java Implements chain-type detection and adjusts findParent behavior accordingly.
engine/schema/src/test/java/org/apache/cloudstack/storage/datastore/db/SnapshotDataStoreDaoImplTest.java Adds tests for usesContentBasedChain behavior across volume/pool conditions.
plugins/storage/volume/linstor/CHANGELOG.md Documents the new incremental snapshot backup support.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@JoaoJandre

Copy link
Copy Markdown
Contributor

I'll try to review this next week if I have the time

@JoaoJandre

Copy link
Copy Markdown
Contributor

Could you share the test cases you made? Since I don't have access to a Linstor storage, I can't test it myself, but if you share your tests I might be able to see potential test cases that have not been covered yet :)

rp- added 4 commits August 6, 2026 09:28
- LinstorBackupSnapshotCommandWrapper: follow coding conventions for the
  createIncrementalQCow2 signature, use KVMStoragePool.getLocalPathFor()
  instead of manual path concatenation, and create the qcow2 overlay via
  QemuImg.create() instead of a hand-rolled qemu-img Script call.
- QemuImg.create: pass the created file's format to -f instead of the
  backing file's format, so an overlay format can differ from its backing
  format (-f qcow2 -F raw). All existing callers use matching formats and
  are unaffected.
- CHANGELOG.md: fix wording of the incremental snapshot entry.
getDiskfulStoragePools only ever returned the storage pool of a single
node, so an endpoint was only looked for on that one node - if its
cloudstack host was down or disabled the operation fell back to a
temporary resource or a diskless attach although other diskful copies
were usable.

Return all diskful storage pools ordered by how suited the copy is (in
use, then active, then inactive) and walk that order when picking a
host. The order matters: thick LVM snapshots on shared storage pools
must be read on the node the volume is active on, so the host selection
no longer shuffles candidates.
endCurrentChainIfNeeded walks the snapshot chain through
findOneBySnapshotId, whose FIND_SNAPSHOT_IN_ZONE query returns an
arbitrary store reference of the snapshot. Content-based chains only
link their image store reference - the primary one stays unlinked - so
for Linstor volumes the walk always saw a chain length of one and
snapshot.delta.max was never enforced: deltas were chained forever and
no full snapshot was ever taken again. Checkpoint-based chains were
unaffected because they carry the parent link on both references.

Order references carrying a chain link (parent or checkpoint) first in
FIND_SNAPSHOT_IN_ZONE, and stop the walk gracefully if a parent has no
references left instead of throwing an NPE.
Cover the incremental (content-diff) snapshot feature end to end:
full -> delta chaining with parent links, snapshot.delta.max chain
rotation, revert to a mid-chain delta, template from a delta (chain
flattening), fallback to a full backup when the parent file is missing
on secondary storage, mid-chain deletion, and that encrypted volumes
are always backed up as full copies.

Chain bookkeeping (parent_snapshot_id, end_of_chain) is not exposed via
the API, so the tests verify it in the DB when reachable and otherwise
fall back to physical-size comparisons against the chain's full backup.
Tests needing host SSH (secondary storage manipulation/inspection)
self-skip without credentials.

The ServiceReady helper the Linstor test modules used to copy around
moves to a shared linstor_test_utils module instead of gaining a third
copy.
@rp-

rp- commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@JoaoJandre Thanks for the review, I addressed your comments and also added the integration tests I ran.

I found another issue, that the snapshot.max.delta wasn't correctly used and fixed that in a commit.

And I also added a fix for Linstors getDiskfulHost selector, that was always only using the first result, instead of all.

@JoaoJandre JoaoJandre left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code overall looks good to me. I'll try to test the incremental snapshots on NFS to make sure no regressions were introduced.

- SnapshotDataStoreDaoImpl: use an explicit JOIN instead of the implicit
  comma join in FIND_SNAPSHOT_IN_ZONE.
- QemuImgTest: cover creating an image whose format differs from its
  backing file's format (qcow2 overlay on a raw backing file), the case
  enabled by passing the created file's format to -f.
@rp- rp- added this to the 4.22.2 milestone Aug 19, 2026
…ered

Add get_guest_template() to linstor_test_utils and use it in the volume,
encrypted-snapshot and incremental-snapshot tests. It picks the purpose-built
~512 MiB "marvin-alpine" USER template registered by
`cs-linstor.py setup-test-template` and falls back to the builtin template
when it is absent, so no extra setup is required.

The builtin CentOS template is an 8 GiB volume that takes ~55s to reach sshd,
and creating an encrypted volume from it forces a full block-level re-encrypt
copy of the whole virtual size (~500s per VM). A small template makes that
negligible.
@rp-

rp- commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

@JoaoJandre is there anything missing from your side?

@JoaoJandre

Copy link
Copy Markdown
Contributor

@JoaoJandre is there anything missing from your side?

This #13746 (review). Although I'm still short on time, I'll try to get it done next week if I can.

@rp-
rp- requested a review from DaanHoogland September 29, 2026 05:59
@rp-

rp- commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@rp- a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19401

@rp-

rp- commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

I thought adding you as reviewer might be enough, but I'll also ping you here @sureshanaparti as I would like to have this included in 4.22.2
Its basically sitting here finished since 3 weeks now...

@JoaoJandre

Copy link
Copy Markdown
Contributor

@JoaoJandre is there anything missing from your side?

This #13746 (review). Although I'm still short on time, I'll try to get it done next week if I can.

@rp- I have been very short on time recently... I might have some time after the COC. But if someone else can test this as well, its ok for me. I just wanted to be sure that no regressions would happen on the current incremental volume snapshot process.

@rp-

rp- commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

@weizhouapache can you have a look at this PR? or at least start the integration tests with it once?

@weizhouapache

Copy link
Copy Markdown
Member

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-17129)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 49606 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13746-t17129-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

@JoaoJandre JoaoJandre left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally got around to testing this. I tested snapshot creation, restore and deletion using kvm incremental snapshots to verify that no regressions were added. I do not have a Linstor env to test the Linstor side.

@rp-
rp- requested a review from weizhouapache October 19, 2026 19:17
@rp- rp- mentioned this pull request Oct 20, 2026
3 of 14 tasks
@rp-

rp- commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

@weizhouapache @DaanHoogland
Can you have a quick look and can we merge it after Joao tested it?

@weizhouapache weizhouapache left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code lgtm

@rp-
rp- merged commit a7697fe into apache:4.22 Oct 21, 2026
26 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 21, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request!

@github-project-automation github-project-automation Bot moved this from Ready to Done in CloudStack Testing Oct 21, 2026
@rp-
rp- deleted the linstor-4.22-incremental-snapshots branch October 21, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

5 participants