Compare commits

..

227 Commits

Author SHA1 Message Date
Pierre Brunet
9829eb58f3 fix: add debug to better understand timeout error 2026-08-07 15:53:52 +02:00
Mathieu
2e5ec8c9e2 fix(xo-server): auth provider did not wait for token update (#10233) 2026-08-06 15:24:21 +02:00
Mathieu
8054d88a88 fix(xo-server): handle 'no-bundle' as legacy licence (#10241) 2026-08-06 15:22:22 +02:00
Thierry Goettelmann
20bedca539 refactor(web-core,lite,web): migrate from modal/drawer packages to the unified overlay system (#10096) 2026-08-05 17:13:15 +02:00
Grandalf
f61019b960 fix(rest-api): coerces strings into numbers in external-router when type="number" (#10213) 2026-08-05 16:22:09 +02:00
Thomas Moraine
be960b36b9 docs(advanced.md) fix broken link to kubernetes page (#10230)
The docs on devops tools and kubernetes was moved
from the Vates VMS doc to the Xen Orchestra doc.
As a result, one link to the kubernetes page broke in the XO doc.
This commit fixes that broken link.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-08-05 09:40:12 +02:00
Mathieu
40dede9e11 feat: release 6.7.1 (#10229) 2026-08-04 15:42:40 +02:00
Mathieu
e9548959c6 feat: technical release (#10228) 2026-08-04 12:05:18 +02:00
Florent BEAUCHAMP
be94d0d083 fix(xo-server/v2v): incorrect uuid for running VM without snapshot (#10194)
* add more details in task log
* fix an issue with VM with more than 10 snapshot, add unit test for this parsing
* fix the real bug: vmware do not update the vmdk description file before the next VM reboot. It means that the uid of the snapshot and vm are the same, thus finishing the transfer with " nothing to transfer" . We are now using the path on source as key + add unit test + add compatibiliy with legacy VM migrated with the uid.
* add e2e test for v2v
2026-08-04 10:41:07 +02:00
Julien PIALET
9c6d10327f doc(RBAC) : added details on POST privilege example (#10168) 2026-08-03 15:20:20 +02:00
spacotte-vates
a94eba4dd1 fix(nbd-client): fix long datamap read nbddisk (#10157)
Customer imported large VM from VMWare (> 3 To), import switched to Qcow2 automatically but the process was stuck while building the qcow2 L1/L2 address tables, where the NbdDisk's hasBlock is called for each l2 table entry which iterated over the disk dataMap which contained ~113,528 ranges. This froze the process for more than 18 minutes after which it was either killed or died due to OOM error.

This fix improves the NbdDisk by:

    Updating processDatamap to throw on overlapping ranges and merge touching extents.
    Override default getBlockIndexesCount with optimized Nbd one.
    Improve hasBlock with a built in cursor to improve performance.

Test file added to cover the new function and updated code.
2026-08-03 11:24:44 +02:00
Florent BEAUCHAMP
a6970b0150 fix(backups/filerestore): restore files from proxy as tgz (#10208)
from ticket 59505

node-tar >= 7 returns a Minipass stream: it has `pipe()` but is not a `node:stream`
instance. xo-proxy serves this stream through Koa, which pipes only bodies passing
instanceof Stream` and JSON-serializes anything else — so a file restore through a
proxy answered a JSON dump of the tar object instead of the archive. Wrapping it in a
real node stream also propagates tar errors to the consumer and aborts the tar job
when the client disconnects.
2026-08-02 14:57:48 +02:00
Bastien Nollet
26cd23179d fix(load-balancer): don't get disabled hosts (#10209)
The load balancer did not take into account that hosts may be in disabled state, which means it's impossible to migrate a VM to it.
This PR fixes that by ignoring these hosts instead.

Fixes ticket 62077
2026-08-02 14:56:58 +02:00
Joris K
03d2af39ad feat(xo6): add host scan PIFs button and job (#10191)
* changelog

* feat(xo6): add host scan PIFs button and job

* add changelog

* missing translation

* review

* review

---------

Co-authored-by: MathieuRA <contact@mathieu-raisin.fr>
2026-07-31 16:10:46 +02:00
Mathieu
be868a307b feat(rest-api): add scan-pifs endpoint (#10187)
* feat(rest-api): add scan-pifs endpoint

* changelog

* add pifs tag to the route

* fix typo

* fix  typo in changelog

* fix

* missing response
2026-07-31 11:50:05 +02:00
Florent BEAUCHAMP
01c00b110e test(immutable-backups): integration test of the new process 2026-07-31 11:28:25 +02:00
Florent BEAUCHAMP
65c32014d6 feat(immutable-backups): better cleanup on start
feat: changelog
2026-07-31 11:28:25 +02:00
Florent BEAUCHAMP
69dfeeaaf6 fix(immutable-backups): ensure shell is in english 2026-07-31 11:28:25 +02:00
Florent BEAUCHAMP
4a1269b898 feat(immutable-backups): run lift on proces start 2026-07-31 11:28:25 +02:00
Marie Garde
498cc7509f feat(xo6/host): host action button start shutdown (#10088)
* feat(xo6): add disable and enable host button

* Add button enable disable

* remove route map file

* Refacto for PR

* Add modal

* refacto for PR

* change key i18n

* fix i18n

* fix route map

* change modale to VtsActionModale

* refacto for MR

* fix type VtsActionModal

* refacto for MR

* add enable disable un treeview

* refacto treeview

* wip

* add shutdown and start button and job

* Update changelog and remove route map

* add check on disable button

* fix job error type

* update button for menuitem

* wip i18n

* add vtsaction modal

* tests ok

* fix build

* refacto for mr

* fix fr.json

* refacto for PR

* add check pending action

* add check host not halted

* fix enable disable button while host shutdown

* returns from MR

* fix changestate host header

* wip

* wip

* add check pending operations

* last fix for PR

* fix menuItem in mobile vue

* remove route map

* returns from MR

* fix start button

* Add changelog
2026-07-31 10:12:52 +02:00
Florent BEAUCHAMP
f98b4c9e4a docs(s3): add ovh as a tier 1 provider (#10206) 2026-07-31 09:40:37 +02:00
Mathieu
cc7ba53b29 feat(lite): 0.24.0 (#10204) 2026-07-30 17:46:20 +02:00
Mathieu
1a795970f9 feat: release 6.7.0 (#10203) 2026-07-30 17:26:51 +02:00
Mathieu
59306dce8f feat: technical release (#10198) 2026-07-30 11:08:08 +02:00
Florent BEAUCHAMP
6ba465be04 fix(xapi): don't try to compute progress on Buffer (#10195)
* fix(xapi): don't try to compute progress on Buffer

introduced by 76b45a9b3b

* port type to vates/types
2026-07-30 10:19:37 +02:00
Mathieu
068c4cdc86 feat: technical release (#10192) 2026-07-29 15:30:14 +02:00
Mathieu
93a7c3b509 fix(docs): fix yarn.lock and broken anchor (#10190)
Introduced by ada3cf745
2026-07-29 14:52:45 +02:00
Florent BEAUCHAMP
07c42bbd1e fix(fs/s3): improve delete multiple performances (#10189) 2026-07-29 11:29:55 +02:00
Florent BEAUCHAMP
ada3cf745c feat(xo-server): serve doc from xo + add backup calculator (#10154)
* feat(xo-server): serve doc locally
* the doc is already distributed from the mono repo
* add a build phase to the global yanr build that build a doc with the
right prefix
* add a http mount doc => embed build

* doc(backups): add a space calculator
* estimate the space needed for backups
* compute the duration protected by immutability
2026-07-29 09:48:41 +02:00
Mathieu
6a0332a784 fix(xapi/host): fix async use in a sync block for smart reboot (#10180)
* fix(xapi/host): fix async use in a sync block for smart reboot
2026-07-29 09:38:38 +02:00
Mathieu
09bd5c3425 feat(openmetrics): expose SR content and UUID (#10149)
Refs XO-2503
2026-07-29 09:24:34 +02:00
Florent BEAUCHAMP
26c36d485e fix(backups): don't hide errors during backup transfers (#10183) 2026-07-28 17:08:19 +02:00
Thierry Goettelmann
087fd5d397 feat(overlay): allow manually aborting overlays (#10167) 2026-07-28 16:47:45 +02:00
Grandalf
4636fcf820 feat(types): type-register-rest-routes (#10171) 2026-07-28 14:51:36 +02:00
Marie Garde
dd58fed217 feat(xo6/xolite): add copyall button xo and lite (#10081) 2026-07-28 13:38:57 +02:00
vates-bot
4aa8a31c25 feat(web-stack/i18n): update Xen Orchestra 6 (#10033) 2026-07-28 11:00:31 +02:00
Olivier Lambert
4ca263c10a feat(xo-server,xo-web/rpu): shut down and restart pinned VMs instead of aborting (#10125)
* feat(xo-server/rpu): opt-in shutdown/restart of pinned VMs instead of aborting
2026-07-28 10:05:43 +02:00
Alina D
8d58147d66 feat(xo6): align vertically all hosts in Treeview (#10153) 2026-07-28 09:53:07 +02:00
Joris K
5849844fca fix(lite): change defer type import on vm creation (#10176) 2026-07-28 09:22:52 +02:00
Mathieu
a5648aede8 fix(web-core): fix several store issues (#10156) 2026-07-28 09:22:45 +02:00
Florent BEAUCHAMP
84ce89d199 fix(disk-transform): implement a guardrail to ensure it does not read block only in parent 2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
9140cd273b fix(disk-transform): fix getBlockIndexesCount
introduced by #10058

the current method assumes that source block are full but they can
be sparse. In the worst case getBlockIndexesCount can be the same as
the source
2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
e710135eda fix(backups): ensure replication never transmit a delta without a base
differentiate when it's expected to transmit a full , and throw an
error on any discrepency
2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
5509f62ccd fix(backups): fix aggregated backup failing instead of falling back to a full
map.foreach((value,key)=> , not map.foreach(key=>
2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
6c4f7336e1 fix(backup-archive): really run tests 2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
65cd607a49 fix(backups): force a full on continuous replication if the base could have been empty 2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
453ffc98d2 fix(backups): fall back to a full backup if it's possible that the base is broken 2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
f8df78c03d feat(backup-archive): add method to get the root of a chain 2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
3613b30aa1 fix(backups): stamp backup after the fix 2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
58332e6b16 doc(backups): more clarification on qcow2 + nbd 2026-07-27 19:45:25 +02:00
Florent BEAUCHAMP
9388fa4bb6 fix(disk-transform): erronous offset in DiskLargerBlock when copying data
the offset must be relative to the start of THIS large block, not to the
start of the disk — otherwise every large block after index 0 lands past
the end of destinationBlock and copy() silently writes nothing (zeroed block).

affect qcow2 incremental backup with NBD disable

NBD + qcow2 and Vhd backups don't go through this code path
2026-07-27 19:45:25 +02:00
Olivier Lambert
f683817dec feat(docs): working cookieless Matomo tracking on a dedicated site
The previous static/js/matomo.js snippet never loaded the tracker
script, so nothing was ever recorded; it also pointed at site 5, the
XO website. Replace it with the client module from the Vates VMS docs
on the new dedicated docs site (23): cookieless page-view tracking that
follows SPA navigations, and instrumentation of the lunr search box
reporting each settled query with its suggestion count, so zero-result
searches surface missing content. The feedback widget mirrors votes and
thumbs-down reasons as doc_feedback events.

Also fills in the Formbricks IDs for the "XO Docs page feedback"
survey, enabling the feedback widget.
2026-07-27 19:25:51 +02:00
Olivier Lambert
51fe6b8920 feat(docs): page feedback widget wired to Formbricks
Swizzles DocItem/Footer to show a 'Was this page helpful?' widget under
every doc article, ported from the Vates VMS docs. Votes post to the
self-hosted Formbricks instance via its public client API; a thumbs-down
asks why (outdated, unclear, missing info, inaccurate) plus optional
free text. The vote is recorded on click so it survives the visitor
leaving before the follow-up. Page path rides along as a hidden field
for per-page filtering.
2026-07-27 19:25:51 +02:00
Mathieu
74b952be2b feat: technical release (#10174) 2026-07-27 15:05:16 +02:00
Florent BEAUCHAMP
efbfb664f7 fix(backup-archives): regenerate cache on discrepencies (#10132)
* ensure cache is generated completely at the end of the backup job
* ensure a broken cache is regenerated even without merge during the cleanup phase
* ensure a broken cache is regenerated on request
2026-07-27 14:19:17 +02:00
Bastien Nollet
1616ec6f41 feat(backups): make fallback cause appear in the task (#10148) 2026-07-27 14:10:03 +02:00
Alina D
246e2e4eba feat(xo6/pool): add connect/disconnect actions on page site (#10075) 2026-07-27 13:26:20 +02:00
Mathieu
bcb91e8b4f feat(rest-api): add POST /srs route to create a storage repository (#9990) 2026-07-27 10:37:55 +02:00
Florent BEAUCHAMP
76b45a9b3b fix(xo-server): ensure import task progress is correctly updated (#10133) 2026-07-27 10:23:29 +02:00
Sylvère
5f42e437c7 feat(lite): add SR connect/disconnect/delete (#10006) 2026-07-27 09:44:23 +02:00
Sylvère
b76294a1b1 feat(web-core): refactor the sidebar system (#10120) 2026-07-27 09:43:16 +02:00
Grandalf
b7bce8e7e7 fix(csp): update Content-Security-Policy to allow connections to GitH… (#10162) 2026-07-27 09:03:17 +02:00
Pierre Brunet
1445f986cf feat: add azure section (#10166)
* feat: linter

* feat: add Azure doc
2026-07-24 17:59:06 +02:00
MlssFrncJrg
83b43a44f6 feat(xo5/hub): open Hub to all plans and gate recipes individually (#10117) 2026-07-24 17:30:04 +02:00
Olivier Lambert
235ff3c455 feat(xo6/new-vm): set HA restart priority at VM creation (#10072) 2026-07-24 14:15:22 +02:00
Alina D
b0194b2c5f feat(xo6): update input version (#10009) 2026-07-24 11:47:22 +02:00
Marie Garde
316122a683 feat(xo6/users): Add Users Table (#10029) 2026-07-24 10:34:04 +02:00
Sandrine Barrucand
954c0aaa9e fix(xo6): add separator between label and hint in table actions (#10143) 2026-07-23 16:19:37 +02:00
Sandrine Barrucand
377e315e02 fix(xo6/host): wrap Network name in PIF side panel (#10155) 2026-07-23 14:54:56 +02:00
Joris K
9b6f882424 fix(xo6/lite): fix some design inconsistency between pages on xo6 and lite (#10109) 2026-07-23 14:23:37 +02:00
Marie Garde
874da48125 feat(xo6): add enable/disable host button (#10074) 2026-07-23 12:02:46 +02:00
Pierre Brunet
e22ee2547b Fix: avoid default value for "private" in normalize-packages script (#10121) 2026-07-23 11:43:21 +02:00
Mathieu
43e45667ab fix(xo-server/rpu): LINSTOR packages update progress no longer exceeds 100% (#10122) 2026-07-23 11:24:23 +02:00
Florent BEAUCHAMP
c05f9bb357 feat(backup): update footer and header at once (#10123) 2026-07-23 11:16:56 +02:00
amouillard-vates
92f8ed4077 feat(lite): add copy info into JSON action to pool networks table (#10083) 2026-07-23 11:10:52 +02:00
MathieuRA
fc6415b4ec fix(xo-server): fix group deletion checking RBAC authorization 2026-07-23 10:52:29 +02:00
MathieuRA
53014d7d18 feat(rest-api): expose acl-roles/:id/groups 2026-07-23 10:52:29 +02:00
MathieuRA
5bb4ac62bf fix(xo-server): delete user don't remove the link between user and acl-role 2026-07-23 10:52:29 +02:00
MathieuRA
b4433e4ecc feat(rest-api): expose acl-roles/:id/users 2026-07-23 10:52:29 +02:00
MathieuRA
fee9bf8334 feat(xo-server): add groupIds and userIds to acl-role object 2026-07-23 10:52:29 +02:00
MathieuRA
1455801282 feat(xo-server): add aclRoleIds property to groups 2026-07-23 10:52:29 +02:00
MathieuRA
bd5937366a feat(rest-api): expose groups/:id/acl-roles 2026-07-23 10:52:29 +02:00
Joris K
d2667ac64b feat(xo6/lite): updated the modal so it can be closed using the X or by clicking outside it (#10137) 2026-07-23 09:06:19 +02:00
Pierre Brunet
de9687e2c0 fix: issue on file overwriting during vhdDirectory creation (#10139) 2026-07-22 15:04:00 +02:00
Florent BEAUCHAMP
c12bd807ed feat(qa-test): add a script to do a load test (#10119)
add new option and script to launch a load test :

    number of vm is configurable
    concurrency is configurable
    number of backup run is configurable

will do multiple run, then a restore
2026-07-22 14:36:50 +02:00
amouillard-vates
7b21690721 feat(xo6/pool): add new VM button and disconnect to pool tree actions (#10046) 2026-07-22 14:19:26 +02:00
Sandrine Barrucand
b719c0a018 refactor(xo6/vif): merge connect/disconnect buttons into a single component (#10112) 2026-07-22 13:25:45 +02:00
Grandalf
8e8afeb1b5 feat(xo-server): add content security policy (#10101) 2026-07-22 12:15:16 +02:00
Elise-F
0a5693345f feat(traffic rule type by VIF): display only the traffic rules for VMs and disable VMs without a VIF. (#9977)
- Show only traffic rules of VMs. Don't include `vm-snapshot` or `vm-template`
- For VM selection, display all VMs and disable those without a VIF, with a
message displayed
2026-07-22 11:01:17 +02:00
Bruno Verachten
ebce0e508f docs(xo): document VBD attach and note ISO mount is JSON-RPC only (#9981) 2026-07-21 16:35:13 +02:00
Olivier Lambert
64f1327f46 feat(rest-api): allow setting HA restart priority on VM creation (#10070)
See #10069
2026-07-21 16:02:08 +02:00
bleader
d2c28450d4 docs(sdn_controller.md): clarify openssl3 update instructions (#9821) 2026-07-21 11:45:09 +02:00
MlssFrncJrg
4eb43fbc36 feat(xo6): resolve objects in tasks names (#9830) 2026-07-21 10:37:36 +02:00
Bastien Nollet
e44034285b fix(backups): fix failed status on successful retry (#10129)
A VM backup log status is calculated depending on its subtask status. Previously, we made that it would be a failure if at least one of its subtask is a failure.

However, in case of successful retries, the intended behaviour is to have failed subtasks, which should not affect the parent task status. The first commit of this PR fixes that.

The second commit simplifies the getStatus function and its usage.
---------

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2026-07-21 09:50:07 +02:00
Thierry Goettelmann
9c1aa104f5 chore(deps): hoist pinia, vite and vue-router to root devDependencies (#10138) 2026-07-21 09:30:21 +02:00
Thierry Goettelmann
4ff8af838f feat(web-core): add hide-permanently package (#10108) 2026-07-21 09:28:07 +02:00
amouillard-vates
101fb46ad1 fix(xo6/site): remove site tree view new vm action (#10135) 2026-07-21 09:04:44 +02:00
Thierry Goettelmann
687dd5f456 feat(web-core): add promise-based overlay system (#10095) 2026-07-20 17:34:09 +02:00
Mathieu
ee6c6b3bbc feat(xo-server): delay load balancer re-enable after RPU (#10111)
XO-2706
2026-07-20 16:51:45 +02:00
Mathieu
54260268ba fix(xo-server): demote successful fallback logs (#10131)
XO-2734
2026-07-20 16:00:13 +02:00
Elise-F
0d53d7621a feat(lite): add the “connect/disconnect” action for a VIF on a VM (#10080) 2026-07-20 15:12:46 +02:00
Florent BEAUCHAMP
c3a2e89c1f feat(disk-cli): add option to show chain info, size, and improve peformances (#10063) 2026-07-20 14:37:20 +02:00
Enée DI IORIO
27fe714e17 feat(web-core): update UiTextArea to V5 (#9855) 2026-07-20 11:10:33 +02:00
Mathieu
5e6c4ad96d fix(xo-server): automatically reconnect to XenServer after connection (#10086)
A server left enabled but without a Xapi instance was never retried by anything in
xo-server. Three paths lead to that state:

    a failed explicit connect: xo-server boots while a pool is down, a user clicks
    "connect" during an outage, or the host-eject path exhausts its 5 attempts;
    a SESSION_INVALID teardown after a pool master restart (_sessionCall calls
    disconnect(), and nothing listens to server:disconnected);
    fixing a server's credentials/address while it is disconnected (server.set never
    triggers a connection).

Once there, the server stays disconnected forever, showing an error that can be
minutes or hours old. Support ends up chasing false "pool is dead" reports, typically
right after a rolling pool update, which is exactly when hosts reboot by design.
2026-07-20 09:42:12 +02:00
Florent BEAUCHAMP
926eb348d3 feat(xo-server): log more data to track memory leaks (#10041)
* feat(xo-server): log more data to track memory leaks

* changelog
2026-07-17 16:35:06 +02:00
Mathieu
bfb6b15cfa fix(xo-server/rpu): retry check_update while updater plugin is busy (#10115) 2026-07-17 16:17:09 +02:00
Grandalf
d05be427fb fix(xo-server): replace hasPermissions (#10093) 2026-07-17 11:48:37 +02:00
Pierre Brunet
76fa4bba51 fix(CI): do not log "ENOENT" when unlink fails after check (#10102)
* fix(CI): do not log "ENOENT" when unlink fails after check

* fix: await for mkdirp before continuing
2026-07-17 10:16:07 +02:00
Grandalf
d02098a397 fix(xo-server): don't log the admin password on server creation (#10094)
XO-2647
Introduced by #3579
2026-07-17 09:43:51 +02:00
Sandrine Barrucand
20c9ed8a41 feat(xo6/vm): add New VM button on Host view (#10048) 2026-07-16 16:18:31 +02:00
Mathieu
2f19fda3a5 feat(xo-server/rpu): assert each host can be evacuated right before its evacuation (#10097) 2026-07-16 15:16:17 +02:00
Mathieu
798bdeb973 fix(xo-server): disable auto power on during rolling pool update/reboot (#10104) 2026-07-16 14:34:50 +02:00
MlssFrncJrg
ac52f8b75e fix(perf-alert): update URL generation for alarms to support v6 routing (#10054)
* fix(perf-alert): update URL generation for alarms to support v6 routing

* changelog entry

* fix changelog

* Apply suggestions from code review

Co-authored-by: Mathieu <goore.csmoviz@gmail.com>

---------

Co-authored-by: Mathieu <goore.csmoviz@gmail.com>
2026-07-16 10:25:24 +02:00
Enée DI IORIO
691af588d3 feat(Lite): remove all coming soon (#10068) 2026-07-15 17:26:22 +02:00
Grandalf
cda3c3a724 perf(xo-server): don't wait for errored br to send data (#10025) 2026-07-15 15:32:21 +02:00
John P. Cooper
f2d6b7f0e4 Update XCP-ng version label in supported_hosts.md to bring into sync with supported status (#10092) 2026-07-15 15:20:56 +02:00
Sandrine Barrucand
1afffb8f75 feat(xo6): add possibility to export VM (#9989) 2026-07-15 14:42:36 +02:00
Mathieu
2b7b6baf02 feat(xo-server-openmetrics): estimate per-VM power consumption from CPU (#10031)
* feat(xo-server-openmetrics): estimate per-VM power consumption from CPU
usage

* refactor(xo-server-openmetrics): move hostPowerWatts undefined guard to
caller

* fix(xo-server-openmetrics): log IPMI power read failure as info
2026-07-15 14:06:49 +02:00
Florent BEAUCHAMP
0cfcb998a2 feat(disk-transform): port getBlockIndexesCount to the relevant classes (#10058) 2026-07-15 14:04:53 +02:00
Sandrine Barrucand
3ffd9b1f56 feat(xo6/vm): in the VM actions, add a separator to Delete, placeholder actions and submenu (#10036) 2026-07-15 13:18:34 +02:00
Mathieu
184d65cef1 feat(xo-server): add crash-resistant on-disk observability for RPU/RPR (#10078)
* feat(xo-server): add crash-resistant on-disk observability for RPU/RPR
(XO-2640)

* fix(xo-server): reconcile RPU heartbeats left pending at boot (XO-2640)

* chore(CHANGELOG): add RPU observability entry (XO-2640)

* docs(rpu): document on-disk traces and heartbeat for troubleshooting

* refactor(xo-server): address RPU observability review feedback
2026-07-15 12:23:36 +02:00
Elise-F
a7ba7c6a89 feat(xo6): add VM duplicate action (#9580) 2026-07-15 12:16:34 +02:00
Sandrine Barrucand
25da6d4c4b fix(xo6/xolite): switch CPU and RAM panels (#10059) 2026-07-15 11:47:20 +02:00
amouillard-vates
8b78ab985d feat(xo6/site): add new VM button and connect pool to site tree actions (#10047) 2026-07-15 11:45:19 +02:00
Joris K
2bee8c279d feat(xo6): add possibility to create VIF (#9677) 2026-07-15 11:26:42 +02:00
Bastien Nollet
5de65620f1 fix(xapi-explore-sr): fix crash on unreferenced VDI (#10113)
Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2026-07-15 11:20:33 +02:00
Olivier Lambert
b078d3d42d feat(xen-api): reconnect to a surviving pool member when the master is unreachable (#10016)
* feat(xen-api): reconnect to a surviving pool member when the master is unreachable

XO follows a pool master change only through HOST_IS_SLAVE redirection, which
needs the contacted host to be up and answering. That covers a graceful master
hand-over (the old master stays up as a slave and redirects us), but not the
case where the master host itself becomes unreachable while another host takes
over as master: HA automatic failover, or a manual emergency transition. The
stored URL then points at a host that no longer responds, login fails with a
transport error instead of HOST_IS_SLAVE, the existing retry never redirects,
and the connection stays stuck on the dead master.

Maintain a set of pool-member addresses (seeded from the configured target and
optional opts.candidateHostnames, refreshed from host.get_all_records on every
session open) and, when the current target is unreachable at the transport
level, probe the other members in parallel. The first to answer wins; if it is
a slave, its login redirects us to the current master through the existing
HOST_IS_SLAVE retry. XapiError responses (including auth failures) are not
failed over, so wrong credentials still fail fast.

* feat(xo-server): persist pool-member addresses so master failover survives a restart

The xen-api failover relies on an in-memory set of pool-member addresses, which
is lost when xo-server restarts: it then only knows the configured host. That is
exactly the address that is dead when XO runs on the very pool whose master just
failed and was HA-restarted onto a survivor, leaving XO unable to reconnect.

Persist the member addresses on the server record (Redis, JSON-encoded like the
existing error field), refresh them whenever the pool object is seen via the
xen-api candidateHostnames getter, and feed them back as opts.candidateHostnames
when (re)building the connection. After a restart the connection can then probe
the surviving members and reconnect to the new master.

* chore(changelog): reconnect to a surviving pool member on master failure

Add the user-facing changelog entry and list xen-api in the packages to
release for the master-failover reconnect feature.

* fix(xen-api): prune ejected hosts from pool-member failover candidates

host.get_all_records is authoritative current membership, so replace the
candidate set on refresh instead of unioning into it. An ejected host is
typically still alive as its own standalone master with the same
credentials, so leaving it in the set could make a failover probe
silently connect to the wrong pool. The currently-connected target is
preserved since host.address may differ from the address XO used to
reach it.

* fix(xen-api): harden surviving-member failover per review

- reject failover probes on sessionId-based connections (no reusable
  credentials to open a session on another host)
- bracket bare IPv6 addresses (host.address / HOST_IS_SLAVE) before using
  them as URL hostnames
- log the probe session out even when the host answers after the probe
  timeout, to avoid leaking a session
- abort the mid-session failover loop when the connection was torn down or
  replaced while waiting (e.g. user disconnected the server)
- xo-server: drop persisted pool-member addresses when the connection
  identity (host/credentials) changes, so a stale slave list can't send a
  failover to the wrong pool

See #10016
2026-07-15 11:04:16 +02:00
Bastien Nollet
913148ec2f chore(changelog): add previously unreleased package (#10106) 2026-07-13 10:48:01 +02:00
Mathieu
44e5ba4cf5 feat(rest-api): add PATCH /rest/v0/vdis/{id} (#9945) 2026-07-10 15:40:05 +02:00
Mathieu
db795ccfe2 feat(rest-api): ability to set transient for host.disable (#10040) 2026-07-10 15:33:08 +02:00
MlssFrncJrg
519b2bd7e1 feat(rest-api): add RBAC check for pool endpoints (#9891) 2026-07-10 15:27:26 +02:00
Grandalf
b5f5e39ddc feat(rest-api): add array type to FieldDefinition (#10055) 2026-07-10 14:33:00 +02:00
Alina D
f3f8519b7a fix(xo6): repair negative other value in backup repository dashboard (#10044) 2026-07-10 09:38:42 +02:00
Bastien Nollet
1d82c373da feat: release 6.6.2 (#10105) 2026-07-09 16:50:11 +02:00
Bastien Nollet
12b380e806 feat: technical release (#10103)
* feat(@xen-orchestra/fs): 4.9.2

* feat(@xen-orchestra/backup-archive): 1.0.4

* feat(@xen-orchestra/backups): 0.73.7

* feat(@xen-orchestra/proxy): 0.31.2

* feat(xo-server): 5.205.2

* chore(CHANGELOG): update next
2026-07-09 14:31:09 +02:00
Pierre Brunet
f8c9c66c40 fix(backups): remove warning for encrypted xva (size cannot be checked for encrypted data) 2026-07-09 12:00:48 +02:00
Pierre Brunet
1afa1e52ed fix(backups): transfer length and maxStreamLength in pipeline() 2026-07-09 12:00:48 +02:00
Florent BEAUCHAMP
b2a48ffacc fix(fs): compute the output stream precisely when encrypted (#10061)
* fix(fs): compute the output stream precisely when encrypted

the current encryption add 12 byte of initialization vector at
the start + 16 bytes of authentication tag at the end

thus, the destination stream can be bigger than the source stream,
and this length is used to compute the optimal chunk size for s3/
azure

from https://xcp-ng.org/forum/topic/12328/error-mirroring-full-backups-to-backblaze-b2

---------

Co-authored-by: Pierre Brunet <pierre.brunet@vates.tech>
2026-07-09 10:35:40 +02:00
Bastien Nollet
a3077cdb62 feat: release 6.6.1 (#10087) 2026-07-08 09:56:44 +02:00
Bastien Nollet
3ba9523dfe feat: technical release (#10084) 2026-07-07 14:37:05 +02:00
spacotte-vates
377581b44a fix(backups): add warn with debug for snapshot pileup issue (#10076) 2026-07-06 14:56:03 +02:00
Thomas Moraine
090ce7e721 docs(xo): refactor navbar navigation to create a unified documentation ecosystem (#10067)
* docs(xo): overhaul navbar to create a unified doc ecosystem

- Clean up the navbar by dropping the redundant GitHub link (retained in footer).
- Relocate "Blog" to the "What's new in XO 6" sidebar.
- Introduce direct navbar links for XCP-ng and Vates VMS.
- Rename "Documentation" to "Xen Orchestra".

This aligns the navigation layout across our three documentation sites, allowing readers to smoothly transition between ecosystems with a fully unified experience.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(xo): move home link from header to footer

To make the navbar cleaner and make space
for the links to the other documentation websites
We're moving the Home link from the header to the footer
and renaming the link to 'About Xen Orchestra'.
Also, we're displaying the 'What's new in XO 6' section again
which now contains a link to the XO blog

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-07-03 10:30:40 +02:00
Thomas Moraine
d53dd3e990 docs(xo): import devops tools documentation from vates vms (#10062) 2026-07-03 09:24:44 +02:00
Thomas Moraine
1b0dbc7d3b docs(xo6): hide categories that dont contain any actual pages (#10066)
Recently we've introduced new sections for the xo6 category of the XO doc.
These categories were meant to be filled later with new pages.
The pages were created as placeholders, but didn't actually contain any text, just titles.
These made readers confused, so this commit
Hides the sections from the sidebar until we update them
With actual content.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-07-03 08:44:14 +02:00
Florent BEAUCHAMP
bb4daac07a fix(xo-server): store rrd as Float32 instead of string (#10039) 2026-07-02 15:57:43 +02:00
spacotte-vates
63f8d624fd fix(backups): improve resume of backup merge failure (#10053)
Introduced by #9765

When a backup merge fails on a VHD file, the footer might be missing. This caused a footer1 !== footer2 error and triggered a full backup while not deleting the previous chain and resulted in all further backup jobs to be displayed as failures even though the new backup chain was working properly.

The fix allows the cleanVm to resume the merge where it stops even if the VHD file footer is missing.
2026-07-01 17:01:38 +02:00
Alina D
373a4c8c6d fix(web-core): make VDI used space in progress bar appear correctly (#10027)
Used space progress bar now shows correctly filled at 100% instead of appearing half-filled

References issue :  XO-1696
2026-07-01 10:43:37 +02:00
Warren Postdam
1ec57392fc docs: update users to add passkeys auth (#9975)
Add configuration elements to allow FIDO2 (passkeys) administrators to configure
with XO by using `auth-oidc` plugin.
2026-07-01 10:24:55 +02:00
Joris K
376bb3603f fix(xo6): fixed an issue where the selected item in the treeview would scroll to the top (#10052) 2026-07-01 09:02:12 +02:00
Bastien Nollet
3bc70f657d feat(lite): 0.23.0 (#10050) 2026-06-30 16:50:07 +02:00
Bastien Nollet
f3b2ac5983 feat: release 6.6.0 (#10049) 2026-06-30 16:20:58 +02:00
Bastien Nollet
eb4c55d82c feat: technical release (#10045)
* feat(@vates/types): 1.28.0

* feat(@xen-orchestra/acl): 1.4.0

* feat(@xen-orchestra/backups): 0.73.5

* feat(@xen-orchestra/proxy): 0.31.0

* feat(@xen-orchestra/rest-api): 0.35.0

* feat(@xen-orchestra/web): 0.55.1

* feat(xo-server): 5.205.0

* feat(xo-server-sdn-controller): 1.4.0

* chore(CHANGELOG): update next

* clear changelog
2026-06-30 12:01:18 +02:00
spacotte-vates
d43ca09b8b feat(rest-api): add 7 rest actions on hosts (#9887) 2026-06-30 11:37:17 +02:00
Grandalf
3ad3ecb244 feat(sdn-controller): add patch routes for vif and network (#9936) 2026-06-30 11:29:29 +02:00
Bastien Nollet
7bbe139b36 fix(backups): fix memory error during delta replication (#10034) 2026-06-29 17:25:17 +02:00
spacotte-vates
235fc07404 feat(rest-api): backup-repository health/bench routes (#9847) 2026-06-29 17:12:51 +02:00
Bastien Nollet
26038fdaec feat: technical release (#10037)
* feat(@xen-orchestra/backups): 0.73.4

* feat(@xen-orchestra/web-core): 0.56.0

* feat(@xen-orchestra/proxy): 0.30.5

* feat(@xen-orchestra/vmware-explorer): 0.14.0

* feat(@xen-orchestra/web): 0.55.0

* feat(xo-server): 5.204.0

* feat(xo-server-netbox): 1.13.1

* feat(xo-web): 5.199.0

* chore(CHANGELOG): update next
2026-06-29 14:38:39 +02:00
Mathieu
4fe505b229 fix(web-core): required disappear after clicking on VtsSelect (#10032) 2026-06-29 14:04:20 +02:00
Sylvère
276c4e5f7e feat(lite): add storage tabs and read-only SR views (#10005) 2026-06-29 13:31:52 +02:00
Florent BEAUCHAMP
91badc593c fix(@xen-orchestra/proxy): invalid parameter in listVmBackups (#10035)
introduced by #9143
2026-06-29 13:01:45 +02:00
Pierre Brunet
130b1612e6 fix(xo-server): add safeguards to xo-server shutdown and update changelog 2026-06-29 10:55:29 +02:00
Pierre Brunet
e8c43dafce fix(s3): stop s3 agents properly after xo-server shutdown 2026-06-29 10:55:29 +02:00
Pierre Brunet
8e0c0e05c0 feat(xo-server-auth-ldap): support LDAP multidomain (#10015) 2026-06-29 10:49:09 +02:00
Enée DI IORIO
71d71b9742 feat(lite): use uiLoader instead of uiSpinner (#10023) 2026-06-29 10:35:45 +02:00
spacotte-vates
34051927a0 fix(encryption): handle missing xen-tools (#9987)
Introduced by 6f68b04

Improve missing xen-tools handling of encryption with better logic and logs.
2026-06-29 09:41:35 +02:00
Sandrine Barrucand
9e6ad45a79 feat(xo6): add possibility to migrate VDI on SR (#9973) 2026-06-29 08:07:21 +02:00
Sandrine Barrucand
52591ec65b feat(xo6/vm): add actions available on VMs tab (#10030) 2026-06-26 17:32:23 +02:00
Mathieu
bf1e992161 feat(rbac): add Storage administrator built-in role template (#9963) 2026-06-26 16:40:52 +02:00
vates-bot
29b9bc46e4 feat(web-stack/i18n): update Xen Orchestra 6 (#9998) 2026-06-26 14:49:43 +02:00
Florent BEAUCHAMP
9d470997e8 fix(xo-server,netbox): properly close hrp stream on error (#10026)
when hrp get a 404, the stream stay ready to be consumed, for example to get
more information on the error

but most of the usage won't handle this case, and will keep the stream in memory
(for example a failing XC patches check) leaking memory

This fixes the calls from xo, and also identifies and fixes the point where we
need body content on 4xx code

relevant code in hrp is here:
https://github.com/JsCommunity/http-request-plus/blob/master/index.js#L149
2026-06-26 14:09:49 +02:00
OlivierFL
836e06843d feat(xo-core): introduce new VtsCardObjectTitle component (#9755) 2026-06-26 12:12:56 +02:00
Joris K
5160ec4c7e fix(web-core/xo6): stream infra tree collections and flatten into a virtualized list (#9986) 2026-06-26 11:33:44 +02:00
Marie Garde
cc88cc395c feat(xo6/vdi): add export VDI action (#9983) 2026-06-26 11:33:14 +02:00
Enée DI IORIO
6ea64e9413 feat(lite): Use web-core's UiLogEntryViewer instead of xo-lite UiRaw (#9995) 2026-06-26 10:28:27 +02:00
Florent BEAUCHAMP
b440f41cd5 fix(xo-server): ignore template memory_static_min is it's less than actual vm memory 2026-06-25 16:29:46 +02:00
Florent BEAUCHAMP
7ea6a5c541 feat(xo-web): show status of previous transfer to help the 2 part transfer 2026-06-25 16:29:46 +02:00
Florent BEAUCHAMP
1987008db1 fix(xo-server): v2v should create a DVDRom is source has one 2026-06-25 16:29:46 +02:00
Florent BEAUCHAMP
4e1e1510e8 fix(xo-server): v2V not updating correctly other config
setting undefined is a no ops, we need to use null to clean a value

this don't have any visible or functionnal impact, but complexify
the build of a filter to look for any non completely migrated VM
2026-06-25 16:29:46 +02:00
Grandalf
e9a8ffe315 feat(xo-server): add esxi.importDisk (#10024)
* feat(xo-server): add esxi.importDisk
2026-06-25 16:17:43 +02:00
OlivierFL
8639e86227 fix: changelog (#10028) 2026-06-25 15:43:26 +02:00
Pierre Brunet
89e1b051cb feat: add failover URIs in ldap domains (#9961) 2026-06-25 14:09:03 +02:00
Florent BEAUCHAMP
c162839d4e docs(immutable-backups): document how to lift the immutability for one VM (#9974) 2026-06-25 11:57:31 +02:00
Enée DI IORIO
d73e9d5e94 chore(lite): remove unused component (#10011) 2026-06-25 10:55:20 +02:00
Sylvère
ab472af351 feat(web-core/web/lite): updated UiPanel to v5 (#9836) 2026-06-25 10:52:35 +02:00
Sylvère
01d8e21538 feat(xo6): warn and link to doc when traffic rules are not in the expected format (#10020)
Co-authored-by: Sylvere <sylvere.armange@vates.tech>
2026-06-25 10:06:35 +02:00
Bastien Nollet
bf2954edb0 fix(backups): uncaught VM error (#10021) 2026-06-25 10:06:19 +02:00
Enée DI IORIO
0fca9b2cde feat(lite): use vtsDivider instead of UiSeparator (#10017) 2026-06-25 10:05:46 +02:00
Alina D
8b9b99b68f feat(xo6): change redirect target (#9984) 2026-06-25 09:22:05 +02:00
Mathieu
c74a83ed06 fix(web): no active job instead no job (#10013) 2026-06-24 16:39:33 +02:00
Bastien Nollet
b96a6dc1e9 feat: technical release (#10022)
* feat(@xen-orchestra/fs): 4.9.1

* feat(@vates/generator-toolbox): 1.1.3

* feat(@xen-orchestra/disk-transform): 1.3.1

* feat(@vates/fuse-vhd): 2.1.3

* feat(xo-common): 0.10.0

* feat(xen-api): 4.7.8

* feat(@vates/types): 1.27.0

* feat(@xen-orchestra/acl): 1.3.0

* feat(@xen-orchestra/backup-archive): 1.0.2

* feat(@xen-orchestra/qcow2): 1.3.1

* feat(@xen-orchestra/xapi): 8.9.0

* feat(@xen-orchestra/backups): 0.73.3

* feat(@xen-orchestra/disk-cli): 2.1.2

* feat(@xen-orchestra/web-core): 0.55.0

* feat(@xen-orchestra/mcp): 1.4.1

* feat(@xen-orchestra/mixins): 0.20.1

* feat(@xen-orchestra/openflow): 0.1.4

* feat(@xen-orchestra/proxy): 0.30.4

* feat(@xen-orchestra/qa-test): 1.1.0

* feat(@xen-orchestra/rest-api): 0.34.0

* feat(@xen-orchestra/upload-ova): 0.1.8

* feat(@xen-orchestra/web): 0.54.0

* feat(xo-acl-resolver): 0.5.3

* feat(xo-server): 5.203.0

* feat(xo-server-audit): 0.15.1

* feat(xo-server-ipmi-sensors): 2.0.1

* feat(xo-server-load-balancer): 0.13.0

* feat(xo-server-netbox): 1.13.0

* feat(xo-server-openmetrics): 1.7.0

* feat(xo-server-sdn-controller): 1.3.3

* feat(xo-web): 5.198.0

* chore(CHANGELOG): update next
2026-06-24 10:18:29 +02:00
Sebastien Marie
a6f2a25aec docs: explain traffic rules migration path (from OpenFlow Protocol to XAPI Plugin) (#9999)
* fix(configuration.md): remove trailing spaces

* feat(sdn-controller.md): explain traffic rules migration path: from OpenFlow Protocol to XAPI Plugin

- add sdn-migration.py python script
- in configuration.md, refer to sdn_controller.md for changing `useDirectChannel`
- in sdn_controller.md, provide documentation on migration path
2026-06-24 08:53:34 +02:00
Sebastien Marie
8f31ff7481 feat(xo-server-sdn-controller): traffic-rules: check the configuration matches the rules format (#10000)
XCPNG-3260

report an error if we detect that the runtime configuration doesn't match the of-rules format stored in xapi database.

the admin is expected to ran migration script by hand to fix it.
2026-06-24 08:52:48 +02:00
Enée DI IORIO
d7bfb171fd feat(lite): Use web-core's UiCardTitle instead of xo-lite version (#9982) 2026-06-23 17:10:25 +02:00
Sylvère
fc45f3e59e feat(web-stack): extract SR types/utils to web-core (#10004) 2026-06-23 17:09:22 +02:00
Florent BEAUCHAMP
cecf1cb6bc feat(xoa): add an option to take a snapshot before upgrade (#9920) 2026-06-23 16:10:29 +02:00
Thomas Moraine
bf0e40977c docs(sdn_controller.md): mention that network-wide rules can now be created in the UI with XO6 (#10019) 2026-06-23 15:33:40 +02:00
Marie Garde
caa174c69c feat(xo6): vdi dedicated page (#9949) 2026-06-23 11:37:52 +02:00
Mathieu
304e219441 fix(rest-api/dashboard): count job sequence schedules for backup computation (#9997) 2026-06-23 10:24:57 +02:00
Enée DI IORIO
9c2739b6dd feat(lite): Use web-core's UiCard instead of xo-lite version (#9980) 2026-06-23 10:16:31 +02:00
Pierre Brunet
7a24dba9c1 fix(backup/cleanVM): avoid useless warning (#9996) 2026-06-23 10:08:32 +02:00
Mathieu
a5d410e0aa feat(xo-server-openmetrics): include XO tags as tags label (#9971)
Fixes #9628
Plane : XO-2244
2026-06-23 10:07:15 +02:00
Thomas Moraine
64401c4040 docs: make a more logical doc structure for xo6 (#9972)
This commit creates a new, refreshed tree structure
for the XO 6 documentation, which we intend
to be more helpful for readers and XO 6 users.
New sections are created, to be completed later.
Additionnally, the ACLv2/RBAC page has been
moved from the Support section to the Security section.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-06-23 09:41:13 +02:00
Pierre Briffaux
93909d9851 feat(xo6/rbac): add admin tab with user management menu (#9947) 2026-06-22 17:25:42 +02:00
Grandalf
5604289f3b test(sdn-controller): add tests (#9957)
Adds rest api endpoints testing for the sdn controller.
Don't forget to set the required SDN_CONTROLLER_VM_ID to a vm that has both a vif and a network.
Run @xen-orchestra/qa-test yarn qa:sdn-controller for testing.
2026-06-22 17:09:39 +02:00
Mathieu
cf569efff2 feat(rest-api): add PATCH /rest/v0/vifs/{id} (#9935) 2026-06-22 15:54:05 +02:00
Sandrine Barrucand
4bde431167 feat(xo6/traffic-rules): add delete button (#9956) 2026-06-18 12:14:56 +02:00
Mathieu
37868e3a74 fix(rest-api): fix vm dashboard replica detection (#9991)
introduced by 680b810886

See https://github.com/vatesfr/xen-orchestra/pull/9806/changes#diff-6f2227ada6bc2da3fd324691bb30b604b7c61a2cdbeebcad7969a8233cd4f0c4R394
2026-06-18 10:40:13 +02:00
Florent BEAUCHAMP
443de97c57 fix(qa-test): metadata test can run so fast there is a datetime collision (#9954) 2026-06-17 17:57:02 +02:00
vates-bot
9e39d0d534 feat(web-stack/i18n): update Xen Orchestra 6 (#9914) 2026-06-17 17:01:24 +02:00
Mathieu
f166037b4d docs(rest-api): update .USAGE.md (#9967)
* docs(rest-api): update .USAGE.md

* Mathieu.P's feedback
2026-06-17 15:19:00 +02:00
Alina D
60c97d3678 feat(xo6): add policy tag colors (#9979) 2026-06-17 15:08:53 +02:00
Joris K
62ba0a5a30 feat(web-core): update table header to remove icons (#9943) 2026-06-17 11:36:50 +02:00
Pierre Brunet
2a5c350a15 fix(disk-cli): update isDisk call parameters (#9988) 2026-06-17 10:55:34 +02:00
Pierre Brunet
4a8b29530b chore: fix changelog (#9993) 2026-06-17 10:15:02 +02:00
Pierre Brunet
9a710e5bf3 feat: update packages (#9969) 2026-06-17 10:02:00 +02:00
Sylvère
8eb6543905 feat(xo6): add the SR connect feature (#9877) 2026-06-16 11:52:29 +02:00
Alina D
b99f686d0c feat(xo6): update CPU labels to vCPU in VM system resource card (#9950) 2026-06-16 10:59:19 +02:00
Stephen Boyd
07b45e157d feat(xo-server-netbox): add option to ignore private IPs (#6306)
Optionally ignore RFC 1918 VM IPs during synchronization
2026-06-16 09:33:39 +02:00
Enée DI IORIO
3e7f003060 feat(lite/xoa-deploy): add confirmation before leaving deployment (#9893)
Add a message if you try to leave the page during XOA deployment.

Unfortunately, it is not possible to have a custom info box for the
`beforeUnload` event. So for homogeneity, `onBeforeRouteLeave` is
`window.confirm` too.
2026-06-16 09:27:00 +02:00
MlssFrncJrg
cf26d3f5b2 feat(rbac): create Administrator template role (#9885) 2026-06-15 15:50:09 +02:00
Florent BEAUCHAMP
60f59509a9 fix(backups): remove inspector (#9966) 2026-06-15 14:31:31 +02:00
Bastien Nollet
801a1f70fe feat(load-balancer): wrap migrations in XO Tasks for readability (#9909)
Use XO Tasks in the load balancer so users can see on the task page what the load balancer does. Now, it creates a task (with no progress) for each VM migration.

The task has a description explaining the reason of the migration, which can be accessed by opening the task log.

XO-992
2026-06-15 13:59:48 +02:00
Enée DI IORIO
b06779655e fix(web-core/VtsStateHero): behavior on mobile (#9363) 2026-06-15 13:46:08 +02:00
Grandalf
d1b99b9e86 feat(rest-api): add forgetRepository endpoint (#9846) 2026-06-15 10:59:09 +02:00
Mathieu
04deb9e328 feat(acl): add Network administrator built-in role template (#9952) 2026-06-12 15:47:22 +02:00
Nathanaël HANNEBERT
8c51de5043 fix(docs): typo preventing building of docs 2026-06-12 11:48:50 +02:00
Florent BEAUCHAMP
de5ea8a888 fix(backups): file level restore : omnibus fixes (#9776) 2026-06-12 11:40:16 +02:00
971 changed files with 37448 additions and 13828 deletions

View File

@@ -4,3 +4,4 @@
# Generated files
/@xen-orchestra/web/src/route-map.d.ts
/@xen-orchestra/lite/src/route-map.d.ts

View File

@@ -1,12 +1,12 @@
import LRU from 'lru-cache'
import Fuse from 'fuse-native'
import LRU from 'lru-cache'
import { createLogger } from '@xen-orchestra/log'
import { VhdSynthetic } from 'vhd-lib'
import { Disposable, fromCallback } from 'promise-toolbox'
const { warn } = createLogger('vates:fuse-vhd')
// build a s stat object from https://github.com/fuse-friends/fuse-native/blob/master/test/fixtures/stat.js
// build a stat object from https://github.com/fuse-friends/fuse-native/blob/master/test/fixtures/stat.js
const stat = st => ({
mtime: st.mtime || new Date(),
atime: st.atime || new Date(),
@@ -20,42 +20,109 @@ const stat = st => ({
export const mount = Disposable.factory(async function* mount(handler, diskPath, mountDir) {
const vhd = yield VhdSynthetic.fromVhdChain(handler, diskPath)
const cache = new LRU({
max: 16, // each cached block is 2MB in size
})
await vhd.readBlockAllocationTable()
const { blockSize } = vhd.header
// A zeroed buffer returned for VHD blocks that are not present in the chain.
// Never mutated — safe to share across all reads.
const EMPTY_BLOCK = Buffer.alloc(blockSize, 0)
// --- coalescing block read queue ---
//
// FUSE read requests from NTFS-3g arrive in small chunks (typically 4 KiB)
// while VHD blocks are 2 MiB. Reading blocks one at a time from the backing
// store (CIFS/NFS/…) bounds libuv threadpool usage to 1 concurrent I/O call.
//
// Coalescing: all pending requests for the same blockId share a single
// readBlock() call. When the block resolves, every waiter is fulfilled at once.
//
// Together these two properties prevent the FUSE-on-FUSE threadpool deadlock:
// concurrent NTFS-3g reads no longer starve the VHD block reads they depend on.
// In-memory block cache. Each VHD block is 2 MiB; 16 blocks = 32 MiB.
// This is the primary tool against the 300× CIFS amplification: NTFS-3g
// reads a 2 MiB block in ~512 sequential 4 KiB FUSE requests. Without a
// cache, each request causes a full 2 MiB CIFS read. With the cache, only
// the first request reads from CIFS; the rest are served from RAM.
const blockCache = new LRU({ max: 64 })
const pendingBlocks = new Map() // blockId → Promise<Buffer>
const readQueue = [] // { blockId, resolve, reject }[]
let queueRunning = false
// log stats every 10 s while there is activity
function enqueueBlock(blockId) {
// 1. serve from cache if available
if (blockCache.has(blockId)) {
return Promise.resolve(blockCache.get(blockId))
}
// 2. coalesce: join an existing in-flight read for this block
if (pendingBlocks.has(blockId)) {
return pendingBlocks.get(blockId)
}
const p = new Promise((resolve, reject) => {
readQueue.push({ blockId, resolve, reject })
})
pendingBlocks.set(blockId, p)
p.then(
data => {
pendingBlocks.delete(blockId)
blockCache.set(blockId, data)
},
() => pendingBlocks.delete(blockId)
)
drainQueue()
return p
}
function drainQueue() {
if (queueRunning) return
queueRunning = true
// async IIFE — errors are forwarded to individual reject() callbacks,
// so the outer floating promise is always fulfilled
;(async () => {
while (readQueue.length > 0) {
const { blockId, resolve, reject } = readQueue.shift()
try {
resolve((await vhd.readBlock(blockId)).data)
} catch (err) {
reject(err)
}
}
queueRunning = false
})()
}
async function readData(buf, len, pos) {
if (len === 0) return 0
const startBlockId = Math.floor(pos / blockSize)
// use (pos + len - 1) so an exact block boundary doesn't include the next block
const endBlockId = Math.floor((pos + len - 1) / blockSize)
let copied = 0
for (let blockId = startBlockId; blockId <= endBlockId; blockId++) {
const data = vhd.containsBlock(blockId) ? await enqueueBlock(blockId) : EMPTY_BLOCK
const offsetStart = blockId === startBlockId ? pos % blockSize : 0
const offsetEnd = blockId === endBlockId ? ((pos + len - 1) % blockSize) + 1 : blockSize
data.copy(buf, copied, offsetStart, offsetEnd)
copied += offsetEnd - offsetStart
}
return copied
}
const fuse = new Fuse(mountDir, {
async readdir(path, cb) {
if (path === '/') {
return cb(null, ['vhd0'])
}
if (path === '/') return cb(null, ['vhd0'])
cb(Fuse.ENOENT)
},
async getattr(path, cb) {
if (path === '/') {
return cb(
null,
stat({
mode: 'dir',
size: 4096,
})
)
}
if (path === '/vhd0') {
return cb(
null,
stat({
mode: 'file',
size: vhd.footer.currentSize,
})
)
}
if (path === '/') return cb(null, stat({ mode: 'dir', size: 4096 }))
if (path === '/vhd0') return cb(null, stat({ mode: 'file', size: vhd.footer.currentSize }))
cb(Fuse.ENOENT)
},
read(path, fd, buf, len, pos, cb) {
if (path === '/vhd0') {
return vhd.readRawData(pos, len, cache, buf).then(cb, error => {
return readData(buf, len, pos).then(cb, error => {
warn('read error', { path, len, pos, error })
cb(Fuse.EIO)
})
@@ -64,7 +131,9 @@ export const mount = Disposable.factory(async function* mount(handler, diskPath,
},
})
return new Disposable(
() => fromCallback(cb => fuse.unmount(cb)),
() => {
return fromCallback(cb => fuse.unmount(cb))
},
fromCallback(cb => fuse.mount(cb))
)
})

View File

@@ -1,6 +1,6 @@
{
"name": "@vates/fuse-vhd",
"version": "2.1.2",
"version": "2.1.3",
"license": "ISC",
"private": false,
"homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@vates/fuse-vhd",
@@ -19,12 +19,12 @@
},
"main": "./index.mjs",
"dependencies": {
"@xen-orchestra/fs": "^4.9.0",
"@xen-orchestra/fs": "^4.9.3",
"@xen-orchestra/log": "^0.7.2",
"fuse-native": "^2.2.6",
"lru-cache": "^7.14.0",
"promise-toolbox": "^0.21.0",
"vhd-lib": "^4.16.0"
"vhd-lib": "^4.16.1"
},
"bin": {
"xo-fuse-vhd": "./cli.mjs"

View File

@@ -1,6 +1,6 @@
{
"name": "@vates/generator-toolbox",
"version": "1.1.2",
"version": "1.1.3",
"main": "dist/index.mjs",
"license": "MIT",
"private": false,
@@ -10,7 +10,7 @@
"rimraf": "^6.0.1",
"@types/node": "^20.6",
"typescript": "~5.6",
"typescript-eslint": "^8.23.0"
"typescript-eslint": "^8.61.0"
},
"scripts": {
"build": "tsc",

View File

@@ -19,6 +19,12 @@ export class NbdDisk extends RandomAccessDisk {
/** @type {number} */
#blockSize
/** @type {number | undefined} */
#hasBlockCursor
/** @type {number | undefined} */
#hasBlockPreviousIndex
constructor(nbdInfos, blockSize, { dataMap } = {}) {
super()
this.#blockSize = blockSize
@@ -27,11 +33,38 @@ export class NbdDisk extends RandomAccessDisk {
}
#processDatamap(rawDataMap) {
return rawDataMap
.filter(({ type }) => type === 0)
const ranges = rawDataMap
.filter(({ type, length }) => type === 0 && length > 0)
.map(({ offset, length }) => ({ offset, length }))
.sort(({ offset: offset1 }, { offset: offset2 }) => offset1 - offset2)
// hasBlock()'s forward-only cursor is only correct if the extents are
// sorted AND disjoint: an earlier extent must never reach into a block
// after a later one, otherwise the cursor could skip it and wrongly report
// a block as empty, dropping data and corrupting the output.
// Touching extents (cur.offset === prevEnd) are merged
const merged = []
for (const range of ranges) {
const last = merged[merged.length - 1]
if (last !== undefined) {
const lastEnd = last.offset + last.length
if (range.offset < lastEnd) {
throw new Error(
`overlapping ranges in data map: [${last.offset}, ${lastEnd}) and [${range.offset}, ${range.offset + range.length})`
)
}
if (range.offset === lastEnd) {
// touching: extend the previous extent
last.length += range.length
continue
}
}
// gap (or first range): new extent
merged.push(range)
}
return merged
}
/**
* @param {number} index
* @returns {Promise<DiskBlock>}
@@ -117,6 +150,34 @@ export class NbdDisk extends RandomAccessDisk {
return [...indexes].sort((a, b) => a - b)
}
/**
* Counts the allocated blocks without materializing the index list.
*
* @returns {number}
*/
getBlockIndexesCount() {
if (!this.#dataMap) {
throw new Error("can't getBlockIndexesCount before init")
}
const blockSize = this.getBlockSize()
let count = 0
let lastCountedBlock = -1
for (const { offset, length } of this.#dataMap) {
const firstBlockIndex = Math.floor(offset / blockSize)
const lastBlockIndex = Math.floor((offset + length - 1) / blockSize)
const from = Math.max(firstBlockIndex, lastCountedBlock + 1)
if (lastBlockIndex >= from) {
count += lastBlockIndex - from + 1
lastCountedBlock = lastBlockIndex
}
}
return count
}
/**
* @param {number} index
* @returns {boolean}
@@ -125,8 +186,32 @@ export class NbdDisk extends RandomAccessDisk {
if (!this.#dataMap) {
throw new Error("can't hasBlock before init")
}
const blockStart = index * this.getBlockSize()
const blockEnd = (index + 1) * this.getBlockSize()
return this.#dataMap.some(({ offset, length }) => offset + length > blockStart && offset < blockEnd)
const blockSize = this.getBlockSize()
const blockStart = index * blockSize
const blockEnd = blockStart + blockSize
let startExtentIndex = 0
if (this.#hasBlockCursor !== undefined && index >= this.#hasBlockPreviousIndex) {
startExtentIndex = this.#hasBlockCursor
} else {
this.#hasBlockCursor = undefined
}
this.#hasBlockPreviousIndex = index
const dataMap = this.#dataMap
const l = dataMap.length
for (let i = startExtentIndex; i < l; i++) {
const { offset, length } = dataMap[i]
if (offset >= blockEnd) {
// extents are sorted: nothing from here on can overlap this block
break
}
if (offset + length > blockStart) {
this.#hasBlockCursor = i
return true
}
}
return false
}
}

View File

@@ -0,0 +1,127 @@
import { describe, it } from 'node:test'
import assert from 'node:assert'
import { NbdDisk } from './NbdDisk.mjs'
const KiB = 1024
const MiB = 1024 * KiB
const TiB = 1024 * 1024 * MiB
const SOURCE_BLOCK_SIZE = 2 * MiB // VHD_BLOCK_SIZE, the block size used when importing from VMware
const CLUSTER_SIZE = 64 * KiB // qcow2 cluster size
// hasBlock() only needs the dataMap (passed through the constructor) and the
// block size, so we can exercise it without a live NBD connection.
describe('NbdDisk.hasBlock', () => {
it('reports allocated and unallocated blocks correctly', () => {
const dataMap = [
{ type: 0, offset: 0, length: SOURCE_BLOCK_SIZE }, // block 0
{ type: 0, offset: 5 * SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE }, // block 5
]
const disk = new NbdDisk({}, SOURCE_BLOCK_SIZE, { dataMap })
assert.strictEqual(disk.hasBlock(0), true)
assert.strictEqual(disk.hasBlock(1), false)
assert.strictEqual(disk.hasBlock(4), false)
assert.strictEqual(disk.hasBlock(5), true)
assert.strictEqual(disk.hasBlock(6), false)
})
it('handles repeated and out-of-order queries with the resume cursor', () => {
const dataMap = [
{ type: 0, offset: 0, length: SOURCE_BLOCK_SIZE }, // block 0
{ type: 0, offset: 5 * SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE }, // block 5
]
const disk = new NbdDisk({}, SOURCE_BLOCK_SIZE, { dataMap })
// repeated indexes (the qcow2 path queries each 2MB block ~32 times as it
// walks 64KB clusters) must return a stable answer
assert.strictEqual(disk.hasBlock(5), true)
assert.strictEqual(disk.hasBlock(5), true)
assert.strictEqual(disk.hasBlock(6), false)
assert.strictEqual(disk.hasBlock(6), false)
// querying backwards must still be correct (falls back to a full scan)
assert.strictEqual(disk.hasBlock(0), true)
assert.strictEqual(disk.hasBlock(5), true)
})
it('invalidates the cursor on a backward miss (no stale skip)', () => {
const dataMap = [
{ type: 0, offset: 3 * SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE }, // block 3
{ type: 0, offset: 10 * SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE }, // block 10
]
const disk = new NbdDisk({}, SOURCE_BLOCK_SIZE, { dataMap })
assert.strictEqual(disk.hasBlock(10), true) // forward hit: cursor parks on the high extent
assert.strictEqual(disk.hasBlock(1), false) // backward miss: must invalidate the cursor
assert.strictEqual(disk.hasBlock(3), true) // lower extent must not be skipped
})
it('throws on overlapping ranges (would break the resume cursor)', () => {
const dataMap = [
{ type: 0, offset: 0, length: SOURCE_BLOCK_SIZE },
{ type: 0, offset: SOURCE_BLOCK_SIZE / 2, length: SOURCE_BLOCK_SIZE }, // overlaps the previous range
]
assert.throws(() => new NbdDisk({}, SOURCE_BLOCK_SIZE, { dataMap }), /overlapping ranges/)
})
it('merges touching ranges and drops zero-length / non type-0 entries', () => {
const dataMap = [
{ type: 0, offset: 2 * SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE }, // [2, 3)
{ type: 0, offset: 0, length: SOURCE_BLOCK_SIZE / 2 }, // [0, .5)
{ type: 0, offset: SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE }, // [1, 2)
{ type: 0, offset: SOURCE_BLOCK_SIZE / 2, length: SOURCE_BLOCK_SIZE / 2 }, // [.5, 1), touches [0, .5)
{ type: 1, offset: 10 * SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE }, // wrong type: ignored
{ type: 0, offset: 7 * SOURCE_BLOCK_SIZE, length: 0 }, // zero length: ignored
]
const disk = new NbdDisk({}, SOURCE_BLOCK_SIZE, { dataMap })
assert.strictEqual(disk.hasBlock(0), true)
assert.strictEqual(disk.hasBlock(1), true)
assert.strictEqual(disk.hasBlock(2), true)
assert.strictEqual(disk.hasBlock(3), false)
assert.strictEqual(disk.hasBlock(7), false) // zero-length range dropped
assert.strictEqual(disk.hasBlock(10), false) // non type-0 range dropped
})
})
describe('NbdDisk.getBlockIndexesCount', () => {
const cases = {
'aligned, disjoint blocks': [
{ type: 0, offset: 0, length: SOURCE_BLOCK_SIZE }, // block 0
{ type: 0, offset: 5 * SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE }, // block 5
],
'a range spanning several blocks': [
{ type: 0, offset: SOURCE_BLOCK_SIZE / 2, length: 3 * SOURCE_BLOCK_SIZE }, // blocks 0,1,2,3
],
'two disjoint extents sharing block 0 (needs dedup)': [
{ type: 0, offset: 0, length: SOURCE_BLOCK_SIZE / 2 }, // [0, .5) => block 0
{ type: 0, offset: 3 * (SOURCE_BLOCK_SIZE / 4), length: SOURCE_BLOCK_SIZE }, // [.75, 1.75) => blocks 0,1
],
'touching sub-block fragments': [
{ type: 0, offset: 0, length: SOURCE_BLOCK_SIZE / 2 },
{ type: 0, offset: SOURCE_BLOCK_SIZE / 2, length: SOURCE_BLOCK_SIZE / 2 },
{ type: 0, offset: SOURCE_BLOCK_SIZE, length: SOURCE_BLOCK_SIZE },
],
empty: [],
}
for (const [name, dataMap] of Object.entries(cases)) {
it(`matches getBlockIndexes().length: ${name}`, () => {
const disk = new NbdDisk({}, SOURCE_BLOCK_SIZE, { dataMap })
assert.strictEqual(disk.getBlockIndexesCount(), disk.getBlockIndexes().length)
})
}
it('matches getBlockIndexes().length on the large sparse map', () => {
const virtualSize = Math.floor(3.5 * TiB)
const NB_RANGES = 113528
const dataMap = []
for (let i = 0; i < NB_RANGES; i++) {
dataMap.push({ type: 0, offset: Math.floor((i / NB_RANGES) * virtualSize), length: CLUSTER_SIZE })
}
const disk = new NbdDisk({}, SOURCE_BLOCK_SIZE, { dataMap })
assert.strictEqual(disk.getBlockIndexesCount(), disk.getBlockIndexes().length)
})
})

View File

@@ -13,7 +13,7 @@
"url": "https://vates.fr"
},
"license": "ISC",
"version": "3.4.0",
"version": "3.5.0",
"engines": {
"node": ">=14.0"
},
@@ -22,10 +22,10 @@
"@vates/async-each": "^1.0.3",
"@vates/read-chunk": "^1.2.1",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/disk-transform": "^1.3.0",
"@xen-orchestra/disk-transform": "^1.3.3",
"@xen-orchestra/log": "^0.7.2",
"promise-toolbox": "^0.21.0",
"xen-api": "^4.7.7"
"xen-api": "^4.8.0"
},
"devDependencies": {
"tap": "^18.7.0",

View File

@@ -1,7 +1,7 @@
{
"name": "@vates/types",
"private": false,
"version": "1.26.0",
"version": "1.30.1",
"main": "./dist/index.mjs",
"exports": {
".": {
@@ -32,7 +32,8 @@
"devDependencies": {
"@eslint/js": "^9.19.0",
"typescript": "~5.6.3",
"typescript-eslint": "^8.23.0"
"typescript-eslint": "^8.61.0",
"@types/express": "^5.0.0"
},
"scripts": {
"build": "tsc",

View File

@@ -744,6 +744,8 @@ export type XapiPoolStats = Record<XoHost['id'], XapiHostStats | { error: Record
export type TrafficRuleProtocol = (typeof TRAFFIC_RULE_PROTOCOLS)[number]
export const SDN_CONTROLLER_OF_RULES_KEY = 'xo:sdn-controller:of-rules'
export const SDN_CONTROLLER_OF_METHOD_KEY = 'xo:sdn-controller:of-method'
export const SDN_CONTROLLER_OF_FORMAT_KEY = 'xo:sdn-controller:of-format'
export const TRAFFIC_RULE_PROTOCOLS = ['ARP', 'ICMP', 'IP', 'TCP', 'UDP'] as const
export const TRAFFIC_RULE_PROTOCOLS_WITH_PORT: readonly TrafficRuleProtocol[] = ['TCP', 'UDP']

View File

@@ -5,3 +5,4 @@ export * from './xo-app.mjs'
export * from './lib/xen-orchestra-xapi.mjs'
export * from './lib/xen-orchestra-acl.mjs'
export * from './lib/complex-matcher.mjs'
export * from './lib/rest-api.mjs'

View File

@@ -0,0 +1,102 @@
import type { NextFunction, Request, Response } from 'express'
import type { VatesTask } from './vates-task.mjs'
import type { XoApp } from '../xo-app.mjs'
export type SecurityName = '*' | 'token' | 'basic' | 'none'
export type FieldDefinition =
| {
type: 'string'
example?: string
optional?: boolean
}
| {
type: 'boolean'
example?: boolean
optional?: boolean
}
| {
type: 'number'
example?: number
optional?: boolean
}
| {
type: 'enum'
enum: string[]
example?: string
optional?: boolean
}
| {
type: 'object'
fields: Record<string, FieldDefinition>
optional?: boolean
}
| {
type: 'array'
items: FieldDefinition
example?: unknown[]
optional?: boolean
}
export type ParamFieldDefinition = Exclude<
FieldDefinition,
{ type: 'boolean' } | { type: 'object' } | { type: 'array' }
>
export type QueryFieldDefinition = Exclude<FieldDefinition, { type: 'object' } | { type: 'array' }>
type RestApi = object & { xoApp: XoApp }
export type CreateAction = <Result>(
callback: (task: VatesTask) => MaybePromise<Result>,
opts: {
sync?: boolean
statusCode?: number
taskProperties: { name: string; [key: string]: unknown }
}
) => Promise<Result | undefined>
export interface BaseRouteDefinition<Middleware> {
method: 'get' | 'post' | 'put' | 'delete' | 'patch'
endpoint: string
description?: string
tags?: string[]
params?: Record<string, ParamFieldDefinition>
query?: Record<string, QueryFieldDefinition>
body?: Record<string, FieldDefinition>
responses?: Array<{
status: number
description: string
schema?: Record<string, FieldDefinition>
}>
middlewares?: Middleware[]
scope?: 'acl'
callback: (params: {
req: Request
res: Response
next: NextFunction
restApi: RestApi
createAction: CreateAction
}) => MaybePromise<unknown>
security?: SecurityName
}
export type MiddlewareDescriptor =
| { name: 'json' | 'urlencoded' | 'text' | 'raw'; options?: Record<string, unknown> }
| { name: 'acl'; acls: LooseAclEntry | LooseAclEntry[] }
type MaybePromise<T> = T | Promise<T>
/**
* Loosely-typed mirror of {@link https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/rest-api/src/middlewares/acl.middleware.mts | AclEntry}
* from `@xen-orchestra/rest-api/src/middlewares/acl.middleware.mts`.
*/
export type LooseAclEntry = {
resource: string
action?: string | ((opts: { req: object; restApi: RestApi }) => string | undefined)
actions?: string[] | ((opts: { req: object; restApi: RestApi }) => string[])
objectId?: string | ((opts: { req: object; restApi: RestApi }) => string)
objectIds?: string[] | ((opts: { req: object; restApi: RestApi }) => string[])
object?: object | ((opts: { req: object; restApi: RestApi }) => MaybePromise<object> | undefined)
objects?: object[] | ((opts: { req: object; restApi: RestApi }) => MaybePromise<object[]> | undefined)
getObject?: (opts: { restApi: RestApi }) => (id: string) => MaybePromise<object>
}
export type PluginRestRouteDefinition = BaseRouteDefinition<MiddlewareDescriptor>

View File

@@ -1,19 +1,24 @@
import { Branded } from '../common.mjs'
import type { XoGroup, XoUser } from '../xo.mjs'
export type XoAclRole =
| {
id: Branded<'acl-v2-role'>
name: string
description?: string
}
| {
id: Branded<'acl-v2-role'>
name: string
description?: string
isTemplate: true
roleTemplateId: number
}
type XoAclBaseRole = {
id: Branded<'acl-v2-role'>
name: string
description?: string
privilegeIds: XoAclBasePrivilege['id'][]
}
type XoAclRegularRole = XoAclBaseRole & {
groupIds: XoGroup['id'][]
userIds: XoUser['id'][]
}
type XoAclTemplateRole = XoAclBaseRole & {
isTemplate: true
roleTemplateId: number
}
export type XoAclRole = XoAclRegularRole | XoAclTemplateRole
export type XoAclSupportedActionsByResource = {
[resource: string]: Record<string, unknown>

View File

@@ -89,6 +89,16 @@ export interface EditVmProps {
xenStoreData?: Record<string, string | null>
}
/** Properties accepted by {@link Xapi.editVif}. */
export interface EditVifProps {
ipv4Allowed?: string[]
ipv6Allowed?: string[]
lockingMode?: VIF_LOCKING_MODE
// in kB/s, `null` removes the limit
rateLimit?: number | null
txChecksumming?: boolean
}
export type XcpPatches = {
changelog?: {
author: string
@@ -169,6 +179,13 @@ export interface Xapi {
deleteVif(vifId: XoVif['id']): Promise<void>
disconnectVif(vifId: XoVif['id']): Promise<void>
exportVmOva(vmRef: XenApiVm['$ref']): Promise<PassThrough>
emergencyShutdownHost(hostId: XoHost['id']): Promise<void>
ejectHostFromPool(hostId: XoHost['id']): Promise<void>
host_smartReboot(
hostRef: XenApiHost['$ref'],
bypassBlockedSuspend?: boolean,
bypassCurrentVmCheck?: boolean
): Promise<void>
migrateVm(
vmId: XoVm['id'],
hostXapi: Xapi,
@@ -212,6 +229,40 @@ export interface Xapi {
}
): Promise<{ vm: XenApiVmWrapped }>
forgetSr(id: XoSr['id']): Promise<void>
forgetHost(hostId: XoHost['id']): Promise<void>
powerOnHost(hostId: XoHost['id']): Promise<void>
rebootHost(hostId: XoHost['id'], force?: boolean): Promise<void>
shutdownHost(hostId: XoHost['id'], opts?: { force?: boolean; bypassEvacuate?: boolean }): Promise<void>
SR_create(params: {
content_type?: XoSr['content_type']
// index signature, not `Record<string, string>`: tsoa emits an empty schema for `Record`
// (https://github.com/lukeautry/tsoa/pull/1858). device_config keys are driver-specific (SR_type).
device_config: {
chapuser?: string
chappassword?: string
device?: string
legacy_mode?: string
location?: string
nfsversion?: string
options?: string
password?: string
SCSIid?: string
server?: string
serverpath?: string
target?: string
targetIQN?: string
username?: string
} & {
[key: string]: string
}
host: XenApiHost['$ref']
name_description?: XoSr['name_description']
name_label: XoSr['name_label']
physical_size?: number
shared: boolean
sm_config?: { [key: string]: string }
type: XoSr['SR_type']
}): Promise<XenApiSr['$ref']>
SR_importVdi(
ref: XenApiSr['$ref'],
stream: Readable,
@@ -235,6 +286,7 @@ export interface Xapi {
props: EditVmProps,
checkLimits?: (limits: Record<string, number>, vm: XenApiVmWrapped) => Promise<void>
): Promise<void>
editVif(id: XoVif['id'], props: EditVifProps): Promise<void>
startVm(
id: XoVm['id'],
opts?: {
@@ -306,6 +358,7 @@ export interface Xapi {
vgpuType?: XoVgpuType['id']
gpuGroup?: XoGpuGroup['id']
copyHostBiosStrings?: boolean
highAvailability?: XoVm['high_availability']
hvmBootFirmware?: 'uefi' | 'bios'
secureBoot?: boolean
},
@@ -358,7 +411,7 @@ export interface Xapi {
): Promise<Readable & { length?: number }>
VDI_importContent(
vdiRef: XenApiVdi['$ref'],
stream: Readable,
stream: Readable | Buffer,
opts: { cancelToken?: unknown; format: SUPPORTED_VDI_FORMAT }
): Promise<void>
VIF_create(
@@ -397,8 +450,12 @@ export interface Xapi {
pathname: string,
params?: { host?: XenApiHost; query?: Record<string, unknown>; task?: boolean | XenApiTask['$ref'] }
): Promise<{ body: Readable }>
clearHost(host: Pick<XenApiHostWrapped, '$ref' | '$pool'>, force?: boolean): Promise<void>
disableHost(hostId: XoHost['id']): Promise<void>
clearHost(
host: Pick<XenApiHostWrapped, '$ref' | '$pool'>,
force?: boolean,
opts?: { transient?: boolean }
): Promise<void>
disableHost(hostId: XoHost['id'], opts?: { transient?: boolean }): Promise<void>
enableHost(hostId: XoHost['id']): Promise<void>
getRecordByUuid<
Type extends WrappedXenApiRecord['$type'],

View File

@@ -493,7 +493,7 @@ export interface XenApiDrTask {
}
type XenApiHostCallMethods = TagCallMethods & {
<T>(method: 'call_plugin',plugin:string, fn: string, args: Record<string, string>): Promise<T>
<T>(method: 'call_plugin', plugin: string, fn: string, args: Record<string, string>): Promise<T>
}
export interface XenApiHost {
$ref: Branded<'host'>
@@ -571,7 +571,9 @@ export interface XenApiHost {
uuid: string
virtual_hardware_platform_versions: number[]
}
export type XenApiHostWrapped = WrapperXenApi<XenApiHost, 'host', XenApiHostCallMethods>
export interface XenApiHostWrapped extends WrapperXenApi<XenApiHost, 'host', XenApiHostCallMethods> {
$restartAgent(): Promise<void>
}
export interface XenApiHostCrashdump {
$ref: Branded<'host_crashdump'>

View File

@@ -1,6 +1,5 @@
import { EventEmitter } from 'node:stream'
import type {
AnyXoBackupJob,
AnyXoJob,
AnyXoLog,
XapiXoRecord,
@@ -16,10 +15,12 @@ import type {
XoServer,
XoTask,
XoUser,
XoVif,
XoVm,
XoVmBackupArchive,
} from './xo.mjs'
import { VatesTask } from './lib/vates-task.mjs'
import type { PluginRestRouteDefinition } from './lib/rest-api.mjs'
import {
Xapi,
XapiHostStats,
@@ -67,6 +68,7 @@ type FeatureCode =
| 'RBAC'
| 'ROLLING_POOL_UPDATE'
| 'ROLLING_POOL_REBOOT'
| 'SMART_REBOOT'
| 'WARM_MIGRATION'
| 'PLUGIN.OPENMETRICS'
@@ -180,6 +182,7 @@ export type XoApp = {
}
// methods ------------
allocIpAddresses(vifId: XoVif['id'], addAddresses?: string[], removeAddresses?: string[]): Promise<void>
addAclV2GroupRole(groupId: XoGroup['id'], roleId: XoAclRole['id']): Promise<XoGroupRole>
addAclV2UserRole(userId: XoUser['id'], roleId: XoAclRole['id']): Promise<XoUserRole>
addUserToGroup: (userId: XoUser['id'], groupId: XoGroup['id']) => Promise<void>
@@ -192,11 +195,13 @@ export type XoApp = {
params?: any
}
) => () => void // eslint-disable-line @typescript-eslint/no-explicit-any
registerRestRoutes: (routes: PluginRestRouteDefinition[], base?: string) => () => void
authenticateUser: (
credentials: { token?: string; username?: string; password?: string },
userData?: { ip?: string },
opts?: { bypassOtp?: boolean; bypassTaskCreation?: boolean }
) => Promise<{ bypassOtp: boolean; expiration: number; user: XoUser }>
backupGuard(poolId: XoPool['id']): Promise<void>
/* Throw if no authorization */
checkFeatureAuthorization(featureCode: FeatureCode): Promise<void>
/* connect a server (XCP-ng/XenServer) */
@@ -229,28 +234,47 @@ export type XoApp = {
url: string
}): Promise<XoBackupRepository>
createUser(params: { name?: string; password?: string; [key: string]: unknown }): Promise<XoUser>
deleteAclV2GroupRole(groupId: XoGroup['id'], roleId: XoAclRole['id']): Promise<boolean>
deleteAclV2GroupRole(
groupId: XoGroup['id'],
roleId: XoAclRole['id'],
opts?: { bypassAuthorization?: boolean }
): Promise<boolean>
deleteAclV2Privilege(privilegeId: XoAclBasePrivilege['id'], options?: { force?: boolean }): Promise<boolean>
deleteAclV2Role(roleId: XoAclRole['id'], options?: { force?: boolean }): Promise<boolean>
deleteGroup(id: XoGroup['id']): Promise<void>
deleteUser(id: XoUser['id']): Promise<void>
detachHostFromPool(hostId: XoHost['id']): Promise<void>
createGroup(params: { name: string; provider?: string; providerGroup?: string }): Promise<XoGroup>
/* disconnect a server (XCP-ng/XenServer) */
disconnectXenServer(id: XoServer['id']): Promise<void>
findEnabledScheduleSequenceFromSchedule(id: XoSchedule['id']): Promise<XoSchedule | undefined>
getAclV2Privilege(id: XoAclBasePrivilege['id']): Promise<XoAclBasePrivilege>
getAclV2Privileges(): Promise<XoAclBasePrivilege[]>
getAclV2RolePrivileges(roleId: XoAclRole['id']): Promise<XoAclBasePrivilege[]>
getAclV2Role(id: XoAclRole['id']): Promise<XoAclRole>
deleteAclV2UserRole(userId: XoUser['id'], roleId: XoAclRole['id']): Promise<boolean>
getAclV2Role(id: XoAclRole['id'], opts?: { bypassAuthorization?: boolean }): Promise<XoAclRole>
deleteAclV2UserRole(
userId: XoUser['id'],
roleId: XoAclRole['id'],
opts?: { bypassAuthorization?: boolean }
): Promise<boolean>
getAclV2GroupRoles(
groupId: XoGroup['id'],
opts?: { bypassAuthorization?: boolean }
): Promise<Exclude<XoAclRole, { isTemplate: true }>[]>
getAclV2Roles(): Promise<XoAclRole[]>
getAclV2UserPrivileges(userId: XoUser['id']): Promise<XoAclBasePrivilege[]>
getAclV2UserRoles(
userId: XoUser['id'],
opts?: { bypassAuthorization?: boolean; fromGroup?: boolean; fromUser?: boolean }
): Promise<XoAclRole[]>
getAllGroups(): Promise<XoGroup[]>
getAllProxies(): Promise<XoProxy[]>
getAllJobs<T extends AnyXoBackupJob['type']>(type: T): Promise<Extract<AnyXoBackupJob, { type: T }>[]>
getAllJobs<T extends AnyXoJob['type']>(type: T): Promise<Extract<AnyXoJob, { type: T }>[]>
getAllJobs(type?: string): Promise<AnyXoJob[]>
getProxy(id: XoProxy['id']): Promise<XoProxy>
getRemote(id: XoBackupRepository['id']): Promise<XoBackupRepository>
getAllRemotes(): Promise<XoBackupRepository[]>
removeRemote(id: XoBackupRepository['id']): Promise<void>
getAllRemotesInfo(): Promise<
Record<
XoBackupRepository['id'],
@@ -308,6 +332,7 @@ export type XoApp = {
backupRepositoryIds: XoBackupRepository['id'][],
opts?: { _forceRefresh?: boolean; vmId: XoVm['id'] }
): Promise<Record<XoBackupRepository['id'], Record<XoVm['id'], XoVmBackupArchive[]>>>
pingRemote(id: XoBackupRepository['id']): Promise<{ success: true }>
/** Allow to add a new server in the DB (XCP-ng/XenServer) */
registerXenServer(
body: Pick<XoServer, 'host' | 'httpProxy' | 'label' | 'username'> & {
@@ -316,13 +341,22 @@ export type XoApp = {
readOnly?: XoServer['readOnly']
}
): Promise<XoServer>
rollingPoolReboot(pool: XoPool, opts?: { parentTask?: VatesTask }): Promise<void>
rollingPoolUpdate(pool: XoPool, opts?: { rebootVm?: boolean; parentTask?: VatesTask }): Promise<void>
rollingPoolReboot(pool: XoPool, opts?: { parentTask?: VatesTask; shutdownPinnedVms?: boolean }): Promise<void>
rollingPoolUpdate(
pool: XoPool,
opts?: { rebootVm?: boolean; parentTask?: VatesTask; shutdownPinnedVms?: boolean }
): Promise<void>
setVmResourceSet(vmId: XoVm['id'], resourceSetId: string | null, force?: boolean): Promise<void>
shareVmResourceSet(vmId: XoVm['id']): Promise<void>
removeUserFromGroup(userId: XoUser['id'], id: XoGroup['id']): Promise<void>
runJob(job: AnyXoJob, schedule: XoSchedule): void
runWithApiContext: (user: XoUser | undefined, fn: () => void) => Promise<unknown>
testRemote(
id: XoBackupRepository['id']
): Promise<
| { success: true; readRate: number; writeRate: number }
| { success: false; step: string; file: string; error: unknown }
>
/** Remove a server from the DB (XCP-ng/XenServer) */
unregisterXenServer(id: XoServer['id']): Promise<void>
updateUser(

View File

@@ -76,7 +76,7 @@ type BaseXoVm = BaseXapiXo & {
expNestedHvm: boolean
isNestedVirtEnabled: boolean
hasVendorDevice: boolean
high_availability: string
high_availability: 'best-effort' | 'restart' | ''
installTime?: number | null
isFirmwareSupported: boolean
memory: {
@@ -222,6 +222,7 @@ export type XoGroup = {
provider?: string
providerGroupId?: string
users: XoUser['id'][]
aclRoleIds: XoAclRole['id'][]
}
export type XoHost = BaseXapiXo & {
@@ -565,6 +566,18 @@ export type XoMirrorBackupJob = BaseXoJob & {
export type XoJob = BaseXoJob & {
type: 'call'
method: string
paramsVector?: {
type: string
items: {
type?: string
values?: {
schedules?: XoSchedule['id'][]
[key: string]: unknown
}[]
[key: string]: unknown
}[]
}
}
export type XoSchedule = {
@@ -601,7 +614,7 @@ export type XoSr = BaseXapiXo & {
VDIs: AnyXoVdi['id'][]
allocationStrategy?: 'thin' | 'thick' | 'unknown'
content_type: string
content_type: 'user' | 'iso' | 'disk'
current_operations: Record<string, STORAGE_OPERATIONS>
id: Branded<'SR'>
inMaintenanceMode: boolean
@@ -612,7 +625,28 @@ export type XoSr = BaseXapiXo & {
shared: boolean
size: number
sm_config: Record<string, string>
SR_type: string
// SM drivers shipped with XCP-ng/XenServer
SR_type:
| 'cephfs'
| 'dummy'
| 'ext'
| 'file'
| 'glusterfs'
| 'hba'
| 'iso'
| 'largeblock'
| 'linstor'
| 'lvm'
| 'lvmofcoe'
| 'lvmohba'
| 'lvmoiscsi'
| 'moosefs'
| 'nfs'
| 'shm'
| 'smb'
| 'udev'
| 'xfs'
| 'zfs'
tags: string[]
type: 'SR'
usage: number

View File

@@ -6,7 +6,7 @@
"main": "dist/index.mjs",
"name": "@xen-orchestra/acl",
"homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@xen-orchestra/acl",
"version": "1.2.0",
"version": "1.5.0",
"license": "AGPL-3.0-or-later",
"private": false,
"type": "module",
@@ -30,10 +30,10 @@
},
"devDependencies": {
"@eslint/js": "^9.19.0",
"@vates/types": "^1.26.0",
"@vates/types": "^1.30.1",
"rimraf": "^6.0.1",
"typescript": "~5.6",
"typescript-eslint": "^8.23.0"
"typescript-eslint": "^8.61.0"
},
"bugs": "https://github.com/vatesfr/xen-orchestra/issues",
"repository": {

View File

@@ -1,6 +1,8 @@
export default {
benchmark: true,
create: true,
read: true,
forget: true,
update: {
enabled: true,
name: true,

View File

@@ -1,14 +1,27 @@
export default {
'allow-vm': true,
detach: true,
disable: true,
'migrate-receive': true,
enable: true,
evacuate: true,
export: {
logs: true,
},
forget: true,
'join-pool': true,
'migrate-receive': true,
read: true,
reboot: {
clean: true,
smart: true,
},
'restart-toolstack': true,
'scan-pifs': true,
shutdown: {
clean: true,
emergency: true,
},
start: true,
update: {
tags: true,
},

View File

@@ -4,5 +4,6 @@ export default {
read: true,
update: {
tags: true,
other_config: true,
},
}

View File

@@ -1,4 +1,5 @@
export default {
create: true,
delete: true,
forget: true,
import: {

View File

@@ -7,6 +7,9 @@ export default {
'migrate-send': true,
read: true,
update: {
name_description: true,
name_label: true,
size: true,
tags: true,
},
}

View File

@@ -4,4 +4,12 @@ export default {
delete: true,
disconnect: true,
read: true,
update: {
allowedIpv4Addresses: true,
allowedIpv6Addresses: true,
lockingMode: true,
rateLimit: true,
txChecksumming: true,
other_config: true,
},
}

View File

@@ -1,6 +1,6 @@
{
"name": "@xen-orchestra/backup-archive",
"version": "1.0.1",
"version": "1.1.1",
"description": "Helper to handle generic backup archives",
"license": "ISC",
"author": "",
@@ -15,29 +15,30 @@
"rimraf": "^6.0.1",
"@types/node": "^22.3.0",
"typescript": "~5.6",
"typescript-eslint": "^8.23.0",
"typescript-eslint": "^8.61.0",
"fs-extra": "^11.1.0",
"sinon": "^18.0.0",
"tar-stream": "^3.1.8",
"@types/tar-stream": "^3.1.4",
"tmp": "^0.2.1",
"@vates/types": "^1.26.0"
"@vates/types": "^1.30.1"
},
"dependencies": {
"@xen-orchestra/log": "^0.7.1",
"@xen-orchestra/fs": "^4.9.0",
"@xen-orchestra/disk-transform": "^1.3.0",
"@xen-orchestra/fs": "^4.9.3",
"@xen-orchestra/disk-transform": "^1.3.3",
"@vates/task": "0.7.0",
"golike-defer": "0.5.1",
"uuid": "^9.0.0",
"promise-toolbox": "^0.21.0",
"@vates/async-each": "^1.0.1",
"vhd-lib": "^4.14.7"
"vhd-lib": "^4.16.1"
},
"scripts": {
"build": "yarn run clean && tsc",
"dev": "tsc --watch",
"postversion": "npm publish --access public",
"test": "yarn run clean && tsc && node --test **/*.test.mjs",
"test-integration": "yarn run clean && tsc && node --test **/*.integ.mjs",
"clean": "rimraf dist/"
},

View File

@@ -21,6 +21,7 @@ import { limitConcurrency } from 'limit-concurrency-decorator'
import assert from 'node:assert/strict'
const defaultMergeLimiter = limitConcurrency(1)
const MERGE_STATE_FILE_RE = /^\.(.+)\.merge\.json$/
/**
* Tracks the disk chain for a single VDI across all backup snapshots.
* Owns merge and deletion decisions for its chain given which disks are still
@@ -84,11 +85,23 @@ export class RemoteDiskLineage {
}
}
const mergeParentPaths = new Set<string>()
for (const filePath of files) {
const match = MERGE_STATE_FILE_RE.exec(basename(filePath))
if (match !== null) {
mergeParentPaths.add(normalize(this.#vdiDir + '/' + match[1]))
}
}
const uuidToPath = new Map<string, string>()
for (const diskPath of this.#diskPaths) {
let disk: RemoteDisk | undefined
try {
disk = await openDisk({ handler: this.#handler, path: diskPath })
if (mergeParentPaths.has(diskPath)) {
disk = await openDisk({ handler: this.#handler, path: diskPath, force: true })
} else {
disk = await openDisk({ handler: this.#handler, path: diskPath })
}
} catch (error) {
if (error?.code === 'NOT_SUPPORTED' && this.#opts.merge) {
throw error
@@ -296,6 +309,8 @@ export class RemoteDiskLineage {
try {
const disk = await openDisk({ handler: this.#handler, path, ignoreBlockIndexes: true })
await disk.unlink({ force: true })
// Remove from tracking so #cleanOrphanDataFiles skips this now-deleted path
this.#unregisterDisk(path)
} catch (error) {
this.#opts.logWarn('failed to delete unused disk', { path, error })
}
@@ -316,8 +331,15 @@ export class RemoteDiskLineage {
await asyncEach(
toMerge,
async ({ chain, isResuming }) => {
const { finalDiskSize, mergeTargetPath } = await limitedMergeChain(chain, isResuming)
const { finalDiskSize, mergeTargetPath } = await limitedMergeChain([...chain], isResuming)
mergedSizes.set(mergeTargetPath, (mergedSizes.get(mergeTargetPath) ?? 0) + finalDiskSize)
// parentPath alias deleted by parentDisk.rename(mergeTargetPath)
// intermediates deleted by childDisk.unlink() when removeUnused=true
// Unregister so #cleanOrphanDataFiles does not read already-deleted aliases
// and produce false "missing target of alias" warnings.
for (const deletedPath of chain.slice(0, -1)) {
this.#unregisterDisk(deletedPath)
}
},
{ concurrency: this.#opts.mergeConcurrency ?? DEFAULT_MERGE_CONCURRENCY }
)

View File

@@ -10,6 +10,11 @@ export interface PartialBackupMetadata {
jobId: string
scheduleId: string
timestamp: number
// 'true' when written by code including the non-NBD qcow2 corruption fix and its force-full
// gate; its absence marks a potentially-corrupt older backup and makes the gate re-probe.
// Preserved (never stamped) across merges here.
// @todo remove this code and the logic associated on 2027 01 01
includeNonNbdQcow2Fix?: boolean
}
export const DEFAULT_MERGE_CONCURRENCY = 1
export const DEFAULT_REMOVE_CONCURRENCY = 4

View File

@@ -39,6 +39,10 @@ export class VmBackupDirectory implements VmBackupInterface {
#checkResult: (CheckResult & { orphans: string[]; linked: string[] }) | undefined = undefined
#uniqueLineages: Map<string, RemoteDiskLineage> | undefined = undefined
// Set by #checkCacheCount(): the on-disk cache did not match the archives found
// on disk, so clean() must regenerate it even when nothing was merged/removed.
#cacheOutOfSync = false
constructor(
handler: RemoteHandlerAbstract,
vmBackupPath: string,
@@ -183,7 +187,7 @@ export class VmBackupDirectory implements VmBackupInterface {
{ concurrency: 2 }
)
if (allMergedSizes.size > 0 || cacheNeedsRegen) {
if (allMergedSizes.size > 0 || cacheNeedsRegen || this.#cacheOutOfSync) {
await this.#regenerateCache()
}
@@ -214,7 +218,8 @@ export class VmBackupDirectory implements VmBackupInterface {
const existingCache = await this.#readCache(cachePath)
const actual = existingCache === undefined ? 0 : Object.keys(existingCache).length
const expected = this.backupArchives.size
if (actual !== expected) {
this.#cacheOutOfSync = actual !== expected
if (this.#cacheOutOfSync) {
this.opts.logWarn('unexpected number of entries in backup cache', { path: cachePath, actual, expected })
}
}

View File

@@ -43,12 +43,19 @@ const isValidTar = async (handler: RemoteHandlerAbstract, size: number, fd: File
return buf.every(byte => byte === 0)
}
export async function isValidXva(handler: RemoteHandlerAbstract, filePath: string): Promise<boolean> {
export async function isValidXva(
handler: RemoteHandlerAbstract,
filePath: string,
metadataSize?: number
): Promise<boolean> {
// size is longer when encrypted + reading part of an encrypted file is not implemented
if (handler.isEncrypted) {
return true
const size = await handler.getSizeOnDisk(filePath)
return size > 20
}
let isValid = true
const fd = await handler.openFile(filePath, 'r')
try {
const size = await handler.getSize(filePath)
@@ -56,13 +63,19 @@ export async function isValidXva(handler: RemoteHandlerAbstract, filePath: strin
// neither a valid gzip nor tar
return false
}
let validSize = true
if (metadataSize !== undefined) {
validSize = metadataSize === size
}
return (await isCompressedFile(handler, fd))
const validDisk = (await isCompressedFile(handler, fd))
? true // compressed files cannot be validated at this time
: await isValidTar(handler, size, fd)
isValid = validSize && validDisk
} finally {
handler.closeFile(fd).catch(noop)
}
return isValid
}
const noop = (): void => {}
@@ -101,28 +114,22 @@ export class VmFullBackupArchive implements VmBackupInterface {
*/
async check(): Promise<CheckResult> {
if (this.isValid === undefined) {
let fileSize = 0
let validDisk = false
let validSize = true
try {
fileSize = await this.handler.getSize(this.xvaPath)
validDisk = await isValidXva(this.handler, this.xvaPath)
this.isValid = await isValidXva(this.handler, this.xvaPath, this.metadata.size)
} catch (error) {
validDisk = false
this.isValid = false
if (error?.code === 'ENOENT') {
this.missingDisk = true
}
this.opts.logWarn('Issue while checking XVA', { error })
}
try {
await this.handler.getSize(`${this.xvaPath}.checksum`)
} catch (error) {
this.opts.logWarn('Checksum file not valid, not blocking', { error })
if (!this.handler.isEncrypted) {
try {
await this.handler.getSize(`${this.xvaPath}.checksum`)
} catch (error) {
this.opts.logWarn('Checksum file not valid, not blocking', { error })
}
}
if (this.metadata.size !== undefined) {
validSize = this.metadata.size === fileSize
}
this.isValid = fileSize > 0 && validSize && validDisk
}
if (!this.isValid) {
if (this.missingDisk) {

View File

@@ -8,13 +8,19 @@ import { getHandler } from '@xen-orchestra/fs'
import { pFromCallback } from 'promise-toolbox'
// eslint-disable-next-line n/no-missing-import
import { VmBackupDirectory } from '../dist/VmBackupDirectory.mjs'
import { RemoteVhdDisk } from '../dist/disks/RemoteVhdDisk.mjs'
import { MergeRemoteDisk } from '../dist/disks/MergeRemoteDisk.mjs'
import { VHDFOOTER, VHDHEADER } from './tests.fixtures.mjs'
import { VhdFile, Constants, VhdDirectory, VhdAbstract } from 'vhd-lib'
import { dirname, basename } from 'node:path'
import { promisify } from 'node:util'
import zlib from 'node:zlib'
import { rimraf } from 'rimraf'
const { beforeEach, afterEach, describe } = test
const gunzip = promisify(zlib.gunzip)
let tempDir, handler, jobId, vdiId, basePath, relativePath, vdiId2, basePath2, relativePath2
const rootPath = 'xo-vm-backups/VMUUID/'
@@ -412,6 +418,118 @@ test('it merges a chain of multiple consecutive orphan ancestors in one pass', a
assert.equal(remainingVhds.includes('child.vhd'), false)
})
test('it resumes, through cleanVm, an interrupted merge whose ancestor lost its footer', async () => {
// Reproduces the production "footer1 !== footer2" failure end-to-end via the full cleanVm
// pipeline. ancestor (full) <- child (differencing, the active backup).
const ancestor = await generateVhd(`${basePath}/ancestor.vhd`, { blocks: [0, 1] })
await generateVhd(`${basePath}/child.vhd`, {
header: { parentUnicodeName: 'ancestor.vhd', parentUuid: ancestor.footer.uuid },
blocks: [2, 3, 4, 5],
})
await handler.writeFile(
`${rootPath}/metadata.json`,
JSON.stringify({ mode: 'delta', size: 12000, vhds: [`${relativePath}/child.vhd`] })
)
// Simulate a merge killed mid-transfer: write 2 of the 4 child blocks into the ancestor
// (which overwrites its end footer) then throw before mergeMetadata() rewrites the footer.
// This leaves the ancestor footerless AND a `.merge.json` state file — the exact post-crash
// state that makes cleanVm take the resume path.
const parent = new RemoteVhdDisk({ handler, path: `${basePath}/ancestor.vhd` })
const child = new RemoteVhdDisk({ handler, path: `${basePath}/child.vhd` })
await parent.init({ force: false })
await child.init({ force: false })
const merger = new MergeRemoteDisk(handler, { removeUnused: true, writeStateDelay: 0 })
let written = 0
const realWriteBlock = parent.writeBlock.bind(parent)
parent.writeBlock = async diskBlock => {
if (written === 2) {
throw new Error('simulated interruption')
}
written++
return realWriteBlock(diskBlock)
}
await assert.rejects(merger.merge(parent, child), /simulated interruption/)
assert.ok(
(await handler.list(basePath)).includes('.ancestor.vhd.merge.json'),
'an interrupted-merge state file should remain'
)
// Full pipeline: cleanVm must recognize the interrupted merge and resume it — opening the
// footerless ancestor through the chain without throwing "footer1 !== footer2".
await VmBackupDirectory.cleanVm(handler, rootPath, {
remove: true,
merge: true,
logInfo: () => {},
logWarn: () => {},
})
// The merge completed: the ancestor was consolidated into child.vhd and the state file is gone.
const remaining = await handler.list(basePath)
assert.equal(remaining.includes('.ancestor.vhd.merge.json'), false, 'state file removed after successful resume')
assert.equal(remaining.includes('ancestor.vhd'), false, 'ancestor consolidated into the merge target')
assert.equal(remaining.includes('child.vhd'), true, 'merge target remains')
// No blocks lost: the resumed merge wrote the blocks (4, 5) that were never written before
// the crash, so the consolidated disk holds the full set.
const merged = new RemoteVhdDisk({ handler, path: `${basePath}/child.vhd` })
await merged.init({ force: false })
try {
assert.deepEqual(merged.getBlockIndexes(), [0, 1, 2, 3, 4, 5], 'all blocks present after resumed merge')
} finally {
await merged.close()
}
// The backup is consistent, so its metadata is preserved.
assert.ok((await handler.list(rootPath)).includes('metadata.json'), 'metadata should be preserved')
})
test('it does not warn "missing target of alias" for aliases deleted by merge', async () => {
// Regression: RemoteDiskLineage.#cleanOrphanDataFiles iterated #diskPaths entries just
// deleted during the same clean() call's merge phase, producing false "missing target of
// alias" warnings
const orphan = await generateVhd(`${basePath}/orphan.vhd`, {
useAlias: true,
blocks: [0],
})
await generateVhd(`${basePath}/child.vhd`, {
useAlias: true,
header: {
parentUnicodeName: 'orphan.vhd.alias.vhd',
parentUuid: orphan.footer.uuid,
},
blocks: [1],
})
await handler.writeFile(
`${rootPath}/metadata.json`,
JSON.stringify({
mode: 'delta',
vhds: [`${relativePath}/child.vhd.alias.vhd`],
})
)
const warnings = []
await VmBackupDirectory.cleanVm(handler, rootPath, {
remove: true,
merge: true,
logWarn: (message, data) => warnings.push({ message, data }),
logInfo: () => {},
})
const missingTargetWarnings = warnings.filter(w => w.message === 'missing target of alias')
assert.equal(
missingTargetWarnings.length,
0,
`"missing target of alias" must not fire after merge; got: ${JSON.stringify(missingTargetWarnings)}`
)
const remaining = await handler.list(basePath)
assert.ok(!remaining.includes('orphan.vhd.alias.vhd'), 'orphan alias must be deleted after merge')
assert.ok(remaining.includes('child.vhd.alias.vhd'), 'child alias must survive as merge target')
})
test('it resumes an interrupted merge with chain field', async () => {
await handler.writeFile(
`${rootPath}/metadata.json`,
@@ -714,3 +832,28 @@ test('it preserves VDI directory when handler.list() throws during lineage init'
const remaining = await handler.list(basePath)
assert.ok(remaining.includes('snapshot.vhd'), 'VHD must survive when handler.list() throws during lineage init')
})
test('it regenerates the cache when its entry count is out of sync, even with nothing to merge/remove', async () => {
// a referenced disk + its metadata, but no cache.json.gz on disk
await generateVhd(`${basePath}/diskA1.vhd`)
await handler.writeFile(
`${rootPath}/metadata.json`,
JSON.stringify({ mode: 'delta', vhds: [`${relativePath}/diskA1.vhd`] })
)
await assert.rejects(handler.readFile(`${rootPath}/cache.json.gz`), { code: 'ENOENT' })
const logged = []
// remove/merge both false: cacheNeedsRegen stays false and no merge happens,
// so only the cache-count mismatch can trigger the regeneration
await VmBackupDirectory.cleanVm(handler, rootPath, {
remove: false,
merge: false,
logInfo: () => {},
logWarn: message => logged.push(message),
})
assert.ok(logged.includes('unexpected number of entries in backup cache'), 'mismatch should be reported')
const cache = JSON.parse((await gunzip(await handler.readFile(`${rootPath}/cache.json.gz`))).toString())
assert.equal(Object.keys(cache).length, 1, 'cache should be regenerated to match the archives on disk')
})

View File

@@ -312,4 +312,12 @@ export class RemoteVhdDiskChain extends RemoteDisk {
async clean(opts = {}) {
throw new Error(`Can't clean a disk chain`)
}
/**
*
* @returns {RemoteVhdDisk|undefined}
*/
getRootDisk() {
return this.#disks[0]
}
}

View File

@@ -15,9 +15,10 @@ import { RemoteVhdDiskChain } from './RemoteVhdDiskChain.mjs'
* @param {string} params.path
* @param {string | undefined} params.until
* @param {boolean} [params.force]
* @param {boolean} [params.ignoreBlockIndexes] - skip reading the block allocation table of each disk (cheaper, but blocks can't be read)
* @returns {Promise<RemoteDisk>}
*/
async function _openDiskChain($defer, { handler, path, until, force = false }) {
async function _openDiskChain($defer, { handler, path, until, force = false, ignoreBlockIndexes = false }) {
/**
* @type {RemoteVhdDisk}
*/
@@ -29,16 +30,18 @@ async function _openDiskChain($defer, { handler, path, until, force = false }) {
$defer.onFailure(() => Promise.all(disks.map(disk => disk.close())))
disk = new RemoteVhdDisk({ handler, path })
await disk.init({ force })
await disk.init({ force, ignoreBlockIndexes })
disks.push(disk)
let foundRootDisk = until === undefined
while (disk.isDifferencing()) {
disk = await disk.openParent()
if (disk.getPath() === until) {
if (disk.getParentPath() === until) {
foundRootDisk = true
await disk.close()
break
}
// Instantiate + init the parent directly (instead of openParent()) so the
// block allocation table read can be skipped when ignoreBlockIndexes is set.
disk = /** @type {RemoteVhdDisk} */ (disk.instantiateParent())
await disk.init({ force, ignoreBlockIndexes })
disks.unshift(disk)
}
if (!foundRootDisk) {
@@ -49,6 +52,6 @@ async function _openDiskChain($defer, { handler, path, until, force = false }) {
}
/**
* @type {(params: { handler: RemoteHandlerAbstract, path: string, until?: string, force?: boolean }) => Promise<RemoteDisk>}
* @type {(params: { handler: RemoteHandlerAbstract, path: string, until?: string, force?: boolean, ignoreBlockIndexes?: boolean }) => Promise<RemoteDisk>}
*/
export const openDiskChain = defer(_openDiskChain)

View File

@@ -8,8 +8,8 @@
"dependencies": {
"@vates/async-each": "^1.0.3",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/backups": "^0.73.2",
"@xen-orchestra/fs": "^4.9.0",
"@xen-orchestra/backups": "^0.73.10",
"@xen-orchestra/fs": "^4.9.3",
"filenamify": "^6.0.0",
"getopts": "^2.2.5",
"lodash": "^4.18.0",

View File

@@ -1,34 +1,23 @@
import { asyncEach } from '@vates/async-each'
import { asyncMap, asyncMapSettled } from '@xen-orchestra/async-map'
import { compose } from '@vates/compose'
import { createLogger } from '@xen-orchestra/log'
import { VhdDirectory, VhdSynthetic } from 'vhd-lib'
import { decorateMethodsWith } from '@vates/decorate-with'
import { deduped } from '@vates/disposable/deduped.js'
import { dirname, join, resolve } from 'node:path'
import { execFile } from 'child_process'
import { mount } from '@vates/fuse-vhd'
import { readdir, lstat } from 'node:fs/promises'
import { synchronized } from 'decorator-synchronized'
import { ZipFile } from 'yazl'
import Disposable from 'promise-toolbox/Disposable'
import fromCallback from 'promise-toolbox/fromCallback'
import fromEvent from 'promise-toolbox/fromEvent'
import groupBy from 'lodash/groupBy.js'
import pDefer from 'promise-toolbox/defer'
import pickBy from 'lodash/pickBy.js'
import reduce from 'lodash/reduce.js'
import * as tar from 'tar'
import zlib from 'zlib'
import { BACKUP_DIR } from './_getVmBackupDir.mjs'
import { VmBackupDirectory } from '@xen-orchestra/backup-archive'
import { fileRestoreDecorators, fileRestoreMethods } from './_fileRestore.mjs'
import { formatFilenameDate } from './_filenameDate.mjs'
import { getTmpDir } from './_getTmpDir.mjs'
import { isMetadataFile } from './_backupType.mjs'
import { isValidXva } from './_isValidXva.mjs'
import { listPartitions, LVM_PARTITION_TYPE_MBR, LVM_PARTITION_TYPE_GPT } from './_listPartitions.mjs'
import { lvs, pvs } from './_lvm.mjs'
import { watchStreamSize } from './_watchStreamSize.mjs'
import { RemoteVhdDisk, openDiskChain } from '@xen-orchestra/backup-archive/disks'
@@ -48,25 +37,6 @@ export const compareTimestamp = (a, b) => a.timestamp - b.timestamp
const noop = Function.prototype
const resolveRelativeFromFile = (file, path) => resolve('/', dirname(file), path).slice(1)
const makeRelative = path => resolve('/', path).slice(1)
const resolveSubpath = (root, path) => resolve(root, makeRelative(path))
async function addZipEntries(zip, realBasePath, virtualBasePath, relativePaths) {
for (const relativePath of relativePaths) {
const realPath = join(realBasePath, relativePath)
const virtualPath = join(virtualBasePath, relativePath)
const stats = await lstat(realPath)
const { mode, mtime } = stats
const opts = { mode, mtime }
if (stats.isDirectory()) {
zip.addEmptyDirectory(virtualPath, opts)
await addZipEntries(zip, realPath, virtualPath, await readdir(realPath))
} else if (stats.isFile()) {
zip.addFile(realPath, virtualPath, opts)
}
}
}
const createSafeReaddir = (handler, methodName) => (path, options) =>
handler.list(path, options).catch(error => {
@@ -76,11 +46,6 @@ const createSafeReaddir = (handler, methodName) => (path, options) =>
return []
})
const debounceResourceFactory = factory =>
function () {
return this._debounceResource(factory.apply(this, arguments))
}
export class RemoteAdapter {
constructor(
handler,
@@ -98,112 +63,6 @@ export class RemoteAdapter {
return this._handler
}
async _findPartition(devicePath, partitionId) {
const partitions = await listPartitions(devicePath)
const partition = partitions.find(_ => _.id === partitionId)
if (partition === undefined) {
throw new Error(`partition ${partitionId} not found`)
}
return partition
}
async *_getLvmLogicalVolumes(devicePath, pvId, vgName) {
yield this._getLvmPhysicalVolume(devicePath, pvId && (await this._findPartition(devicePath, pvId)))
debug('activate LVM volume group', { vgName })
await fromCallback(execFile, 'vgchange', ['-ay', vgName])
try {
debug('get LVM volume group name and path', { vgName })
yield lvs(['lv_name', 'lv_path'], vgName)
} finally {
debug('deactivate LVM volume group', { vgName })
await fromCallback(execFile, 'vgchange', ['-an', vgName])
}
}
async *_getLvmPhysicalVolume(devicePath, partition) {
const args = []
if (partition !== undefined) {
args.push('-o', partition.start * 512, '--sizelimit', partition.size)
}
args.push('--show', '-f', devicePath)
debug('attach loop device', { devicePath, partition })
const path = (await fromCallback(execFile, 'losetup', args)).trim()
try {
debug('list LVM physical volume', { path })
await fromCallback(execFile, 'pvscan', ['--cache', path])
yield path
} finally {
try {
const vgNames = await pvs('vg_name', path)
debug('deactivate LVM volume groups', { vgNames })
await fromCallback(execFile, 'vgchange', ['-an', ...vgNames])
} finally {
debug('detach loop device', { path })
await fromCallback(execFile, 'losetup', ['-d', path])
}
}
}
async *_getPartition(devicePath, partition) {
// the norecovery option is necessary because if the partition is dirty,
// mount will try to fix it which is impossible if because the device is read-only
const options = ['loop', 'ro', 'norecovery']
if (partition !== undefined) {
const { size, start } = partition
options.push(`sizelimit=${size}`)
if (start !== undefined) {
options.push(`offset=${start * 512}`)
}
}
const path = yield getTmpDir()
const mount = options => {
debug('mount device', { devicePath, mountPath: path })
return fromCallback(execFile, 'mount', [
`--options=${options.join(',')}`,
`--source=${devicePath}`,
`--target=${path}`,
])
}
// `norecovery` option is used for ext3/ext4/xfs, if it fails it might be
// another fs, try without
try {
await mount([...options, 'norecovery'])
} catch (error) {
await mount(options)
}
try {
yield path
} finally {
debug('umount device', { devicePath, mountPath: path })
await fromCallback(execFile, 'umount', ['--lazy', path])
}
}
_listLvmLogicalVolumes(devicePath, partition, results = []) {
return Disposable.use(this._getLvmPhysicalVolume(devicePath, partition), async path => {
const lvs = await pvs(['lv_name', 'lv_path', 'lv_size', 'vg_name'], path)
const partitionId = partition !== undefined ? partition.id : ''
lvs.forEach((lv, i) => {
const name = lv.lv_name
if (name !== '') {
results.push({
id: `${partitionId}/${lv.vg_name}/${name}`,
name,
size: lv.lv_size,
})
}
})
return results
})
}
// check if we will be allowed to merge a vhd created in this adapter
// with the vhd at path `path`
async isMergeableParent(packedParentUid, path) {
@@ -221,34 +80,6 @@ export class RemoteAdapter {
})
}
fetchPartitionFiles(diskId, partitionId, paths, format) {
const { promise, reject, resolve } = pDefer()
Disposable.use(
async function* () {
const path = yield this.getPartition(diskId, partitionId)
let outputStream
if (format === 'tgz') {
outputStream = tar.c({ cwd: path, gzip: true }, paths.map(makeRelative))
} else if (format === 'zip') {
const zip = new ZipFile()
await addZipEntries(zip, path, '', paths.map(makeRelative))
zip.end()
;({ outputStream } = zip)
} else {
throw new Error('unsupported format ' + format)
}
resolve(outputStream)
await fromEvent(outputStream, 'end')
}.bind(this)
).catch(error => {
warn(error)
reject(error)
})
return promise
}
async #removeVmBackupsFromCache(backups) {
await asyncEach(
Object.entries(
@@ -386,76 +217,6 @@ export class RemoteAdapter {
return this.useVhdDirectory()
}
async *#getDiskLegacy(diskId) {
const RE_VHDI = /^vhdi(\d+)$/
const handler = this._handler
const diskPath = handler.getFilePath('/' + diskId)
const mountDir = yield getTmpDir()
debug('mount VHD (vhdimount)', { diskPath, mountPath: mountDir })
await fromCallback(execFile, 'vhdimount', [diskPath, mountDir])
try {
let max = 0
let maxEntry
const entries = await readdir(mountDir)
entries.forEach(entry => {
const matches = RE_VHDI.exec(entry)
if (matches !== null) {
const value = +matches[1]
if (value > max) {
max = value
maxEntry = entry
}
}
})
if (max === 0) {
throw new Error('no disks found')
}
yield `${mountDir}/${maxEntry}`
} finally {
debug('umount VHD (fusermount)', { diskPath, mountPath: mountDir })
await fromCallback(execFile, 'fusermount', ['-uz', mountDir])
}
}
async *getDisk(diskId) {
if (this._useGetDiskLegacy) {
yield* this.#getDiskLegacy(diskId)
return
}
const handler = this._handler
// this is a disposable
const mountDir = yield getTmpDir()
// this is also a disposable
yield mount(handler, diskId, mountDir)
// this will yield disk path to caller
yield `${mountDir}/vhd0`
}
// partitionId values:
//
// - undefined: raw disk
// - `<partitionId>`: partitioned disk
// - `<pvId>/<vgName>/<lvName>`: LVM on a partitioned disk
// - `/<vgName>/lvName>`: LVM on a raw disk
async *getPartition(diskId, partitionId) {
const devicePath = yield this.getDisk(diskId)
if (partitionId === undefined) {
return yield this._getPartition(devicePath)
}
const isLvmPartition = partitionId.includes('/')
if (isLvmPartition) {
const [pvId, vgName, lvName] = partitionId.split('/')
const lvs = yield this._getLvmLogicalVolumes(devicePath, pvId !== '' ? pvId : undefined, vgName)
return yield this._getPartition(lvs.find(_ => _.lv_name === lvName).lv_path)
}
return yield this._getPartition(devicePath, await this._findPartition(devicePath, partitionId))
}
// if we use alias on this remote, we have to name the file alias.vhd
getVhdFileName(baseName) {
if (this.#useAlias()) {
@@ -496,56 +257,6 @@ export class RemoteAdapter {
return backups
}
listPartitionFiles(diskId, partitionId, path) {
return Disposable.use(this.getPartition(diskId, partitionId), async rootPath => {
path = resolveSubpath(rootPath, path)
const entriesMap = {}
await asyncEach(
await readdir(path),
async name => {
try {
const stats = await lstat(`${path}/${name}`)
if (stats.isDirectory()) {
entriesMap[name + '/'] = {}
} else if (stats.isFile()) {
entriesMap[name] = {}
}
} catch (error) {
if (error == null || error.code !== 'ENOENT') {
throw error
}
}
},
{ concurrency: 1 }
)
return entriesMap
})
}
listPartitions(diskId) {
return Disposable.use(this.getDisk(diskId), async devicePath => {
const partitions = await listPartitions(devicePath)
if (partitions.length === 0) {
try {
// handle potential raw LVM physical volume
return await this._listLvmLogicalVolumes(devicePath, undefined, partitions)
} catch (error) {
return []
}
}
const results = []
await asyncMapSettled(partitions, partition =>
partition.type === LVM_PARTITION_TYPE_MBR || partition.type === LVM_PARTITION_TYPE_GPT
? this._listLvmLogicalVolumes(devicePath, partition, results)
: results.push(partition)
)
return results
})
}
async listPoolMetadataBackups() {
const handler = this._handler
const safeReaddir = createSafeReaddir(handler, 'listPoolMetadataBackups')
@@ -604,6 +315,11 @@ export class RemoteAdapter {
fn(cache)
await this._writeCache(path, cache)
} else {
const regenerated = await this.#getCacheableDataListVmBackups(dirname(path))
if (regenerated !== undefined) {
await this._writeCache(path, regenerated)
}
}
}
@@ -672,8 +388,7 @@ export class RemoteAdapter {
return
}
// detached async action, will not reject
this._writeCache(path, backups)
await this._writeCache(path, backups)
return backups
}
@@ -738,7 +453,7 @@ export class RemoteAdapter {
return path
}
async writeVhd(path, disk, { validator = noop, writeBlockConcurrency } = {}) {
async writeVhd(path, disk, { validator = noop, writeBlockConcurrency, uuid, parentUuid, parentPath } = {}) {
const handler = this._handler
if (this.useVhdDirectory()) {
@@ -750,11 +465,19 @@ export class RemoteAdapter {
concurrency: writeBlockConcurrency,
validator,
compression: 'brotli',
uuid,
parentUuid,
parentPath,
},
})
} else {
const stream = await toVhdStream(disk)
const size = await this.outputStream(path, stream, { validator, checksum: false })
const stream = await toVhdStream(disk, { uuid, parentUuid, parentPath })
const size = await this.outputStream(path, stream, {
validator,
// no checksum for VHDs, because they will be invalidated by
// merges and chains
checksum: false,
})
await validator(path)
return size
}
@@ -932,26 +655,8 @@ Object.assign(RemoteAdapter.prototype, {
isValidXva,
})
decorateMethodsWith(RemoteAdapter, {
_getLvmLogicalVolumes: compose([
Disposable.factory,
[deduped, (devicePath, pvId, vgName) => [devicePath, pvId, vgName]],
debounceResourceFactory,
]),
// File-level-restore methods live in ./_fileRestore.mjs; mix them onto the prototype
// before decorating so decorateMethodsWith can wrap them.
Object.assign(RemoteAdapter.prototype, fileRestoreMethods)
_getLvmPhysicalVolume: compose([
Disposable.factory,
[deduped, (devicePath, partition) => [devicePath, partition?.id]],
debounceResourceFactory,
]),
_getPartition: compose([
Disposable.factory,
[deduped, (devicePath, partition) => [devicePath, partition?.id]],
debounceResourceFactory,
]),
getDisk: compose([Disposable.factory, [deduped, diskId => [diskId]], debounceResourceFactory]),
getPartition: Disposable.factory,
})
decorateMethodsWith(RemoteAdapter, fileRestoreDecorators)

View File

@@ -0,0 +1,126 @@
import { strict as assert } from 'node:assert'
import test from 'node:test'
import { execFile } from 'node:child_process'
import { copyFile, mkdtemp, rm, writeFile } from 'node:fs/promises'
import { Stream } from 'node:stream'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { promisify } from 'node:util'
import { randomBytes } from 'node:crypto'
import Disposable from 'promise-toolbox/Disposable'
import { RemoteAdapter } from './RemoteAdapter.mjs'
// losetup/dmsetup/pvcreate/vgcreate/lvcreate/mkfs all require root (CAP_SYS_ADMIN).
const skip =
typeof process.getuid !== 'function' || process.getuid() !== 0 ? 'requires root (losetup/dmsetup/lvm)' : false
const pExec = promisify(execFile)
const uniqueName = () => `xotest${randomBytes(6).toString('hex')}`
async function withLoop(imagePath, fn) {
const loop = (await pExec('losetup', ['--show', '-f', imagePath])).stdout.trim()
try {
return await fn(loop)
} finally {
await pExec('losetup', ['-d', loop]).catch(() => {})
}
}
// Build a self-contained LVM image: a 64 MiB file holding one PV / VG / ext4 LV.
async function createLvmImage(imagePath, vgName, lvName) {
await pExec('truncate', ['-s', '64M', imagePath])
await withLoop(imagePath, async loop => {
await pExec('pvcreate', ['-f', '-y', loop])
await pExec('vgcreate', [vgName, loop])
await pExec('lvcreate', ['-y', '-l', '100%FREE', '-n', lvName, vgName])
await pExec('mkfs.ext4', ['-q', `/dev/${vgName}/${lvName}`])
await pExec('vgchange', ['-an', vgName])
})
}
// xo-proxy answers this stream through Koa, which pipes only bodies passing `instanceof Stream`
// and JSON-serializes anything else: node-tar >= 7 returns a duck-typed (Minipass) stream, which
// silently turned a proxied file restore into a JSON dump of the tar object. Every format must
// resolve with a real node stream.
test('fetchPartitionFiles resolves with a node:stream for every format', async t => {
const tmp = await mkdtemp(join(tmpdir(), 'xo-flr-fetch-'))
t.after(() => rm(tmp, { recursive: true, force: true }))
await writeFile(join(tmp, 'file.txt'), 'hello')
const adapter = new RemoteAdapter({})
// at this point of a real restore the partition is already mounted
adapter.getPartition = Disposable.factory(async function* () {
yield tmp
})
for (const [format, magic] of [
['tgz', Buffer.from([0x1f, 0x8b])],
['zip', Buffer.from('PK')],
]) {
const stream = await adapter.fetchPartitionFiles('disk', 'partition', ['/file.txt'], format)
assert.ok(stream instanceof Stream, `${format}: expected a node:stream instance`)
const chunks = []
for await (const chunk of stream) {
chunks.push(chunk)
}
const archive = Buffer.concat(chunks)
assert.deepEqual(archive.subarray(0, magic.length), magic, `${format}: unexpected archive header`)
}
})
test('LVM file-restore against real losetup/dmsetup/lvm', { skip }, async t => {
const tmp = await mkdtemp(join(tmpdir(), 'xo-flr-'))
const adapter = new RemoteAdapter({})
const vg = uniqueName()
const lv = 'root_lv'
const image = join(tmp, 'disk.raw')
const clone = join(tmp, 'clone.raw')
const plain = join(tmp, 'plain.raw')
t.after(async () => {
await rm(tmp, { recursive: true, force: true })
// drop any stale online-cache entries pointing at the now-deleted images
await pExec('pvscan', ['--cache']).catch(() => {})
})
await createLvmImage(image, vg, lv)
await t.test('detects the PV and renames the VG to a unique name', async () => {
await Disposable.use(adapter._getLvmPhysicalVolume(image, undefined), async ({ originalVgName, vgName }) => {
assert.equal(originalVgName, vg)
assert.match(vgName, /^xo[0-9a-f]{16}$/)
assert.notEqual(vgName, vg)
})
})
await t.test('two clones of the same VM (shared PVID) list concurrently without collision', async () => {
await copyFile(image, clone) // exact clone → identical PVID + VG name + VG UUID
const [a, b] = await Promise.all([
adapter._listLvmLogicalVolumes(image, undefined, []),
adapter._listLvmLogicalVolumes(clone, undefined, []),
])
assert.equal(a.length, 1)
assert.equal(b.length, 1)
// the human-readable name is the (shared) original; the id carries the unique renamed VG
assert.equal(a[0].name, `${vg}/${lv}`)
assert.equal(b[0].name, `${vg}/${lv}`)
assert.notEqual(a[0].id, b[0].id)
})
await t.test('a non-LVM partition is rejected without building a snapshot', async () => {
await pExec('truncate', ['-s', '16M', plain])
await pExec('mkfs.ext4', ['-q', plain])
await assert.rejects(
Disposable.use(adapter._getLvmPhysicalVolume(plain, undefined), async () => {}),
/no LVM physical volume/
)
})
await t.test('leaves no dangling xo-pv-* dm-snapshots behind', async () => {
const { stdout } = await pExec('dmsetup', ['ls']).catch(() => ({ stdout: '' }))
assert.equal(/xo-pv-/.test(stdout), false, `leftover snapshots:\n${stdout}`)
})
})

View File

@@ -0,0 +1,485 @@
// File-level-restore (FLR) methods for RemoteAdapter.
//
// These were extracted from RemoteAdapter.mjs to keep that file focused on backup
// CRUD/cache/metadata. They are mixed back onto `RemoteAdapter.prototype` via
// `Object.assign` and decorated via `decorateMethodsWith` in RemoteAdapter.mjs, so `this`
// is the RemoteAdapter instance at call time (`this._handler`, `this._debounceResource`,
// `this._useGetDiskLegacy` keep working unchanged). The logger keeps the original
// `xo:backups:RemoteAdapter` namespace so existing debug filters are unaffected.
import { asyncEach } from '@vates/async-each'
import { asyncMapSettled } from '@xen-orchestra/async-map'
import { compose } from '@vates/compose'
import { createLogger } from '@xen-orchestra/log'
import { deduped } from '@vates/disposable/deduped.js'
import { randomBytes } from 'node:crypto'
import { Readable } from 'node:stream'
import { join, resolve } from 'node:path'
import { execFile } from 'child_process'
import { finished } from 'node:stream/promises'
import { lstat, open, readdir, unlink } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { mount } from '@vates/fuse-vhd'
import { ZipFile } from 'yazl'
import Disposable from 'promise-toolbox/Disposable'
import fromCallback from 'promise-toolbox/fromCallback'
import pDefer from 'promise-toolbox/defer'
import * as tar from 'tar'
import { getTmpDir } from './_getTmpDir.mjs'
import {
listPartitions,
LINUX_DATA_PARTITION_TYPE_GPT,
LINUX_DATA_PARTITION_TYPE_MBR,
LVM_PARTITION_TYPE_GPT,
LVM_PARTITION_TYPE_MBR,
} from './_listPartitions.mjs'
import { lvs, pvs } from './_lvm.mjs'
const { debug, warn } = createLogger('xo:backups:RemoteAdapter')
const noop = Function.prototype
// Restrict LVM scanning to a single device. Backup PVs are clones: the very same PVID
// appears at once on the raw loop device, the dm-snapshot overlaid on it, and every
// other restored copy of the same VM. An unscoped pvs/pvscan/vgimportclone/vgchange then
// aborts with "duplicate PV ... for PVID ..." and the VG can neither be renamed nor
// activated. Accepting only the device in hand removes every duplicate from LVM's view.
export const lvmOnlyDevice = devicePath => `devices { global_filter=[ "a|^${devicePath}$|", "r|.*|" ] }`
// Partition-type predicates (exported for unit tests).
export const isLvmPartitionType = type => type === LVM_PARTITION_TYPE_MBR || type === LVM_PARTITION_TYPE_GPT
// Some installers (e.g. Ubuntu subiquity) place an LVM PV on a generic Linux-data partition
// instead of the LVM type, so those are the only non-LVM types worth the (expensive) probe.
export const isProbeableForLvm = type =>
type === LINUX_DATA_PARTITION_TYPE_MBR || type === LINUX_DATA_PARTITION_TYPE_GPT
// Build the partition entries for a PV's logical volumes (exported for unit tests).
// Shows "ubuntu-vg/ubuntu-lv" for readability; skips unnamed LVs.
export const toLvPartitions = (partitionId, originalVgName, lvItems) =>
lvItems
.filter(lv => lv.lv_name !== '')
.map(lv => ({
id: `${partitionId}/${lv.vg_name}/${lv.lv_name}`,
name: originalVgName !== undefined ? `${originalVgName}/${lv.lv_name}` : lv.lv_name,
size: lv.lv_size,
}))
const makeRelative = path => resolve('/', path).slice(1)
const resolveSubpath = (root, path) => resolve(root, makeRelative(path))
async function addZipEntries(zip, realBasePath, virtualBasePath, relativePaths) {
for (const relativePath of relativePaths) {
const realPath = join(realBasePath, relativePath)
const virtualPath = join(virtualBasePath, relativePath)
const stats = await lstat(realPath)
const { mode, mtime } = stats
const opts = { mode, mtime }
if (stats.isDirectory()) {
zip.addEmptyDirectory(virtualPath, opts)
await addZipEntries(zip, realPath, virtualPath, await readdir(realPath))
} else if (stats.isFile()) {
zip.addFile(realPath, virtualPath, opts)
}
}
}
const debounceResourceFactory = factory =>
function () {
return this._debounceResource(factory.apply(this, arguments))
}
// legacy disk mount via vhdimount; extracted from a private method so it can live in this
// module (mixin methods cannot reference class-private members).
async function* getDiskLegacy(handler, diskId) {
const RE_VHDI = /^vhdi(\d+)$/
const diskPath = handler.getFilePath('/' + diskId)
const mountDir = yield getTmpDir()
debug('mount VHD (vhdimount)', { diskPath, mountPath: mountDir })
await fromCallback(execFile, 'vhdimount', [diskPath, mountDir])
try {
let max = 0
let maxEntry
const entries = await readdir(mountDir)
entries.forEach(entry => {
const matches = RE_VHDI.exec(entry)
if (matches !== null) {
const value = +matches[1]
if (value > max) {
max = value
maxEntry = entry
}
}
})
if (max === 0) {
throw new Error('no disks found')
}
yield `${mountDir}/${maxEntry}`
} finally {
debug('umount VHD (fusermount)', { diskPath, mountPath: mountDir })
await fromCallback(execFile, 'fusermount', ['-uz', mountDir])
}
}
// Mixed onto RemoteAdapter.prototype — `this` is the RemoteAdapter instance.
export const fileRestoreMethods = {
async _findPartition(devicePath, partitionId) {
const partitions = await listPartitions(devicePath)
const partition = partitions.find(_ => _.id === partitionId)
if (partition === undefined) {
throw new Error(`partition ${partitionId} not found`)
}
return partition
},
async *_getLvmLogicalVolumes(devicePath, pvId, requestedVgName) {
const { vgName, lvmConfig } = yield this._getLvmPhysicalVolume(
devicePath,
pvId && (await this._findPartition(devicePath, pvId))
)
// vgName is the unique name vgimportclone just assigned to this PV; prefer it over the
// (possibly stale) name embedded in the partitionId. lvmConfig scopes every command to
// this device so a colliding/duplicate VG on the host or another copy can't shadow it.
const effectiveVgName = vgName ?? requestedVgName
debug('activate LVM volume group', { effectiveVgName, requestedVgName })
await fromCallback(execFile, 'vgchange', ['--config', lvmConfig, '-ay', effectiveVgName])
try {
debug('get LVM logical volumes', { effectiveVgName })
yield lvs(['lv_name', 'lv_path'], '--config', lvmConfig, effectiveVgName)
} finally {
debug('deactivate LVM volume group', { effectiveVgName })
await fromCallback(execFile, 'vgchange', ['--config', lvmConfig, '-an', effectiveVgName])
}
},
async *_getLvmPhysicalVolume(devicePath, partition) {
const loopArgs = []
if (partition !== undefined) {
loopArgs.push('-o', partition.start * 512, '--sizelimit', partition.size)
}
loopArgs.push('--show', '-f', devicePath)
debug('attach loop device', { devicePath, partition })
const loopDevice = (await fromCallback(execFile, 'losetup', loopArgs)).trim()
let cowPath, cowLoop, mapperName, vgName, lvmConfig
try {
// Cheap pre-check, scoped to this loop so a duplicate PVID (another copy of the same
// VM restored concurrently) can't make it fail: is there an LVM PV here at all?
// Non-PV partitions (/boot, EFI, raw disks) skip the whole dm-snapshot machinery.
const [originalVgName] = (
await pvs('vg_name', '--config', lvmOnlyDevice(loopDevice), loopDevice).catch(() => [])
).filter(Boolean)
if (originalVgName === undefined) {
const where = partition !== undefined ? ` partition ${partition.id}` : ''
throw new Error(`no LVM physical volume on ${devicePath}${where}`)
}
// The backup is read-only, so overlay a writable dm-snapshot to let vgimportclone
// rewrite metadata, and rename the VG to a unique name so concurrent clones (identical
// PVID and VG name) don't collide on device-mapper node names. Every LVM command is
// scoped to the snapshot device (lvmConfig) to keep duplicate PVIDs out of LVM's view.
mapperName = `xo-pv-${randomBytes(4).toString('hex')}`
const mapperPath = `/dev/mapper/${mapperName}`
lvmConfig = lvmOnlyDevice(mapperPath)
// ~4 MB sparse COW is enough to hold the LVM metadata rewrites
cowPath = join(tmpdir(), `${mapperName}.cow`)
const fh = await open(cowPath, 'w')
await fh.truncate(4 * 1024 * 1024)
await fh.close()
cowLoop = (await fromCallback(execFile, 'losetup', ['--show', '-f', cowPath])).trim()
const sectors = (await fromCallback(execFile, 'blockdev', ['--getsz', loopDevice])).trim()
await fromCallback(execFile, 'dmsetup', [
'create',
mapperName,
'--table',
`0 ${sectors} snapshot ${loopDevice} ${cowLoop} P 8`,
])
// Unique random VG name: list and mount each query/yield the name from the device, so
// it need not be deterministic — only collision-free across concurrent clones.
vgName = `xo${randomBytes(8).toString('hex')}`
debug('import LVM volume group with unique name via dm-snapshot', { vgName, originalVgName })
await fromCallback(execFile, 'vgimportclone', ['--config', lvmConfig, '--basevgname', vgName, mapperPath])
yield { path: mapperPath, originalVgName, vgName, lvmConfig }
} finally {
if (mapperName !== undefined) {
// best-effort deactivate (a no-op if it was never activated) before removing the snapshot
await fromCallback(execFile, 'vgchange', ['--config', lvmConfig, '-an', vgName]).catch(noop)
debug('remove dm-snapshot', { mapperName })
await fromCallback(execFile, 'dmsetup', ['remove', mapperName]).catch(err =>
warn('failed to remove dm-snapshot', { mapperName, error: err })
)
}
if (cowLoop !== undefined) {
await fromCallback(execFile, 'losetup', ['-d', cowLoop]).catch(noop)
}
if (cowPath !== undefined) {
await unlink(cowPath).catch(noop)
}
debug('detach loop device', { loopDevice })
await fromCallback(execFile, 'losetup', ['-d', loopDevice])
}
},
async *_getPartition(devicePath, partition) {
const options = ['loop', 'ro']
if (partition !== undefined) {
const { size, start } = partition
options.push(`sizelimit=${size}`)
if (start !== undefined) {
options.push(`offset=${start * 512}`)
}
}
const path = yield getTmpDir()
const mount = options => {
debug('mount device', { devicePath, mountPath: path })
return fromCallback(execFile, 'mount', [
`--options=${options.join(',')}`,
`--source=${devicePath}`,
`--target=${path}`,
]).catch(error => {
if (error.stderr) {
error.message = `${error.message}: ${error.stderr.trim()}`
}
throw error
})
}
// norecovery prevents mount from attempting journal replay on a read-only device (ext3/ext4/xfs).
// Other filesystems don't support it, so fall back without it on failure.
try {
await mount([...options, 'norecovery'])
} catch (error) {
await mount(options)
}
try {
yield path
} finally {
debug('umount device', { devicePath, mountPath: path })
await fromCallback(execFile, 'umount', ['--lazy', path])
}
},
_listLvmLogicalVolumes(devicePath, partition, results = []) {
return Disposable.use(
this._getLvmPhysicalVolume(devicePath, partition),
async ({ path, originalVgName, lvmConfig }) => {
const lvItems = await pvs(['lv_name', 'lv_path', 'lv_size', 'vg_name'], '--config', lvmConfig, path)
const partitionId = partition !== undefined ? partition.id : ''
results.push(...toLvPartitions(partitionId, originalVgName, lvItems))
return results
}
)
},
fetchPartitionFiles(diskId, partitionId, paths, format) {
const { promise, reject, resolve } = pDefer()
const self = this
Disposable.use(async function* () {
const path = yield self.getPartition(diskId, partitionId)
let outputStream
if (format === 'tgz') {
// process one entry at a time with { job: 1}. node-tar defaults to 4
// concurrent jobs, which on a FUSE-backed restore mount means up to 4
// simultaneous reads. Those saturate the libuv threadpool and starve
// the underlying vhd/CIFS reads NTFS-3g depends on (FUSE-on-FUSE
// threadpool deadlock). Serializing keeps a worker free for them.
const pack = tar.c({ cwd: path, gzip: true, jobs: 1 }, paths.map(makeRelative))
// node-tar >= 7 returns a Minipass stream: it has `pipe()` but is not a `node:stream`
// instance. xo-proxy serves this stream through Koa, which pipes only bodies passing
// `instanceof Stream` and JSON-serializes anything else — so a file restore through a
// proxy answered a JSON dump of the tar object instead of the archive. Wrapping it in a
// real node stream also propagates tar errors to the consumer and aborts the tar job
// when the client disconnects.
outputStream = Readable.from(pack, { objectMode: false })
resolve(outputStream)
} else if (format === 'zip') {
const zip = new ZipFile()
// Resolve with the stream before enumeration so the client can start
// receiving data immediately — addZipEntries over FUSE/S3 can take
// minutes for large trees (e.g. node_modules) and would otherwise
// appear as a freeze with no response sent.
outputStream = zip.outputStream
resolve(zip.outputStream)
await addZipEntries(zip, path, '', paths.map(makeRelative))
zip.end()
} else {
throw new Error('unsupported format ' + format)
}
await finished(outputStream).catch(noop)
}).catch(error => {
warn(error)
reject(error)
})
return promise
},
async *getDisk(diskId) {
if (this._useGetDiskLegacy) {
yield* getDiskLegacy(this._handler, diskId)
return
}
const handler = this._handler
// this is a disposable
const mountDir = yield getTmpDir()
// this is also a disposable
yield mount(handler, diskId, mountDir)
// this will yield disk path to caller
yield `${mountDir}/vhd0`
},
// partitionId values:
//
// - undefined: raw disk
// - `<partitionId>`: partitioned disk
// - `<pvId>/<vgName>/<lvName>`: LVM on a partitioned disk
// - `/<vgName>/lvName>`: LVM on a raw disk
async *getPartition(diskId, partitionId) {
const devicePath = yield this.getDisk(diskId)
if (partitionId === undefined) {
debug(
'no partition specified, attempting raw disk mount — call listPartitions first if disk has partitions or LVM'
)
return yield this._getPartition(devicePath)
}
const isLvmPartition = partitionId.includes('/')
if (isLvmPartition) {
const [pvId, vgName, lvName] = partitionId.split('/')
const lvs = yield this._getLvmLogicalVolumes(devicePath, pvId !== '' ? pvId : undefined, vgName)
return yield this._getPartition(lvs.find(_ => _.lv_name === lvName).lv_path)
}
return yield this._getPartition(devicePath, await this._findPartition(devicePath, partitionId))
},
listPartitionFiles(diskId, partitionId, path) {
return Disposable.use(this.getPartition(diskId, partitionId), async rootPath => {
path = resolveSubpath(rootPath, path)
const entriesMap = {}
await asyncEach(
await readdir(path),
async name => {
try {
const stats = await lstat(`${path}/${name}`)
if (stats.isDirectory()) {
entriesMap[name + '/'] = {}
} else if (stats.isFile()) {
entriesMap[name] = {}
}
} catch (error) {
if (error == null || error.code !== 'ENOENT') {
throw error
}
}
},
{ concurrency: 1 }
)
return entriesMap
})
},
listPartitions(diskId) {
return Disposable.use(this.getDisk(diskId), async devicePath => {
// partx may return empty on FUSE-backed files (vhd0); a loop device
// presents proper block-device semantics that partx reads reliably.
// losetup may itself fail if the FUSE mount isn't fully ready yet —
// fall back to the direct path so listPartitions returns [] instead of throwing.
let loopForParts, partitions
try {
loopForParts = (await fromCallback(execFile, 'losetup', ['--show', '-f', devicePath])).trim()
partitions = await listPartitions(loopForParts)
} catch (error) {
debug('partition probe via loop device failed, falling back to direct path', { error })
partitions = await listPartitions(devicePath)
} finally {
if (loopForParts !== undefined) {
await fromCallback(execFile, 'losetup', ['-d', loopForParts]).catch(noop)
}
}
if (partitions.length === 0) {
try {
// handle potential raw LVM physical volume
return await this._listLvmLogicalVolumes(devicePath, undefined, partitions)
} catch (error) {
return []
}
}
const results = []
await asyncMapSettled(partitions, async partition => {
if (isLvmPartitionType(partition.type)) {
return this._listLvmLogicalVolumes(devicePath, partition, results)
}
// Only generic Linux-data partitions can hide an LVM PV (subiquity-style); other
// types — BIOS boot, EFI, swap, … — are never PVs, so list them directly without
// the (expensive) loop + dm-snapshot + vgimportclone probe.
if (!isProbeableForLvm(partition.type)) {
results.push(partition)
return
}
const lvResults = []
try {
await this._listLvmLogicalVolumes(devicePath, partition, lvResults)
} catch (error) {
debug('LVM probe failed for Linux-data partition, treating as regular partition', {
partition,
error,
})
}
if (lvResults.length > 0) {
results.push(...lvResults)
} else {
results.push(partition)
}
})
return results
})
},
}
// Applied by RemoteAdapter.mjs via decorateMethodsWith(RemoteAdapter, fileRestoreDecorators).
// debounceResourceFactory/deduped reference `this._debounceResource` at call time.
export const fileRestoreDecorators = {
_getLvmLogicalVolumes: compose([
Disposable.factory,
[deduped, (devicePath, pvId, vgName) => [devicePath, pvId, vgName]],
debounceResourceFactory,
]),
_getLvmPhysicalVolume: compose([
Disposable.factory,
[deduped, (devicePath, partition) => [devicePath, partition?.id]],
debounceResourceFactory,
]),
_getPartition: compose([
Disposable.factory,
[deduped, (devicePath, partition) => [devicePath, partition?.id]],
debounceResourceFactory,
]),
getDisk: compose([Disposable.factory, [deduped, diskId => [diskId]], debounceResourceFactory]),
getPartition: Disposable.factory,
}

View File

@@ -0,0 +1,61 @@
import { strict as assert } from 'node:assert'
import test from 'node:test'
import { isLvmPartitionType, isProbeableForLvm, lvmOnlyDevice, toLvPartitions } from './_fileRestore.mjs'
import {
LINUX_DATA_PARTITION_TYPE_GPT,
LINUX_DATA_PARTITION_TYPE_MBR,
LVM_PARTITION_TYPE_GPT,
LVM_PARTITION_TYPE_MBR,
} from './_listPartitions.mjs'
// well-known GPT type GUIDs that must NOT be probed/treated as LVM
const EFI = 'c12a7328-f81f-11d2-ba4b-00a0c93ec93b'
const BIOS_BOOT = '21686148-6449-6e6f-744e-656564454649'
const SWAP = '0657fd6d-a4ab-43c4-84e5-0933c84b4f4f'
test('lvmOnlyDevice builds an accept-only global_filter for the device', () => {
assert.equal(
lvmOnlyDevice('/dev/mapper/xo-pv-abcd1234'),
'devices { global_filter=[ "a|^/dev/mapper/xo-pv-abcd1234$|", "r|.*|" ] }'
)
assert.equal(lvmOnlyDevice('/dev/loop7'), 'devices { global_filter=[ "a|^/dev/loop7$|", "r|.*|" ] }')
})
test('isLvmPartitionType matches only the LVM partition types', () => {
assert.equal(isLvmPartitionType(LVM_PARTITION_TYPE_MBR), true)
assert.equal(isLvmPartitionType(LVM_PARTITION_TYPE_GPT), true)
assert.equal(isLvmPartitionType(LINUX_DATA_PARTITION_TYPE_GPT), false)
assert.equal(isLvmPartitionType(EFI), false)
assert.equal(isLvmPartitionType(undefined), false)
})
test('isProbeableForLvm matches only generic Linux-data types', () => {
assert.equal(isProbeableForLvm(LINUX_DATA_PARTITION_TYPE_MBR), true)
assert.equal(isProbeableForLvm(LINUX_DATA_PARTITION_TYPE_GPT), true)
// never probe these
assert.equal(isProbeableForLvm(LVM_PARTITION_TYPE_GPT), false)
assert.equal(isProbeableForLvm(BIOS_BOOT), false)
assert.equal(isProbeableForLvm(EFI), false)
assert.equal(isProbeableForLvm(SWAP), false)
assert.equal(isProbeableForLvm(undefined), false)
})
test('toLvPartitions builds entries, skips unnamed LVs, shows the original VG name', () => {
const lvItems = [
{ lv_name: 'ubuntu-lv', lv_path: '/dev/xohash/ubuntu-lv', lv_size: '1000', vg_name: 'xohash' },
// a PV with no LV yields a row with an empty lv_name → must be skipped
{ lv_name: '', lv_path: '', lv_size: '0', vg_name: 'xohash' },
{ lv_name: 'swap_1', lv_path: '/dev/xohash/swap_1', lv_size: '500', vg_name: 'xohash' },
]
assert.deepEqual(toLvPartitions('part3', 'ubuntu-vg', lvItems), [
{ id: 'part3/xohash/ubuntu-lv', name: 'ubuntu-vg/ubuntu-lv', size: '1000' },
{ id: 'part3/xohash/swap_1', name: 'ubuntu-vg/swap_1', size: '500' },
])
})
test('toLvPartitions falls back to the bare LV name when the original VG name is unknown', () => {
const lvItems = [{ lv_name: 'data', lv_path: '/dev/xohash/data', lv_size: '42', vg_name: 'xohash' }]
// raw-disk PV → empty partitionId
assert.deepEqual(toLvPartitions('', undefined, lvItems), [{ id: '/xohash/data', name: 'data', size: '42' }])
})

View File

@@ -16,6 +16,44 @@ import { toQcow2Stream } from '@xen-orchestra/qcow2'
const ensureArray = value => (value === undefined ? [] : Array.isArray(value) ? value : [value])
const orderedMemoryLimits = ['memory_static_min', 'memory_dynamic_min', 'memory_dynamic_max', 'memory_static_max']
// The dynamic memory range MUST respect this inequality at any moment: static_min <= dynamic_min <= dynamic_max <= static_max.
// We must update these properties in the right order to avoid XAPI error.
// The order depends on the values. It can be an increase, a decrease or a mix of both, so any order could be required.
export async function updateMemoryFields(xapi, targetVm, vmRecord) {
const memoryValues = {}
for (const key of orderedMemoryLimits) {
memoryValues[key] = {
currentValue: targetVm[key],
newValue: vmRecord[key] ?? targetVm[key],
}
}
while (await updateNextMemoryField(xapi, memoryValues, targetVm.$ref)) {
/* execute until all memory fields are updated */
}
}
// Update one more memory field if needed, then return a boolean describing if a field was updated or if all fields are up to date
async function updateNextMemoryField(xapi, memoryValues, vmRef) {
for (let i = 0; i < orderedMemoryLimits.length; i++) {
const currentField = memoryValues[orderedMemoryLimits[i]]
const nextField = i === orderedMemoryLimits.length - 1 ? undefined : memoryValues[orderedMemoryLimits[i + 1]]
if (
currentField.newValue !== currentField.currentValue &&
(nextField === undefined || currentField.newValue <= nextField.currentValue)
) {
// no need to check that previousField.currentValue <= currentField.newValue, as we can deduce it
await xapi.setField('VM', vmRef, orderedMemoryLimits[i], currentField.newValue)
await xapi.barrier()
currentField.currentValue = currentField.newValue
return true
}
}
return false
}
export async function exportIncrementalVm(
vm,
baseVdis = {},
@@ -160,6 +198,9 @@ export const importIncrementalVm = defer(async function importIncrementalVm(
let vmRef
if (isUpdate) {
vmRef = targetRef
await updateMemoryFields(xapi, targetVm, vmRecord)
await Promise.all([
xapi.setFields('VM', vmRef, {
actions_after_crash: vmRecord.actions_after_crash,
@@ -171,10 +212,6 @@ export const importIncrementalVm = defer(async function importIncrementalVm(
HVM_boot_params: vmRecord.HVM_boot_params,
HVM_boot_policy: vmRecord.HVM_boot_policy,
HVM_shadow_multiplier: vmRecord.HVM_shadow_multiplier,
memory_dynamic_max: vmRecord.memory_dynamic_max,
memory_dynamic_min: vmRecord.memory_dynamic_min,
memory_static_max: vmRecord.memory_static_max,
memory_static_min: vmRecord.memory_static_min,
name_label: vmRecord.name_label,
name_description: vmRecord.name_description,
order: vmRecord.order,

View File

@@ -0,0 +1,122 @@
import { describe, it } from 'node:test'
import assert from 'node:assert/strict'
import { updateMemoryFields } from './_incrementalVm.mjs'
const MOCK_REF = 'mockRef'
function mockXapiVm(initialMemoryValues) {
const res = {
...initialMemoryValues,
updates: [],
barrier: () => {},
setField: (type, ref, field, value) => {
assert(type === 'VM')
assert(ref === MOCK_REF)
res.updates.push({ field, value })
res[field] = value
assert(res.memory_static_min <= res.memory_dynamic_min)
assert(res.memory_dynamic_min <= res.memory_dynamic_max)
assert(res.memory_dynamic_max <= res.memory_static_max)
},
}
return res
}
const tests = [
{
label: 'handles a memory increase',
targetVm: {
memory_static_min: 2,
memory_dynamic_min: 4,
memory_dynamic_max: 4,
memory_static_max: 8,
},
vmRecord: {
memory_static_min: 16,
memory_dynamic_min: 16,
memory_dynamic_max: 16,
memory_static_max: 16,
},
expectedUpdates: [
{ field: 'memory_static_max', value: 16 },
{ field: 'memory_dynamic_max', value: 16 },
{ field: 'memory_dynamic_min', value: 16 },
{ field: 'memory_static_min', value: 16 },
],
},
{
label: 'handles a memory decrease',
targetVm: {
memory_static_min: 4,
memory_dynamic_min: 8,
memory_dynamic_max: 16,
memory_static_max: 16,
},
vmRecord: {
memory_static_min: 2,
memory_dynamic_min: 2,
memory_dynamic_max: 2,
memory_static_max: 2,
},
expectedUpdates: [
{ field: 'memory_static_min', value: 2 },
{ field: 'memory_dynamic_min', value: 2 },
{ field: 'memory_dynamic_max', value: 2 },
{ field: 'memory_static_max', value: 2 },
],
},
{
label: 'handles a more complex case',
targetVm: {
memory_static_min: 1,
memory_dynamic_min: 2,
memory_dynamic_max: 8,
memory_static_max: 16,
},
vmRecord: {
memory_static_min: 4,
memory_dynamic_min: 4,
memory_dynamic_max: 4,
memory_static_max: 4,
},
expectedUpdates: [
{ field: 'memory_dynamic_min', value: 4 },
{ field: 'memory_static_min', value: 4 },
{ field: 'memory_dynamic_max', value: 4 },
{ field: 'memory_static_max', value: 4 },
],
},
{
label: 'avoids updating identical value',
targetVm: {
memory_static_min: 1,
memory_dynamic_min: 2,
memory_dynamic_max: 4,
memory_static_max: 8,
},
vmRecord: {
memory_static_min: 1,
memory_dynamic_min: 2,
memory_dynamic_max: 4,
memory_static_max: 8,
},
expectedUpdates: [],
},
]
describe('updateMemoryFields() should update in the right order', () => {
for (const test of tests) {
it(test.label, async () => {
const mock = mockXapiVm(test.targetVm)
test.targetVm.$ref = MOCK_REF
await updateMemoryFields(mock, test.targetVm, test.vmRecord)
assert.equal(mock.updates.length, test.expectedUpdates.length)
for (let i = 0; i < mock.updates.length; i++) {
assert.equal(mock.updates[i].value, test.expectedUpdates[i].value)
assert.equal(mock.updates[i].field, test.expectedUpdates[i].field)
}
})
}
})

View File

@@ -29,6 +29,15 @@ export const LVM_PARTITION_TYPE_MBR = 0x8e
// GPT LVM type
export const LVM_PARTITION_TYPE_GPT = 'e6d6d379-f507-44c2-a23c-238f2a3df928'
// Generic "Linux filesystem data" types. These are the only non-LVM-typed
// partitions worth probing for a hidden LVM PV, because some installers (e.g.
// Ubuntu subiquity) place a PV on a partition left with this generic type.
// Other types (BIOS boot, EFI, swap, …) are never LVM PVs.
// MBR Linux native
export const LINUX_DATA_PARTITION_TYPE_MBR = 0x83
// GPT Linux filesystem data
export const LINUX_DATA_PARTITION_TYPE_GPT = '0fc63daf-8483-4772-8e79-3d69d8477de4'
const parsePartxLine = createParser({
keyTransform: key => (key === 'UUID' ? 'id' : key.toLowerCase()),
valueTransform: (value, key) => {

View File

@@ -33,6 +33,13 @@ export const VM_UUID = 'xo:backup:vm'
// allows direction-agnostic discovery of common base VDIs (e.g. for failback/reverse replication)
export const CONTENT_KEY = 'xo:backup:contentKey'
// in `other_config` of a replicated VM and its VDIs
// set to the string 'true' on every write done by code that includes the non-NBD qcow2
// corruption fix (fix(disk-transform): erroneous DiskLargerBlock copy offset). Its absence
// on a qcow2 full means the replica may be silently corrupt and its chain must be re-based
// with a clean full. Named after the specific fix so future fixes get their own flag.
export const INCLUDE_NON_NBD_QCOW2_FIX = 'xo:backup:includeNonNbdQcow2Fix'
async function listVdiRefs(xapi, vmRef) {
return xapi.VM_getDisks(vmRef)
}
@@ -90,6 +97,7 @@ export function resetVmOtherConfig(xapi, vmRef) {
[DATETIME]: null,
[DELTA_CHAIN_LENGTH]: null,
[EXPORTED_SUCCESSFULLY]: null,
[INCLUDE_NON_NBD_QCOW2_FIX]: null,
[JOB_ID]: null,
[SCHEDULE_ID]: null,
[VM_UUID]: null,
@@ -120,6 +128,10 @@ export async function setVmOtherConfig(xapi, vmRef, { timestamp, jobId, schedule
xapi.setFieldEntries(type, ref, 'other_config', {
[REPLICATED_TO_SR_UUID]: srUuid,
[DATETIME]: formatDateTime(timestamp),
// stamped on every write: its presence marks the object as produced by code
// that includes the non-NBD qcow2 corruption fix, so a chain rooted in a pre-fix
// qcow2 full (missing this flag) can be detected and re-based with a clean full
[INCLUDE_NON_NBD_QCOW2_FIX]: 'true',
[JOB_ID]: jobId,
[SCHEDULE_ID]: scheduleId,
[VM_UUID]: vmUuid,

View File

@@ -137,6 +137,7 @@ export const VmsRemote = class RemoteVmsBackupRunner extends Abstract {
task.warning(`Retry the VM mirror backup due to an error`, {
attempt: nTriesByVmId[vmUuid],
error: taskError.message,
isRetry: true,
})
queue.add(vmUuid)
})

View File

@@ -101,10 +101,14 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
const handleVm = vmUuid => {
const getVmTask = () => {
if (taskByVmId[vmUuid] === undefined) {
const started = taskByVmId[vmUuid] !== undefined
if (!started) {
taskByVmId[vmUuid] = new Task(taskStart)
}
return taskByVmId[vmUuid]
return {
task: taskByVmId[vmUuid],
started,
}
}
const vmBackupFailed = async (error, task) => {
if (isLastRun) {
@@ -114,6 +118,7 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
task.warning(`Retry the VM backup due to an error`, {
attempt: nTriesByVmId[vmUuid],
error: error.message,
isRetry: true,
})
queue.add(vmUuid)
}
@@ -135,7 +140,7 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
taskStart.properties.name_label = vm.name_label
}
const task = getVmTask()
const { task } = getVmTask()
// error has to be caught in the task to prevent its failure, but handled outside the task to execute another task.run()
let taskError
return task
@@ -174,12 +179,19 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
task.success(result)
} else {
// ending the task with error or not ending the task
vmBackupFailed(taskError, task)
return vmBackupFailed(taskError, task)
}
})
.catch(noop) // errors are handled by logs
}),
error => vmBackupFailed(error, getVmTask())
error => {
const { task: vmTask, started } = getVmTask()
if (!started) {
// the task is not started (except if it's a retry), and an unstarted task can't be failed
vmTask.start()
}
return vmBackupFailed(error, vmTask)
}
)
}
const { concurrency } = settings

View File

@@ -1,10 +1,13 @@
import { asyncMap } from '@xen-orchestra/async-map'
import { createLogger } from '@xen-orchestra/log'
import { Task } from '@vates/task'
import { DIR_XO_POOL_METADATA_BACKUPS } from '../RemoteAdapter.mjs'
import { forkStreamUnpipe } from './_forkStreamUnpipe.mjs'
import { formatFilenameDate } from '../_filenameDate.mjs'
const { debug } = createLogger('xo:backups:PoolMetadataBackup')
export const PATH_DB_DUMP = '/pool/xmldbdump'
export class PoolMetadataBackup {
@@ -31,7 +34,22 @@ export class PoolMetadataBackup {
const poolDir = `${DIR_XO_POOL_METADATA_BACKUPS}/${schedule.id}/${pool.$id}`
const dir = `${poolDir}/${formatFilenameDate(timestamp)}`
const tasks = Object.values(pool.$xapi.objects.indexes.type.task ?? {})
const pendingTasks = tasks.filter(task => task.status === 'pending')
debug('pool tasks before export', {
total: tasks.length,
pending: pendingTasks.length,
oldestPendingCreated: pendingTasks.map(task => task.created).sort()[0],
})
const stream = (await this._exportPoolMetadata()).body
const startedAt = Date.now()
let bytes = 0
stream.once('data', () => debug('first byte received', { afterMs: Date.now() - startedAt }))
stream.on('data', chunk => (bytes += chunk.length))
stream.once('end', () => debug('stream ended', { bytes, durationMs: Date.now() - startedAt }))
stream.once('error', error => debug('stream errored', { error, bytes, durationMs: Date.now() - startedAt }))
const fileName = `${dir}/data`
const metadata = JSON.stringify(

View File

@@ -10,6 +10,9 @@ export default function createStreamThrottle(rate) {
}
const group = new ThrottleGroup({ rate })
return function throttleStream(stream) {
return pipeline(stream, group.createThrottle(), noop)
const throttled = pipeline(stream, group.createThrottle(), noop)
throttled.length = stream.length
throttled.maxStreamLength = stream.maxStreamLength
return throttled
}
}

View File

@@ -0,0 +1,58 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { Readable } from 'node:stream'
import createStreamThrottle from './_createStreamThrottle.mjs'
const makeStream = (chunks, props = {}) => {
const stream = Readable.from(chunks)
Object.assign(stream, props)
return stream
}
const readAll = async stream => {
const chunks = []
for await (const chunk of stream) {
chunks.push(chunk)
}
return Buffer.concat(chunks.map(c => (typeof c === 'string' ? Buffer.from(c) : c))).toString()
}
describe('createStreamThrottle', () => {
it('rate=0 returns identity, stream data and props intact', async () => {
const throttleStream = createStreamThrottle(0)
const stream = makeStream(['hello world'], { length: 11, maxStreamLength: 20 })
const throttled = throttleStream(stream)
assert.strictEqual(throttled, stream)
assert.strictEqual(throttled.length, 11)
assert.strictEqual(throttled.maxStreamLength, 20)
assert.strictEqual(await readAll(throttled), 'hello world')
})
it('rate>0 passes data through and preserves length and maxStreamLength', async () => {
const throttleStream = createStreamThrottle(1024 * 1024)
const stream = makeStream(['hello world'], { length: 11, maxStreamLength: 20 })
const throttled = throttleStream(stream)
assert.strictEqual(throttled.length, 11)
assert.strictEqual(throttled.maxStreamLength, 20)
assert.strictEqual(await readAll(throttled), 'hello world')
})
it('rate>0 handles undefined length and maxStreamLength', async () => {
const throttleStream = createStreamThrottle(1024 * 1024)
const stream = makeStream(['hello world'])
const throttled = throttleStream(stream)
assert.strictEqual(throttled.length, undefined)
assert.strictEqual(throttled.maxStreamLength, undefined)
assert.strictEqual(await readAll(throttled), 'hello world')
})
// regression: maxStreamLength was lost after throttling, causing S3 to ignore it and cap at 50GB
it('rate>0 preserves maxStreamLength so S3 can compute correct partSize', async () => {
const throttleStream = createStreamThrottle(1024 * 1024)
const stream = makeStream(['hello world'], { maxStreamLength: 11 })
const throttled = throttleStream(stream)
assert.strictEqual(throttled.maxStreamLength, 11)
assert.strictEqual(await readAll(throttled), 'hello world')
})
})

View File

@@ -99,6 +99,7 @@ class IncrementalRemoteVmBackupRunner extends AbstractRemote {
writer =>
writer.transfer({
deltaExport: forkDeltaExport(incrementalExport, writer.constructor.name),
includeNonNbdQcow2Fix: metadata.includeNonNbdQcow2Fix,
isVhdDifferencing,
timestamp: metadata.timestamp,
vm: metadata.vm,
@@ -106,16 +107,6 @@ class IncrementalRemoteVmBackupRunner extends AbstractRemote {
}),
'writer.transfer()'
)
// this will update parent name with the needed alias
await this._callWriters(
writer =>
writer.updateUuidAndChain({
isVhdDifferencing,
timestamp: metadata.timestamp,
vdis: incrementalExport.vdis,
}),
'writer.updateUuidAndChain()'
)
await this._callWriters(writer => writer.cleanup(), 'writer.cleanup()')
// for healthcheck

View File

@@ -73,6 +73,10 @@ export const IncrementalXapi = class IncrementalXapiVmBackupRunner extends Abstr
writer =>
writer.transfer({
deltaExport: forkDeltaExport(deltaExport, writer.constructor.name),
// freshly exported from XAPI by code that has the fix and the force-full gate
// (writer.checkBaseVdis ran before this transfer), so the chain is verified/forced
// clean; the tip's flag lets the next run skip re-probing
includeNonNbdQcow2Fix: true,
isVhdDifferencing,
timestamp,
vm,
@@ -80,15 +84,6 @@ export const IncrementalXapi = class IncrementalXapiVmBackupRunner extends Abstr
}),
'writer.transfer()'
)
await this._callWriters(
writer =>
writer.updateUuidAndChain({
isVhdDifferencing,
timestamp,
vdis: deltaExport.vdis,
}),
'writer.updateUuidAndChain()'
)
if (isFull) {
await setVmDeltaChainLength(this._xapi, exportedVm.$ref, 0)

View File

@@ -213,6 +213,35 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
vmUuid: vm.uuid,
})
await setVmSnapshotContentKeys(xapi, snapshotRef)
// Verify that setVmOtherConfig actually tagged every disk we expected.
// If the snapshot exposes fewer 'Disk' VBDs than the live VM has, the
// tagging call silently skipped some VDIs and the resulting snapshot
// will be invisible to retention/cleanup on the next run.
const liveDisks = await xapi.VM_getDisks(vm.$ref)
const snapshotDisks = await xapi.VM_getDisks(snapshotRef)
if (liveDisks.length !== snapshotDisks.length) {
const snapshotVbdRefs = await xapi.getField('VM', snapshotRef, 'VBDs')
const snapshotVbds = await xapi.getRecords('VBD', snapshotVbdRefs)
warn('inconsistent snapshot tagging detected', {
vmUuid: vm.uuid,
vmName: vm.name_label,
snapshotRef,
jobId: this._jobId,
timestamp: this.timestamp,
expectedDiskCount: liveDisks.length,
actualDiskCount: snapshotDisks.length,
liveDisks,
snapshotDisks,
snapshotVbds: snapshotVbds.map(vbd => ({
uuid: vbd.uuid,
type: vbd.type,
VDI: vbd.VDI,
currently_attached: vbd.currently_attached,
})),
})
}
const snapshot = await xapi.getRecord('VM', snapshotRef)
await snapshot.set_name_label(this._getSnapshotNameLabel(vm))
// reload data to ensure it is up to date with the new name label

View File

@@ -22,7 +22,7 @@ export class AggregatedIncrementalRemoteWriter extends AbstractAggregatedRemoteW
}
// update the parameter with our best candidate
// that can be an empty map
baseUuidToSrcVdi.forEach(key => {
baseUuidToSrcVdi.forEach((_, key) => {
if (!selectedCopy.has(key)) {
baseUuidToSrcVdi.delete(key)
}
@@ -50,13 +50,8 @@ export class AggregatedIncrementalRemoteWriter extends AbstractAggregatedRemoteW
await Promise.all(this.writers.map(writer => writer.prepare({ isFull })))
}
// write only on the main writer
transfer({ isVhdDifferencing, timestamp, deltaExport, vm, vmSnapshot }) {
return this.mainWriter.transfer({ isVhdDifferencing, timestamp, deltaExport, vm, vmSnapshot })
}
// chain only on the main writer
updateUuidAndChain({ isVhdDifferencing, vdis }) {
return this.mainWriter.updateUuidAndChain({ isVhdDifferencing, vdis })
transfer(args) {
return this.mainWriter.transfer(args)
}
// remove the backups and remove the entries

View File

@@ -25,7 +25,7 @@ export class AggregatedIncrementalXapiWriter extends AbstractAggregatedXapiWrite
}
// update the parameter with our best candidate
// that can be an empty map
baseUuidToSrcVdi.forEach(key => {
baseUuidToSrcVdi.forEach((_, key) => {
if (!selectedCopy.has(key)) {
baseUuidToSrcVdi.delete(key)
}
@@ -59,11 +59,6 @@ export class AggregatedIncrementalXapiWriter extends AbstractAggregatedXapiWrite
return this.mainWriter.transfer(args)
}
// chain only on the main writer
updateUuidAndChain(args) {
return this.mainWriter.updateUuidAndChain(args)
}
// remove the backups and remove the entries
async cleanup() {
debug('cleanup')

View File

@@ -2,11 +2,11 @@ import assert from 'node:assert'
import mapValues from 'lodash/mapValues.js'
import { asyncEach } from '@vates/async-each'
import { asyncMap } from '@xen-orchestra/async-map'
import { chainVhd, openVhd } from 'vhd-lib'
import { createLogger } from '@xen-orchestra/log'
import { decorateClass } from '@vates/decorate-with'
import { defer } from 'golike-defer'
import { dirname, basename } from 'node:path'
import { relativeFromFile } from '@xen-orchestra/fs/path'
import { Task } from '@vates/task'
import { formatFilenameDate } from '../../_filenameDate.mjs'
@@ -16,15 +16,18 @@ import { MixinRemoteWriter } from './_MixinRemoteWriter.mjs'
import { AbstractIncrementalWriter } from './_AbstractIncrementalWriter.mjs'
import { checkVhd } from './_checkVhd.mjs'
import { packUuid } from './_packUuid.mjs'
import { Disposable } from 'promise-toolbox'
import { openDiskChain } from '@xen-orchestra/backup-archive/disks'
import { VDI_FORMAT_QCOW2 } from '@xen-orchestra/xapi'
import { diskHasData } from './_diskHasData.mjs'
const { warn } = createLogger('xo:backups:DeltaBackupWriter')
export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrementalWriter) {
#parentVdiPaths
#vhds
#parentUuids
async checkBaseVdis(baseUuidToSrcVdi) {
this.#parentVdiPaths = {}
this.#parentUuids = {}
const { handler } = this._adapter
const adapter = this._adapter
@@ -32,6 +35,7 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
await asyncMap(baseUuidToSrcVdi, async ([baseUuid, srcVdiUuid]) => {
let parentDestPath
let parentUuid
const vhdDir = `${vdisDir}/${srcVdiUuid}`
try {
const vhds = await handler.list(vhdDir, {
@@ -47,6 +51,7 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
try {
if (await adapter.isMergeableParent(packedBaseUuid, path)) {
parentDestPath = path
parentUuid = packedBaseUuid
}
} catch (error) {
warn('checkBaseVdis', { error })
@@ -59,11 +64,77 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
if (parentDestPath === undefined) {
baseUuidToSrcVdi.delete(baseUuid)
} else {
this.#parentVdiPaths[vhdDir] = parentDestPath
try {
// A delta is only chained onto a qcow2 base when NBD is enabled; without NBD a qcow2
// export falls back to a full, so there is no suspect base to worry about.
if (this._settings.preferNbd && !(await this.#isBaseChainable(parentDestPath))) {
Task.warning('forcing a full: qcow2 base predates the non-NBD qcow2 fix and looks corrupt', {
parentDestPath,
})
baseUuidToSrcVdi.delete(baseUuid)
return
}
this.#parentVdiPaths[vhdDir] = parentDestPath
this.#parentUuids[vhdDir] = parentUuid
} catch (error) {
warn('checkBaseVdis: unexpected error, forcing a full', { error, parentDestPath })
baseUuidToSrcVdi.delete(baseUuid)
}
}
})
}
// Decide whether it is safe to chain a delta onto `parentDestPath` (the chain tip).
// A qcow2 full exported before the DiskLargerBlock 20260724 fix can be silently corrupt (block 0
// valid, following allocated blocks all zero); chaining a delta onto it would extend a
// broken chain, so we detect that case and let the caller fall back to a full.
async #isBaseChainable(parentDestPath) {
// parentDestPath is the chain tip = most-recent backup; its filename is stable under
// merges (merges rename the OLDER disks), so unlike the root it maps reliably to its
// metadata json.
const parentDate = basename(parentDestPath).replace(/\.(alias\.)?vhd$/, '')
let metadata
try {
metadata = await this._adapter.readVmBackupMetadata(`${this._vmBackupDir}/${parentDate}.json`)
} catch (error) {
warn('checkBaseVdis: cannot read parent metadata, will probe', { error, parentDestPath })
}
if (metadata?.includeNonNbdQcow2Fix === true) {
// written by code that has the fix and this gate, so its chain was already
// verified/forced clean — no disk open needed
// we can't have includeNonNbdQcow2Fix = false since it would have restarted a clean chain
return true
}
const isQcow2 =
metadata === undefined ||
Object.values(metadata.vdis ?? {}).some(vdi => vdi?.sm_config?.['image-format'] === VDI_FORMAT_QCOW2)
if (!isQcow2) {
// only qcow2 exports can carry this corruption
return true
}
return this.#baseHasData(parentDestPath)
}
// Probe the first data blocks of the chain root for the DiskLargerBlock corruption
// signature. Returns false only when enough allocated blocks past the first are all zero.
async #baseHasData(parentDestPath) {
const { handler } = this._adapter
// open the chain (with block allocation tables) and probe its root full. This only runs
// once per chain — afterwards the includeNonNbdQcow2Fix flag on the tip skips it — so
// reading the deltas' BATs here is negligible and lets us reuse the root disk directly.
const chain = await openDiskChain({ handler, path: parentDestPath })
try {
const root = chain.getRootDisk()
if (root === undefined) {
throw new Error(`can't resolve root of chain ${parentDestPath}`)
}
return await diskHasData(root)
} finally {
await chain.close()
}
}
async beforeBackup() {
await super.beforeBackup()
return this._cleanVm({ merge: true, remove: true })
@@ -126,48 +197,6 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
}
}
async updateUuidAndChain({ isVhdDifferencing, vdis }) {
assert.notStrictEqual(
this.#vhds,
undefined,
'_transfer must be called before updateUuidAndChain for incremental backups'
)
const parentVdiPaths = this.#parentVdiPaths
const { handler } = this._adapter
const vhds = this.#vhds
await asyncEach(Object.entries(vdis), async ([id, vdi]) => {
const isDifferencing = isVhdDifferencing[id]
const path = `${this._vmBackupDir}/${vhds[id]}`
if (isDifferencing) {
assert.notStrictEqual(
parentVdiPaths,
undefined,
'checkbasevdi must be called before updateUuidAndChain for incremental backups'
)
const parentPath = parentVdiPaths[dirname(path)]
// we are in a incremental backup
// we already computed the chain in checkBaseVdis
assert.notStrictEqual(parentPath, undefined, 'A differential VHD must have a parent')
// forbid any kind of loop
assert.ok(basename(parentPath) < basename(path), `vhd must be sorted to be chained`)
// re-chainVhd is mandatory
// since the parent may be a alias or not
// and the child may be the other
await chainVhd(handler, parentPath, handler, path)
}
// set the correct UUID in the VHD if needed
await Disposable.use(openVhd(handler, path), async vhd => {
if (!vhd.footer.uuid.equals(packUuid(vdi.uuid))) {
vhd.footer.uuid = packUuid(vdi.uuid)
await vhd.readBlockAllocationTable() // required by writeFooter()
await vhd.writeFooter()
}
})
})
}
async _deleteOldEntries() {
const adapter = this._adapter
const oldEntries = this._oldEntries
@@ -178,7 +207,7 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
}
}
async _transfer($defer, { isVhdDifferencing, timestamp, deltaExport, vm, vmSnapshot }) {
async _transfer($defer, { includeNonNbdQcow2Fix, isVhdDifferencing, timestamp, deltaExport, vm, vmSnapshot }) {
const adapter = this._adapter
const job = this._job
const scheduleId = this._schedule.id
@@ -186,10 +215,8 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
const jobId = job.id
const handler = adapter.handler
const basename = formatFilenameDate(timestamp)
// update this.#vhds before eventually skipping transfer, so that
// updateUuidAndChain has all the mandatory data
const vhds = (this.#vhds = mapValues(
const filenameDate = formatFilenameDate(timestamp)
const vhds = mapValues(
deltaExport.vdis,
vdi =>
`vdis/${jobId}/${
@@ -198,8 +225,8 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
// don't do delta for it
vdi.uuid
: vdi.$snapshot_of$uuid
}/${adapter.getVhdFileName(basename)}`
))
}/${adapter.getVhdFileName(filenameDate)}`
)
let metadataContent = await this._isAlreadyTransferred(timestamp)
if (metadataContent !== undefined) {
@@ -209,6 +236,11 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
}
metadataContent = {
// marks whether the disk data is known-good w.r.t. the non-NBD qcow2 corruption bug.
// A fresh XAPI export by fixed code passes `true`; the mirror runner propagates the
// source backup's value (copying corrupt source data must NOT become "fixed"). Its
// absence flags a potentially-corrupt older disk (used by cleanVm and the force-full gate).
includeNonNbdQcow2Fix,
isVhdDifferencing,
jobId,
mode: job.mode,
@@ -229,12 +261,24 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
Object.entries(deltaExport.disks),
async ([diskRef, disk]) => {
const path = `${this._vmBackupDir}/${vhds[diskRef]}`
const vdi = deltaExport.vdis[diskRef]
let parentUuid, parentPath
if (isVhdDifferencing[diskRef]) {
const parentDestPath = this.#parentVdiPaths[dirname(path)]
parentUuid = this.#parentUuids[dirname(path)]
assert.notStrictEqual(parentDestPath, undefined, 'A differential VHD must have a parent')
// forbid any kind of loop
assert.ok(basename(parentDestPath) < basename(path), `vhd must be sorted to be chained`)
parentPath = relativeFromFile(path, parentDestPath)
}
const transferred = await adapter.writeVhd(path, disk, {
// no checksum for VHDs, because they will be invalidated by
// merges and chains
checksum: false,
validator: tmpPath => checkVhd(handler, tmpPath),
writeBlockConcurrency: this._config.writeBlockConcurrency,
uuid: packUuid(vdi.uuid),
parentUuid,
parentPath,
})
size += transferred
},

View File

@@ -1,3 +1,4 @@
import assert from 'node:assert'
import humanFormat from 'human-format'
import { asyncMapSettled } from '@xen-orchestra/async-map'
@@ -20,6 +21,7 @@ import {
DATETIME,
VM_UUID,
CONTENT_KEY,
INCLUDE_NON_NBD_QCOW2_FIX,
resetVmOtherConfig,
} from '../../_otherConfig.mjs'
import { formatFilenameDate } from '../../_filenameDate.mjs'
@@ -27,8 +29,9 @@ import { XapiDiskSource } from '@xen-orchestra/xapi'
import { asyncEach } from '@vates/async-each'
import { createLogger } from '@xen-orchestra/log'
import { VM_POWER_STATE } from '@vates/types'
import { diskHasData } from './_diskHasData.mjs'
const { debug } = createLogger('xo:backups:IncrementalXapiWriter')
const { debug, warn } = createLogger('xo:backups:IncrementalXapiWriter')
export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWriter) {
// Map of source VDI UUID (COPY_OF) → validated active VDI on the target SR.
@@ -93,6 +96,45 @@ export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWr
)
}
if (target !== undefined) {
// A delta is only chained onto a qcow2 base when NBD is enabled; without NBD a qcow2
// export falls back to a full, so there is no suspect base.
if (this._settings.preferNbd) {
// Every copy written by fixed code carries the flag, so the marker on the most recent
// link is enough: if it is there, this chain was already verified (or re-based) by a
// run that had the fix, and older links need no re-check.
const suspect = target.other_config[INCLUDE_NON_NBD_QCOW2_FIX] !== 'true'
if (suspect) {
const chain = this.#diskReplicaSnapshots(target)
// probe the OLDEST link (least-masked view of the corruption — a later delta could
// have overwritten the corrupt blocks and hidden the damage on a more recent snapshot).
// Re-replicating a full is expensive, so only force one when it actually looks corrupt.
let base = target
assert.notStrictEqual(base.other_config[DATETIME], undefined)
for (const vdi of chain) {
const datetime = vdi.other_config[DATETIME]
assert.notStrictEqual(datetime, undefined)
if (datetime < base.other_config[DATETIME]) {
base = vdi
}
}
let chainable
try {
chainable = await this.#baseVdiHasData(base.$ref)
} catch (error) {
// can't read the base to decide → don't chain onto it
warn('checkBaseVdis: error probing replicated base, forcing a full', { error, baseVdi: base.uuid })
chainable = false
}
if (!chainable) {
Task.warning('forcing a full: replicated qcow2 chain predates the non-NBD qcow2 fix and looks corrupt', {
baseVdi: base.uuid,
})
// drop it so neither the snapshot nor the legacy fallback reuses it → full
baseUuidToSrcVdi.delete(baseUuid)
continue
}
}
}
snapshotCandidates.set(baseUuid, target)
}
}
@@ -133,6 +175,42 @@ export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWr
}
}
}
// All replica snapshots of a given disk share the same active target VDI ($snapshot_of). Return
// them (same job+vm) so the caller can inspect the whole chain: any link missing the fix marker
// makes the chain suspect, and the oldest link (by DATETIME) is the least-masked base to probe.
// Scoped by $snapshot_of, so history before a CR VM recreation (which resets it) is not included.
#diskReplicaSnapshots(target) {
const activeUuid = target.$snapshot_of?.uuid
const snapshots = []
for (const vdi of this._sr.$VDIs) {
if (
vdi?.managed &&
vdi?.is_a_snapshot &&
vdi.$snapshot_of?.uuid === activeUuid &&
vdi.other_config[JOB_ID] === this._job.id &&
vdi.other_config[VM_UUID] === this._vmUuid
) {
snapshots.push(vdi)
}
}
return snapshots
}
// Probe a replicated base VDI for the DiskLargerBlock corruption signature (block 0 valid,
// following allocated blocks all zero). Uses an export (not NBD): block flow is guaranteed
// to be in index order, so block 0 comes first and inspecting the first few blocks is enough.
// Returns false only when no allocated non-zero block is found among the first ones.
async #baseVdiHasData(vdiRef) {
const source = new XapiDiskSource({ xapi: this._sr.$xapi, vdiRef, preferNbd: false })
try {
await source.init()
return await diskHasData(source)
} finally {
await source.close()
}
}
/**
* 6.3+ snapshot-based validation: for each snapshot candidate, check whether
* the active VDI has diverged from the snapshot. Returns a baseVdisBySourceUuid
@@ -235,9 +313,6 @@ export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWr
return { baseVdisBySourceUuid, targetVmRef }
}
updateUuidAndChain() {
// nothing to do, the chaining is not modified in this case
}
prepare({ isFull }) {
// create the task related to this export and ensure all methods are called in this context
const task = new Task({
@@ -343,6 +418,10 @@ export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWr
if (baseDeltaVdiUuid !== undefined) {
// reuse the validated mapping built by checkBaseVdis
vdi.baseVdi = this.#baseVdisBySourceUuid.get(baseDeltaVdiUuid)
// BASE_DELTA_VDI is set, so the export is differencing. Without a base to clone,
// importIncrementalVm would create a blank VDI and write only the changed blocks
// into it, silently producing a corrupt replica: fail the job instead.
assert.notStrictEqual(vdi.baseVdi, undefined, `no validated target base for ${baseDeltaVdiUuid}`)
} else {
// first replication of this disk (full, no base)
vdi.baseVdi = undefined

View File

@@ -5,10 +5,6 @@ export class AbstractIncrementalWriter extends AbstractWriter {
throw new Error('Not implemented')
}
updateUuidAndChain() {
throw new Error('Not implemented')
}
cleanup() {
throw new Error('Not implemented')
}
@@ -20,8 +16,6 @@ export class AbstractIncrementalWriter extends AbstractWriter {
async transfer({ deltaExport, ...other }) {
try {
return await this._transfer({ deltaExport, ...other })
} catch (err) {
console.error({ err })
} finally {
// ensure all sources are properly closed
for (const disk of Object.values(deltaExport.disks)) {

View File

@@ -0,0 +1,26 @@
// Probe a disk for the DiskLargerBlock corruption signature (block 0 valid,
// following allocated blocks all zero). Uses an export (not NBD): block flow is guaranteed
// to be in index order, so block 0 comes first and inspecting the first few blocks is enough.
// Returns false only when no allocated non-zero block is found among the first ones.
export async function diskHasData(source) {
const empty = Buffer.alloc(source.getBlockSize(), 0)
const MAX_EMPTY = 10
let nbEmpty = 0
// diskBlocks() closes the source when the iteration ends or is broken out of
for await (const { index, data } of source.diskBlocks()) {
if (index === 0) {
// block 0 stays valid even in the corruption signature — skip it
continue
}
if (!data.equals(empty)) {
// an allocated block holds real data → not corrupt
return true
}
if (++nbEmpty >= MAX_EMPTY) {
break
}
}
// no allocated non-zero block among the first ones → looks corrupt (or empty; a full is cheap then)
return false
}

View File

@@ -230,7 +230,6 @@ Settings are described in [`@xen-orchestra/backups/\_runners/VmsXapi.mjs``](http
- `checkBaseVdis(baseUuidToSrcVdi, baseVm)`
- `prepare({ isFull })`
- `transfer({ timestamp, deltaExport, sizeContainers })`
- `updateUuidAndChain({ isVhdDifferencing, vdis })`
- `cleanup()`
- `healthCheck()` // is not executed if no health check sr or tag doesn't match
- **Full**

View File

@@ -8,7 +8,7 @@
"type": "git",
"url": "https://github.com/vatesfr/xen-orchestra.git"
},
"version": "0.73.2",
"version": "0.73.10",
"engines": {
"node": ">=14.18"
},
@@ -25,19 +25,19 @@
"@vates/compose": "^2.1.0",
"@vates/decorate-with": "^2.1.0",
"@vates/disposable": "^0.1.6",
"@vates/fuse-vhd": "^2.1.2",
"@vates/generator-toolbox": "^1.1.2",
"@vates/nbd-client": "^3.4.0",
"@vates/fuse-vhd": "^2.1.3",
"@vates/generator-toolbox": "^1.1.3",
"@vates/nbd-client": "^3.5.0",
"@vates/parse-duration": "^0.1.1",
"@vates/task": "^0.7.0",
"@vates/types": "^1.26.0",
"@vates/types": "^1.30.1",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/disk-transform": "^1.3.0",
"@xen-orchestra/fs": "^4.9.0",
"@xen-orchestra/disk-transform": "^1.3.3",
"@xen-orchestra/fs": "^4.9.3",
"@xen-orchestra/log": "^0.7.2",
"@xen-orchestra/qcow2": "^1.3.0",
"@xen-orchestra/qcow2": "^1.3.1",
"@xen-orchestra/template": "^0.1.1",
"@xen-orchestra/backup-archive": "^1.0.1",
"@xen-orchestra/backup-archive": "^1.1.1",
"app-conf": "^3.0.0",
"compare-versions": "^6.0.0",
"d3-time-format": "^4.1.0",
@@ -55,8 +55,8 @@
"tar": "^7.5.3",
"uuid": "^9.0.0",
"value-matcher": "^0.2.0",
"vhd-lib": "^4.16.0",
"xen-api": "^4.7.7",
"vhd-lib": "^4.16.1",
"xen-api": "^4.8.0",
"yazl": "^2.5.1"
},
"devDependencies": {
@@ -67,7 +67,7 @@
"typescript": "^5.9.3"
},
"peerDependencies": {
"@xen-orchestra/xapi": "^8.8.0"
"@xen-orchestra/xapi": "^8.10.2"
},
"license": "AGPL-3.0-or-later",
"author": {

View File

@@ -4,22 +4,10 @@ import { fork } from 'child_process'
const { warn } = createLogger('xo:backups:backupWorker')
const PATH = new URL('_backupWorker.mjs', import.meta.url).pathname
const DEFAULT_INSPECTOR_PORT = 9229
export function runBackupWorker(params, onLog) {
return new Promise((resolve, reject) => {
// run Node inspector on port+1 if --inspect or --inspect-brk
const inspectArg = process.execArgv.find(arg => arg.startsWith('--inspect') || arg.startsWith('--inspect-brk'))
const execArgv = inspectArg
? [
inspectArg.replace(/^(--inspect(-brk)?)(=([^:]+:)?(\d+))?$/, (_, prefix, brk, _fullMatch, host, port) => {
const basePort = port ? parseInt(port) : DEFAULT_INSPECTOR_PORT
return `${prefix}=${host || ''}${basePort + 1}`
}),
]
: []
const worker = fork(PATH, [], { execArgv })
const worker = fork(PATH, [])
worker.on('exit', (code, signal) => reject(new Error(`worker exited with code ${code} and signal ${signal}`)))
worker.on('error', reject)

View File

@@ -18,7 +18,7 @@
"preferGlobal": true,
"dependencies": {
"golike-defer": "^0.5.1",
"xen-api": "^4.7.7"
"xen-api": "^4.8.0"
},
"scripts": {
"postversion": "npm publish"

View File

@@ -3,8 +3,8 @@ $ xo-disk-cli --help
Usage: xo-disk-cli <command> <handler-url> <path> [options]
Commands:
info Show disk info (virtual size, uid, parent uid, block size)
list List all disks at a path and display their properties in a table
info Show disk info (virtual size, uid, parent uid, block size); --chain shows the full parent chain, --size adds the size on disk
list List all disks at a path and display their properties in a table; --size adds the size on disk
transform Convert a disk to another format and write to stdout (raw | vhd | qcow2)
```
@@ -15,7 +15,7 @@ Commands:
Display metadata for a single disk file.
```
xo-disk-cli info <handler-url> <disk-path>
xo-disk-cli info <handler-url> <disk-path> [--chain] [--size]
```
Output includes: UID, parent UID (for differencing disks), virtual size, and block size.
@@ -29,16 +29,66 @@ Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd
Block size: 2.00 MiB (2097152 bytes)
```
By default the disk is opened without reading its block allocation table, which
is cheap but does not allow computing the size on disk. Pass `--size` to read
the block allocation table and print the size on disk as an extra line:
```
$ xo-disk-cli info file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd --size
Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd
UID: <uuid>
Parent UID: <parent-uuid>
Virtual size: 8.00 GiB (8589934592 bytes)
Block size: 2.00 MiB (2097152 bytes)
Size on disk: 128.00 MiB (134217728 bytes)
```
With `--chain`, the disk's full parent chain is opened and each disk is printed
from the root (base) disk down to the given leaf disk. `--chain` and `--size`
can be combined:
```
$ xo-disk-cli info file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd --chain
Disk chain: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd (2 disks, root → leaf)
Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/base.vhd
UID: <base-uuid>
Parent UID: (none)
Virtual size: 8.00 GiB (8589934592 bytes)
Block size: 2.00 MiB (2097152 bytes)
Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd
UID: <uuid>
Parent UID: <base-uuid>
Virtual size: 8.00 GiB (8589934592 bytes)
Block size: 2.00 MiB (2097152 bytes)
```
### `list`
List all disk files found at a directory path and display their properties in a table. Disks are sorted so that each child appears directly after its parent. When a disk's parent is the row immediately above, the parent UID column shows `↑` for quick chain health assessment.
```
xo-disk-cli list <handler-url> <dir-path>
xo-disk-cli list <handler-url> <dir-path> [--size]
```
By default disks are opened without reading their block allocation table (cheap),
so the "Size on disk" column is omitted:
```
$ xo-disk-cli list file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/
┌──────────────┬──────────────────────────────────────┬──────────────┬─────────────┬──────────────────────────────────────┐
│ File │ UID │ Virtual size │ Differencing │ Parent UID │
├──────────────┼──────────────────────────────────────┼──────────────┼─────────────┼──────────────────────────────────────┤
│ base.vhd │ xxxxxxxx-... │ 8.00 GiB │ no │ (none) │
│ snapshot.vhd │ yyyyyyyy-... │ 8.00 GiB │ yes │ ↑ │
└──────────────┴──────────────────────────────────────┴──────────────┴─────────────┴──────────────────────────────────────┘
```
Pass `--size` to read each disk's block allocation table and add the "Size on disk" column:
```
$ xo-disk-cli list file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/ --size
┌──────────────┬──────────────────────────────────────┬─────────────┬──────────────┬─────────────┬──────────────────────────────────────┐
│ File │ UID │ Size on disk│ Virtual size │ Differencing │ Parent UID │
├──────────────┼──────────────────────────────────────┼─────────────┼──────────────┼─────────────┼──────────────────────────────────────┤

View File

@@ -21,8 +21,8 @@ $ xo-disk-cli --help
Usage: xo-disk-cli <command> <handler-url> <path> [options]
Commands:
info Show disk info (virtual size, uid, parent uid, block size)
list List all disks at a path and display their properties in a table
info Show disk info (virtual size, uid, parent uid, block size); --chain shows the full parent chain, --size adds the size on disk
list List all disks at a path and display their properties in a table; --size adds the size on disk
transform Convert a disk to another format and write to stdout (raw | vhd | qcow2)
```
@@ -33,7 +33,7 @@ Commands:
Display metadata for a single disk file.
```
xo-disk-cli info <handler-url> <disk-path>
xo-disk-cli info <handler-url> <disk-path> [--chain] [--size]
```
Output includes: UID, parent UID (for differencing disks), virtual size, and block size.
@@ -47,16 +47,66 @@ Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd
Block size: 2.00 MiB (2097152 bytes)
```
By default the disk is opened without reading its block allocation table, which
is cheap but does not allow computing the size on disk. Pass `--size` to read
the block allocation table and print the size on disk as an extra line:
```
$ xo-disk-cli info file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd --size
Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd
UID: <uuid>
Parent UID: <parent-uuid>
Virtual size: 8.00 GiB (8589934592 bytes)
Block size: 2.00 MiB (2097152 bytes)
Size on disk: 128.00 MiB (134217728 bytes)
```
With `--chain`, the disk's full parent chain is opened and each disk is printed
from the root (base) disk down to the given leaf disk. `--chain` and `--size`
can be combined:
```
$ xo-disk-cli info file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd --chain
Disk chain: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd (2 disks, root → leaf)
Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/base.vhd
UID: <base-uuid>
Parent UID: (none)
Virtual size: 8.00 GiB (8589934592 bytes)
Block size: 2.00 MiB (2097152 bytes)
Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd
UID: <uuid>
Parent UID: <base-uuid>
Virtual size: 8.00 GiB (8589934592 bytes)
Block size: 2.00 MiB (2097152 bytes)
```
### `list`
List all disk files found at a directory path and display their properties in a table. Disks are sorted so that each child appears directly after its parent. When a disk's parent is the row immediately above, the parent UID column shows `↑` for quick chain health assessment.
```
xo-disk-cli list <handler-url> <dir-path>
xo-disk-cli list <handler-url> <dir-path> [--size]
```
By default disks are opened without reading their block allocation table (cheap),
so the "Size on disk" column is omitted:
```
$ xo-disk-cli list file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/
┌──────────────┬──────────────────────────────────────┬──────────────┬─────────────┬──────────────────────────────────────┐
│ File │ UID │ Virtual size │ Differencing │ Parent UID │
├──────────────┼──────────────────────────────────────┼──────────────┼─────────────┼──────────────────────────────────────┤
│ base.vhd │ xxxxxxxx-... │ 8.00 GiB │ no │ (none) │
│ snapshot.vhd │ yyyyyyyy-... │ 8.00 GiB │ yes │ ↑ │
└──────────────┴──────────────────────────────────────┴──────────────┴─────────────┴──────────────────────────────────────┘
```
Pass `--size` to read each disk's block allocation table and add the "Size on disk" column:
```
$ xo-disk-cli list file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/ --size
┌──────────────┬──────────────────────────────────────┬─────────────┬──────────────┬─────────────┬──────────────────────────────────────┐
│ File │ UID │ Size on disk│ Virtual size │ Differencing │ Parent UID │
├──────────────┼──────────────────────────────────────┼─────────────┼──────────────┼─────────────┼──────────────────────────────────────┤

View File

@@ -1,7 +1,7 @@
{
"private": false,
"name": "@xen-orchestra/disk-cli",
"version": "2.1.1",
"version": "2.2.0",
"description": "CLI tool to inspect and manage disks in backup repositories",
"type": "module",
"bin": {
@@ -20,13 +20,13 @@
},
"dependencies": {
"@vates/async-each": "^1.0.3",
"@xen-orchestra/backups": "^0.73.2",
"@xen-orchestra/backup-archive": "^1.0.1",
"@xen-orchestra/disk-transform": "^1.3.0",
"@xen-orchestra/fs": "^4.9.0",
"@xen-orchestra/qcow2": "^1.3.0",
"@xen-orchestra/backups": "^0.73.10",
"@xen-orchestra/backup-archive": "^1.1.1",
"@xen-orchestra/disk-transform": "^1.3.3",
"@xen-orchestra/fs": "^4.9.3",
"@xen-orchestra/qcow2": "^1.3.1",
"promise-toolbox": "^0.21.0",
"vhd-lib": "^4.16.0"
"vhd-lib": "^4.16.1"
},
"devDependencies": {
"@types/node": "^22.3.0",

View File

@@ -1,22 +1,84 @@
import Disposable from 'promise-toolbox/Disposable'
import { getSyncedHandler } from '@xen-orchestra/fs'
import { openDisposableDisk, RemoteDisk } from '@xen-orchestra/backup-archive/disks'
import { getSyncedHandler, type RemoteHandler } from '@xen-orchestra/fs'
import { openDisposableDisk, openDiskChain, RemoteDisk, DiskDisposable } from '@xen-orchestra/backup-archive/disks'
import { formatBytes } from '../utils.mjs'
export async function infoCommand(handlerUrl: string, diskPath: string, _extraArgs: string[]): Promise<void> {
await Disposable.use(getSyncedHandler({ url: handlerUrl }), async handler => {
await Disposable.use(openDisposableDisk({ handler, path: diskPath }), async (disk: RemoteDisk) => {
const isDifferencing = disk.isDifferencing()
const virtualSize = disk.getVirtualSize()
const blockSize = disk.getBlockSize()
const uid = disk.getUuid()
const parentUid = isDifferencing ? disk.getParentUuid() : null
type DiskInfoLine = {
path: string
uid: string
parentUid: string | null
virtualSize: number
blockSize: number
// Only computed with --size; requires reading the block allocation table.
sizeOnDisk?: number
}
console.log(`Disk info: ${diskPath}`)
console.log(` UID: ${uid}`)
console.log(` Parent UID: ${parentUid ?? '(none)'}`)
console.log(` Virtual size: ${formatBytes(virtualSize)} (${virtualSize} bytes)`)
console.log(` Block size: ${formatBytes(blockSize)} (${blockSize} bytes)`)
})
function readDiskInfo(disk: RemoteDisk, path: string, showSize: boolean): DiskInfoLine {
const isDifferencing = disk.isDifferencing()
return {
path,
uid: disk.getUuid(),
parentUid: isDifferencing ? disk.getParentUuid() : null,
virtualSize: disk.getVirtualSize(),
blockSize: disk.getBlockSize(),
sizeOnDisk: showSize ? disk.getSizeOnDisk() : undefined,
}
}
function printDiskInfo(info: DiskInfoLine): void {
console.log(`Disk info: ${info.path}`)
console.log(` UID: ${info.uid}`)
console.log(` Parent UID: ${info.parentUid ?? '(none)'}`)
console.log(` Virtual size: ${formatBytes(info.virtualSize)} (${info.virtualSize} bytes)`)
console.log(` Block size: ${formatBytes(info.blockSize)} (${info.blockSize} bytes)`)
if (info.sizeOnDisk !== undefined) {
console.log(` Size on disk: ${formatBytes(info.sizeOnDisk)} (${info.sizeOnDisk} bytes)`)
}
}
// Opens the full chain (root → leaf) as a disposable, reusing the same chain
// walker as the `transform` command.
async function openDisposableChain(
handler: RemoteHandler,
path: string,
ignoreBlockIndexes: boolean
): Promise<DiskDisposable> {
const chain = await openDiskChain({ handler, path, ignoreBlockIndexes })
return { value: chain, dispose: () => chain.close() }
}
export async function infoCommand(handlerUrl: string, diskPath: string, extraArgs: string[]): Promise<void> {
const showChain = extraArgs.includes('--chain')
const showSize = extraArgs.includes('--size')
// Reading the block allocation table is costly, so only do it when the size on
// disk is requested; every other field comes from the footer/header.
const ignoreBlockIndexes = !showSize
await Disposable.use(getSyncedHandler({ url: handlerUrl }), async handler => {
if (!showChain) {
const info = await Disposable.use(
openDisposableDisk({ handler, path: diskPath, ignoreBlockIndexes }),
(disk: RemoteDisk) => readDiskInfo(disk, diskPath, showSize)
)
printDiskInfo(info)
return
}
// Resolve the full lineage (root → leaf), then print each disk's info.
const paths = await Disposable.use(openDisposableChain(handler, diskPath, ignoreBlockIndexes), chain =>
chain.getPaths()
)
console.log(`Disk chain: ${diskPath} (${paths.length} disk${paths.length === 1 ? '' : 's'}, root → leaf)\n`)
for (const [index, path] of paths.entries()) {
const info = await Disposable.use(openDisposableDisk({ handler, path, ignoreBlockIndexes }), (disk: RemoteDisk) =>
readDiskInfo(disk, path, showSize)
)
printDiskInfo(info)
if (index < paths.length - 1) {
console.log()
}
}
})
}

View File

@@ -3,13 +3,18 @@ import { getSyncedHandler } from '@xen-orchestra/fs'
import { isDisk, openDisposableDisk, RemoteDisk } from '@xen-orchestra/backup-archive/disks'
import { formatBytes, renderTable } from '../utils.mjs'
import { asyncEach } from '@vates/async-each'
const HEADERS = ['File', 'UID', 'Size on disk', 'Virtual size', 'Differencing', 'Parent UID']
// The "Size on disk" column is only shown with --size: computing it requires
// reading the block allocation table of every disk, which is costly.
function buildHeaders(showSize: boolean): string[] {
return ['File', 'UID', ...(showSize ? ['Size on disk'] : []), 'Virtual size', 'Differencing', 'Parent UID']
}
export type OkDiskInfo = {
ok: true
filename: string
uid: string
sizeOnDisk: string
sizeOnDisk?: string
virtualSize: string
differencing: boolean
parentUid: string | null
@@ -54,9 +59,10 @@ export function sortDisks(disks: DiskInfo[]): DiskInfo[] {
return [...sorted, ...errDisks]
}
export function toTableRow(disk: DiskInfo, prevDisk: DiskInfo | undefined): string[] {
export function toTableRow(disk: DiskInfo, prevDisk: DiskInfo | undefined, showSize: boolean): string[] {
if (!disk.ok) {
return [disk.filename, `(error: ${disk.error})`, '-', '-', '-', '-']
// filename + error message, then a placeholder for every remaining column.
return [disk.filename, `(error: ${disk.error})`, ...Array(showSize ? 4 : 3).fill('-')]
}
const parentIsAbove = disk.differencing && prevDisk?.ok && prevDisk.uid === disk.parentUid
@@ -64,17 +70,19 @@ export function toTableRow(disk: DiskInfo, prevDisk: DiskInfo | undefined): stri
return [
disk.filename,
disk.uid,
disk.sizeOnDisk,
...(showSize ? [disk.sizeOnDisk ?? '-'] : []),
disk.virtualSize,
disk.differencing ? 'yes' : 'no',
disk.differencing ? (parentIsAbove ? '↑' : (disk.parentUid ?? '')) : '(none)',
]
}
export async function listCommand(handlerUrl: string, dirPath: string, _extraArgs: string[]): Promise<void> {
export async function listCommand(handlerUrl: string, dirPath: string, extraArgs: string[]): Promise<void> {
const showSize = extraArgs.includes('--size')
await Disposable.use(getSyncedHandler({ url: handlerUrl }), async handler => {
const entries = await handler.list(dirPath, { prependDir: true })
const diskPaths = entries.filter(entry => isDisk(handler, entry))
const diskPaths = entries.filter(entry => isDisk(entry))
if (diskPaths.length === 0) {
console.log(`No disks found at ${dirPath}`)
@@ -88,14 +96,14 @@ export async function listCommand(handlerUrl: string, dirPath: string, _extraArg
const filename = diskPath.slice(diskPath.lastIndexOf('/') + 1)
try {
const disk = await Disposable.use(
openDisposableDisk({ handler, path: diskPath }),
openDisposableDisk({ handler, path: diskPath, ignoreBlockIndexes: !showSize }),
async (disk: RemoteDisk) => {
const differencing = disk.isDifferencing()
return {
ok: true as const,
filename,
uid: disk.getUuid(),
sizeOnDisk: formatBytes(disk.getSizeOnDisk()),
sizeOnDisk: showSize ? formatBytes(disk.getSizeOnDisk()) : undefined,
virtualSize: formatBytes(disk.getVirtualSize()),
differencing,
parentUid: differencing ? disk.getParentUuid() : null,
@@ -111,7 +119,7 @@ export async function listCommand(handlerUrl: string, dirPath: string, _extraArg
)
const sorted = sortDisks(disks)
const rows = sorted.map((disk, i) => toTableRow(disk, sorted[i - 1]))
console.log(renderTable(HEADERS, rows))
const rows = sorted.map((disk, i) => toTableRow(disk, sorted[i - 1], showSize))
console.log(renderTable(buildHeaders(showSize), rows))
})
}

View File

@@ -42,6 +42,7 @@ declare module '@xen-orchestra/backup-archive/disks/openDiskChain.mjs' {
handler: RemoteHandler
path: string
until?: string
ignoreBlockIndexes?: boolean
}): Promise<RandomAccessDisk>
}
@@ -52,8 +53,10 @@ declare module '@xen-orchestra/backup-archive/disks' {
export interface RemoteDisk extends RandomAccessDisk {
getSizeOnDisk(): number
getPath(): string
getPaths(): string[]
getUuid(): string
getParentUuid(): string
getParentPath(): string
getMaxBlockCount(): number
}
@@ -62,13 +65,20 @@ declare module '@xen-orchestra/backup-archive/disks' {
dispose: () => Promise<void>
}
export function isDisk(handler: RemoteHandler, path: string): boolean
export function isDisk(path: string): boolean
export function openDisposableDisk(params: { handler: RemoteHandler; path: string }): Promise<DiskDisposable>
export function openDisposableDisk(params: {
handler: RemoteHandler
path: string
force?: boolean
ignoreBlockIndexes?: boolean
}): Promise<DiskDisposable>
// Returns a RemoteVhdDiskChain, which is a RemoteDisk exposing getPaths() over the whole lineage.
export function openDiskChain(params: {
handler: RemoteHandler
path: string
until?: string
}): Promise<RandomAccessDisk>
ignoreBlockIndexes?: boolean
}): Promise<RemoteDisk>
}

View File

@@ -14,8 +14,8 @@ const COMMANDS: Record<string, CommandFn> = {
function showHelp(exitCode: number): never {
const commandDescriptions: Record<string, string> = {
info: 'Show disk info (virtual size, uid, parent uid, block size)',
list: 'List all disks at a path and display their properties in a table',
info: 'Show disk info (virtual size, uid, parent uid, block size); --chain shows the full parent chain, --size adds the size on disk',
list: 'List all disks at a path and display their properties in a table; --size adds the size on disk',
transform: 'Convert a disk to another format and write to stdout (raw | vhd | qcow2)',
}
@@ -30,7 +30,9 @@ ${commandList}
Examples:
xo-disk-cli info file:///mnt/backups /xo-vm-backups/uuid/disk.alias.vhd
xo-disk-cli info file:///mnt/backups /xo-vm-backups/uuid/disk.alias.vhd --chain --size
xo-disk-cli list file:///mnt/backups /xo-vm-backups/uuid/
xo-disk-cli list file:///mnt/backups /xo-vm-backups/uuid/ --size
xo-disk-cli transform file:///mnt/backups /xo-vm-backups/uuid/disk.vhd raw > disk.img
xo-disk-cli transform file:///mnt/backups /xo-vm-backups/uuid/disk.vhd qcow2 > disk.qcow2
`)

View File

@@ -4,25 +4,30 @@
* Focus:
* - Handler and disk are always disposed, even when disk methods throw.
* - Errors from broken disks are propagated (not swallowed).
* - By default disks are opened with ignoreBlockIndexes: true (cheap); --size
* opens with ignoreBlockIndexes: false and prints the size on disk.
* - --chain opens the whole lineage and prints each disk.
*
* Design: mock.module() is called once at the top level. The mock for openDisk
* reads `currentDisk` at call-time via closure, so individual tests can inject
* failure scenarios simply by reassigning the variable before calling the command.
* Design: mock.module() is called once at the top level. The mocks read mutable
* closure variables at call-time so individual tests can inject scenarios simply
* by reassigning a variable before calling the command.
*/
import { describe, test, mock, beforeEach } from 'node:test'
import assert from 'node:assert/strict'
// ---------------------------------------------------------------------------
// Spies — created once and reused; reset in beforeEach.
// Spies & mutable state — created once and reused; reset in beforeEach.
// ---------------------------------------------------------------------------
const handlerDispose = mock.fn(() => Promise.resolve())
const diskDispose = mock.fn(() => Promise.resolve())
const chainClose = mock.fn(() => Promise.resolve())
const baseDisk = {
isDifferencing: () => false,
getVirtualSize: () => 1024 * 1024 * 1024,
getBlockSize: () => 2 * 1024 * 1024,
getSizeOnDisk: () => 128 * 1024 * 1024,
getUuid: () => 'aaaa-bbbb',
getParentUuid: () => 'pppp-qqqq',
}
@@ -30,9 +35,20 @@ const baseDisk = {
// Reassigned per-test to inject different disk behaviours.
let currentDisk: typeof baseDisk = { ...baseDisk }
// Paths returned by the chain (root → leaf); reassigned per-test.
let chainPaths: string[] = ['/vdis/root.vhd', '/vdis/leaf.vhd']
// Records the options each openDisposableDisk call received.
const openDiskParams: Array<{ path: string; ignoreBlockIndexes?: boolean }> = []
const openChainParams: Array<{ path: string; ignoreBlockIndexes?: boolean }> = []
const openDiskChainSpy = mock.fn((params: { path: string; ignoreBlockIndexes?: boolean }) => {
openChainParams.push(params)
return Promise.resolve({ getPaths: () => chainPaths, close: chainClose })
})
// ---------------------------------------------------------------------------
// Module mocks — registered once. openDisk reads `currentDisk` at call-time
// so tests only need to reassign the variable, not re-mock the module.
// Module mocks — registered once. Implementations read mutable state at call-time.
// ---------------------------------------------------------------------------
mock.module('@xen-orchestra/fs', {
@@ -43,14 +59,31 @@ mock.module('@xen-orchestra/fs', {
mock.module('@xen-orchestra/backup-archive/disks', {
namedExports: {
openDisposableDisk: () => Promise.resolve({ value: currentDisk, dispose: diskDispose }),
openDisposableDisk: (params: { path: string; ignoreBlockIndexes?: boolean }) => {
openDiskParams.push(params)
return Promise.resolve({ value: currentDisk, dispose: diskDispose })
},
openDiskChain: openDiskChainSpy,
},
})
// Import the module under test AFTER mocks are registered so it receives the
// mocked implementations. The same infoCommand reference is reused for all tests.
// mocked implementations.
const { infoCommand } = await import('../../commands/info.mjs')
// Capture console.log output without polluting test output.
async function captureLog(fn: () => Promise<void>): Promise<string> {
const lines: string[] = []
const orig = console.log
console.log = (...args: unknown[]) => lines.push(args.map(String).join(' '))
try {
await fn()
} finally {
console.log = orig
}
return lines.join('\n')
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
@@ -59,11 +92,16 @@ describe('info command', () => {
beforeEach(() => {
handlerDispose.mock.resetCalls()
diskDispose.mock.resetCalls()
chainClose.mock.resetCalls()
openDiskChainSpy.mock.resetCalls()
openDiskParams.length = 0
openChainParams.length = 0
currentDisk = { ...baseDisk }
chainPaths = ['/vdis/root.vhd', '/vdis/leaf.vhd']
})
test('disposes handler and disk on success', async () => {
await infoCommand('file:///test', '/vdis/disk.vhd', [])
await captureLog(() => infoCommand('file:///test', '/vdis/disk.vhd', []))
assert.strictEqual(handlerDispose.mock.callCount(), 1, 'handler must be disposed once')
assert.strictEqual(diskDispose.mock.callCount(), 1, 'disk must be disposed once')
})
@@ -89,4 +127,63 @@ describe('info command', () => {
}
await assert.rejects(() => infoCommand('file:///test', '/vdis/disk.vhd', []), /corrupt header/)
})
test('by default opens the disk with ignoreBlockIndexes: true and hides size on disk', async () => {
const output = await captureLog(() => infoCommand('file:///test', '/vdis/disk.vhd', []))
assert.strictEqual(openDiskParams.length, 1)
assert.strictEqual(openDiskParams[0].ignoreBlockIndexes, true, 'block indexes must be skipped by default')
assert.ok(!output.includes('Size on disk'), 'size on disk must not be printed without --size')
})
test('--size opens the disk with ignoreBlockIndexes: false and prints size on disk', async () => {
const output = await captureLog(() => infoCommand('file:///test', '/vdis/disk.vhd', ['--size']))
assert.strictEqual(openDiskParams[0].ignoreBlockIndexes, false, 'block indexes must be read with --size')
assert.ok(output.includes('Size on disk'), 'size on disk must be printed with --size')
})
test('without --chain, openDiskChain is never called', async () => {
await captureLog(() => infoCommand('file:///test', '/vdis/disk.vhd', []))
assert.strictEqual(openDiskChainSpy.mock.callCount(), 0, 'openDiskChain must not be called without --chain')
})
test('--chain: opens the chain, opens each disk in it, and disposes everything', async () => {
await captureLog(() => infoCommand('file:///test', '/vdis/leaf.vhd', ['--chain']))
assert.strictEqual(openDiskChainSpy.mock.callCount(), 1, 'openDiskChain must be called once')
assert.strictEqual(chainClose.mock.callCount(), 1, 'chain must be closed after enumerating paths')
// One openDisposableDisk (and thus one dispose) per disk in the chain.
assert.strictEqual(diskDispose.mock.callCount(), chainPaths.length, 'each chain disk must be opened and disposed')
assert.strictEqual(handlerDispose.mock.callCount(), 1, 'handler must be disposed once')
})
test('--chain defaults to ignoreBlockIndexes: true for both the chain and each disk', async () => {
await captureLog(() => infoCommand('file:///test', '/vdis/leaf.vhd', ['--chain']))
assert.strictEqual(openChainParams[0].ignoreBlockIndexes, true, 'chain walk must skip block indexes by default')
assert.ok(
openDiskParams.every(p => p.ignoreBlockIndexes === true),
'each chain disk must be opened with ignoreBlockIndexes: true'
)
})
test('--chain --size reads block indexes and prints size on disk for each disk', async () => {
const output = await captureLog(() => infoCommand('file:///test', '/vdis/leaf.vhd', ['--chain', '--size']))
assert.strictEqual(openChainParams[0].ignoreBlockIndexes, false, 'chain walk must read block indexes with --size')
assert.ok(
openDiskParams.every(p => p.ignoreBlockIndexes === false),
'each chain disk must be opened with ignoreBlockIndexes: false'
)
const sizeLines = output.split('\n').filter(l => l.includes('Size on disk'))
assert.strictEqual(sizeLines.length, chainPaths.length, 'size on disk must be printed for each disk')
})
test('--chain: chain is closed even when opening a disk in it throws', async () => {
currentDisk = {
...baseDisk,
getUuid: () => {
throw new Error('corrupt header')
},
}
await assert.rejects(() => infoCommand('file:///test', '/vdis/leaf.vhd', ['--chain']), /corrupt header/)
assert.strictEqual(chainClose.mock.callCount(), 1, 'chain must be closed even on error')
assert.strictEqual(handlerDispose.mock.callCount(), 1, 'handler must be disposed even on error')
})
})

View File

@@ -27,6 +27,9 @@ const baseDisk = {
getParentUuid: () => null as unknown as string,
}
// Records the options each openDisposableDisk call received.
const openDiskParams: Array<{ path: string; ignoreBlockIndexes?: boolean }> = []
// Reassigned per-test to control what handler.list() returns (or throws).
let currentListImpl: () => Promise<string[]> = async () => []
@@ -51,9 +54,10 @@ mock.module('@xen-orchestra/fs', {
mock.module('@xen-orchestra/backup-archive/disks', {
namedExports: {
isDisk: (_h: unknown, p: string) => p.endsWith('.vhd'),
openDisposableDisk: ({ path }: { path: string }) => {
if (brokenPaths.has(path)) throw new Error('disk header corrupted')
isDisk: (p: string) => p.endsWith('.vhd'),
openDisposableDisk: (params: { path: string; ignoreBlockIndexes?: boolean }) => {
openDiskParams.push(params)
if (brokenPaths.has(params.path)) throw new Error('disk header corrupted')
return Promise.resolve({ value: { ...baseDisk }, dispose: diskDispose })
},
},
@@ -86,6 +90,7 @@ describe('list command', () => {
handlerDispose.mock.resetCalls()
diskDispose.mock.resetCalls()
brokenPaths.clear()
openDiskParams.length = 0
currentListImpl = async () => ['/dir/disk.vhd']
})
@@ -122,4 +127,22 @@ describe('list command', () => {
const output = await captureLog(() => listCommand('file:///test', '/dir', []))
assert.ok(output.includes('No disks found'))
})
test('by default opens disks with ignoreBlockIndexes: true and hides the size column', async () => {
const output = await captureLog(() => listCommand('file:///test', '/dir', []))
assert.ok(
openDiskParams.every(p => p.ignoreBlockIndexes === true),
'disks must be opened with ignoreBlockIndexes: true by default'
)
assert.ok(!output.includes('Size on disk'), 'the size column must be hidden without --size')
})
test('--size opens disks with ignoreBlockIndexes: false and shows the size column', async () => {
const output = await captureLog(() => listCommand('file:///test', '/dir', ['--size']))
assert.ok(
openDiskParams.every(p => p.ignoreBlockIndexes === false),
'disks must be opened with ignoreBlockIndexes: false with --size'
)
assert.ok(output.includes('Size on disk'), 'the size column must appear with --size')
})
})

View File

@@ -2,6 +2,10 @@ import { describe, test } from 'node:test'
import assert from 'node:assert/strict'
import { sortDisks, toTableRow, type DiskInfo } from '../commands/list.mjs'
// These tests assert on the full-width row (with the "Size on disk" column), so
// they always exercise toTableRow with showSize = true.
const toTableRowWithSize = (disk: DiskInfo, prevDisk: DiskInfo | undefined) => toTableRow(disk, prevDisk, true)
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
@@ -106,43 +110,43 @@ describe('sortDisks', () => {
describe('toTableRow', () => {
test('base disk shows "(none)" in parent uid column', () => {
const row = toTableRow(ok('a'), undefined)
const row = toTableRowWithSize(ok('a'), undefined)
assert.strictEqual(row[5], '(none)')
})
test('differencing disk with no previous disk shows parentUid', () => {
const row = toTableRow(ok('b', 'a'), undefined)
const row = toTableRowWithSize(ok('b', 'a'), undefined)
assert.strictEqual(row[5], 'a')
})
test('differencing disk whose parent is not the previous row shows parentUid', () => {
const row = toTableRow(ok('b', 'a'), ok('x'))
const row = toTableRowWithSize(ok('b', 'a'), ok('x'))
assert.strictEqual(row[5], 'a')
})
test('differencing disk whose parent is immediately above shows ↑', () => {
const row = toTableRow(ok('b', 'a'), ok('a'))
const row = toTableRowWithSize(ok('b', 'a'), ok('a'))
assert.strictEqual(row[5], '↑')
})
test('error disk shows error message in second column', () => {
const row = toTableRow(err('bad.vhd'), undefined)
const row = toTableRowWithSize(err('bad.vhd'), undefined)
assert.ok(row[1].includes('error'))
assert.ok(row[1].includes('disk header corrupted'))
})
test('differencing column shows "yes" for differencing disk', () => {
assert.strictEqual(toTableRow(ok('b', 'a'), undefined)[4], 'yes')
assert.strictEqual(toTableRowWithSize(ok('b', 'a'), undefined)[4], 'yes')
})
test('differencing column shows "no" for base disk', () => {
assert.strictEqual(toTableRow(ok('a'), undefined)[4], 'no')
assert.strictEqual(toTableRowWithSize(ok('a'), undefined)[4], 'no')
})
test('parent with multiple children: at least one child shows parent UID instead of ↑', () => {
const sorted = sortDisks([ok('c2', 'a'), ok('a'), ok('c1', 'a')])
const rows = sorted.map((disk, i) => toTableRow(disk, sorted[i - 1]))
const rows = sorted.map((disk, i) => toTableRowWithSize(disk, sorted[i - 1]))
// Find rows for the two children
const childRows = rows.filter((_row, i) => {

View File

@@ -1,6 +1,6 @@
{
"name": "@xen-orchestra/disk-transform",
"version": "1.3.0",
"version": "1.3.3",
"main": "./dist/index.mjs",
"license": "MIT",
"private": false,
@@ -10,7 +10,7 @@
"rimraf": "^6.0.1",
"@types/node": "^22.3.0",
"typescript": "~5.6",
"typescript-eslint": "^8.23.0"
"typescript-eslint": "^8.61.0"
},
"scripts": {
"build": "tsc",
@@ -34,6 +34,6 @@
"node": ">=20.17"
},
"dependencies": {
"@vates/generator-toolbox": "^1.1.2"
"@vates/generator-toolbox": "^1.1.3"
}
}

View File

@@ -31,11 +31,15 @@ export class DiskLargerBlock extends RandomAccessDisk {
throw new Error('Method not implemented.')
}
async readBlock(index: number): Promise<DiskBlock> {
if (!this.hasBlock(index)) {
throw new Error(`Block ${index} not present in this disk`)
}
// @todo handle partial block at the end
const source = this.source
const destinationBlock = Buffer.alloc(this.getBlockSize(), 0)
const destinationBlockData = Buffer.alloc(this.getBlockSize(), 0)
const blockRatio = this.#blockSize / source.getBlockSize()
for (let i = index * blockRatio; i < (index + 1) * blockRatio; i++) {
const firstSourceBlockIndex = index * blockRatio
for (let i = firstSourceBlockIndex; i < firstSourceBlockIndex + blockRatio; i++) {
let data: Buffer | undefined
if (source.hasBlock(i)) {
data = (await source.readBlock(i)).data
@@ -51,12 +55,12 @@ export class DiskLargerBlock extends RandomAccessDisk {
}
}
if (data !== undefined) {
data.copy(destinationBlock, i * source.getBlockSize())
data.copy(destinationBlockData, (i - firstSourceBlockIndex) * source.getBlockSize())
}
}
return {
index,
data: destinationBlock,
data: destinationBlockData,
}
}
@@ -66,19 +70,26 @@ export class DiskLargerBlock extends RandomAccessDisk {
getBlockIndexes(): Array<number> {
const maxBlock = Math.ceil(this.getVirtualSize() / this.getBlockSize())
const blockRatio = this.getBlockSize() / this.source.getBlockSize()
const indexes = []
for (let i = 0; i < maxBlock; i++) {
for (let j = 0; j < blockRatio; j++) {
if (this.source.hasBlock(i * blockRatio + j)) {
indexes.push(i)
break
}
if (this.hasBlock(i)) {
indexes.push(i)
}
}
return indexes
}
getBlockIndexesCount(): number {
let count = 0
const maxBlock = Math.ceil(this.getVirtualSize() / this.getBlockSize())
for (let i = 0; i < maxBlock; i++) {
if (this.hasBlock(i)) {
count++
}
}
return count
}
hasBlock(index: number): boolean {
const source = this.source
let maxBlockSource = Math.ceil(source.getVirtualSize() / source.getBlockSize())

View File

@@ -1,4 +1,4 @@
import assert from 'node:assert'
import assert, { strictEqual } from 'node:assert'
import { test } from 'node:test'
import { DiskLargerBlock } from './DiskLargerBlock.mjs'
import { RandomAccessDisk, DiskBlock, Disk } from './Disk.mjs'
@@ -45,7 +45,7 @@ class MockDisk extends RandomAccessDisk {
this.closed = true
}
isDifferencing(): boolean {
return this.isDiff
return this.parentDisk !== null || this.isDiff
}
instantiateParent(): RandomAccessDisk {
if (!this.parentDisk) throw new Error('No parent disk available')
@@ -118,6 +118,29 @@ test('readBlock with simple block mapping', async () => {
assert(result.data.subarray(512).equals(block2))
})
test('readBlock at index > 0 places source data at block-relative offsets', async () => {
// Regression test: a large block after index 0 must be assembled from its own
// source blocks at offsets relative to the start of that large block.
// With distinct, non-zero patterns this catches the case where copy() targets an
// absolute (disk-wide) offset and silently produces a zeroed block.
const blockC = createPatternBuffer(512, 'C')
const blockD = createPatternBuffer(512, 'D')
const source = new MockDisk(512, 4096, [
[2, blockC],
[3, blockD],
])
await source.init()
const disk = new DiskLargerBlock(source, 1024) // blockRatio = 2, so large block 1 = source blocks 2 & 3
await disk.init()
const result = await disk.readBlock(1)
assert.strictEqual(result.index, 1)
assert.strictEqual(result.data.length, 1024)
assert(result.data.subarray(0, 512).equals(blockC))
assert(result.data.subarray(512).equals(blockD))
})
test('hasBlock behavior', async () => {
const source = new MockDisk(512, 2048, [
[0, Buffer.alloc(512)],
@@ -214,6 +237,34 @@ test('close propagation', async () => {
await disk.close()
assert.strictEqual(source.closed, true)
})
test('getBlockIndexesCount', async () => {
const block = createPatternBuffer(512, 'BLOCK')
let source = new MockDisk(512, 4096, [
[2, block],
[3, block],
])
await source.init()
let disk = new DiskLargerBlock(source, 1024)
await disk.init()
assert.strictEqual(disk.getBlockIndexesCount(), 1)
assert.strictEqual(disk.getBlockIndexesCount(), disk.getBlockIndexes().length)
await disk.close()
source = new MockDisk(512, 4096, [
[0, block],
[3, block],
])
await source.init()
disk = new DiskLargerBlock(source, 1024)
await disk.init()
assert.strictEqual(disk.getBlockIndexesCount(), 2)
assert.strictEqual(disk.getBlockIndexesCount(), disk.getBlockIndexes().length)
await disk.close()
})
test('generator', async () => {
const source = new MockDisk(512, 1024)
@@ -225,3 +276,29 @@ test('generator', async () => {
assert.strictEqual(block.data.length, 1024)
}
})
test('readblock missing block throws', async () => {
const parentBlock = createPatternBuffer(512, 'PARENT')
const childBlock = createPatternBuffer(512, 'CHILD')
const sourceParent = new MockDisk(512, 2048, [
[0, parentBlock],
[1, parentBlock],
[2, parentBlock],
[3, parentBlock],
])
await sourceParent.init()
const source = new MockDisk(512, 2048, [[2, childBlock]])
source.parentDisk = sourceParent
await source.init()
const disk = new DiskLargerBlock(source, 1024)
await disk.init()
assert.strictEqual(disk.getBlockIndexesCount(), 1)
for await (const block of disk.diskBlocks()) {
assert.strictEqual(block.index, 1)
assert.ok(block.data.slice(0, 512).equals(childBlock), 'missing child data ')
assert.ok(block.data.slice(512, 1024).equals(parentBlock), 'missing parent data')
}
await assert.rejects(() => disk.readBlock(0))
})

View File

@@ -42,6 +42,9 @@ export class DiskPassthrough extends Disk {
getBlockIndexes(): Array<number> {
return this.source.getBlockIndexes()
}
getBlockIndexesCount(): number {
return this.source.getBlockIndexesCount()
}
hasBlock(index: number): boolean {
return this.source.hasBlock(index)
}
@@ -102,6 +105,9 @@ export abstract class RandomDiskPassthrough extends RandomAccessDisk {
getBlockIndexes(): Array<number> {
return this.source.getBlockIndexes()
}
getBlockIndexesCount(): number {
return this.source.getBlockIndexesCount()
}
hasBlock(index: number): boolean {
return this.source.hasBlock(index)
}

View File

@@ -49,6 +49,9 @@ export class NegativeDisk extends RandomAccessDisk {
getBlockIndexes(): Array<number> {
return this.#child.getBlockIndexes()
}
getBlockIndexesCount(): number {
return this.#child.getBlockIndexesCount()
}
hasBlock(index: number): boolean {
return this.#child.hasBlock(index)
}

View File

@@ -1,7 +1,7 @@
{
"private": false,
"name": "@xen-orchestra/fs",
"version": "4.9.0",
"version": "4.9.3",
"license": "AGPL-3.0-or-later",
"description": "The File System for Xen Orchestra backups.",
"homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@xen-orchestra/fs",
@@ -21,11 +21,11 @@
},
"dependencies": {
"@aws-sdk/abort-controller": "^3.272.0",
"@aws-sdk/client-s3": "^3.54.0",
"@aws-sdk/lib-storage": "^3.54.0",
"@aws-sdk/client-s3": "^3.1065.0",
"@aws-sdk/lib-storage": "^3.1065.0",
"@aws-sdk/middleware-apply-body-checksum": "^3.58.0",
"@aws-sdk/node-http-handler": "^3.54.0",
"@azure/storage-blob": "^12.26.0",
"@azure/storage-blob": "^12.32.0",
"@vates/async-each": "^1.0.3",
"@vates/coalesce-calls": "^0.1.0",
"@vates/decorate-with": "^2.1.0",
@@ -45,11 +45,11 @@
"xo-remote-parser": "^0.10.1"
},
"devDependencies": {
"@babel/cli": "^7.0.0",
"@babel/core": "^7.0.0",
"@babel/plugin-proposal-decorators": "^7.1.6",
"@babel/plugin-proposal-function-bind": "^7.0.0",
"@babel/preset-env": "^7.8.0",
"@babel/cli": "^7.29.0",
"@babel/core": "^7.29.0",
"@babel/plugin-proposal-decorators": "^7.29.0",
"@babel/plugin-proposal-function-bind": "^7.29.0",
"@babel/preset-env": "^7.29.0",
"cross-env": "^7.0.2",
"dotenv": "^16.0.0",
"rimraf": "^6.0.1",

View File

@@ -19,6 +19,7 @@ function getEncryptor(algorithm = DEFAULT_ENCRYPTION_ALGORITHM, key) {
algorithm: 'none',
key: 'none',
ivLength: 0,
authTagLength: 0,
encryptData: buffer => buffer,
encryptStream: stream => stream,
decryptData: buffer => buffer,
@@ -127,6 +128,7 @@ function getEncryptor(algorithm = DEFAULT_ENCRYPTION_ALGORITHM, key) {
id: algorithm,
algorithm,
key,
authTagLength,
ivLength,
encryptData,
encryptStream,

View File

@@ -292,7 +292,14 @@ export default class RemoteHandlerAbstract {
async outputStream(path, input, { checksum = true, dirMode, maxStreamLength, streamLength, validator } = {}) {
path = normalizePath(path)
let checksumStream
info('will output stream ', { path, streamLength, maxStreamLength, isEncrypted: this.isEncrypted })
if (this.isEncrypted) {
info('stream is encrypted', { path, streamLength, maxStreamLength })
const overhead = this.#encryptor.ivLength + this.#encryptor.authTagLength
if (streamLength !== undefined) streamLength += overhead
if (maxStreamLength !== undefined) maxStreamLength += overhead
info('stream is updated ', { path, streamLength, maxStreamLength, overhead })
}
input = this.#encryptor.encryptStream(input)
if (checksum) {
checksumStream = createChecksumStream()

View File

@@ -40,6 +40,8 @@ export default class S3Handler extends RemoteHandlerAbstract {
#bucket
#dir
#s3
#httpAgent
#httpsAgent
#immutable = false
#maxPartSize
#maxPartNumber
@@ -77,13 +79,13 @@ export default class S3Handler extends RemoteHandlerAbstract {
region,
requestHandler: new NodeHttpHandler({
socketTimeout: 600000,
httpAgent: new HttpAgent({
httpAgent: (this.#httpAgent = new HttpAgent({
keepAlive: true,
}),
httpsAgent: new HttpsAgent({
})),
httpsAgent: (this.#httpsAgent = new HttpsAgent({
rejectUnauthorized: !allowUnauthorized,
keepAlive: true,
}),
})),
}),
// from https://github.com/aws/aws-sdk-js-v3/issues/6810
// some non AWS services like backblaze or cloudflare don't expect the new headers
@@ -100,6 +102,12 @@ export default class S3Handler extends RemoteHandlerAbstract {
return 's3'
}
async _forget() {
this.#s3.destroy()
this.#httpAgent.destroy()
this.#httpsAgent.destroy()
}
#makeCopySource(path) {
return join(this.#bucket, this.#dir, path)
}
@@ -433,7 +441,11 @@ export default class S3Handler extends RemoteHandlerAbstract {
new DeleteObjectsCommand({
Bucket: this.#bucket,
Delete: {
Objects: result.Contents ?? [],
// only keep the key: `ETag` and `Size` are also part of `ObjectIdentifier`, where they
// mean "delete only if it still matches", and passing the whole listed object would
// serialize them in the request. Beyond making the deletion conditional, some
// providers reject these elements with a `MalformedXML` error.
Objects: (result.Contents ?? []).map(({ Key }) => ({ Key })),
},
})
)

View File

@@ -157,7 +157,7 @@ export abstract class RemoteHandlerAbstract {
abstract getInfo(): Promise<RemoteInfoResult>
abstract getSize(file: FileDescriptor): Promise<number>
abstract getSizeOnDisk(file: FileArg): Promise<number>
abstract getSizeOnDisk(file: FileDescriptor): Promise<number>
abstract lock(path: string): Promise<LockDisposer>
abstract test(): Promise<TestResult>
abstract isImmutable(): boolean

View File

@@ -38,8 +38,7 @@ immutabilityDuration = "7d"
## CLI commands
- **`xo-immutable-remote`**: Start the watching daemon. Must be kept running to protect new backups reliably.
- **`xo-lift-remote-immutability`**: Manually trigger a lifting pass. If `liftEvery` is set in the config, the process continues running and repeats the check on that interval.
- **`xo-immutable-remote`**: Start the watching daemon. Must be kept running to protect new backups reliably. It lifts expired immutability on startup, then every `liftEvery`.
## How protection works
@@ -86,6 +85,8 @@ The expiry reference is the **datetime in the filename**, not the file's `mtime`
On the **first lift run after startup**, all backup files are scanned unconditionally (full scan). This catches orphaned immutable files left by a previous partial or interrupted lock. Subsequent runs use a fast-path: only backups whose `.json` sentinel is currently immutable are processed.
The first run also walks the disk directories (`xo-vm-backups/<vmUUID>/vdis/<jobId>/<vdiId>/`) directly and lifts any expired disk found there, still using the datetime in its own filename. Both the locking and the regular lifting name a backup's disks from its `<datetime>.json`, so a disk stops being reachable that way once XO's retention has deleted that json — or once a merge has renamed the disk, since the surviving `data/<datetime>.vhd` then carries the datetime of the older backup its blocks came from. Such a disk would otherwise stay immutable forever, and an immutable disk prevents XO from ever merging or deleting the backups of that VDI.
## Troubleshooting
### Some files are still immutable after the duration expired
@@ -100,13 +101,33 @@ Restart `xo-immutable-remote`. The first lift run after startup always performs
chattr -i -R /path/to/remote/on/fileserver/
```
### Make one VM temporarily mutable again
If one or a few VM had an issue and need manual cleanup, you can manually lift the immutability for one VM.
1. Stop the `xo-immutable-remote` service.
2. As root on the file server, run:
```bash
chattr -i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>
```
3. Run the backup job, it will fix what is repairable and purge the non recoverable backups
4. As root on the file server, run:
```bash
chattr +i /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/*.json
chattr +i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/vdis/*/*/*.vhd
chattr +i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/vdis/*/*/data/*.vhd
```
Lock the disks, **not** the directories that hold them: `chattr +i -R …/vdis/` also locks `vdis`, the job and VDI directories and `data`, and an immutable directory rejects the files of the next backup, so every later run of that VM would fail. Depending on whether the remote uses VHD directories, one of the two commands above may report that it matched nothing.
Note that this VM will be mutable between step 2 and 4, it's up to you to document and test the backup after.
### Increasing the immutability duration
Change the setting. The next lift cycle will use the new duration; files already past the old duration but within the new one will remain protected.
### Reducing the immutability duration
Change the setting, then run `xo-lift-remote-immutability` to apply immediately, or wait for the next scheduled `liftEvery` cycle.
Change the setting, then restart `xo-immutable-remote` to apply immediately, or wait for the next scheduled `liftEvery` cycle.
### Why are my incremental backups not marked as protected in XO?

View File

@@ -54,8 +54,7 @@ immutabilityDuration = "7d"
## CLI commands
- **`xo-immutable-remote`**: Start the watching daemon. Must be kept running to protect new backups reliably.
- **`xo-lift-remote-immutability`**: Manually trigger a lifting pass. If `liftEvery` is set in the config, the process continues running and repeats the check on that interval.
- **`xo-immutable-remote`**: Start the watching daemon. Must be kept running to protect new backups reliably. It lifts expired immutability on startup, then every `liftEvery`.
## How protection works
@@ -102,6 +101,8 @@ The expiry reference is the **datetime in the filename**, not the file's `mtime`
On the **first lift run after startup**, all backup files are scanned unconditionally (full scan). This catches orphaned immutable files left by a previous partial or interrupted lock. Subsequent runs use a fast-path: only backups whose `.json` sentinel is currently immutable are processed.
The first run also walks the disk directories (`xo-vm-backups/<vmUUID>/vdis/<jobId>/<vdiId>/`) directly and lifts any expired disk found there, still using the datetime in its own filename. Both the locking and the regular lifting name a backup's disks from its `<datetime>.json`, so a disk stops being reachable that way once XO's retention has deleted that json — or once a merge has renamed the disk, since the surviving `data/<datetime>.vhd` then carries the datetime of the older backup its blocks came from. Such a disk would otherwise stay immutable forever, and an immutable disk prevents XO from ever merging or deleting the backups of that VDI.
## Troubleshooting
### Some files are still immutable after the duration expired
@@ -116,13 +117,33 @@ Restart `xo-immutable-remote`. The first lift run after startup always performs
chattr -i -R /path/to/remote/on/fileserver/
```
### Make one VM temporarily mutable again
If one or a few VM had an issue and need manual cleanup, you can manually lift the immutability for one VM.
1. Stop the `xo-immutable-remote` service.
2. As root on the file server, run:
```bash
chattr -i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>
```
3. Run the backup job, it will fix what is repairable and purge the non recoverable backups
4. As root on the file server, run:
```bash
chattr +i /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/*.json
chattr +i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/vdis/*/*/*.vhd
chattr +i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/vdis/*/*/data/*.vhd
```
Lock the disks, **not** the directories that hold them: `chattr +i -R …/vdis/` also locks `vdis`, the job and VDI directories and `data`, and an immutable directory rejects the files of the next backup, so every later run of that VM would fail. Depending on whether the remote uses VHD directories, one of the two commands above may report that it matched nothing.
Note that this VM will be mutable between step 2 and 4, it's up to you to document and test the backup after.
### Increasing the immutability duration
Change the setting. The next lift cycle will use the new duration; files already past the old duration but within the new one will remain protected.
### Reducing the immutability duration
Change the setting, then run `xo-lift-remote-immutability` to apply immediately, or wait for the next scheduled `liftEvery` cycle.
Change the setting, then restart `xo-immutable-remote` to apply immediately, or wait for the next scheduled `liftEvery` cycle.
### Why are my incremental backups not marked as protected in XO?

View File

@@ -16,13 +16,13 @@
"xo-immutable-remote": "./dist/protectRemotes.mjs"
},
"license": "AGPL-3.0-or-later",
"version": "3.0.3",
"version": "3.0.4",
"engines": {
"node": ">=20.0.0"
},
"dependencies": {
"@vates/async-each": "^1.0.3",
"@xen-orchestra/backups": "^0.73.2",
"@xen-orchestra/backups": "^0.73.10",
"@xen-orchestra/log": "^0.7.2",
"app-conf": "^3.0.0",
"execa": "^5.0.0",

View File

@@ -8,17 +8,18 @@ export async function makeImmutable(dirPath: string): Promise<void> {
// chattr processes all paths even when some are missing (it does not abort on the first
// error), so every existing path is correctly lifted.
//
// Per-path "while trying to stat" messages (ENOENT) are silently ignored regardless of
// locale — these are always expected for optional paths (.xva, .alias.vhd, …) that are
// absent in delta backups. chattr localises the error description but "while trying to
// stat" is always emitted in English, so we match on that suffix only.
// Per-path "while trying to stat" messages (ENOENT) are silently ignored — these are always
// expected for optional paths (.xva, .alias.vhd, …) that are absent in delta backups. chattr
// translates that whole message, so it is run under `LC_ALL=C`: without it, every batch
// containing an optional missing path fails on a localised system, which means no backup ever
// gets locked there.
//
// Any other error (e.g. EAGAIN on NFS when the file is still open, permission denied) is
// retried up to 3 times with a 1 s delay before being re-thrown.
async function execChattrWithMissingFiles(args: string[]): Promise<void> {
for (let attempt = 0; ; attempt++) {
try {
await execa('chattr', args)
await execa('chattr', args, { env: { LC_ALL: 'C' } })
return
} catch (err) {
const stderr = (err as ExecaError).stderr ?? ''

View File

@@ -0,0 +1,177 @@
import { describe, it } from 'node:test'
import assert from 'node:assert/strict'
import fs from 'node:fs/promises'
import path from 'node:path'
import { tmpdir } from 'node:os'
import { rimraf } from 'rimraf'
import execa from 'execa'
import * as Directory from './directory.mjs'
import { liftRemoteImmutability } from './liftProtection.mjs'
// ---------------------------------------------------------------------------
// Constants
// ---------------------------------------------------------------------------
const ONE_DAY_MS = 24 * 60 * 60 * 1000
const VM_UUID = 'aaaaaaaa-0000-0000-0000-000000000001'
const JOB_UUID = 'bbbbbbbb-0000-0000-0000-000000000001'
const VDI_UUID = 'cccccccc-0000-0000-0000-000000000001'
/** Well past any reasonable immutabilityDuration. */
const EXPIRED_DATE = '20240115T120000Z'
/** Inside the window: the expiry reference is the datetime in the name, not the file's dates. */
const RECENT_DATE = new Date().toISOString().replace(/[-:.]/g, '').slice(0, 15) + 'Z'
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
// Recursively lift all immutable flags inside `root`, then delete the tree.
// Must be run as root since chattr requires elevated privileges.
async function cleanupRoot(root: string): Promise<void> {
try {
await execa('chattr', ['-i', '-R', root])
} catch (_err) {}
await rimraf(root)
}
async function makeRoot(): Promise<string> {
return await fs.mkdtemp(path.join(tmpdir(), 'immut-lift-test-'))
}
function vmDirOf(root: string): string {
return path.join(root, 'xo-vm-backups', VM_UUID)
}
function vdiDirOf(root: string): string {
return path.join(vmDirOf(root), 'vdis', JOB_UUID, VDI_UUID)
}
// Creates `vdis/<jobId>/<vdiId>/<datetime>.alias.vhd` plus the VHD directory it points at, the way
// XO lays them out, and returns both paths along with one block file inside the VHD directory.
async function writeDisk(root: string, datetime: string) {
const vdiDir = vdiDirOf(root)
const dataDir = path.join(vdiDir, 'data')
const vhdDir = path.join(dataDir, `${datetime}.vhd`)
const blockFile = path.join(vhdDir, 'blocks', '0', '0')
await fs.mkdir(path.dirname(blockFile), { recursive: true })
await fs.writeFile(blockFile, 'block')
await fs.writeFile(path.join(vhdDir, 'footer'), 'footer')
const alias = path.join(vdiDir, `${datetime}.alias.vhd`)
await fs.writeFile(alias, path.relative(vdiDir, vhdDir))
return { alias, vhdDir, blockFile }
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
describe('liftProtection/liftRemoteImmutability', async () => {
it('lifts expired disks that no metadata json names any more', async () => {
const root = await makeRoot()
try {
// Retention deletes the json first and leaves the disks to the next cleanVm, so nothing names
// them: this is the state the sentinel-driven pass cannot reach.
const { alias, vhdDir, blockFile } = await writeDisk(root, EXPIRED_DATE)
await Directory.makeImmutableBatch([alias, vhdDir])
assert.strictEqual(await Directory.isImmutable(alias), true, 'alias should start immutable')
assert.strictEqual(await Directory.isImmutable(vhdDir), true, 'VHD directory should start immutable')
await liftRemoteImmutability(root, ONE_DAY_MS, true)
assert.strictEqual(await Directory.isImmutable(alias), false, 'alias should be lifted')
assert.strictEqual(await Directory.isImmutable(vhdDir), false, 'VHD directory should be lifted')
// A VHD directory is useless to XO unless its blocks are writable too: the merge moves block
// files out of it.
assert.strictEqual(await Directory.isImmutable(blockFile), false, 'blocks should be lifted too')
// and the files are really usable again
await fs.unlink(blockFile)
await fs.unlink(alias)
} finally {
await cleanupRoot(root)
}
})
it('keeps disks inside the immutability window locked', async () => {
const root = await makeRoot()
try {
const { alias, vhdDir, blockFile } = await writeDisk(root, RECENT_DATE)
await Directory.makeImmutableBatch([alias, vhdDir])
await liftRemoteImmutability(root, ONE_DAY_MS, true)
assert.strictEqual(await Directory.isImmutable(alias), true, 'recent alias must stay immutable')
assert.strictEqual(await Directory.isImmutable(vhdDir), true, 'recent VHD directory must stay immutable')
assert.strictEqual(await Directory.isImmutable(blockFile), true, 'recent blocks must stay immutable')
await assert.rejects(fs.unlink(alias), { code: 'EPERM' })
} finally {
await cleanupRoot(root)
}
})
it('does not sweep the disk directories outside of a full scan', async () => {
const root = await makeRoot()
try {
const { alias, vhdDir } = await writeDisk(root, EXPIRED_DATE)
await Directory.makeImmutableBatch([alias, vhdDir])
await liftRemoteImmutability(root, ONE_DAY_MS, false)
assert.strictEqual(await Directory.isImmutable(alias), true, 'alias should be untouched')
assert.strictEqual(await Directory.isImmutable(vhdDir), true, 'VHD directory should be untouched')
} finally {
await cleanupRoot(root)
}
})
it('lifts a disk whose name does not match the datetime of the backup referencing it', async () => {
const root = await makeRoot()
try {
// Shape left by a merge: the alias of the recent backup points at the VHD directory of the
// older one, whose blocks it now holds.
const { vhdDir, blockFile } = await writeDisk(root, EXPIRED_DATE)
const alias = path.join(vdiDirOf(root), `${RECENT_DATE}.alias.vhd`)
await fs.writeFile(alias, path.relative(vdiDirOf(root), vhdDir))
await fs.unlink(path.join(vdiDirOf(root), `${EXPIRED_DATE}.alias.vhd`))
await Directory.makeImmutableBatch([alias, vhdDir])
await liftRemoteImmutability(root, ONE_DAY_MS, true)
// The VHD directory carries an expired datetime, so it is released and stays mergeable — the
// reason mtime cannot be the reference: every merge would push it forward and it would never
// expire.
assert.strictEqual(await Directory.isImmutable(vhdDir), false, 'merged VHD directory should be lifted')
assert.strictEqual(await Directory.isImmutable(blockFile), false, 'its blocks should be lifted')
assert.strictEqual(await Directory.isImmutable(alias), true, 'the recent alias keeps its lock')
} finally {
await cleanupRoot(root)
}
})
it('still lifts through the metadata json when it is present', async () => {
const root = await makeRoot()
try {
const { alias, vhdDir } = await writeDisk(root, EXPIRED_DATE)
const json = path.join(vmDirOf(root), `${EXPIRED_DATE}.json`)
await fs.writeFile(json, '{}')
await Directory.makeImmutableBatch([json, alias, vhdDir])
// not a full scan: the fast path applies, and the json is immutable so it is picked up
await liftRemoteImmutability(root, ONE_DAY_MS, false)
assert.strictEqual(await Directory.isImmutable(json), false, 'json should be lifted')
assert.strictEqual(await Directory.isImmutable(alias), false, 'alias should be lifted')
assert.strictEqual(await Directory.isImmutable(vhdDir), false, 'VHD directory should be lifted')
} finally {
await cleanupRoot(root)
}
})
})

View File

@@ -118,6 +118,68 @@ async function liftExpiredVmBackups(root: string, immutabilityDuration: number,
})
}
// `liftExpiredVmBackups` names the files of a backup from the datetime of its `<datetime>.json`,
// like the watcher does when locking them. A disk can outlive that name:
//
// - retention deletes the metadata json first and leaves the disks to the next `cleanVm`, so
// nothing names them any more;
// - a merge renames a disk, and the surviving `data/<datetime>.vhd` keeps the name of the older
// backup its blocks came from.
//
// Such a disk would stay immutable forever — a full scan does not help, it only bypasses the
// `isImmutable` fast-path — and an immutable disk blocks every future merge and deletion in its VDI
// directory. So walk the VDI directories themselves and lift what is expired, using the datetime in
// each entry's own name, which is the reference the other passes use too.
//
// Only runs on the first pass after startup: it recovers states that should not happen, and looking
// hourly would not make it more effective.
async function liftExpiredOrphanDisks(root: string, immutabilityDuration: number): Promise<void> {
const threshold = Date.now() - immutabilityDuration
const vmDirs = await listDirs(join(root, 'xo-vm-backups'))
debug('scanning VDI directories for expired disks', { count: vmDirs.length })
await asyncEach(vmDirs, async vmDir => {
// disks live under `vdis/<jobId>/<vdiId>/`, so two levels of directories to walk first
const jobDirs = await listDirs(join(vmDir, 'vdis'))
const vdiDirs = (await Promise.all(jobDirs.map(jobDir => listDirs(jobDir)))).flat()
await asyncEach(vdiDirs, async vdiDir => {
try {
// `<datetime>.vhd` (flat VHD or alias) and `<datetime>.alias.vhd`
const entries = await fsp.readdir(vdiDir, { withFileTypes: true })
const candidates = entries.map(entry => join(vdiDir, entry.name))
// `data/<datetime>.vhd` VHD directories. They are locked recursively, but the attribute is
// set on the directory itself too, so its own entry is enough to decide — no need to look
// at the blocks inside.
if (entries.some(entry => entry.name === 'data' && entry.isDirectory())) {
const dataDir = join(vdiDir, 'data')
candidates.push(...(await fsp.readdir(dataDir)).map(name => join(dataDir, name)))
}
const expired: string[] = []
for (const path of candidates) {
// Entries without a datetime — `data` itself, merge state files — are left alone: there
// is nothing to compare them to.
const datetime = extractDatetime(basename(path))
if (datetime === undefined) continue
const backupTimestamp = parseDatetime(datetime)
if (backupTimestamp === undefined || backupTimestamp > threshold) continue
expired.push(path)
}
if (expired.length === 0) return
debug('lifting expired disks', { vdiDir, count: expired.length })
// Recursive: a VHD directory must be released together with its blocks, and `-R` is a no-op
// on the flat VHDs and aliases sharing the batch.
await Directory.liftImmutabilityBatch(expired)
} catch (err) {
// `cleanVm` may be merging or deleting these files at the same time
if (err.code !== 'ENOENT') warn('error lifting expired disks', { err, vdiDir })
}
})
})
}
// Walk `xo-config-backups/<scheduleId>/<datetime>/metadata.json` files and
// lift immutability on any backup directory whose metadata mtime is expired.
async function liftExpiredConfigBackups(root: string, immutabilityDuration: number, fullScan: boolean): Promise<void> {
@@ -180,6 +242,12 @@ export async function liftRemoteImmutability(
liftExpiredConfigBackups(root, immutabilityDuration, fullScan),
liftExpiredPoolBackups(root, immutabilityDuration, fullScan),
])
// after the passes above: every disk still named by a metadata json has been handled, so this one
// only has the leftovers to look at
if (fullScan) {
await liftExpiredOrphanDisks(root, immutabilityDuration)
}
}
// Lift immutability on all expired backups across every configured remote.

View File

@@ -13,6 +13,7 @@ for (const [remoteId, remote] of Object.entries(remotes)) {
}
info('setup watcher for immutability lifting')
await liftImmutability(remotes).catch(error => warn('error while lifting immutability', error))
setInterval(async () => {
await liftImmutability(remotes).catch(error => warn('error while lifting immutability', error))
}, liftEvery)

View File

@@ -1,14 +1,32 @@
# ChangeLog
## **next**
## **0.24.0** (2026-07-30)
- [Host/dashboard] Switch CPU and RAM panels order to match Pool dashboard layout (PR [#10059](https://github.com/vatesfr/xen-orchestra/pull/10059))
- Remove all "coming soon" disabled button placeholders from network, VM, and pool components (PR [#10068](https://github.com/vatesfr/xen-orchestra/pull/10068))
- [VM/Network] add possibility to “connect/disconnect” a VIF on a VM (PR [#10080](https://github.com/vatesfr/xen-orchestra/pull/10080))
- [Pool/networks] Add the possibility to copy information from one or more networks in JSON format (PR [#10083](https://github.com/vatesfr/xen-orchestra/pull/10083))
- Fix some design inconsistency between pages (PR [#10109](https://github.com/vatesfr/xen-orchestra/pull/10109))
## **0.23.0** (2026-06-30)
- Update the UiTitle component to use the one from web-core (PR [#9869](https://github.com/vatesfr/xen-orchestra/pull/9869))
- [Pool/System] Add `Reboot VM on internal shutdown` in pool's system tab (PR [#9962](https://github.com/vatesfr/xen-orchestra/pull/9962))
- Update the UiCard component to use the one from web-core (PR [#9980](https://github.com/vatesfr/xen-orchestra/pull/9980))
- Update the UiCardTitle component to use the one from web-core (PR [#9982](https://github.com/vatesfr/xen-orchestra/pull/9982))
- Replacement of the UiSeparator component with VtsDivider from web-core (PR [#10017](https://github.com/vatesfr/xen-orchestra/pull/10017))
- Update side panels (PR [#9836](https://github.com/vatesfr/xen-orchestra/pull/9836))
- [XOA deploy] Update log visualization component (PR [#9995](https://github.com/vatesfr/xen-orchestra/pull/9995))
- [SidePanels] Add and use new `VtsCardObjectTitle` component to display object title and ID in side panels (PR [#9755](https://github.com/vatesfr/xen-orchestra/pull/9755))
- Replacement of the UiSpinner component with UiLoader from web-core (PR [#10023](https://github.com/vatesfr/xen-orchestra/pull/10023))
- [Pool,Host/Storage] Add Storage tabs (PR [#10005](https://github.com/vatesfr/xen-orchestra/pull/10005))
- [Pool,Host/Storage] Storage repositories can now be connected, disconnected and deleted from the Storage tabs (PR [#10006](https://github.com/vatesfr/xen-orchestra/pull/10006))
## **0.22.0** (2026-05-28)
- [System] Update system pages layout ([user feedback](https://feedback.vates.tech/posts/37/xo6-ui-very-messy-columns-width-in-vm-system-far-better-in-xo5)) (PR [#9746](https://github.com/vatesfr/xen-orchestra/pull/9746))
- Introduce robots.txt for avoid browser indexing (PR [#9858](https://github.com/vatesfr/xen-orchestra/pull/9858))
- [XOA deploy] Add confirmation before leaving the page during deployment (PR [#9893](https://github.com/vatesfr/xen-orchestra/pull/9893))
## **0.20.0** (2026-04-01)

View File

@@ -1,6 +1,6 @@
{
"name": "@xen-orchestra/lite",
"version": "0.22.0",
"version": "0.24.0",
"type": "module",
"scripts": {
"dev": "GIT_HEAD=$(git rev-parse HEAD) vite",
@@ -28,13 +28,13 @@
"@types/file-saver": "^2.0.7",
"@types/lodash-es": "^4.17.12",
"@types/node": "^18.19.80",
"@vates/types": "^1.26.0",
"@vates/types": "^1.30.1",
"@vitejs/plugin-vue": "^6.0.0",
"@vue/tsconfig": "^0.9.0",
"@vueuse/core": "^14.0.0",
"@vueuse/math": "^14.0.0",
"@vueuse/shared": "^14.0.0",
"@xen-orchestra/web-core": "^0.54.0",
"@xen-orchestra/web-core": "^0.58.0",
"complex-matcher": "^1.1.1",
"d3-time-format": "^4.1.0",
"decorator-synchronized": "^0.6.0",
@@ -62,7 +62,7 @@
"vite": "^8.0.0",
"vue": "~3.5.13",
"vue-i18n": "^11.1.2",
"vue-router": "^5.0.0",
"vue-router": "5.0.6",
"vue-tsc": "~2.2.8",
"zx": "^7.2.3"
},

View File

@@ -11,8 +11,7 @@
</main>
</div>
</div>
<VtsModalList />
<VtsDrawerList />
<VtsOverlayList />
<VtsTooltipList />
</template>
@@ -23,8 +22,7 @@ import AppNavigation from '@/components/AppNavigation.vue'
import { useUnreachableHosts } from '@/composables/unreachable-hosts.composable'
import { usePoolStore } from '@/stores/xen-api/pool.store'
import { useXenApiStore } from '@/stores/xen-api.store'
import VtsDrawerList from '@core/components/drawer/VtsDrawerList.vue'
import VtsModalList from '@core/components/modal/VtsModalList.vue'
import VtsOverlayList from '@core/components/overlay/VtsOverlayList.vue'
import VtsTooltipList from '@core/components/tooltip-list/VtsTooltipList.vue'
import { useChartTheme } from '@core/composables/chart-theme.composable'
import { useUiStore } from '@core/stores/ui.store'

View File

@@ -20,8 +20,7 @@ import UiActionButton from '@/components/ui/UiActionButton.vue'
import UiFilter from '@/components/ui/UiFilter.vue'
import UiFilterGroup from '@/components/ui/UiFilterGroup.vue'
import type { Filters } from '@/types/filter'
import { useModal } from '@core/packages/modal/use-modal.ts'
import { computed } from 'vue'
import { useOverlay } from '@core/packages/overlay/use-overlay.ts'
import { useI18n } from 'vue-i18n'
const { availableFilters } = defineProps<{
@@ -36,18 +35,25 @@ const emit = defineEmits<{
const { t } = useI18n()
const openFilterModal = useModal((editedFilter?: string) => ({
component: import('@/components/modals/CollectionFilterModal.vue'),
props: {
editedFilter,
availableFilters: computed(() => availableFilters),
const { open } = useOverlay({
component: () => import('@/components/modals/CollectionFilterModal.vue'),
events: {
onCancel: true,
},
onConfirm: async newFilter => {
if (editedFilter !== undefined) {
emit('removeFilter', editedFilter)
}
})
emit('addFilter', newFilter)
},
}))
function openFilterModal(editedFilter?: string) {
return open({
props: { editedFilter, availableFilters },
events: {
onConfirm: newFilter => {
if (editedFilter !== undefined) {
emit('removeFilter', editedFilter)
}
emit('addFilter', newFilter)
},
},
})
}
</script>

View File

@@ -4,7 +4,7 @@
<div class="inputs">
<UiInput v-if="newFilter.isAdvanced" v-model="filterContent" accent="brand" class="advanced-input" />
<template v-else>
<VtsSelect :id="builderPropertySelectId" accent="brand" :icon="currentFilterIcon" />
<VtsSelect :id="builderPropertySelectId" accent="brand" />
<VtsSelect v-if="hasComparisonSelect" :id="builderComparisonSelectId" accent="brand" />
<VtsSelect v-if="currentFilter?.type === 'enum'" :id="builderValueSelectId" accent="brand" />
<UiInput v-else-if="hasValueInput" v-model="newFilter.builder.value" accent="brand" />
@@ -20,7 +20,6 @@
<script lang="ts" setup>
import UiActionButton from '@/components/ui/UiActionButton.vue'
import { buildComplexMatcherNode } from '@/libs/complex-matcher.utils'
import { getFilterIcon } from '@/libs/utils'
import type { Filter, FilterComparisons, FilterComparisonType, Filters, FilterType, NewFilter } from '@/types/filter'
import VtsSelect from '@core/components/select/VtsSelect.vue'
import UiInput from '@core/components/ui/input/UiInput.vue'
@@ -104,8 +103,6 @@ function getDefaultComparisonType(property: string) {
const currentFilter = computed<Filter | undefined>(() => availableFilters[builderProperty.value])
const currentFilterIcon = computed(() => getFilterIcon(currentFilter.value))
const hasValueInput = computed(() =>
currentFilter.value ? ['string', 'number'].includes(currentFilter.value?.type) : false
)

View File

@@ -24,8 +24,7 @@ import UiFilter from '@/components/ui/UiFilter.vue'
import UiFilterGroup from '@/components/ui/UiFilterGroup.vue'
import type { ActiveSorts, Sorts } from '@/types/sort'
import VtsIcon from '@core/components/icon/VtsIcon.vue'
import { useModal } from '@core/packages/modal/use-modal.ts'
import { computed } from 'vue'
import { useOverlay } from '@core/packages/overlay/use-overlay.ts'
import { useI18n } from 'vue-i18n'
const { availableSorts } = defineProps<{
@@ -41,11 +40,17 @@ const emit = defineEmits<{
const { t } = useI18n()
const openFormModal = useModal({
component: import('@/components/modals/CollectionSorterModal.vue'),
props: { availableSorts: computed(() => availableSorts) },
onConfirm: ({ property, isAscending }) => emit('addSort', property as TProperty, isAscending),
const { open } = useOverlay({
component: () => import('@/components/modals/CollectionSorterModal.vue'),
events: {
onConfirm: ({ property, isAscending }) => emit('addSort', property as TProperty, isAscending),
onCancel: true,
},
})
function openFormModal() {
return open({ props: { availableSorts } })
}
</script>
<style lang="postcss" scoped>

View File

@@ -1,5 +1,5 @@
<template>
<UiSpinner v-if="!isReady" />
<UiLoader v-if="!isReady" />
<template v-else-if="record !== undefined">
<RouterLink v-if="objectRoute" :to="objectRoute">
{{ record.name_label }}
@@ -10,13 +10,13 @@
</template>
<script generic="T extends ObjectType" lang="ts" setup>
import UiSpinner from '@/components/ui/UiSpinner.vue'
import type { ObjectType, ObjectTypeToRecord } from '@/libs/xen-api/xen-api.types'
import type { XapiContext } from '@/stores/xen-api/create-xapi-store-config'
import { useHostStore } from '@/stores/xen-api/host.store'
import { usePoolStore } from '@/stores/xen-api/pool.store'
import { useSrStore } from '@/stores/xen-api/sr.store'
import { useVmStore } from '@/stores/xen-api/vm.store'
import UiLoader from '@core/components/ui/loader/UiLoader.vue'
import { computed, watch } from 'vue'
import { type RouteLocationTyped, type RouteRecordName } from 'vue-router'

View File

@@ -1,15 +1,15 @@
<template>
<div v-if="!isReady" class="wrapper-spinner">
<UiSpinner class="spinner" />
<UiLoader class="loader" />
</div>
<ObjectNotFoundView v-else-if="isRecordNotFound" :id />
<slot v-else />
</template>
<script generic="T extends XenApiRecord<ObjectType>, I extends T['uuid']" lang="ts" setup>
import UiSpinner from '@/components/ui/UiSpinner.vue'
import type { ObjectType, XenApiRecord } from '@/libs/xen-api/xen-api.types'
import ObjectNotFoundView from '@/pages/object-not-found.vue'
import UiLoader from '@core/components/ui/loader/UiLoader.vue'
import { computed } from 'vue'
import { useRoute } from 'vue-router'
@@ -32,7 +32,7 @@ const isRecordNotFound = computed(() => props.isReady && !props.uuidChecker(id.v
height: 100%;
}
.spinner {
.loader {
color: var(--color-brand-txt-base);
display: flex;
margin: auto;

View File

@@ -40,7 +40,7 @@
<UiCard>
<UiCardTitle>
Logs
<template #right>
<template #info>
<UiButton
v-if="eventsLog.length > 0"
size="medium"
@@ -79,8 +79,6 @@ import StoryEventParams from '@/components/component-story/StoryEventParams.vue'
import StoryPropParams from '@/components/component-story/StoryPropParams.vue'
import StorySettingParams from '@/components/component-story/StorySettingParams.vue'
import StorySlotParams from '@/components/component-story/StorySlotParams.vue'
import UiCard from '@/components/ui/UiCard.vue'
import UiCardTitle from '@/components/ui/UiCardTitle.vue'
import {
isEventParam,
isModelParam,
@@ -96,6 +94,8 @@ import MenuList from '@core/components/menu/MenuList.vue'
import TabItem from '@core/components/tab/TabItem.vue'
import TabList from '@core/components/tab/TabList.vue'
import UiButton from '@core/components/ui/button/UiButton.vue'
import UiCard from '@core/components/ui/card/UiCard.vue'
import UiCardTitle from '@core/components/ui/card-title/UiCardTitle.vue'
import UiCounter from '@core/components/ui/counter/UiCounter.vue'
import 'highlight.js/styles/github-dark.css'
import { uniqueId, upperFirst } from 'lodash-es'

View File

@@ -87,7 +87,7 @@ import useSortedCollection from '@/composables/sorted-collection.composable'
import type { PropParam } from '@/libs/story/story-param'
import VtsIcon from '@core/components/icon/VtsIcon.vue'
import { vTooltip } from '@core/directives/tooltip.directive'
import { useModal } from '@core/packages/modal/use-modal.ts'
import { useOverlay } from '@core/packages/overlay/use-overlay.ts'
import { useVModel } from '@vueuse/core'
import { toRef } from 'vue'
@@ -111,10 +111,16 @@ const params = useSortedCollection(toRef(props, 'params'), (p1, p2) => {
const model = useVModel(props, 'modelValue', emit)
const openRawValueModal = useModal((code: string) => ({
component: import('@/components/modals/CodeHighlightModal.vue'),
props: { code },
}))
const { open } = useOverlay({
component: () => import('@/components/modals/CodeHighlightModal.vue'),
events: {
onClose: true,
},
})
function openRawValueModal(code: string) {
return open({ props: { code } })
}
</script>
<style lang="postcss" scoped>

View File

@@ -4,7 +4,7 @@
<script lang="ts" setup>
import FormInput from '@/components/form/FormInput.vue'
import { useModal } from '@core/packages/modal/use-modal.ts'
import { useOverlay } from '@core/packages/overlay/use-overlay.ts'
import { useVModel } from '@vueuse/core'
import { computed } from 'vue'
@@ -20,11 +20,17 @@ const model = useVModel(props, 'modelValue', emit)
const jsonValue = computed(() => JSON.stringify(model.value, undefined, 2))
const openJsonModal = useModal({
component: import('@/components/modals/JsonEditorModal.vue'),
props: { initialValue: computed(() => jsonValue.value) },
onConfirm: newValue => {
model.value = JSON.parse(newValue)
const { open } = useOverlay({
component: () => import('@/components/modals/JsonEditorModal.vue'),
events: {
onConfirm: (newValue: string) => {
model.value = JSON.parse(newValue)
},
onCancel: true,
},
})
function openJsonModal() {
return open({ props: { initialValue: jsonValue.value } })
}
</script>

View File

@@ -12,6 +12,9 @@
<RouterTab :to="{ name: '/host/[uuid]/network', params: { uuid } }">
{{ t('network') }}
</RouterTab>
<RouterTab :to="{ name: '/host/[uuid]/storage', params: { uuid } }">
{{ t('storage') }}
</RouterTab>
<RouterTab :to="{ name: '/host/[uuid]/tasks', params: { uuid } }" disabled>
{{ t('tasks') }}
</RouterTab>

View File

@@ -1,58 +1,10 @@
<template>
<UiPanel :class="{ 'mobile-drawer': uiStore.isSmall }">
<template #header>
<div :class="{ 'action-buttons-container': uiStore.isSmall }">
<UiButtonIcon
v-if="uiStore.isSmall"
v-tooltip="t('action:close')"
size="small"
variant="tertiary"
accent="brand"
icon="fa:angle-left"
@click="emit('close')"
/>
<div class="action-buttons">
<UiButton
v-tooltip="t('coming-soon!')"
disabled
size="medium"
variant="tertiary"
accent="brand"
left-icon="fa:edit"
>
{{ t('action:edit') }}
</UiButton>
<UiButton
v-tooltip="t('coming-soon!')"
disabled
size="medium"
variant="tertiary"
accent="danger"
left-icon="fa:trash"
>
{{ t('action:delete') }}
</UiButton>
</div>
</div>
</template>
<template #default>
<VtsSidePanel :has-selection="!!pif" @close="emit('close')">
<template v-if="pif" #default>
<!-- PIF -->
<UiCard class="card">
<UiCardTitle>{{ isBond ? t('bond') : t('pif') }}</UiCardTitle>
<VtsCardObjectTitle :id="pif.uuid" :label="isBond ? t('bond') : t('pif')" />
<div class="content">
<!-- UUID -->
<VtsCardRowKeyValue>
<template #key>
{{ t('uuid') }}
</template>
<template #value>
{{ pif.uuid }}
</template>
<template #addons>
<VtsIcon v-if="pif.management" v-tooltip="t('management')" name="status:primary-circle" size="medium" />
<VtsCopyButton :value="pif.uuid" />
</template>
</VtsCardRowKeyValue>
<!-- NETWORK -->
<VtsCardRowKeyValue>
<template #key>
@@ -75,6 +27,7 @@
</template>
<template #value>
{{ pif.device }}
<VtsIcon v-if="pif.management" v-tooltip="t('management')" name="status:primary-circle" size="medium" />
</template>
<template #addons>
<VtsCopyButton :value="pif.device" />
@@ -138,14 +91,7 @@
</template>
<template #addons>
<VtsCopyButton :value="ip" />
<UiButtonIcon
v-if="index === 0 && ipAddresses.length > 1"
v-tooltip="t('coming-soon!')"
disabled
icon="fa:ellipsis"
size="small"
accent="brand"
/>
<VtsCopyAllMenuItem v-if="index === 0 && ipAddresses.length > 1" :values="ipAddresses" />
</template>
</VtsCardRowKeyValue>
</div>
@@ -229,14 +175,7 @@
</template>
<template v-if="device" #addons>
<VtsCopyButton :value="device" />
<UiButtonIcon
v-if="index === 0 && bondDevices.length > 1"
v-tooltip="t('coming-soon!')"
disabled
icon="fa:ellipsis"
size="small"
accent="brand"
/>
<VtsCopyAllMenuItem v-if="index === 0 && bondDevices.length > 1" :values="bondDevicesFiltered" />
</template>
</VtsCardRowKeyValue>
</div>
@@ -282,7 +221,7 @@
</div>
</UiCard>
</template>
</UiPanel>
</VtsSidePanel>
</template>
<script setup lang="ts">
@@ -291,24 +230,23 @@ import { useNetworkStore } from '@/stores/xen-api/network.store'
import { usePifMetricsStore } from '@/stores/xen-api/pif-metrics.store'
import { usePifStore } from '@/stores/xen-api/pif.store'
import VtsCardRowKeyValue from '@core/components/card/VtsCardRowKeyValue.vue'
import VtsCardObjectTitle from '@core/components/card-object-title/VtsCardObjectTitle.vue'
import VtsCopyAllMenuItem from '@core/components/copy-all-menu-item/VtsCopyAllMenuItem.vue'
import VtsCopyButton from '@core/components/copy-button/VtsCopyButton.vue'
import VtsIcon from '@core/components/icon/VtsIcon.vue'
import VtsSidePanel from '@core/components/panel/VtsSidePanel.vue'
import VtsStatus from '@core/components/status/VtsStatus.vue'
import VtsTag from '@core/components/tag/VtsTag.vue'
import UiButton from '@core/components/ui/button/UiButton.vue'
import UiButtonIcon from '@core/components/ui/button-icon/UiButtonIcon.vue'
import UiCard from '@core/components/ui/card/UiCard.vue'
import UiCardTitle from '@core/components/ui/card-title/UiCardTitle.vue'
import UiPanel from '@core/components/ui/panel/UiPanel.vue'
import UiTagsList from '@core/components/ui/tag/UiTagsList.vue'
import { vTooltip } from '@core/directives/tooltip.directive'
import { useUiStore } from '@core/stores/ui.store.ts'
import humanFormat from 'human-format'
import { computed } from 'vue'
import { useI18n } from 'vue-i18n'
const { pif } = defineProps<{
pif: XenApiPif
pif?: XenApiPif
}>()
const emit = defineEmits<{
@@ -318,21 +256,32 @@ const emit = defineEmits<{
const { getByOpaqueRef: getPifMetricsByOpaqueRef, getPifCarrier } = usePifMetricsStore().subscribe()
const { getByOpaqueRef: getNetworkByOpaqueRef } = useNetworkStore().subscribe()
const { getBondsDevices, isBondMaster } = usePifStore().subscribe()
const uiStore = useUiStore()
const { t } = useI18n()
const ipAddresses = computed(() => [pif.IP, ...pif.IPv6].filter(ip => ip))
const ipAddresses = computed(() => {
if (pif === undefined) {
return []
}
const network = computed(() => getNetworkByOpaqueRef(pif.network))
return [pif.IP, ...pif.IPv6].filter(ip => ip)
})
const network = computed(() => (pif !== undefined ? getNetworkByOpaqueRef(pif.network) : undefined))
const networkPurpose = computed(() => (network.value?.purpose?.[0] ? t('on') : t('off')))
const status = computed(() => (pif.currently_attached ? 'connected' : 'disconnected'))
const status = computed(() => (pif?.currently_attached ? 'connected' : 'disconnected'))
const physicalInterfaceStatus = computed(() => (getPifCarrier(pif) ? 'connected' : 'physically-disconnected'))
const physicalInterfaceStatus = computed(() =>
pif !== undefined && getPifCarrier(pif) ? 'connected' : 'physically-disconnected'
)
const ipConfigurationMode = computed(() => {
if (pif === undefined) {
return t('none')
}
switch (pif.ip_configuration_mode) {
case 'Static':
return t('static')
@@ -343,11 +292,19 @@ const ipConfigurationMode = computed(() => {
}
})
const bondDevices = computed(() => getBondsDevices(pif))
const bondDevices = computed(() => (pif !== undefined ? getBondsDevices(pif) : []))
const isBond = computed(() => isBondMaster(pif))
const bondDevicesFiltered = computed(() => {
return bondDevices.value.filter((bondDevice): bondDevice is string => bondDevice !== undefined)
})
const isBond = computed(() => (pif !== undefined ? isBondMaster(pif) : false))
const speed = computed(() => {
if (pif === undefined) {
return ''
}
const speed = getPifMetricsByOpaqueRef(pif.metrics)?.speed || 0
const speedInBytes = speed * 1000000
@@ -384,21 +341,4 @@ const speed = computed(() => {
content: '-';
}
}
.mobile-drawer {
position: fixed;
inset: 0;
.action-buttons-container {
display: flex;
justify-content: space-between;
align-items: center;
width: 100%;
}
}
.action-buttons {
display: flex;
align-items: center;
}
</style>

View File

@@ -2,45 +2,9 @@
<div class="host-pif-table">
<UiTitle>
{{ t('pifs') }}
<template #action>
<UiButton
v-tooltip="t('coming-soon!')"
disabled
left-icon="fa:plus"
variant="secondary"
accent="brand"
size="medium"
>
{{ t('scan-pifs') }}
</UiButton>
</template>
</UiTitle>
<div class="container">
<div class="table-actions">
<UiQuerySearchBar @search="(value: string) => (searchQuery = value)" />
<UiTableActions :title="t('table-actions')">
<UiButton
v-tooltip="t('coming-soon!')"
disabled
left-icon="fa:edit"
variant="tertiary"
accent="brand"
size="medium"
>
{{ t('action:edit') }}
</UiButton>
<UiButton
v-tooltip="t('coming-soon!')"
disabled
left-icon="fa:trash"
variant="tertiary"
accent="danger"
size="medium"
>
{{ t('action:delete') }}
</UiButton>
</UiTableActions>
</div>
<UiQuerySearchBar @search="(value: string) => (searchQuery = value)" />
<VtsTable :state :pagination-bindings sticky="right">
<thead>
<tr>
@@ -63,14 +27,11 @@ import { useNetworkStore } from '@/stores/xen-api/network.store'
import { usePifStore } from '@/stores/xen-api/pif.store'
import VtsRow from '@core/components/table/VtsRow.vue'
import VtsTable from '@core/components/table/VtsTable.vue'
import UiButton from '@core/components/ui/button/UiButton.vue'
import UiQuerySearchBar from '@core/components/ui/query-search-bar/UiQuerySearchBar.vue'
import UiTableActions from '@core/components/ui/table-actions/UiTableActions.vue'
import UiTitle from '@core/components/ui/title/UiTitle.vue'
import { usePagination } from '@core/composables/pagination.composable'
import { useRouteQuery } from '@core/composables/route-query.composable'
import { useTableState } from '@core/composables/table-state.composable'
import { vTooltip } from '@core/directives/tooltip.directive'
import { icon } from '@core/icons'
import { usePifColumns } from '@core/tables/column-sets/pif-columns'
import { logicNot } from '@vueuse/math'
@@ -167,7 +128,6 @@ const { HeadCells, BodyCells } = usePifColumns({
<style scoped lang="postcss">
.host-pif-table,
.table-actions,
.container {
display: flex;
flex-direction: column;
@@ -176,8 +136,7 @@ const { HeadCells, BodyCells } = usePifColumns({
.host-pif-table {
gap: 2.4rem;
.container,
.table-actions {
.container {
gap: 0.8rem;
}
}

View File

@@ -1,16 +1,20 @@
<template>
<VtsModal accent="info" dismissible>
<UiModal accent="info" @dismiss="emit('close')">
<template #content>
<CodeHighlight :code />
</template>
</VtsModal>
</UiModal>
</template>
<script lang="ts" setup>
import CodeHighlight from '@/components/CodeHighlight.vue'
import VtsModal from '@core/components/modal/VtsModal.vue'
import UiModal from '@core/components/ui/modal/UiModal.vue'
defineProps<{
code: any
}>()
const emit = defineEmits<{
close: []
}>()
</script>

Some files were not shown because too many files have changed in this diff Show More