Compare commits

..

604 Commits

Author SHA1 Message Date
unel
abc3018b66 feat(uiObjectLink): replace all uiObjectLink by uiLink 2026-04-27 14:22:45 +02:00
unel
423f00fefa rebase 2026-04-27 14:22:45 +02:00
unel
7456ab9faf feedback 2026-04-27 14:22:02 +02:00
unel
4d30d43a66 add changelog 2026-04-27 14:22:01 +02:00
unel
d1f3efc57f feat(xo6): rebase and resolve comment 2026-04-27 14:22:01 +02:00
unel
1d881fe8cc feat(xo6/uiLink): wrap uiLink on side panel 2026-04-27 14:22:01 +02:00
unel
7a9de0de57 feat(xo6/uiLink): add text ellipsis on uiLink 2026-04-27 14:21:56 +02:00
Joris K
699fb196fb feat(xo6): add possibility to remove a snapshot (#9749) 2026-04-27 10:25:14 +02:00
Mathieu
00e0bee372 fix(xo-server): fix default-template still in xo after deletion (#9760)
When VM_destroy is called on a default template, it is first converted to a simple template (and then to a VM).

After converting a default template to a simple template, the object received after the xapiObjectToXo transformation has a different ID (the default template's ID = pool.uuid + template.uuid, and non default template ID = template.uuid).

Since this object does not already exist in the XO server cache(because the ID is new), XO treats it as a new object and not as an update, so the old object with ID pool.uuid + template.uuid stay in the cache
2026-04-27 08:39:51 +02:00
Florent BEAUCHAMP
da045e0de1 fix(backups): use a delta transfer even if the VM is modified (#9709)
* fix(backups): use a delta transfer even if the VM is modified
* test(backup): fix date test without timezone
* refactor(backups): extract method that check which snapshot can be reused for replica target
* reset before launching the replication
2026-04-24 19:30:51 +02:00
Bastien Nollet
fdb82ed68a feat: technical release (#9768) 2026-04-24 14:38:15 +02:00
Pierre Brunet
112c22e748 fix: add chainPhysicalUsage property in vdi object (#9708)
*add  chainPhysicalUsage property to vdi
2026-04-23 16:29:55 +02:00
Thomas Moraine
c540a49643 doc(users.md): recommend against using local users in prod environments (#9761) 2026-04-23 15:13:14 +02:00
Joris K
f6484a0695 feat(xo6): add possibility to delete a VBD (#9698) 2026-04-21 16:34:13 +02:00
Joris K
5593f27110 feat(xo6): add possibility to remove VDIs (#9689) 2026-04-21 15:41:15 +02:00
Joris K
81813c7e5b feat(xo6): add possibility to remove VIFs (#9601) 2026-04-21 15:19:06 +02:00
Mathieu
0bf0a468b5 fix(rest-api): fix other_config being ignored on VDI creation (#9695) 2026-04-21 12:35:33 +02:00
Pierre Brunet
e78275f046 docs: fix typo (#9739) 2026-04-21 11:39:49 +02:00
spacotte-vates
89c7551293 feat(rest-api): add connect/disconnect vif actions (#9643) 2026-04-21 10:25:24 +02:00
Mathieu
ed30291993 fix(rest-api): fix double slash in the href property (#9741)
Introduced by dd5f0cdaa9
2026-04-21 10:07:52 +02:00
Pierre Donias
9765435471 feat(xo-web/servers): remind users to only add pool masters (#9742) 2026-04-20 15:43:02 +02:00
spacotte-vates
e2a4bd2231 fix(changelog): add version 6.2.3 (#9743) 2026-04-20 15:10:02 +02:00
Florent BEAUCHAMP
c6ccd987e2 fix(xo-server): TLSSocket memory leak — stoppable + http-server-plus incompatibility (#9725)
xo-server uses two libraries for its web server:

    http-server-plus — a multi-protocol server facade that multiplexes HTTP and HTTPS listeners behind a single EventEmitter interface ( introduced here JsCommunity/http-server-plus@ec63f3f )
    stoppable — wraps the server to track open connections for graceful shutdown, using an internal reqsPerSocket Map (server._pendingSockets) (introduced here : 782505b )


Fix

@vates/http-server-plus (fork of JsCommunity/http-server-plus): add 'secureConnection' to the forwardedEvents list so that webServer.on('secureConnection', handler) is propagated to underlying https.Server instances at listen() time.

xo-server/src/index.mjs: register the missing handler that stoppable was supposed to add but never did:
2026-04-20 14:38:37 +02:00
Mathieu
45f810c044 feat(openmetrics): add VM status and uptime metrics (#9684)
* feat(openmetrics): add VM status and uptime metrics

* refactor(openmetrics): address review feedback on VM status/uptime
2026-04-20 14:09:28 +02:00
Bastien Nollet
fbba05bbde feat(backups): use XO Tasks (#9734)
Change the backups so they stop using their own task system, and use the same system as XO Tasks.

For the moment, it shouldn't affect the users, but when the previous backup logs will be old enough to be erased, this will allow us to have shorter loading times for backup logs.

XO-52
2026-04-17 17:49:18 +02:00
OlivierFL
9dc52f2e5b fix(xo-6/network-form): toRef causes TS error on build (#9736) 2026-04-17 17:33:06 +02:00
Mathieu
271d123395 fix(rest-api): SSE MiB limit instead percent (#9731) 2026-04-17 16:58:26 +02:00
Enée DI IORIO
88a89cc3a9 fix(changelog): fix PR link formatting in changelogs (#9732) 2026-04-17 11:46:47 +02:00
Florent BEAUCHAMP
5f7fab9d34 test(backups): replication and full mirror backups
* test(replication): test the new snapshot based replication

* fix(tests): use mergeBackupsSynchronously to ensure the backups don't fails with lock already heled error

* fix: ensure the repository that will be purged start empty

* test(backup): add test for full mirror backups
2026-04-17 11:25:38 +02:00
spacotte-vates
8df8ef0af8 feat(backups): log merge size (#9679)
XO-2050

Add the merged size in the cleanVm task log
2026-04-17 09:58:12 +02:00
Mathieu
6fbb961e7f fix(xo-server): fix 408 timeout when copying a VM with compression (#9699)
* fix compression not used on front whe copying a VM
* ensure the first bytes are ready before starting to import the VM 

---------

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2026-04-17 09:56:39 +02:00
Mathieu
6b1a2afc9c docs(@vates/types): document vm.videoram (#9694) 2026-04-16 16:05:28 +02:00
Pierre Donias
3ed8c7b689 feat(netbox): introduce sub-platforms with major version number (#9644)
Fixes #7773
See Zammad#53857

On Netbox >= 4.4 (which introduced platform hierarchies):
- If the major version number is available: create a versioned child platform (e.g. "Debian 12") with the distro-only platform (e.g. "Debian") as its parent, and assign the child to the VM
- Otherwise: keep previous behaviour and assign the VM to the distro-only platform
2026-04-16 10:31:32 +02:00
Mathieu
db9c4815b1 feat(openmetrics): add 9 missing host RRD metrics (#9696) 2026-04-15 19:27:03 +02:00
Mathieu
3e21edad4e chore: upgrade dependencies with known security vulnerabilities (#9640)
* chore: upgrade dependencies with known security vulnerabilities

* Update Node.js version in CI workflow

* chore: upgrade additional security dependencies

* chore: stabilize yarn.lock path-to-regexp entry

* chore: bump minimum version ranges for security-patched dependencies

* chore: update CHANGELOG.unreleased.md with security upgrades

* chore: revert vite upgrade to 6.4.1 (6.4.2 breaks vue-tsc)

* chore: fix vite upgrade and pin vue to 3.5.28
2026-04-15 16:56:43 +02:00
vates-bot
1b12aebb47 feat(web-stack/i18n): update Xen Orchestra 6 (#9649) 2026-04-15 14:43:26 +02:00
p-bo
2e009535aa fix(i18n): shutdown -> reboot in force reboot dialog (as in French) (#9683)
99d19012ad/%40xen-orchestra/web-core/lib/locales/fr.json (L485)
2026-04-15 14:36:13 +02:00
Elise-F
5af32683ce feat(web-core): update default tab functionality (#9651) 2026-04-15 12:40:20 +02:00
Mathieu
0fef765ce6 fix(rest-api): fix memory-leak on subscription (#9707) 2026-04-15 12:18:06 +02:00
Pierre Donias
4508e01644 fix(CHANGELOG): markdown, URLs, labels, ... (#9723) 2026-04-15 11:56:25 +02:00
Pierre Briffaux
f60203a927 fix(web): fix lost focus on vdi name input (#9685) 2026-04-15 11:29:11 +02:00
Mathieu
5ebefb0af2 feat(openmetrics): add VDI disk size and physical usage metrics (#9680)
Fixes XO-1995
2026-04-15 09:52:26 +02:00
Thomas Moraine
fa8844037f docs(xen-orchestra): upgrade docusaurus to v3.10.0 (#9682) 2026-04-15 09:40:19 +02:00
Bastien Nollet
2098c02e7f chore(xo-web): remove unused function (#9691) 2026-04-15 09:37:11 +02:00
Mathieu
c4763d5eec feat(rest-api,mcp): add ?markdown=true output format to REST API (#9624) 2026-04-15 09:30:20 +02:00
Elise-F
0b78cab0b3 fix(web-core): match npm package version (#9721)
web-core 0.49.0 was accidentally published to npm.
This commit bumps the version to match what is already on npm.
2026-04-14 18:18:38 +02:00
Elise-F
bbf3e1e446 feat: release 6.3.3 (#9720) 2026-04-14 17:19:55 +02:00
Elise-F
d94de6090b feat: technical release (#9716) 2026-04-14 15:31:22 +02:00
Bastien Nollet
816b60f11b docs(backups): extra details (#9693) 2026-04-14 14:20:07 +02:00
Florent BEAUCHAMP
6b36b41d68 fix(xo-web): toggle distributed replication (#9715)
introduced by #9433
2026-04-14 11:44:26 +02:00
spacotte-vates
e9caeafd7a fix(bakups): fix merge resume issue when child is disk chain (#9668)
* fix(bakups): fix merge resume issue when child is disk chain
2026-04-14 11:04:23 +02:00
Pierre Brunet
06c3933756 test(backups): add unit test to check #9667 (#9670) 2026-04-13 18:30:17 +02:00
Mathieu
30ff04d88a docs(mcp): add multi-instance configuration (#9705) 2026-04-13 18:25:27 +02:00
Florent BEAUCHAMP
43d86038b1 fix(backups): Fix "Storage_error ([S(Illegal_transition);[[S(Activated);S(RO)];[S(Activated);S(RW)]]])" (#9702)
* don't connect NBD server if we only want the block list
* wait for the disk to be disconnected from dom0 before trying to write into it
2026-04-13 16:46:23 +02:00
Thomas Moraine
415207d1ba docs(intro_backup.md): fix broken youtube embed (#9701) 2026-04-13 10:41:57 +02:00
spacotte-vates
b21e5e8c53 fix(backups): fix vhd directory merge speed regression (#9692)
* fix(backups): fix vhd directory merge speed regression

* add changelog
2026-04-13 09:35:40 +02:00
OlivierFL
aa605639e4 feat(xo-6): network creation forms (#9629) 2026-04-10 17:35:21 +02:00
Elise-F
8ecaa6d817 fix(xo6): increase SSE ping timeout (#9681)
Introduced by #9375
2026-04-09 10:14:54 +02:00
Mathieu
76655a9e05 fix(@xen-orchestra/immutable-backups): fix missing prepublishOnly entry (#9687)
See zammad#51952
2026-04-08 16:12:20 +02:00
Elise-F
58b56a0749 fix(XO6): 'please wait' page not displayed on dev pages (#9686) 2026-04-08 14:41:50 +02:00
Thomas Moraine
99d19012ad docs(xo): replace some XO5 screenshots with the XO6 versions (#9678) 2026-04-07 09:24:38 +02:00
Elise-F
c1e5f2e8d7 feat: release XO 6.3.2 (#9676) 2026-04-03 15:21:41 +02:00
Elise-F
372a2be21d feat: technical release (#9673) 2026-04-03 10:47:59 +02:00
Florent BEAUCHAMP
3acd35142c fix(backups): don't fail incremental replication when the VM has VTPM (#9671) 2026-04-03 10:06:44 +02:00
Florent BEAUCHAMP
28c5e5f138 fix(backups): schedule reference is missing (#9667)
introduced by #9635
2026-04-02 14:47:00 +02:00
Elise-F
a6c508be46 feat(lite): 0.20.0 (#9666) 2026-04-01 16:27:01 +02:00
Elise-F
875ceb7cab feat: release XO 6.3.1 (#9665) 2026-04-01 15:08:02 +02:00
Thomas Moraine
d98dd78168 docs(manage-infrastructure.md): fix formatting errors in info panels (#9664)
Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-04-01 13:07:04 +02:00
Elise-F
df0c69c023 feat: technical release (#9663) 2026-04-01 10:37:53 +02:00
Mathieu
df7ffbb90c fix(web-core): fix objects store not updated (#9660)
* fix(web-core): fix objects store not updated
2026-04-01 10:10:23 +02:00
Mathieu
8553f1cd08 fix(rest-api): fix no such VDI for /vms/:id/(dashboard|vdis) (#9662)
* fix(rest-api): fix no such VDI for /vms/:id/(dashboard|vdis)
2026-04-01 10:07:10 +02:00
Florent BEAUCHAMP
41de7528bf fix(backups): full when going from 6.2 to 6.3 (#9661) 2026-04-01 09:48:55 +02:00
Elise-F
bfb0a052ef feat: release XO 6.3.0 (#9658) 2026-03-31 16:37:31 +02:00
Elise-F
5bd04b0736 feat: technical release (#9656) 2026-03-31 14:52:50 +02:00
Mathieu
4015987b03 fix(web-core): fix performance issue when lot of new events (#9627) 2026-03-31 12:05:00 +02:00
Elise-F
25db113349 feat: technical release (#9654) 2026-03-31 10:00:56 +02:00
spacotte-vates
b8637097c2 fix(backups): fix double close on merge (#9636)
* fix(backups): fix double close on merge

* set dispose to promise instead of empty func
2026-03-31 09:20:06 +02:00
Mathieu
9226c51df7 fix(rest-api): cannot read properties of undefined for vms/:id/dashboard (#9650)
Fix forum#12031
2026-03-30 20:07:45 +02:00
Joris K
14328ad62a feat(web-core): add stepper component (#9569) 2026-03-30 17:19:29 +02:00
Florent BEAUCHAMP
ac8dfeb7f5 feat(backups): improve incremental replication (#9635) 2026-03-30 17:04:16 +02:00
OlivierFL
bce3c88915 fix(xo-core): wrong translation key on PT locale (#9652)
Introduced by 658de40
2026-03-30 15:55:53 +02:00
DustyArmstrong
658de40f9e fix(web-core): fix some grammar issues on the XO 6 Dashboard (#9647) 2026-03-30 15:31:16 +02:00
Pierre Donias
b413db14e7 fix(xo-server-netbox): avoid mutating TYPES_WITH_UUID constant (#9607) 2026-03-30 15:24:58 +02:00
Enée DI IORIO
e6e5031964 feat(xo6/lite): add bug tools download button (#9419) 2026-03-30 14:35:45 +02:00
Florent BEAUCHAMP
484b9a02a3 docs(openmetrics): add documentation for xo metrics (#9631)
introduced by 9535
2026-03-30 14:34:32 +02:00
OlivierFL
a7829414b0 chore(web-core/query-builder): add query builder documentation (#9541) 2026-03-30 10:56:05 +02:00
vates-bot
39acc6b3a0 feat(web-stack/i18n): update Xen Orchestra 6 (#9554) 2026-03-30 10:55:00 +02:00
Mathieu
465c00b71d fix(xo-stack): return null instead of 0 for every NaN stats (#9634) 2026-03-27 17:26:55 +01:00
Mathieu
6814f6b2ef feat(qa-test): migrate xo-qatest into monorepo as @xen-orchestra/qa-test (#9626)
* feat(qa-test): migrate xo-qatest into monorepo as @xen-orchestra/qa-test
2026-03-27 14:35:03 +01:00
Elise-F
cc4cc1796b feat: technical release (#9646) 2026-03-27 14:22:03 +01:00
Florent BEAUCHAMP
1d81c0b26e fix(xapi): add missing package to changelog (#9645)
introduced by #9596
2026-03-27 13:46:38 +01:00
Elise-F
04a6fb3805 feat: technical release (#9642) 2026-03-27 10:23:50 +01:00
Florent BEAUCHAMP
1302618f90 fix(backup): compute the restore size for incremental restore (#9632)
introduced by #8432
2026-03-26 11:48:44 +01:00
Elise-F
7a39e45ffb feat(xo6): add please wait page for first connection in xo server (#9568) 2026-03-26 11:27:16 +01:00
Florent BEAUCHAMP
4bbfa77ec8 feat(xo5/sr): add preferred-image-format field (#9596) 2026-03-26 10:43:52 +01:00
Elise-F
57414dafce feat(xo6): implement vm snapshot tab page (#9521) 2026-03-26 10:03:19 +01:00
Elise-F
e28e768c72 Fix(xo6): remove scrollbar from console tab (#9609) 2026-03-26 09:14:24 +01:00
Joris K
8e5809a675 feat(xo6): add backup replication on vm dashboard (#9545) 2026-03-25 14:55:54 +01:00
Mathieu
56b17ce7f6 feat(rest-api): expose create internal and bonded network (#9611) 2026-03-25 11:25:06 +01:00
Mathieu
27a119f037 refactor(mcp): split monolithic index.mts into modular file structure (#9621)
XO-2221
2026-03-25 11:14:48 +01:00
Elise-F
60ba5070c9 fix(XO6): update watch to avoid reset template select in new VM (#9603) 2026-03-24 17:33:01 +01:00
Florent BEAUCHAMP
0efa078190 fix(backups): vhd directory have empty names (#9633) 2026-03-24 17:05:49 +01:00
Mathieu
c3dcb8fd50 feat(mcp): add list_srs and get_sr_details tools for SR (#9599)
* feat(mcp): add list_srs and get_sr_details tools for storage
repositories

* docs(mcp): add list_srs and get_sr_details to documentation

* fix(mcp): extract expected tool names into a constant in tests

* docs(mcp): add link to live filter syntax documentation for all filter
parameters
2026-03-24 16:39:58 +01:00
Florent BEAUCHAMP
0286f7786a fix(immutable-backups): decrease resource consumption, add tests and typing (#9548) 2026-03-24 16:00:01 +01:00
Florent BEAUCHAMP
f2b1931408 feat(disk-cli): implement disk cli tooling (#9551)
* feat(disk-cli): implement disk cli tooling

* changelog

* fix test path

* fix following review

* fix following rebase

* ci

* fix following rebase

* more tests
2026-03-24 14:35:05 +01:00
Florent BEAUCHAMP
f546849494 feat(xo-server): use index for getAllUnhealthyVdiChainsLength (#9622) 2026-03-24 10:37:18 +01:00
Florent BEAUCHAMP
6ae70a424a fix(backups): correctly rename replicated VMs (#9606) 2026-03-24 09:16:43 +01:00
Florent BEAUCHAMP
41751a8599 fix(backups): use the snapshot of the source (#9612)
* fix(backups): use the snapshot of the source

introduced by 9524

use only the vdi from the source VM as the source on backup
this bug could only occurs if the replication was intra pool, if
not the different xapi was filtering the results
2026-03-24 09:16:16 +01:00
Joris K
7ea8640ad0 fix(xo6/lite): fixed duplicated ip addresses in vm networks tab (#9547) 2026-03-23 16:29:41 +01:00
Enée DI IORIO
2994761178 fix(xo-lite/header): Fix menu button on large screen (#9558) 2026-03-23 10:12:56 +01:00
Mathieu
d1736e01f7 feat(mcp): add list_networks and get_network_details tools (#9595) 2026-03-20 09:38:59 +01:00
Mathieu
a2e3343b6e fix(rest-api/dashboard): exlude ISO SRs from dashboard (#9608) 2026-03-19 14:07:54 +01:00
Bastien Nollet
e325c6af51 docs(load-balancer): add info on migration cooldown (#9604) 2026-03-18 10:09:40 +01:00
Florent BEAUCHAMP
598ab85446 feat(vmware-explorer): make V2V import cancellable (#9552) 2026-03-17 18:17:08 +01:00
Pierre Brunet
1591749a23 fix(fs/s3): add check before using DeleteObjectsCommand (#9598)
fallback to delete object by object on multiple delete error  to handle backends that can't handle this command
2026-03-16 16:56:33 +01:00
Mathieu
800fd97996 feat(openmetrics): add missing VBD and DCMI metrics (#9563)
XO-1491
2026-03-16 11:37:19 +01:00
Pierre Brunet
45aee36ae7 typescript: Add types in several packages (#9583)
* feat: add types for fs (#9515)

* feat: add types for fs

* fix: update from review

* fix: update devDep

* feat: add types from @xen-orchestra/fs/path

* feat: export types based on async-each

* fix: remove ts-expect-error because async-each is now typed

* fix: export backups types and packages

* fix: update tsconfig async-each and fix types in MergeRemoteDisk

* fix: extractIdsFromSimplePattern from backups

* fix: package.json

* feat: export vhd-lib types

* feat(disks): export types

* fix(backups/disks): make params optional

* fix(backups/disks): small update in jsdoc

* bump: update yarn.lock

* fix: update vhd-lib exports

* fix(vhd-lib): update package exports

* fix: revert to previous import

* fix: new approach without exports

* fix: revert types except for asyncEach, openDisk

* fix(disks): import in vhd-lib

* fix(remoteVhdDisk): update typedef
2026-03-16 10:55:18 +01:00
Mathieu
2aff87d6d6 feat(mcp): add list_vdis tool to list virtual disks (#9559) 2026-03-13 15:54:40 +01:00
Mathieu
449e79de21 feat(rest-api): add disable/enable action endpoints for hosts (#9532) 2026-03-13 10:03:56 +01:00
MlssFrncJrg
9d657ac7cc feat(@xen-orchestra/rest-api): expose forget SR (#9505) 2026-03-12 16:12:43 +01:00
Enée DI IORIO
f4cc40233c fix(xo6/backups): backupLog transfer size calculation (#9496) 2026-03-12 15:09:19 +01:00
MlssFrncJrg
2cd5ba0319 feat(@xen-orchestra/rest-api): expose vms/:id/actions/clone (#9453) 2026-03-11 15:58:18 +01:00
Mathieu
2444861082 fix(rest-api/xo6): cpu at VM creation was not working (#9591)
* fix(rest-api/xo6): cpu at VM creation was not working

* changelog
2026-03-11 15:07:56 +01:00
Florent BEAUCHAMP
7ddff121de feat(fs): make part size configurable for s3 (#9561)
from https://xcp-ng.org/forum/topic/10587/s3-backup-maximum-number-of-parts/
2026-03-11 09:54:08 +01:00
Florent BEAUCHAMP
c1e9b83adc feat(backups): symetrical backups (#9524) 2026-03-11 09:34:09 +01:00
Mathieu
18d9b33c87 feat(mcp): support token authentication via XO_TOKEN (#9577) 2026-03-11 09:20:40 +01:00
Bastien Nollet
673e729afe fix(xo-server): incorrect import (#9589) 2026-03-10 19:25:55 +01:00
Florent BEAUCHAMP
4e0655dc3d fix(backups): better handling of generator cleanup (#9557) 2026-03-10 15:55:59 +01:00
bleader
a78870b473 docs(sdn-controller): mention xapi plugin is in BETA (#9556) 2026-03-10 15:42:16 +01:00
Thomas Moraine
0bf309a4ad docs(sdn_controller.md): clarify requirements for regenerating certificates (#9584) 2026-03-10 09:54:01 +01:00
spacotte-vates
5bdd76a955 feat: release 6.2.2 (#9579) 2026-03-09 14:41:09 +01:00
Pierre Brunet
e11530ea7c feat: technical release (#9576) 2026-03-09 10:10:36 +01:00
Florent BEAUCHAMP
c70f911b5e fix(changelog): remove package from release list (#9575)
introduced during rebase by #9572
2026-03-09 09:56:22 +01:00
Mathieu
82f047ef9d fix(xo-server): proxy issue if hostname defined (#9572)
fix #9500
2026-03-09 09:49:24 +01:00
Mathieu
57aec1ab2d fix(xo-server): race condition on XOA WebSocket during plugin loading (#9402) 2026-03-09 09:42:53 +01:00
Pierre Donias
385fcf3810 fix(xo-server/vm): allocate resource set quotas when not using DMC (#9566)
When editing VM RAM, the high-level `memory` field dispatches to `memoryMax` or
`_memory` depending on whether the VM uses DMC. Both paths need `limitName:
'memory'` on their spec so the resource set quota counter is correctly updated,
but `_memory` was missing it, causing memory resizes on non-DMC VMs to silently
create a stale `_memory` counter instead of updating the actual memory quota.
2026-03-09 09:37:24 +01:00
Florent BEAUCHAMP
6699bd1f5d fix(backups): remove snapshot of vm with cdrommounted (#9571) 2026-03-06 17:28:50 +01:00
Pierre Brunet
844c07e166 feat: technical release (#9570) 2026-03-06 14:49:29 +01:00
spacotte-vates
7676978c78 fix(backups): fix out of range error on merge (#9567)
Fixes out of range error during merge that was happening on delta backups when the child disk was bigger than the parent disk but no blocks where written in the "new" space.

Also added a test for this error.
2026-03-06 14:17:17 +01:00
Elise-F
7054fefdbd Style(xo6): update color for status paused VM icon (#9562) 2026-03-05 11:21:06 +01:00
Enée DI IORIO
acd84649ed feat(xo6): add backupRuns and BackupArchives card on vm dashboard (#9303) 2026-03-05 10:32:07 +01:00
Florent BEAUCHAMP
7460bf93d1 feat(openmetrics): add XO related metrics (#9535) 2026-03-04 10:30:20 +01:00
Mathieu
61e4a77cab fix(openmetrics): convert latency from µs to seconds instead of ms (#9550) 2026-03-03 10:01:56 +01:00
vates-bot
55130311f3 feat(web-stack/i18n): update Xen Orchestra 6 (#9531) 2026-03-03 09:18:35 +01:00
Florent BEAUCHAMP
f0cbd1bd72 docs(backup): document distributed backups (#9528)
* docs(backup): document distributed backups

* Apply suggestions from MPT code review

Co-authored-by: Mathieu <goore.csmoviz@gmail.com>

---------

Co-authored-by: Mathieu <goore.csmoviz@gmail.com>
2026-03-03 09:11:36 +01:00
Enée DI IORIO
561e2349e9 feat(xo6-lite/dashboard): add reactivity on resize (#9378) 2026-03-02 14:59:51 +01:00
Enée DI IORIO
7c83ed63f1 fix(web-core/treeItemLabel): fix treeview tooltip overlap (#9319) 2026-03-02 10:41:26 +01:00
Enée DI IORIO
9525abdee5 feat(xo6/Lite): add secureBoot support (#9423) 2026-02-27 15:09:49 +01:00
spacotte-vates
c6172b7b74 feat: release 6.2.1 (#9546) 2026-02-27 14:42:27 +01:00
Pierre Brunet
1b865bbf89 feat: technical release (#9544)
* feat(@xen-orchestra/backups): 0.69.2

* feat(@xen-orchestra/proxy): 0.29.49

* feat(xo-server): 5.197.3

* chore(CHANGELOG): update next

* fix: update changelog
2026-02-27 11:56:10 +01:00
spacotte-vates
815f4be76c fix(backups): changelog (#9543) 2026-02-27 11:44:13 +01:00
spacotte-vates
53f5c63fa2 fix(backups): race condition on merge (#9542) 2026-02-27 11:29:42 +01:00
spacotte-vates
4ffd1b4e13 feat(lite): 0.19.0 (#9540) 2026-02-26 18:29:55 +01:00
spacotte-vates
d25d15efd7 feat: release 6.2.0 (#9539) 2026-02-26 17:46:07 +01:00
Pierre Brunet
322205d203 feat: technical release (#9537) 2026-02-26 14:57:42 +01:00
Mathieu
89494457a1 fix(mcp): use bin wrapper to fix npx permission denied (#9536)
Fix npx @xen-orchestra/mcp failing with "Permission denied".

tsc does not preserve the executable bit on compiled files, so bin pointing to ./dist/index.mjs (mode 644) caused the error. This adds a bin/xo-mcp.mjs wrapper tracked by git with mode 755, following the same pattern as other packages (@xen-orchestra/xapi, @xen-orchestra/proxy, etc.).
2026-02-26 13:34:08 +01:00
Pierre Briffaux
6bc2844092 feat(xo6/new-vm): add custom user-config to new VM creation (#9427) 2026-02-26 13:15:30 +01:00
Mathieu
53c15bc9e4 docs(mcp) : documentation (#9529)
docs(mcp): add README and Docusaurus documentation page
2026-02-25 17:19:38 +01:00
Pierre Brunet
1dd8c1603f Release (#9534) 2026-02-25 16:19:54 +01:00
Mathieu
52513fc898 feat(xo-server): add transient option to host.disable (#9503)
Add optional transient parameter (boolean) to host.disable API endpoint.

When transient=true, passes auto_enable=false to XAPI, keeping the host disabled across reboots and toolstack restarts (requires XAPI >= 25.31.0). Default behavior unchanged (backward compatible).

XAPI 25.31.0 introduced a new auto_enable parameter to Host.disable (xen-api#6652). When set to false, the host stays disabled regardless of toolstack restarts, host reboots, or consider_enabling_host calls. Only an explicit Host.enable can re-activate it.
2026-02-25 15:54:26 +01:00
spacotte-vates
ed508ec04f feat: technical release (#9533)
* feat(@xen-orchestra/backups): 0.69.1

* feat(complex-matcher): 1.1.0

* feat(@xen-orchestra/web-core): 0.44.0

* feat(@xen-orchestra/mcp): 1.0.0

* feat(@xen-orchestra/proxy): 0.29.48

* feat(@xen-orchestra/rest-api): 0.26.0

* feat(@xen-orchestra/web): 0.42.0

* feat(xo-server): 5.197.1

* feat(xo-web): 5.194.0

* chore(CHANGELOG): update next

---------

Co-authored-by: Pierre Brunet <pierre.brunet@vates.tech>
Co-authored-by: Pierre Donias <pierre.donias@gmail.com>
2026-02-25 15:39:25 +01:00
MathieuRA
21823bf430 feat: add rest-api debug 2026-02-25 14:13:30 +01:00
Florent BEAUCHAMP
e36e1012e2 fix(xo-web): reducing polling for patches and license 2026-02-25 14:13:30 +01:00
Elise-F
3fcf582087 feat(new VM): add ssh key to vm creation form (#9413) 2026-02-25 10:24:29 +01:00
vates-bot
f1389cff6e feat(web-stack/i18n): update Xen Orchestra 6 (#9462) 2026-02-25 09:39:32 +01:00
Mathieu
438a7cc746 feat(mcp): add @xen-orchestra/mcp package (#9519)
This PR adds the new @xen-orchestra/mcp package — an MCP (Model Context Protocol) server that enables AI assistants (Claude Desktop, Claude Code, or any MCP-compatible client) to query a Xen Orchestra infrastructure in read-only mode.
2026-02-24 17:10:32 +01:00
Florent BEAUCHAMP
31e2ff1b02 feat(backups): only check size for full backups (#9527)
The size is computed from the blocksize * nb allocated blocks during the backups
which is not aligned with the real size on disk
Checking the file on disk is too costly for vhd directory
This will be eventually reimplemented by the next cleanVm rework
2026-02-24 17:04:58 +01:00
Thomas Moraine
44a23fb33f docs(sdn_controller.md): document openssl 3 + sdn upgrade path (#9525)
This pull request documents the manual steps that will be needed for the SDN controller, with the incoming March 2026 update of XCP-ng that will include OpenSSL3.
2026-02-24 13:21:26 +01:00
Pierre Donias
f8501c1a1e feat(xo-web/patches): warn about updating XOSTOR before installing patches (#9517) 2026-02-24 12:07:44 +01:00
spacotte-vates
6cab502260 feat(rest-api): sr scan route (#9514) 2026-02-24 11:05:18 +01:00
Pierre Donias
157e7fe757 feat(xo-web/licenses): display bundle name next to license name (#9512) 2026-02-23 16:20:46 +01:00
OlivierFL
0866a5db81 feat(xo-stack): implement Query Builder (#9488) 2026-02-23 15:35:23 +01:00
Pierre Donias
1c245acca5 feat(xo-web/network): reconfigure management PIF (#9510) 2026-02-23 14:36:18 +01:00
spacotte-vates
58c2c518a3 feat: technical release (#9523)
* feat(@xen-orchestra/fs): 4.6.7

* feat(@vates/types): 1.20.0

* feat(@xen-orchestra/backups): 0.69.0

* feat(@xen-orchestra/backups-cli): 1.1.8

* feat(@xen-orchestra/immutable-backups): 1.0.30

* feat(@xen-orchestra/web-core): 0.43.0

* feat(@xen-orchestra/proxy): 0.29.47

* feat(@xen-orchestra/rest-api): 0.25.0

* feat(@xen-orchestra/vmware-explorer): 0.11.0

* feat(@xen-orchestra/web): 0.41.0

* feat(xo-server): 5.197.0

* feat(xo-server-load-balancer): 0.12.0

* feat(xo-server-netbox): 1.11.0

* feat(xo-server-openmetrics): 1.2.0

* feat(xo-server-sdn-controller): 1.2.1

* feat(xo-web): 5.193.0

* chore(CHANGELOG): update next

---------

Co-authored-by: Pierre Brunet <pierre.brunet@vates.tech>
2026-02-23 10:23:57 +01:00
Elise-F
755ac108f2 feat(xo6): adds a banner when access to the xo server fails (#9375) 2026-02-20 17:12:29 +01:00
Bastien Nollet
f88c0e8be2 feat(rest-api): expose POST vdis (#9492) 2026-02-20 16:24:18 +01:00
Philippe Coval
109e37695c fix(sdn-controller): Support openssl-3 for ca/cert generation (#9507)
OpenSSL-3 (on xcp-ng) requires CA set to true in basicConstraints.

"All CAs should have the CA flag set to true."

For context, here are additional resources:

- https://docs.openssl.org/3.0/man1/openssl-verification-options/#basic-constraints
- https://docs.openssl.org/1.0.2/man1/x509v3_config/#basic-constraints
- https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.9

To connect to OpenSSL 3, certificates should be re-generated if they
already exist (just remove xo-server/data/sdn-controller).

Renewed certs were tested on XCP-ng host on "OpenSSL 1.0.2k-fips"
without regression, so it is safe to assume those ext are designed to
be backward compatible.

Observed issue:

    openssl verify \
      -CAfile .../xo-server/data/sdn-controller/ca-cert.pem \
      .../xo-server/data/sdn-controller/client-cert.pem

    C=XX, L=Default City, O=Default Company LTD
    error 79 at 1 depth lookup: invalid CA certificate
    error .../xo-server/data/sdn-controller/client-cert.pem: \
      verification failed

The expected output is now:

    .../xo-server/data/sdn-controller/client-cert.pem: OK

The expected output is now:

    .../xo-server/data/sdn-controller/client-cert.pem: OK

It was tested along ovsdb-server (Open vSwitch) 2.17.7 :

    openssl s_client \
      -connect  10.1.75.62:6640 \
      -cert .../xo-server/data/sdn-controller/client-cert.pem \
      -key .../xo-server/data/sdn-controller/client-key.pem

    (...)
    CONNECTED(00000003)
    Can't use SSL_get_servername
    (...)
    verify error:num=18:self-signed certificate
    (...)
    SSL handshake has read 2713 bytes and written 3799 bytes
    Verification error: self-signed certificate
    (...)
    New, TLSv1.2, Cipher is AES128-GCM-SHA256
    (...)
    {"id":"echo","method":"echo","params":[]}

Extra info at:

- https://xapi-project.github.io/new-docs/design/pool-certificates/

Origin: https://github.com/vatesfr/xen-orchestra/pull/9507
Relate-to: XCPNG-2710
Relate-to: XO-1986
Signed-off-by: Philippe Coval <philippe.coval@vates.tech>
2026-02-20 09:34:13 +01:00
spacotte-vates
a4b480de4a feat(rest-api): add relaim_space action to sr (#9486) 2026-02-19 17:26:12 +01:00
spacotte-vates
db8addbc45 fix(backups): fix RemoteVhdDisk & DiskChain parent and directory issue (#9508)
* fix(backups): fix RemoteVhdDisk & DiskChain parent and directory issue
2026-02-18 14:06:05 +01:00
Sebastien Marie
9dbc090116 fix(xo-server-sdn-controller): communicate with enabled hosts only (#9442)
permit to have working traffic rules when a pool is in a 'degraded' mode (for
example, one host temporary down or unreachable).

currently, exceptions occurs and nothing happens (no rules added or deleted).
with it, unreachable hosts are ignored and reachable hosts properly updated.
2026-02-17 13:31:44 +01:00
OlivierFL
353678e754 feat(xo-6/internal-links): normalize internal links (#9276) 2026-02-17 08:20:01 +01:00
Florent BEAUCHAMP
a313986694 feat(backups): implements distributed backup and replications (#9433)
The deletion of old backups is done globally
For Full backups/replication:
 * retention is computed on the total of the target
 * choose the storage with the most free size ( if possible, since some backup repositories don't have any size used/ free )
 * one vm is written on the same target ( xva, index, checksum )
 * healthcheck ( if enabled ) will run only on the target that received the data
 * the cleanup of old backups/replication is done on the right target ( but considering the full lenght across all the targets )

For incremental: 
 * delta chain are continued on the same target 
 
For mirror : 
if a vm has multiple backups to transfer during the same run, then
they will all go to teh same remote
 
UX : 
* show a toggle on remote/sr selectors
* toggle is disabled if 1 or less targets are selected
* for backup and mirror backup forms
* show the details in the job list
  

More details on the PR page
2026-02-16 18:17:03 +01:00
Florent BEAUCHAMP
983ac2d2c9 docs(config): explain the config files priorities (#9463) 2026-02-16 18:11:50 +01:00
spacotte-vates
4e50858526 feat(@xen-orchestra/backups): refactor backup merge code (#9300)
this merge code leverage the new classes introduced to handle qcow2, improving our flexibility, and future uses.

this also improve the performance on "block mode" backups (s3/ azure and other storage with the option), since it reliably updates the block allocation table only at the end of the merge instead of each time a new block is detected
2026-02-16 18:04:12 +01:00
Mathieu
c497ae3d38 fix(web): use the store's object type and not type from vates/types (#9421) 2026-02-16 17:01:26 +01:00
spacotte-vates
9d9b5d5e33 feat(rest-api): add plug and unplug pbd routes to rest api (#9477) 2026-02-16 16:52:04 +01:00
OlivierFL
5834070e77 fix(xo-6/xo-lite): missing required prop for UiRadioButtonGroup in new VM creation form (#9504) 2026-02-16 16:50:40 +01:00
Enée DI IORIO
5c5f27f92b fix(xo-stack): fix VM creation on small screen (#9344) 2026-02-16 16:16:11 +01:00
Cyrille G.
8fc04bc03b fix(hub/recipe): remove DHCP option for the Kubernetes recipe (#9454) 2026-02-16 10:20:11 +01:00
Pierre Brunet
87135c7469 feat: add qcow2 support in vdi creation (#9493)
Add qcow2 image-format when size is higher than 2TB. It will still be a VHD if PBD.preferred-image-formats do not list qcow2
2026-02-16 09:56:08 +01:00
Mathieu
fb030a590e docs(openmetrics): update documentation for host uptime, host status and is_control_domain (#9497)
docs(openmetrics): update documentation for host uptime, host status and
is_control_domain
2026-02-16 09:55:02 +01:00
Florent BEAUCHAMP
0be2392519 feat(xo-server/V2V): automatically take a snapshot if needed (#9471) 2026-02-12 17:07:56 +01:00
Florent BEAUCHAMP
546d42427c feat(xo-web): remember V2V connections settings (#9490) 2026-02-12 15:42:47 +01:00
Mathieu
fc86fda13d fix(rest-api): exclude removable storage from top 5 SRs usage (#9495) 2026-02-12 14:04:12 +01:00
Elise-F
b44bed6d10 feat(web-core): new ui rich radio button component created (#9198) 2026-02-12 11:53:35 +01:00
Mathieu
4e66889233 fix(xo-server-openmetrics): bind to localhost instead of 127.0.0.1 (#9489) 2026-02-12 09:51:42 +01:00
Pierre Donias
10bb1741e7 chore: split changelog file (#9390) 2026-02-11 14:57:35 +01:00
Mathieu
a0f890f301 feat(openmetrics): add host uptime metric (#9449) 2026-02-11 11:08:43 +01:00
Pierre Donias
839a484017 fix(xo-web/host/patches): remove obsolete yum install command instruction (#9451) 2026-02-11 10:34:02 +01:00
MlssFrncJrg
dc9e029bdb feat(netbox): support Netbox v4.5.x (#9445)
Fixes #9379
See https://github.com/netbox-community/netbox/releases
2026-02-11 10:31:47 +01:00
Mathieu
f203b1f1e1 feat(rest-api): expose vms/:id/actions/migrate (#9414) 2026-02-11 09:55:51 +01:00
Mathieu
7399a9fec5 feat(openmetrics): add xcp_host_status metric (#9457) 2026-02-10 16:39:40 +01:00
Olivier Lambert
c69b2336f6 feat(xo-server-load-balancer): add migration cooldown to prevent VM flapping (#9388) 2026-02-10 16:11:27 +01:00
Florent BEAUCHAMP
80ab29de29 feat(xo-server): warm migration works with non running VM 2026-02-10 16:08:10 +01:00
Florent BEAUCHAMP
1449a8482e feat(xo-server): vm.warmMigration now return the uuid 2026-02-10 16:08:10 +01:00
Mathieu
ec9806e081 feat(rest-api): add objectType to tasks from actions (#9429) 2026-02-10 15:22:03 +01:00
Pierre Brunet
c6c31524e1 build: bump all dependencies (#9481) 2026-02-10 14:42:37 +01:00
Mathieu
49fce9b61f feat(openmetrics): add is_control_domain label to differentiate dom0 VMs (#9474) 2026-02-10 14:41:42 +01:00
Thomas Moraine
f52a35c093 docs(metadata_backup): clarify the two types of metadata backups (#9487) 2026-02-10 14:11:42 +01:00
Mathieu
9d74585bd4 fix(@xen-orchestra/web): fix build issue (#9482)
Introduced by #7851
2026-02-10 10:59:37 +01:00
Mathieu
12f4360100 feat: release 6.1.2 (#9485) 2026-02-10 10:01:17 +01:00
Pierre Brunet
771ce8c6d5 fix(backups/s3): add fallback if deleteObjects unsupported (#9450) 2026-02-10 09:29:53 +01:00
Mathieu
41c9a6c7c2 feat: release 6.1.2 (#9480) 2026-02-09 09:43:51 +01:00
Florent BEAUCHAMP
7e08a016bd fix(backups): ensure vm created by backups are removable by backups : halthcheck/replication (#9473) 2026-02-06 15:15:37 +01:00
Mathieu
91c5d98489 feat: release 6.1.1 (#9470) 2026-02-05 16:59:00 +01:00
Mathieu
b895f48a4f feat: technical release (#9469) 2026-02-05 16:01:47 +01:00
Mathieu
d68d878747 fix(changelog): add missing package to release (#9468) 2026-02-05 15:46:55 +01:00
Mathieu
5155a9d5f1 feat: technical release (#9467) 2026-02-05 14:58:41 +01:00
Florent BEAUCHAMP
7437427fae fix(backups): refresh the snapshot name after removing the temp name (#9466)
introduced by #9434
2026-02-05 13:13:39 +01:00
spacotte-vates
a16b4cb540 feat(rest-api): add post vifs and delete vifs/:id routes to rest api (#9393) 2026-02-05 12:05:26 +01:00
Florent BEAUCHAMP
21508c97bc fix(immutable-backups): really protect pool metadata
Apply suggestion from @MathieuRA

Co-authored-by: Mathieu <70369997+MathieuRA@users.noreply.github.com>

Apply suggestion from @MathieuRA

Co-authored-by: Mathieu <70369997+MathieuRA@users.noreply.github.com>
2026-02-05 10:16:03 +01:00
Florent BEAUCHAMP
e8ff60c8bc docs(immutable-backups): fix app name in documentation
alos changelog that wasn't in previous PR
2026-02-05 10:16:03 +01:00
Mathieu
153efccb85 fix(rest-api): pools/:id/actions/create_vm install.repository was ignored (#9461) 2026-02-05 10:10:39 +01:00
Joris K
a99e1aa72d fix(xo6/lite): only count running vms for cpu provisioning in host dashboard (#9459) 2026-02-05 09:27:20 +01:00
Mathieu
dd5f0cdaa9 fix(rest-api): fix backup-archives href (#9460)
* fix(rest-api): fix backup-archives href

* changelog

---------
2026-02-04 17:10:37 +01:00
Mathieu
ea142b50a4 fix(rest-api): only count running vms for cpu provisioning in pool dashboard (#9456)
See Forum#101359
2026-02-04 17:05:16 +01:00
Pierre Briffaux
5b5a0a72cc fix(web-core): enable legacy clipboard support for http (#9426) 2026-02-04 16:41:16 +01:00
Joris K
84a16ef85f fix(xo6): fix issue with alarms display (#9458) 2026-02-04 11:18:38 +01:00
vates-bot
b983e6d4fa feat(web-stack/i18n): update Xen Orchestra 6 (#9440) 2026-02-04 10:40:27 +01:00
Joris K
a0e771f09c feat(xo6): update site dashboard (#8964) 2026-02-03 17:51:06 +01:00
Florent BEAUCHAMP
9bd91aea43 feat(fs): autodetect object lock/immutability on s3 (#9431)
* add a setting to declare remote immutable ?immutable=true in the URL 
* autodetect object lock/immutabiltiy on s3
2026-02-03 17:12:15 +01:00
Florent BEAUCHAMP
0a042d2f54 fix(xapi): correctly block operations
blocked operation on vm creation does not seem to work

also importIncrementalVm was not unlocking it at the end
2026-02-03 15:59:24 +01:00
Florent BEAUCHAMP
c610951450 fix(backups): set other_config on incremental replication vm creation
this will ensure that any partial transfer will ben cleaned on next run
2026-02-03 15:59:24 +01:00
Florent BEAUCHAMP
a831ac7b21 chore(backups): remove old todo 2026-02-03 15:59:24 +01:00
Florent BEAUCHAMP
d0f71d4ec7 fix(backups): don't leave dangling snapshots (#9434)
when the connection to the host break between the snapshot() and
the update of other config
this could leave snapshot that will never be cleaned by any backup job
The fix is that the snapshot will be named with a specific name, it
will cause an issue if a user reuse this name
2026-02-03 15:55:11 +01:00
Joris K
c1a9bc82fb feat(xo6/vm): add more actions button to delete VM and take snapshot (#9410) 2026-02-03 13:05:49 +01:00
Dan Pollak
601cc3e9fd fix(immutable-backups): Fix typo in sample config file (#9444)
* fix(immutable-backups): default config.toml missing quotes

Signed-off-by: Dan Pollak <danpollak2@gmail.com>

* fix(immutable-backups): Locate config file

Signed-off-by: Dan Pollak <danpollak2@gmail.com>

* Update CHANGELOG

Signed-off-by: Dan Pollak <danpollak2@gmail.com>

* Revert "fix(immutable-backups): Locate config file"

This reverts commit e9e367474d.

* Update CHANGELOG.unreleased.md

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@gmail.com>

---------

Signed-off-by: Dan Pollak <danpollak2@gmail.com>
Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@gmail.com>
2026-02-03 13:03:09 +01:00
MathieuRA
79d1f42dd7 fix(rest-api): do not create auth task when using auth token 2026-02-03 09:20:25 +01:00
MathieuRA
64c366838a fix(rest-api): don't send message updates when subscribing to alarm 2026-02-03 09:20:25 +01:00
MathieuRA
85b3162be6 fix(rest-api): set memory limit per sse client 2026-02-03 09:20:25 +01:00
Mathieu
eb020e250b docs(openmetrics): add SR capacity metrics and complete VM metrics (#9438)
documentation
2026-01-30 16:50:55 +01:00
Olivier Lambert
56142085e5 feat(xo6/settings): add various themes (#9387) 2026-01-30 15:21:00 +01:00
Mathieu
2838bde875 feat(lite): 0.18.0 (#9437) 2026-01-29 17:56:53 +01:00
Mathieu
fa110ed9c9 feat: release 6.1.0 (#9436) 2026-01-29 17:31:28 +01:00
Mathieu
d3187c06b5 feat: technical release (#9432) 2026-01-29 10:57:33 +01:00
Florent BEAUCHAMP
2f41ccffe8 feat(xapi): try to disconnect vdi if a vm_destroy fail (#9415)
happenned during a backup for HK
only solve case where the disk can be manually disconnected from DOM0
2026-01-29 10:02:59 +01:00
Elise-F
a9e45fcb8e feat(xo6/Lite): add VTPM support (#9389) 2026-01-29 09:46:41 +01:00
Pierre Briffaux
1e69c25ba5 fix(xo6/new-vm): fixed disk naming from template (#9294) 2026-01-28 16:13:58 +01:00
Joris K
9f852365fc feat(xo6/vm): add VMs actions to manage lifecycle (#9403) 2026-01-28 16:01:57 +01:00
Enée DI IORIO
05824cb8a6 feat(web-core): normalize icons creation from design system (#8989) 2026-01-28 15:29:40 +01:00
vates-bot
638ee77aff feat(web-stack/i18n): update Xen Orchestra 6 (#9330) 2026-01-28 10:36:59 +01:00
Bastien Nollet
08a560a8da fix(xo-server): fix NFSv4 ISO SR creation (#9425) 2026-01-28 10:02:44 +01:00
Elise-F
e77a0e43f1 feat(xo6): remove text ellipsis on sides panel (#9238) 2026-01-27 18:29:35 +01:00
Florent BEAUCHAMP
8069adf26d feat(xo-server): expose an api method to export a disk from vmware 2026-01-27 16:04:05 +01:00
Florent BEAUCHAMP
1241977599 fix(vhd-cli): raw need to load BAT before working 2026-01-27 16:04:05 +01:00
Florent BEAUCHAMP
4ca314fbc7 fix(nbd-client): discrepency when using hasblock/getblockindexes 2026-01-27 16:04:05 +01:00
Mathieu
13ab1cce86 feat(rest-api): add POST /vdis/{id}/actions/migrate endpoint (#9408) 2026-01-27 10:58:28 +01:00
Mathieu
9f7797fdda feat(rest-api): add VBD connect/disconnect endpoints (#9399) 2026-01-27 10:25:04 +01:00
Mathieu
21767145bd feat: technical release (#9422) 2026-01-23 14:44:23 +01:00
OlivierFL
6805e16108 refactor(xo6): project architecture (#9371) 2026-01-23 12:02:43 +01:00
Mathieu
ecd49e69a7 fix(xo-server-openmetrics): add sr_name label to VM disk metrics (#9353)
Introduced by 0005a4f4cc1990e9a20eb48becb5943b5bf15c43
2026-01-23 10:24:29 +01:00
Florent BEAUCHAMP
a5f5a0273e fix(boackups): reverting a VM from a backup shouldn't leave dangling other_config (#9398) 2026-01-23 10:18:52 +01:00
Mathieu
ddc9bf3211 feat(rest-api): add DELETE /vbds/{id} endpoint (#9394) 2026-01-22 14:31:25 +01:00
Mathieu
75db97f9ea feat(@vates/types,xo-server-openmetrics): use centralized XoApp type (#9404)
XO-1810
2026-01-22 11:19:49 +01:00
Mathieu
07c1c02f03 feat(rest-api): add secureBoot param for vm creation (#9417) 2026-01-22 11:13:42 +01:00
Mathieu
dbe8e0ce26 fix(rest-api): vms/:id/start fix request body being ignored (#9416)
introduced by 0202b56d33
2026-01-21 17:42:20 +01:00
Florent BEAUCHAMP
82cd861479 feat(backups): expose LTR tags to VM backups (#9364) 2026-01-21 17:34:50 +01:00
Pierre Briffaux
32f8c1a25f feat(xo-lite): scroll to selected item in treeview (#9332) 2026-01-21 16:52:06 +01:00
Florent BEAUCHAMP
9e7813319a fix(vhd-lib/qcow2): better handling of unaligned blocks (#9407) 2026-01-21 15:52:33 +01:00
Mathieu
caba00a2b9 feat(rest-api): add createVtpm parameter to VM creation endpoint (#9412) 2026-01-21 15:25:03 +01:00
Pierre Briffaux
927561123a feat(xo6/treeview): scroll to selected item in treeview (#9268) 2026-01-21 11:51:25 +01:00
Mathieu
097c2bd1d9 feat(rest-api): add POST /vbds endpoint to create VBDs (#9391) 2026-01-20 14:36:24 +01:00
MlssFrncJrg
5288b6d848 feat(xo6/vm): add 'Change State' button on VM page (#9317) 2026-01-20 11:54:01 +01:00
Pierre Brunet
263059be22 fix: update node-tar to handle CVE-2026-23745 (#9406)
* fix: update node-tar to handle https://github.com/advisories/GHSA-8qq5-rm4j-mr97

* chore: update changelog
2026-01-20 11:42:57 +01:00
Florent BEAUCHAMP
c83df8a342 fix(backups): more filtering on disk snapshot before starting (#9405)
from ticket 33058
2026-01-19 11:20:09 +01:00
Bastien Nollet
4f959e2d22 fix(backup-reports): prevent succesful backups being sometimes repor… (#9400)
* fix(backup-reports): prevent succesfull backups being sometimes reported as interrupted

* increase delay after user feedback

* change comment according to delay change

---------

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2026-01-19 11:06:36 +01:00
Florent BEAUCHAMP
ee6df8c979 chore(backups): remove unused and broken code (#9367)
note the missing await for the adapter._handler.readFile
this method is correctly implemented in MixinRemoteWriter
2026-01-19 10:58:16 +01:00
Mathieu
0a0ae5dc7e feat(xo-server/rest-api): add management interface reconfigure API (#9369) 2026-01-14 16:29:43 +01:00
Mathieu
58b6215c41 fix(@xen-orchestra/rest-api): check length instead of undefined (#9401)
Introduced by #9395
2026-01-14 15:17:59 +01:00
Mathieu
0e56c5e2bd fix(@xen-orchestra/rest-api): add isEmpty prop for xoa dashboard (#9395) 2026-01-14 11:27:28 +01:00
Florent BEAUCHAMP
0b44b1b52b fix(backups): size estimation of full backups (#9396) 2026-01-14 10:38:52 +01:00
Bastien Nollet
4c7b8b194f feat(vhd-cli): prevent using invalid options (#9386) 2026-01-13 16:52:36 +01:00
Mathieu
ac2ed8880a feat(@vates/types): expose XoApp (#9322) 2026-01-12 15:27:53 +01:00
Raven
6fc30c899d docs(xo): Add instructions for fetching deps on RHEL 10 distros (#9383)
RHEL 10 and co. (AlmaLinux, Rocky, CentOS) stopped shipping redis and
moved some packages around. Update the docs to reflect that.
2026-01-12 11:56:48 +01:00
Florent BEAUCHAMP
05e1298eec fix(backups): rework snapshot listing (#9377) 2026-01-12 08:17:54 +01:00
Mathieu
d20561fadc fix(xo-server-openmetrics): use correct timestamp property name in computeVmCpuUsageFallback (#9384)
Introduced by 29fa4b2
2026-01-09 14:12:28 +01:00
Mathieu
53a3e2a70f feat(xo-server-openmetrics): add SR capacity metrics (#9360)
* feat(xo-server-openmetrics): add SR capacity metrics

* fix(xo-server-openmetrics): refactor SR types and add host labels for
local SRs

---------

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2026-01-09 10:25:10 +01:00
Mathieu
2c90eb5ba8 fix(rest-api): string and not 'string' (#9382) 2026-01-09 10:24:37 +01:00
Mathieu
c3e78ebdfd fix(rest-api): fix getVmVdis and enhance the type (#9381) 2026-01-09 10:24:20 +01:00
Mathieu
264b93d502 feat(open-metrics): compute vm_cpu_usage from per-core metrics (#9331)
* feat(open-metrics): compute vm_cpu_usage from per-core metrics

* refactor(xo-server-openmetrics): combine conditional checks in
computeVmCpuUsageFallback
2026-01-09 10:23:26 +01:00
Elise-F
d0abf8810c feat(web-core): update UI title component (#9292) 2026-01-09 10:09:25 +01:00
Mathieu
bdd9b26651 fix(rest-api): various vm dashboard fixes (#9380) 2026-01-08 17:30:12 +01:00
Elise-F
66d5014755 feat(xo6): removal borders of sidebar in mobile (#9366) 2026-01-08 16:49:22 +01:00
mathieuRA
8e2d9f1022 feat(xo6): use define-job for server creation 2026-01-07 16:04:21 +01:00
mathieuRA
566bdeda63 feat(xo6): use define-job for VM creation 2026-01-07 16:04:21 +01:00
mathieuRA
e86d71fc31 feat(xo6): implement async action monitoring 2026-01-07 16:04:21 +01:00
Mathieu
4827f33305 feat(@xen-orchestra/rest-api): update xoa dashboard endpoint (#9207)
XO-1462
2026-01-07 14:39:48 +01:00
Elise-F
9b6d1089f4 feat(web-core): update text logo with custom font (#9298) 2026-01-06 16:01:34 +01:00
Elise-F
d1f8f61791 feat: release 6.0.3 (#9374) 2026-01-06 15:20:32 +01:00
Elise-F
dbf00ba0cd feat: technical patch release (#9373) 2026-01-06 14:33:41 +01:00
Pierre Brunet
44a0d14e88 fix (XO Hub): read getResourceCatalog removed during refactor (#9362) 2026-01-06 14:08:48 +01:00
Elise-F
0b52a05bb3 feat: technical patch release (#9372) 2026-01-06 11:51:13 +01:00
Florent BEAUCHAMP
407591d74d fix(vhd-lib): more logs on backup edge cases (#9326)
* more logs for vdi attached to multiples VMs failing _removeUnusedSnapshots
* more logs for incomplete reading of a disk
2026-01-06 10:57:41 +01:00
Enée DI IORIO
0b29586626 fix(xo6/site-dashboard): replace raw text by translated text in Pools Status (#9329) 2026-01-05 16:38:07 +01:00
Florent BEAUCHAMP
42eb8664be fix(xo-server/v2v): more block alignment fixes (#9293)
from #45336
2026-01-05 16:01:59 +01:00
Florent BEAUCHAMP
1d8c429532 fix(backups): cleanup data on restore (#9361) 2026-01-05 12:10:44 +01:00
Joris K
c51df2239f fix(xo6/host): correction of a translation key (#9359) 2026-01-05 10:34:06 +01:00
Mathieu
b89c26459c feat: release 6.0.2 (#9356) 2025-12-23 15:41:32 +01:00
Mathieu
a1333c07d7 feat: technical release (#9355) 2025-12-23 14:45:33 +01:00
Mathieu
5e8e5beba1 fix(xo-server): fix netdata url (#9354) 2025-12-23 14:04:08 +01:00
Mathieu
9169e11f35 fix(xo-server): fix don't call the updater with secure protocol (#9349) 2025-12-23 11:46:28 +01:00
Mathieu
6908001bd8 fix(xo-server): bypass session auth for /openmetrics routes 2025-12-22 18:02:29 +01:00
Mathieu
53ec9a62a8 fix(xo-server-openmetrics): comply with OpenMetrics 1.0 specification 2025-12-22 18:02:29 +01:00
Joris K
4a0590b7e9 fix(web core): fix task item tree (#9352)
Introduced by 3eb9efe

Fix an issue on taskitem tree only on firefox
2025-12-22 11:22:10 +01:00
Thierry Goettelmann
d358aed897 fix(web-core/lite): UiInput v-model modifiers and reactivity (#9282)
Fixes randomly blank input
2025-12-22 11:03:15 +01:00
Florent BEAUCHAMP
5fcb6db32e docs(xo5): use absolute path (#9347) 2025-12-19 17:29:54 +01:00
Mathieu
1de60c52b4 docs(openmetrics): add user documentation for Prometheus integration (#9333) 2025-12-19 14:17:38 +01:00
Elise-F
55243b65a2 feat: release XO 6.0.1 (#9345) 2025-12-19 14:03:18 +01:00
Elise-F
0247b3b9b3 feat: technical release (#9343) 2025-12-19 11:48:25 +01:00
Mathieu
6dae628e0d Fix/openmetrics hardcode server config (#9342)
* docs(changelog): add entry for xo-server-openmetrics security fix

* Apply suggestion from @fbeauchamp

---------

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2025-12-19 11:28:51 +01:00
Mathieu
7827b27d63 fix(xo-server): fix http.proxies when redirectToHttps=true (#9339)
introduced by 7c1764a39e
fix [forum#11681](https://xcp-ng.org/forum/post/100852)
2025-12-19 11:11:35 +01:00
Mathieu
86ddff46a9 fix(xo-server-openmetrics): hardcode port and bind address for security (#9337) 2025-12-19 10:35:50 +01:00
Pierre Donias
476f4e5f1f fix(xo-server/signin-otp): pass xo5Mount param to Pug template (#9340) 2025-12-19 10:12:24 +01:00
Elise-F
8448b4d5a6 feat: release 6.0.0 (#9336) 2025-12-18 16:36:07 +01:00
Elise-F
28952a0a4a feat: technical release (#9334) 2025-12-18 12:21:33 +01:00
Pierre Brunet
19f973fcd3 fix: replace deprecated endpoint call (#9316)
Call to deprecated route restore/logs replaced by restore-logs/
2025-12-18 09:20:48 +01:00
OlivierFL
d9fe9b603f fix(xo-6/account-menu): license url was not redirecting properly to XO 5 (#9321) 2025-12-17 17:49:46 +01:00
OlivierFL
13a462429e fix(xo-6/xo-lite/new-vm-form): change "memory" section to "resource management" for better clarity (#9309) 2025-12-17 17:45:08 +01:00
Mathieu
bfe788e096 fix(@xen-orchestra/web-core): fix race condition on reactive store (#9307)
Introduced by 5d28710ab3
2025-12-17 17:37:15 +01:00
Mathieu
980e4b5887 Feature openmetrics (#9323) 2025-12-17 17:17:53 +01:00
Florent BEAUCHAMP
0d8cb990c6 fix(xen-api): don't eat relevant error during import (#9327)
don't eat relevant error during import

SR contains the full list of the VDI on the SR
host contains the full list of the VM on the HOST
2025-12-17 16:40:30 +01:00
vates-bot
7794adfb89 feat(web-stack/i18n): update Xen Orchestra 6 (#9305) 2025-12-17 16:14:27 +01:00
Mathieu
fb67e8eea7 fix(xo-server): report 0 RAM usage for halted hosts (#9272) 2025-12-17 16:01:17 +01:00
Pierre Brunet
1316a811f5 feat(plugins/backup-report): add custom subject in plugin settings (#8973) 2025-12-17 15:41:11 +01:00
Joris K
c6f47f9ef3 feat(xo6/vm): implement tasks view and side panel information (#9313) 2025-12-17 15:31:45 +01:00
Joris K
c2a95467f8 feat(xo6/host): implement tasks view and side panel information (#9311) 2025-12-17 15:25:45 +01:00
Joris K
4d58ff885a feat(xo6/pool) implement tasks view and side panel information (#9312) 2025-12-17 15:15:08 +01:00
Florent BEAUCHAMP
3892eee7a0 fix(xo-server): changelog after xosan removing (#9302) 2025-12-17 15:01:33 +01:00
Joris K
15b9e13ed2 feat(xo6/site): implement tasks view and side panel information (#9063) 2025-12-17 14:50:34 +01:00
Mathieu
e504b8af0f fix(@xen-orchestra/rest-api): fix duplicated entries with ndjson (#9320) 2025-12-17 12:02:22 +01:00
Mathieu
efc751c54e fix(xo-server): check certificate instead port to define secure protocol (#9324) 2025-12-17 11:35:50 +01:00
Mathieu
8b4c953ab0 docs(configuration): update doc for XO5 by default (#9296) 2025-12-17 09:44:50 +01:00
OlivierFL
32b3c0b5fc refactor(web-stack/translations): remove duplicates, improve pluralization, improve actions, clean translations (#9295) 2025-12-16 15:51:57 +01:00
Mathieu
7c1764a39e fix(xo-server): fix proxied urls when using ssl (#9314)
introduced by 03cea8b24d

Fixed [forum#11681](https://xcp-ng.org/forum/topic/11681/xo5-breaks-after-defaulting-to-xo6-from-source/16?_=1765875445829)
2025-12-16 11:23:54 +01:00
Pierre Brunet
b248cddece fix: perf-alert unload (#9306) 2025-12-16 10:19:50 +01:00
Florent BEAUCHAMP
df2dc3a710 fix(xapi): don't fail detecting image format on suspend disk (#9308)
from #48758

suspend_vdi  disk created for snapshot with memory are snapshot
but don't have a parent, thus failing the check 

introduced by c870bb653a
2025-12-15 17:08:26 +01:00
Elise-F
227063341e feat: technical release (#9310) 2025-12-15 16:00:02 +01:00
Mathieu
a4ab2e8b06 feat(@xen-orchestra/rest-api): add reactivity on task store (#9271) 2025-12-15 10:51:58 +01:00
Mathieu
9f3872a5f1 feat(@xen-orchestra/rest-api): ability to subscribe to task changes (#9269) 2025-12-12 18:07:39 +01:00
mathieuRA
e2d727abe5 fix(@xen-orchestra/rest-api): fix cannot read property of undefined on VM dashboard 2025-12-12 17:27:35 +01:00
mathieuRA
11720d96f1 fix(@xen-orchestra/rest-api): fix cannot convert undefined or null to object 2025-12-12 17:27:35 +01:00
vates-bot
859ebb892d feat(web-stack/i18n): update Xen Orchestra 6 (#9243) 2025-12-12 13:06:18 +01:00
Pierre Brunet
035ee8fb8e refactor: remove all xosan components (#9248)
XOSAN is not used anymore by clients. We can safely remove it from code base to clean it up.
2025-12-11 17:14:04 +01:00
Mathieu
bc69046bdd feat: technical release (#9301) 2025-12-11 15:22:37 +01:00
spacotte-vates
debde3f16e fix(xo-server): fix guiRoute generation (#9299) 2025-12-11 15:09:08 +01:00
Joris K
3eb9efe1f2 feat(web-core): add UiTaskItem and UiTaskList components (#9009) 2025-12-10 17:18:57 +01:00
Mathieu
7cf9726876 feat: technical release (#9297) 2025-12-10 15:08:46 +01:00
Pierre Briffaux
64d9f8c39e feat(xo6): update links to XO-5 (#9214) 2025-12-10 14:38:02 +01:00
Mathieu
03cea8b24d feat(xo-server): set XO6 the default page (#9212) 2025-12-10 14:29:49 +01:00
Bastien Nollet
4b4c9b91f3 fix(xo-web): allow offline delta backups (#9228) 2025-12-10 13:19:09 +01:00
Mathieu
907ee7d88c feat(@xen-orchestra/web): display modal of the first connection (#9230) 2025-12-10 12:52:01 +01:00
Enée DI IORIO
dbdc4cac4a fix(web-core): fix pagination placement (#9264) 2025-12-10 10:16:33 +01:00
Bastien Nollet
19c0b958fc fix(backups): fix incorrect info on mirror backups (#9286)
From forum#11623
2025-12-10 09:39:03 +01:00
Florent BEAUCHAMP
e5d2e83de6 fix(backups): cleanup .vhd.checksum files (#9291) 2025-12-10 09:19:58 +01:00
spacotte-vates
85596da792 fix(xo-server): improve handling of xapi snapshots (#9231)
In some cases, when converting an xapi object into an xo object, the xapi.is_a_snapshot param may be set to false even tho the xapi.snapshotOf refers to another xapi object. Handle this case better.
2025-12-10 09:16:18 +01:00
Pierre Brunet
230749a3f3 feat(fs/s3): DeleteObjectsCommand (#9281) 2025-12-10 09:08:35 +01:00
Bastien Nollet
b667bc8a23 fix(@xen-orchestra/fs): avoid retrying on some statuses (#9188) 2025-12-09 17:50:32 +01:00
Mathieu
76abf1ed73 docs(restapi.md): update REST API docs (#9284) 2025-12-09 14:18:51 +01:00
Mathieu
54930ee07f fix(@xen-orchestra/rest-api): vmProtected -> vmProtection (#9288) 2025-12-09 11:53:18 +01:00
Mathieu
90295ca924 feat(@xen-orchestra/web): redirect non admin to xo5 (#9219) 2025-12-09 11:48:04 +01:00
Mathieu
e281c536d3 feat: release 5.113.2 (#9290) 2025-12-09 11:32:27 +01:00
Mathieu
efa5b1a71e feat(@xen-orchestra/rest-api): display URL of the openApi spec (#9285) 2025-12-09 11:10:10 +01:00
Mathieu
62aaa21ced feat: technical release (#9289) 2025-12-09 10:19:09 +01:00
Florent BEAUCHAMP
dfbf654159 fix(xo-server-perf-alert): improve email alert subject (#9283) 2025-12-08 18:03:26 +01:00
spacotte-vates
4bc614ffd1 fix(xo-server-auth-oidc): fix groups import on string (#9280)
Introduced in 6cd0710

For some OIDC auth provider, when the user only has 1 group, the idToken groups param is a string and not an array.
2025-12-08 15:17:09 +01:00
Thierry Goettelmann
1640a6ae00 feat(web-stack): enhance tree lib and add persistence to XO6 navigation tree (#9277) 2025-12-05 15:56:05 +01:00
Thierry Goettelmann
66b677af28 feat: migrate XO Lite tables. introduces new tables. clean code. (#9265) 2025-12-03 17:31:24 +01:00
Thierry Goettelmann
655bda4789 feat(web-stack): table system iteration (composition over config) (#9247) 2025-12-02 17:57:54 +01:00
Thierry Goettelmann
07e605162a refactor(web-core): merge busy prop into type in VtsStateHero (#9273) 2025-12-02 17:36:29 +01:00
Pierre Donias
ce98c3deba fix(web/vdis): xo 5 fallback URL query should start with a "?" (#9267)
Introduced by #9232
See https://xcp-ng.org/forum/post/100170
2025-12-02 10:24:44 +01:00
MlssFrncJrg
71fa8924d7 feat: release 5.113.1 (#9263) 2025-11-28 13:01:09 +01:00
MlssFrncJrg
9c78018662 feat: technical release (#9262)
* feat(xo-server-perf-alert): 1.0.3

* chore(CHANGELOG): update next
2025-11-28 12:16:37 +01:00
Florent BEAUCHAMP
72f31d412d fix(xo-server-perf-alert): compute core average if cpu_usage is missing (#9258) 2025-11-28 11:51:55 +01:00
MlssFrncJrg
ef60450dc8 feat(lite): 0.17.0 (#9261) 2025-11-28 10:11:57 +01:00
MlssFrncJrg
f4988c00af feat(lite): 0.17.0 (#9260) 2025-11-27 18:06:40 +01:00
MlssFrncJrg
43c9f574fd feat: release 5.113.0 (#9259) 2025-11-27 17:36:38 +01:00
MlssFrncJrg
a15eeeed5b feat: technical release (#9257) 2025-11-27 16:29:04 +01:00
Florent BEAUCHAMP
14a72a2612 fix(xo-server-perf-alert): better handling of VM not loaded in xapi and undefined (#9256) 2025-11-27 16:16:35 +01:00
Mathieu
68554b3e0a fix(@xen-orchestra/core): predicate early return if undefined (#9255)
introduced by 5d28710ab3
2025-11-27 16:05:35 +01:00
MlssFrncJrg
91d5107e1d feat: technical release (#9254)
* feat(@xen-orchestra/web): 0.34.1

* feat(xo-server-perf-alert): 1.0.1

* chore(CHANGELOG): update next
2025-11-27 15:03:51 +01:00
Florent BEAUCHAMP
8e46b83d3e fix(xo-server-perf-alert): copy template to dist and fix imports (#9253) 2025-11-27 14:47:08 +01:00
Mathieu
6105681df2 fix(@xen-orchestra/web): add missing bios_strings field (#9252)
introduced by 0c92bf559a
2025-11-27 14:41:32 +01:00
Thomas Moraine
05713d16c0 docs(proxy.md): explain error message for incompatible proxies (#9245)
Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2025-11-27 13:24:06 +01:00
MlssFrncJrg
1c6bc86794 Release (#9251)
* feat(vhd-lib): 4.14.5

* feat(@vates/nbd-client): 3.2.2

* feat(@xen-orchestra/web-core): 0.35.0

* feat(@xen-orchestra/rest-api): 0.21.1

* feat(@xen-orchestra/vmware-explorer): 0.10.6

* feat(@xen-orchestra/web): 0.34.0

* feat(xo-server): 5.193.1

* feat(xo-server-auth-oidc): 0.4.0

* feat(xo-server-perf-alert): 1.0.0

* feat(xo-web): 5.191.0

* chore(CHANGELOG): update next

* feat(vhd-cli): 1.0.0

* add missing lib
2025-11-27 13:23:09 +01:00
Florent BEAUCHAMP
21b4d1ee34 docs(xo-server): guidelines (#8583)
Co-authored-by: Mathieu <70369997+MathieuRA@users.noreply.github.com>

Co-authored-by: Bastien Nollet <150058367+b-Nollet@users.noreply.github.com>

Co-authored-by: Mathieu <goore.csmoviz@gmail.com>

---------

Co-authored-by: Mathieu <70369997+MathieuRA@users.noreply.github.com>
Co-authored-by: Bastien Nollet <150058367+b-Nollet@users.noreply.github.com>
Co-authored-by: Mathieu <goore.csmoviz@gmail.com>
2025-11-27 11:45:22 +01:00
MlssFrncJrg
0c92bf559a feat(xo6/web): add boot firmware at vm creation (#9158) 2025-11-27 10:51:36 +01:00
Mathieu
5d28710ab3 feat(@xen-orchestra/web): implement reactivity for sub-stores (#9250) 2025-11-27 09:42:29 +01:00
Thierry Goettelmann
3126688d5e fix(web): charts reload every 30 seconds (#8939) 2025-11-27 09:24:51 +01:00
Elise-F
e70fe39934 feat(xo6): implement vdis tab page (#9232) 2025-11-26 18:33:41 +01:00
Enée DI IORIO
0af02941ed feat(xo6/pool/hosts): refactoring pool hosts page (#9218) 2025-11-26 17:54:10 +01:00
Florent BEAUCHAMP
779876ede9 feat(xo-server-perf-alert): improve performances (#9070) 2025-11-26 17:25:55 +01:00
Enée DI IORIO
dd8bb763e5 fix(xo-stack/resourceManagement): fix resourceManagement card if controllerMemory is undefined (#9197) 2025-11-26 17:04:42 +01:00
Enée DI IORIO
5e6c19b39f fix(web-core): add tooltip on UiTag (#9184) 2025-11-26 16:05:40 +01:00
Mathieu
dbb76cce6e fix(@xen-orchestra/rest-api): status code 400 instead of 500 for CM parse error (#9244)
See https://team.vates.fr/vates/pl/uu1c3r4bzbnx7gjon8xf6wmzsc
XO-1569
2025-11-26 15:15:53 +01:00
Mathieu
5cf1de5356 feat(xo6): implement reactivity for XAPI objects (#9183) 2025-11-26 14:20:50 +01:00
OlivierFL
bed27e5ffa fix(xo-6/dashboard): prevent double scrollbar in alarms lists (#9246) 2025-11-26 12:00:42 +01:00
Mathieu
bff4e4d4e2 fix(@xen-orchestra/rest-api/backup-archive): fix cannot convert undefined or null to object (#9240)
introduced by 544590a
fix zammad#48178
2025-11-26 09:47:45 +01:00
Enée DI IORIO
97d202dae0 feat(web-core): update UiLabel from V1 to V3 (#9201) 2025-11-26 09:40:20 +01:00
spacotte-vates
460bbc4569 fix(nbd-client): fix esxi 6 warm migration (#9223) 2025-11-26 09:21:19 +01:00
Bastien Nollet
708089622d feat(vhd-cli): accept all types of remotes (#9235) 2025-11-26 09:02:08 +01:00
OlivierFL
1818a031af feat(xo6/header): add DC Scope and DC NetScope buttons (#9242) 2025-11-26 08:23:56 +01:00
Mathieu
5cc6154bd2 fix(@xen-orchestra/rest-api): remove unused import (#9241) 2025-11-25 16:14:15 +01:00
spacotte-vates
6cd0710f89 feat(xo-server-auth-oidc): import user groups from OIDC (#9206) 2025-11-25 16:11:47 +01:00
Cyrille G.
65858c9ce7 feat(hub/recipes): add support for XO CCM and custom CIDR configurations in Pyrgos form (#9236) 2025-11-25 12:10:27 +01:00
Florent BEAUCHAMP
6277a12c4a fix(nbd-client): nbddisk must emit full size block (#9233) 2025-11-25 11:57:41 +01:00
Thomas Moraine
16de2e9fca docs(configuration.md): document changing the default XO interface (#9225) 2025-11-25 10:56:59 +01:00
Pierre Donias
39fa0f041f feat(xo-web/disk): (dis)connectVbd → (dis)connectVbds (#9211)
See Zammad#47676

Always use bulk methods for connecting or disconnecting a VBD so that
confirmation modals behaviours are mutualized.
2025-11-25 10:51:52 +01:00
vates-bot
ca3a7540f4 feat(web-stack/i18n): update Xen Orchestra 6 (#9165) 2025-11-25 10:40:59 +01:00
MlssFrncJrg
0bac1d3e0e feat: technical release (#9237) 2025-11-24 17:30:39 +01:00
MlssFrncJrg
99f221ccbf feat(xo-6): inform user about VM creation limitations (#9203) 2025-11-24 14:51:52 +01:00
Pierre Brunet
f24952f967 chore(xo-web): update marked to more recent version (#9227)
Fix https://github.com/vatesfr/xen-orchestra/security/dependabot/27
2025-11-24 14:36:08 +01:00
Pierre Brunet
566b210d1e chore(xo-web): update ansi_up (#9226) 2025-11-24 13:55:45 +01:00
Florent BEAUCHAMP
5fc2e4e8e5 fix(nbd-client): wrong comment leads to bad code (#9222) 2025-11-24 10:44:58 +01:00
Mathieu
65de89c7ee fix(@xen-orchestra/rest-api): don't return job for /backup-jobs endpoint (#9210)
* fix(@xen-orchestra/rest-api): don't return job for /backup-jobs endpoint
2025-11-24 10:37:06 +01:00
Bastien Nollet
1c55ca61e2 feat(xo-server-load-balancer): affinity tags (#9116)
Add Affinity rule option to the load balancer plugin.
This options ensures some VMs are positioned on the same host, when it's possible.

It's important to be careful while configuring the Affinity tags and Anti-affinity tags fields, otherwise it may lead to conflicts between the two behaviours. In such case, affinity takes precedence over anti-affinity.
2025-11-24 10:29:29 +01:00
Mathieu
ed5bd4d42e Feat/use vif move for network change (#9221)
* feat(vif): use VIF.move for network changes instead of delete+create
2025-11-24 09:56:00 +01:00
Julien Porschen
488eec9d1c fix(backups): handle LVM on GPT partitions #7029 (#9182)
* fix(backups): handle LVM on GPT partitions on file level restore #7029
* fix(backups): ignore swap partition of file level restore
2025-11-24 09:51:28 +01:00
Elise-F
89d45caac9 feat(xo6): fetch xo gui routes (#9138) 2025-11-24 08:51:44 +01:00
Mathieu
2592f2fc2c feat(xo-server-usage-report): add RAM/CPU allocation to reports (#9224) 2025-11-20 17:28:35 +01:00
spacotte-vates
87547a2130 fix(@xen-orchestra/rest-api): async actions return json (#9209) 2025-11-20 16:12:21 +01:00
Enée DI IORIO
5349f2ad09 fix(web-core): remove secondary cancel button on search input (#9189)
fix(web-core/uiInput): remove secondary cancel button on search input
2025-11-19 17:02:19 +01:00
Joris K
d34025fd25 feat(xo6/host): update vms view and implement new page vms and sidepanel (#9193) 2025-11-19 08:59:56 +01:00
Joris K
c53758ddc5 feat(xo6/pool) update vms view and implement new page vms and sidepanel (#9196) 2025-11-19 08:39:22 +01:00
Elise-F
90d1207a3c feat(xo6): creating setting page (#9175) 2025-11-18 17:16:15 +01:00
Joris K
9f478f97a2 feat(xo6/site): implement vms page (#9145) 2025-11-18 17:05:59 +01:00
Pierre Briffaux
e86bd71819 fix(web-core/MenuList): add overflow scroll to prevent content clipping (#9215) 2025-11-18 17:04:08 +01:00
Pierre Brunet
53c1a33106 security: update nodemailer (#9217)
Fix Nodemailer: Email to an unintended domain can occur due to Interpretation Conflict: https://github.com/vatesfr/xen-orchestra/security/dependabot/246
2025-11-18 09:23:27 +01:00
Pierre Brunet
7c06abc99f chore(saml): update GW tutorial (#9119)
Changes thanks to the forum posts here:
https://xcp-ng.org/forum/topic/6174/saml-auth-with-azure-ad/29?_=1760599896992
2025-11-18 09:06:23 +01:00
Pierre Brunet
2a5b0ffd1a security: update package xml2js in xo-web and xo-vmdk-to-vhd (#9216)
Update xml2js in xo-web and xo-vmdk-to-vhd for security purpose
Fix https://github.com/vatesfr/xen-orchestra/security/dependabot/84
2025-11-18 09:04:54 +01:00
Pierre Brunet
e494007b2a security: update cookie package (#9220)
Update cookie package in xo-server for security
Fix https://github.com/vatesfr/xen-orchestra/security/dependabot/167

Proxy check tested
2025-11-18 09:03:48 +01:00
OlivierFL
c2fb0470fc feat(xo-6/host): add storage repositories page and table (#9137) 2025-11-17 15:46:42 +01:00
OlivierFL
5fb5e639cd feat(xo-6/pool): add storage repositories page (#9134)
* fix reactivity issue in progress.util

* update VtsProgressBar component to display progress bar with no ruler

* create and use ProgressCell in Table
2025-11-17 15:01:42 +01:00
OlivierFL
da3733a3a9 feat(xo-6/backups): update wordings for backups modes (#9199) 2025-11-17 11:10:57 +01:00
Florent BEAUCHAMP
bd7ba64b68 fix(vmware-explorer): force vddk library path (#9208)
fix ticket 47466
2025-11-14 16:29:34 +01:00
Enée DI IORIO
436e2ea168 feat(xo6/site) implement hosts page (#9128) 2025-11-14 16:09:32 +01:00
Mathieu
dcbd550961 fix(@xen-orchestra/rest-api): fix eventListener not reattached (#9191) 2025-11-14 11:56:50 +01:00
Thierry Goettelmann
8811bb4bac feat(xo6): migrate typing to @vates/types (#9186) 2025-11-14 09:09:33 +01:00
Joris K
2d06677887 fix(xo6/host/vm): fix issues with condition on dashboards, and chart translation (#9204) 2025-11-13 09:23:37 +01:00
Mathieu
6cfefc91e4 feat(@xen-orchestra/rest-api): expose vms/:id/dashboard (#9143) 2025-11-12 18:01:41 +01:00
Florent BEAUCHAMP
8dc7254128 fix(backups): getOldEntries must accept entries without id or timestamp (#9202)
introduced by #9180 https://github.com/vatesfr/xen-orchestra/pull/9180
2025-11-12 17:42:18 +01:00
Pierre Donias
18a41cd055 feat(xo-web/menu): add link to XO 6 (#9187) 2025-11-12 15:31:06 +01:00
spacotte-vates
a7ad70bbed feat(rest-api): add vm.tags to backups-archives (#9190) 2025-11-12 11:10:17 +01:00
Mathieu
fa0202f5b3 docs(xo): add Gmail App Password and Microsoft 365 auth guidance (#9194)
* docs(xo): add Gmail App Password and Microsoft 365 auth guidance

* fix(docs): update email authentication guidance and improve clarity

* fix(docs): clarify backup-reports plugin configuration step

Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>

---------

Co-authored-by: Mathieu Piton <mathieu.piton@vates.tech>
Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>
2025-11-10 09:32:40 +01:00
Elise-F
88b88187e9 feat(xo6): updated ui of input search (#9156)
* feat(xO6): udpate ui of input search

* updated changelog file

* feat(web-core): updated story of ui input component

* feedbacks on PR
2025-11-07 17:21:02 +01:00
Mathieu
ade825df55 fix(@xen-orchestra/xapi): await VBD.plug (#9192)
introduced by d353dc622c
2025-11-07 15:07:21 +01:00
Thierry Goettelmann
042c9c645b chore(types): complete exiting xo type definitions (#9174) 2025-11-06 10:05:51 +01:00
Dan Pollak
6bf5f4b7d4 feat(xo-server/recover-account): supply password via CLI (#9173)
Allow password to be passed as a parameter
2025-11-05 16:40:41 +01:00
Bastien Nollet
063f499ee0 fix(docs): fix long-term retention doc (#9178) 2025-11-05 15:39:03 +01:00
Mathieu
e4d484e84f feat(xo-server-usage-report): add OS information to usage reports (#9179)
---------

Co-authored-by: Mathieu Piton <mathieu.piton@vates.tech>
2025-11-05 14:24:29 +01:00
Florent BEAUCHAMP
05c6f87a12 fix(backups): long term retention taking the most recent values (#9180) 2025-11-05 11:01:43 +01:00
Pierre Donias
8a390057b6 fix(backups): typos and spelling (#9154)
See #8593

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2025-11-05 10:20:58 +01:00
Alexander Sennhauser
82d65e3c68 describe correct tag for backup health check (#9010) 2025-11-05 10:17:17 +01:00
Pierre Donias
58f029dcbe feat(xo-web/VM/backup): warn that backup job affects all VMs (#9155)
See Zammad#45581
2025-11-04 15:58:32 +01:00
Mathieu
7112c33142 feat(@xen-orchestra/rest-api): expose events (#9130) 2025-11-04 15:05:06 +01:00
Thomas Moraine
08c2923386 docs(xo): document immutability (#9135)
Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2025-11-04 11:50:41 +01:00
Thomas Moraine
ed5be40099 docs(manage_infrastructure.md): update container/docker information (#9140)
This commit renames and updates the **Docker management** section in the Xen Orchestra documentation. We explain a but better how containers are supported in Xen Orchestra, and remove links and references to the now-removed Docker support section, which was triggering a Docusaurus warning at build time.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2025-11-04 09:40:28 +01:00
Bastien Nollet
8c038bac34 feat: release 5.112.1 (#9177) 2025-11-03 14:37:36 +01:00
Bastien Nollet
961b505cfb feat: technical release (#9176) 2025-11-03 12:14:53 +01:00
Florent BEAUCHAMP
32556029cd feat(backups): export qcow2 disk as qcow2 (#9170)
fix full backups for delta
2025-11-03 10:54:11 +01:00
Mathieu
65013190e3 fix(xo-server/rpu): update linstor-satellite and linstor-controller (#9171) 2025-11-03 10:29:33 +01:00
OlivierFL
2dd451a7d9 fix(xo-6/backup-settings): reactivity issue when using useXoBackupJobSettingsUtils (#9169)
Introduced by e547447
2025-10-31 14:27:16 +01:00
Pierre Brunet
73a8ba59ba fix(xo-web): update easyvirt forms for DC Net Scope (#9107) 2025-10-31 11:07:33 +01:00
Bastien Nollet
285d1da361 feat(lite): 0.16.0 (#9167) 2025-10-30 18:24:47 +01:00
Bastien Nollet
4d77b79ce9 feat: release 5.112.0 (#9166) 2025-10-30 17:55:30 +01:00
Bastien Nollet
268033439b feat: technical release (#9164) 2025-10-30 15:25:41 +01:00
Florent BEAUCHAMP
df3a365eb7 fix(xapi): use qcow2 exports only for bigger disks (#9163) 2025-10-30 15:01:05 +01:00
MlssFrncJrg
61ee81ea3a fix(@vates/types): add packages to release in changelog (#9161)
introduced by cfb64f8
2025-10-30 12:50:37 +01:00
vates-bot
2789b339cb feat(web-stack/i18n): update Xen Orchestra 6 (#9095) 2025-10-30 11:59:16 +01:00
Olivier Floch
c895f05486 feat(xo6): introduce constants.ts file to avoid duplications 2025-10-30 11:17:43 +01:00
Olivier Floch
ed3f50cb1a feat(xo6/backups): add breadcrumb in backups pages header 2025-10-30 11:17:43 +01:00
Olivier Floch
6bb90082b9 feat(xo-core): add UiBreadCrumb component 2025-10-30 11:17:43 +01:00
Florent BEAUCHAMP
c870bb653a fix(backups): use qcow2 for export of qcow2 disks (#9160) 2025-10-30 09:20:35 +01:00
Florent BEAUCHAMP
b11484ce6d fix(xo-web): image format in disk tab of vm (#9157) 2025-10-29 16:04:02 +01:00
MlssFrncJrg
cfb64f8e0d feat(@xen-orchestra/rest-api): add boot firmware for VM creation (#9147) 2025-10-29 15:51:11 +01:00
Enée DI IORIO
d6e0f70510 feat(xo-6/backups) add backup targets page (#9048) 2025-10-29 15:33:56 +01:00
Pierre Donias
b47a7055a2 feat(netbox): support Netbox v4.4.x (#9153)
Fixes #8958
2025-10-29 15:26:19 +01:00
MlssFrncJrg
a101e9e02d feat(@xen-orchestra/rest-api): expose gui routes (#9133) 2025-10-29 15:04:11 +01:00
Enée DI IORIO
e547447bb4 feat(xo-6/backups): add backup configuration page (#9008) 2025-10-29 14:49:44 +01:00
Florent BEAUCHAMP
5d92f813f6 feat(xo-server): support import of disk > 2TB toward qcow2 SR (#9148) 2025-10-29 10:13:44 +01:00
Florent BEAUCHAMP
ab4e20b0a5 fix(vhd-lib): error while handling open error (#9151)
fix an error Cannot read properties of undefined while handling an error raised during a vhd file error.
2025-10-29 10:04:25 +01:00
Mathieu
ff4b087be8 feat(xo-cli): add system status download functionality for hosts and pools (#9141)
Add two new API methods to download XCP-ng/XenServer system status (bug reports) via xo-cli:

    host.getSystemStatus - Download system status from a single host
    pool.getSystemStatuses - Download system status from all hosts in a pool in parallel


---------

Co-authored-by: Mathieu Piton <mathieu.piton@vates.tech>
2025-10-29 09:29:20 +01:00
Elise-F
09208ba0fb feat(xo6): added loader in treeview when search is triggered (#9142)
* feat(xo6): added loader in treeview when search is triggered

* update changelog file

* fix(xo6): code cleanup following PR feedback

* fix order packages in changelog file
2025-10-29 09:23:54 +01:00
Elise-F
896e9386a9 feat(xo6): added new links in the user menu and customized it (#9126)
* feat(xo6): added new links in the user menu and customized it

* udpate Changelog file

* fix(xo6): code cleanup following PR feedback

* fix order packages in changelog file
2025-10-29 09:15:10 +01:00
Olivier Lambert
87471d90c4 feat(@xen-orchestra/xapi): expose IPMI sensors for Lenovo hardware (#9149)
---------

Signed-off-by: Olivier Lambert <olivier.lambert@vates.fr>
Co-authored-by: Mathieu <70369997+MathieuRA@users.noreply.github.com>
Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2025-10-28 19:15:37 +01:00
Enée DI IORIO
740fc03a04 feat(xo-6/backups): add backed up vms page (#9018)
Co-authored-by: Olivier Floch <olivier.floch@vates.fr>
2025-10-28 17:08:08 +01:00
Florent BEAUCHAMP
45ef65b251 fix(qcow2): number/bigint comparison (#9150) 2025-10-28 15:47:27 +01:00
Florent BEAUCHAMP
bf705e0a98 feat(xo-web): add merge synchronously option to mirror backups (#9118) 2025-10-28 14:15:30 +01:00
Thomas Moraine
989864314f docs: upgrade docusaurus from version 3.8 to 3.9.2 (#9073)
docs: upgrade docusaurus from version 3.8 to 3.9

The onBrokenMarkdownLinks option is deprecated and has been replaced by siteConfig.hooks.onBrokenMarkdownLinks. However, since the option was configured to 'warn' and is the default behavior for this option, deleting the onBrokenMarkdownLinks seems like easier

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2025-10-28 12:20:53 +01:00
MlssFrncJrg
0a377f9568 feat(@xen-orchestra/rest-api): expose ping (#9129) 2025-10-28 10:10:27 +01:00
Anthoine Bourgeois
cb96de6d11 feat: Qcow2Disk: support sub-cluster feature
Signed-off-by: Anthoine Bourgeois <anthoine.bourgeois@vates.tech>
2025-10-28 09:43:37 +01:00
Anthoine Bourgeois
7e032e9f84 fix: Qcow2Disk: fix host cluster offset mask
Signed-off-by: Anthoine Bourgeois <anthoine.bourgeois@vates.tech>
2025-10-28 09:43:37 +01:00
Bastien Nollet
51855c22d4 feat: technical release (#9146) 2025-10-27 12:13:55 +01:00
MathieuRA
676b059485 fix(xo-lite): set default cores-per-socket value 2025-10-27 11:41:38 +01:00
MathieuRA
81d0d28819 fix(xo-server): set default cores-per-socket value 2025-10-27 11:41:38 +01:00
Pierre Brunet
9123921676 Chore: update rimraf packages to fix glob ending support (#9046)
feat: update rimraf
2025-10-27 09:48:17 +01:00
Mathieu
544590a860 feat(@xen-orchestra/rest-api): expose backup-archives (#8982)
XO-1468
2025-10-24 17:29:10 +02:00
OlivierFL
63e7eb881c feat(xo-6/backups): add backup runs page (#9007) 2025-10-24 11:09:35 +02:00
Bastien Nollet
030d697392 fix(xo-server-load-balancer): correctly avoid migrating tagged VMs (#9139)
Introduced by 9ef05b8afe
2025-10-23 15:45:32 +02:00
Florent BEAUCHAMP
af56933587 fix(backups): wrong error on close before init (#9125)
this happens if the disk init fails, for example when trying
to read qcow2 disk from a non qcow2 SR
2025-10-22 15:56:21 +02:00
bleader
ec0bdb3286 docs(sdn-controller): document OpenFlow XAPI plugin usage (#9114)
- State to use a single XOA instance with SDN Controller plugin enabled
- Cleanup existing OpenFlow documentation
- Document XAPI plugin usage, including xo-cli commands
- Add configuration to toggle between direct channel and XAPI plugin
2025-10-21 16:46:14 +02:00
OlivierFL
70055960d0 feat(xo-core): update logical properties wording in CSS guidelines (#9132) 2025-10-21 11:12:52 +02:00
Pierre Donias
a9ab882b75 fix(fatfs): typos and spelling (#9131)
See #8593
2025-10-20 16:49:04 +02:00
Elise-F
bce7408f1f feat(xo6): add loader on input search treeview when is triggered (#9122)
* feat(xo6): add loader on input search treeview when is triggered

* udpate Changelog file

* fix(xo6): removed condition on props icon in UiInput

* fix(xo6): removed debounced on clear
2025-10-20 15:20:12 +02:00
Mathieu
c8abbb259e fix(@xen-orchestra/rest-api): add bad request response documentation (#9108) 2025-10-20 14:07:38 +02:00
Thomas Moraine
35cc755bf0 docs(xo): specify configuration file path (#9127)
Several sections of the Xen Orchestra Appliance documentation mention modifying the config file, without specifying the actual file path. This commit updates those references to explicitly include /etc/xo-server/config.toml, which helps users locate and edit the correct configuration file more easily.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2025-10-20 12:10:09 +02:00
Bastien Nollet
5750b58362 fix(xo-web/Size): fix saving Size component (#9121)
Changing the value of a `Size` component by editing the field and pressing
'Enter' was executing twice the `_save()` function in parallel (which calls the
`onChange` function).

In some cases like editing a running VM memory limit, this caused errors (trying
to open two modals at the same time, and trying to restart the VM twice)
2025-10-20 10:28:28 +02:00
Joris K
8a95941297 fix(xo6): fix formatted date in backup jobs (#9124) 2025-10-20 08:45:17 +02:00
Florent BEAUCHAMP
8289150460 feat(vmware-explore): add more debug when failing to get thumbprint (#9012)
from ticket 45117
2025-10-17 14:25:44 +02:00
Florent BEAUCHAMP
87df162944 chores(all): remove jest (#8956)
jest test are unmaintained and flacky, they increase the dependencies surface
2025-10-17 14:24:05 +02:00
Thierry Goettelmann
e9eda42d7d feat(web-stack): new VtsStatus component (#9052) 2025-10-17 10:14:59 +02:00
Thierry Goettelmann
d942d144ca feat(web-core): add objectIcon helper (#9053) 2025-10-17 09:40:50 +02:00
Elise-F
d829816f62 fix(xo6): display user menu above tree structure (#9115)
* fix(xo6): display user menu above tree structure

* add CHANGELOG
2025-10-16 16:47:01 +02:00
Florent BEAUCHAMP
8b3b23e995 fix(backups): throw an error on non unique target for replication (#9117)
from ticket 40151
2025-10-16 15:49:00 +02:00
Cécile Morange
52807ed15f docs(xo): change link to the xcp-ng docs, add some infos in HA section (#9112)
* docs(xo): change link to the xcp-ng docs, add some infos in HA section

* Update docs/docs/manage_infrastructure.md

Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>

* Update docs/docs/manage_infrastructure.md

Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>

* Update docs/docs/manage_infrastructure.md

Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>

---------

Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>
2025-10-16 14:20:58 +02:00
Florent BEAUCHAMP
596f9ac536 fix(vhd-lib): better error handling
ensure disk is really closed, and that error are caught
2025-10-16 11:30:25 +02:00
Florent BEAUCHAMP
d58efc7b2d fix(xapi): implement Throttling on Xapi/Remote read 2025-10-16 11:30:25 +02:00
Florent BEAUCHAMP
73aa66a219 fix(xapi): implement Timeout on Xapi read 2025-10-16 11:30:25 +02:00
Florent BEAUCHAMP
93c8903953 fix(backups): put back rate speed to full backup, replication and mirror 2025-10-16 11:30:25 +02:00
Florent BEAUCHAMP
dc8c455a0b feat(generator-toolbox): handle unlimited throttle (speed at 0) 2025-10-16 11:30:25 +02:00
Thierry Goettelmann
d601558bf4 feat(web-core): new table system (#9054) 2025-10-16 10:39:22 +02:00
Mathieu
d5cee13e25 fix(@xen-orchestra/rest-api): use middleware instead of route for users/me redirection (#9103)
Introduced by #8985
2025-10-16 09:41:36 +02:00
Florent BEAUCHAMP
151d7ed22a fix(immutable-backups): double delete that can block immut lifting (#9104) 2025-10-15 17:39:11 +02:00
OlivierFL
3d29348337 feat(xo-6/collections): implement virtual lists for Treeview, QuickTasks and Alarms collections in dashboards (#9077) 2025-10-15 17:17:32 +02:00
Thomas Moraine
62ab5a2f5b docs(xo): remove mentions of xscontainer from the xo doc (#9113)
docs(manage_infrastructure.md): remove content related to xscontainer, which is deprecated

Since XSContainer is no longer supported on XCP-ng 8.3 or XenServer and
we currently have no alternative, instructions related to XScontainer in
the XO doc are removed

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2025-10-15 14:33:31 +02:00
Pierre Brunet
c2d1c66a59 fix(plugin/auth-saml): add default config in SAML (#9093) 2025-10-14 16:30:58 +02:00
Bastien Nollet
730d0ff41d fix(xo-web/Toggle): swallow undefined errors thrown by confirm (#8953)
Updating the `Toggle` component to avoid throwing an error coming from a
`confirm` modal, which ends up appearing in the console.

This also changes a bit the behaviour of the `Toggle` component, which now
awaits its `onChange` function.
2025-10-14 15:11:18 +02:00
Mathieu
64d164300b feat(@xen-orchestra/rest-api): expose pbds (#9106) 2025-10-14 14:44:26 +02:00
Mathieu
04eeb78df7 fix(@vates/types): fix XoSr allocationStrategy (#9105) 2025-10-14 10:44:33 +02:00
Mathieu
6bdfbfb953 feat(@xen-orchestra/rest-api): expose users/authentication_tokens (#9102) 2025-10-14 09:48:31 +02:00
Mathieu
e6573e26a9 feat(xo-server/rest-api): remove legacy code (#9101) 2025-10-14 09:41:28 +02:00
Mathieu
5f5cb9b45d feat(@xen-orchestra/rest-api): expose vdis/:id.(vhd|raw) (#9038)
* feat(@xen-orchestra/rest-api): expose vdi(s|-snapshots)/:id.(vhd|raw)
2025-10-10 14:39:31 +02:00
MlssFrncJrg
940e879a1c feat(@xen-orchestra/rest-api): expose vm-controllers/:id/tags/:tag (#9097) 2025-10-10 11:36:11 +02:00
MlssFrncJrg
861567d0db feat(@xen-orchestra/rest-api): expose vm-templates/:id/tags/:tag (#9099) 2025-10-10 11:23:57 +02:00
MlssFrncJrg
8e536a4c32 feat(@xen-orchestra/rest-api): expose vm-snapshots/:id/tags/:tag (#9098) 2025-10-10 11:10:08 +02:00
MlssFrncJrg
e100736349 feat(@xen-orchestra/rest-api): expose srs/:id/tags/:tag (#9089) 2025-10-10 11:03:17 +02:00
MlssFrncJrg
98e4bcccfa feat(@xen-orchestra/rest-api): expose vdis/:id/tags/:tag (#9094) 2025-10-10 10:43:35 +02:00
MlssFrncJrg
938b34343d feat(@xen-orchestra/rest-api): expose vdi-snapshots/:id/tags/:tag (#9091) 2025-10-10 10:31:03 +02:00
MlssFrncJrg
e1ecc2d6d4 feat(@xen-orchestra/rest-api): remove vifs/:id/tags/:tag (#9096) 2025-10-10 10:22:12 +02:00
Pierre Donias
afabf35c66 fix(xo-web/Copiable): remove gap in hover box (#9059)
See Zammad#45442

Wrap "Copy to clipboard" button with a span to make the hoverable zone
continuous between the copiable element and the button. Without this, the 1ex
gap between the 2 was sometimes enough to make the button disappear before the
user had a chance to click it.
2025-10-10 10:19:50 +02:00
MlssFrncJrg
8e0f48bddd feat(@xen-orchestra/rest-api): remove vbds/:id/tags/:tag (#9090) 2025-10-10 10:13:34 +02:00
MlssFrncJrg
70dc82e621 feat(@xen-orchestra/rest-api): expose srs/:id/tasks (#9086) 2025-10-10 10:03:18 +02:00
Mathieu
2928813613 feat(@xen-orchestra/rest-api): expose vms/:id/tags/:tag (#9092) 2025-10-10 09:54:57 +02:00
MlssFrncJrg
a86d07adea feat(@xen-orchestra/rest-api): expose vbds/:id/tasks (#9085) 2025-10-10 09:48:00 +02:00
Mathieu
a282e10069 feat(@xen-orchestra/rest-api): expose pools/:id/tasks (#9080) 2025-10-09 16:39:13 +02:00
MlssFrncJrg
8a70196fa9 feat(@xen-orchestra/rest-api): expose pools/:id/tags/:tag (#9088) 2025-10-09 14:40:04 +02:00
MlssFrncJrg
1d8ae5eea8 feat(@xen-orchestra/rest-api): expose networks/:id/tags/:tag (#9087) 2025-10-09 12:54:03 +02:00
Mathieu
055bce7d91 feat(@xen-orchestra/rest-api): expose hosts/:id/tags/:tag (#9037) 2025-10-09 10:47:54 +02:00
Thomas Moraine
1ee07eb06f docs(xo): document object storage support list (#9068)
Adds a page to the Xen Orchestra documentation to list the compatible S3-based
storage support providers for Xen orchestra, with the different support tiers
and appropriate caveats.
2025-10-09 09:53:50 +02:00
Joris K
6a5eb12051 fix(xo6/site): issue in backups tab (#9064)
* fix(xo6/site): issue in backups tab

* add changelog
2025-10-08 17:50:05 +02:00
Pierre Brunet
fbf3f1bc7f chore: update doc for SAML due to breaking changes in config (#9084) 2025-10-08 17:13:31 +02:00
MlssFrncJrg
dedfa91669 feat(@xen-orchestra/rest-api): expose vdis-snapshots/:id/tasks (#9082) 2025-10-08 15:47:16 +02:00
MlssFrncJrg
f3480f6939 feat(@xen-orchestra/rest-api): expose vdis/:id/tasks (#9079) 2025-10-08 15:38:35 +02:00
Mathieu
e48166846c feat(@xen-orchestra/rest-api): expose vbds/:id/messages (#9029) 2025-10-08 15:23:05 +02:00
Mathieu
f7a79a0be3 feat(@xen-orchestra/rest-api): expose hosts/:id/tasks (#9074) 2025-10-08 15:02:42 +02:00
Mathieu
f0f1fd1715 feat(@xen-orchestra/rest-api): expose networks/:id/tasks (#9076) 2025-10-08 14:56:47 +02:00
MlssFrncJrg
2e4c45b207 fix(xo-server/rest-api): fix indentation (#9081) 2025-10-08 14:49:44 +02:00
Mathieu
72a3f0a9da feat(@xen-orchestra/rest-api): expose pifs/:id/tasks (#9078) 2025-10-08 14:45:05 +02:00
MlssFrncJrg
2d6f784f0a feat(@xen-orchestra/rest-api): expose vifs/:id/tasks (#9075) 2025-10-08 14:19:48 +02:00
MlssFrncJrg
614449d540 feat(@xen-orchestra/rest-api): migrate vm-controllers tasks (#9069) 2025-10-08 14:12:57 +02:00
Thierry Goettelmann
53ff333778 feat(web-core): introducing toComputed helper (+ migrate codebase) (#8960) 2025-10-08 11:49:12 +02:00
Thierry Goettelmann
4207954982 feat(web-stack): migrate modal system to web-core + enhancements + new UiModal (#8887) 2025-10-08 11:48:46 +02:00
Mathieu
6af64c7095 feat(@xen-orchestra/rest-api): expose groups/:id/tasks (#9072)
XO-1187
2025-10-08 11:05:28 +02:00
Mathieu
7d03560a3e feat(@xen-orchestra/rest-api): expose users/:id/tasks (#9066)
XO-1188
2025-10-08 10:26:06 +02:00
Julien Fontanet
9f9519ef15 docs(xoa): use HTTP for alternative install (#9071)
Introduced by 51b06b1177

Fixes #9067
2025-10-07 20:29:29 +02:00
MlssFrncJrg
0838b112de feat(@xen-orchestra/rest-api): migrate vm-templates tasks (#9004) 2025-10-07 17:29:48 +02:00
Pierre Donias
d6e4456538 fix(xo-web/VM/ActionBar): do not hide actions if user is admin (#9061)
See Zammad#44366

Admin users should always be able to perform actions on VMs even if the VM was
created with Self Service.
2025-10-07 15:57:06 +02:00
Mathieu
022dc0bf2e feat(@xen-orchestra/rest-api): expose servers/:id/tasks (#9065) 2025-10-07 14:55:51 +02:00
MlssFrncJrg
49909df1d2 feat(@xen-orchestra/rest-api): migrate vm-snapshots tasks (#9005) 2025-10-07 14:35:05 +02:00
Mathieu
7c7d7a938e feat(@xen-orchestra/rest-api): expose vm-controllers/:id/messages (#9050) 2025-10-07 14:16:14 +02:00
Mathieu
a67ad48012 feat(@xen-orchestra/rest-api): expose vifs/:id/messages (#9049) 2025-10-06 16:29:27 +02:00
Mathieu
6c6d7a377f feat(@xen-orchestra/rest-api): expose vdis/:id/messages (#9044) 2025-10-06 16:24:11 +02:00
Mathieu
d3006026ca feat(@xen-orchestra/rest-api): expose vdi-snapshots/:id/messages (#9043) 2025-10-06 16:04:41 +02:00
Mathieu
62cdd56691 feat(@xen-orchestra/rest-api): expose networks/:id/messages (#9023) 2025-10-06 15:58:25 +02:00
Mathieu
775e8a8b12 feat(@xen-orchestra/rest-api): expose pifs/:id/messages (#9021) 2025-10-06 15:49:44 +02:00
Mathieu
f5676705cc fix(@xen-orchestra/rest-api): move getTasksObject out of a service (#9058)
Introduced #8955
2025-10-06 15:40:29 +02:00
Pierre Donias
0bad96418b feat: release 5.111.1 (#9062) 2025-10-06 15:26:01 +02:00
Pierre Donias
3222025088 feat: technical release (#9060) 2025-10-06 14:15:47 +02:00
Pierre Brunet
86f3096c1b chore: update packages (#9042)
- passport-saml -> @node-saml/passport-saml
- connect-flash -> typed-flash
2025-10-06 10:42:44 +02:00
Florent BEAUCHAMP
d7fe4f1729 fix(vmware-explorer): thumbprint is case insensitive (#9045)
* fix(vmware-explorer): thumbprint is case insensitive

depending on version ? it can answer a text starting woth sha1 or SHA1
2025-10-03 13:47:27 +02:00
Florent BEAUCHAMP
64a2666645 fix(asyncEach): handle iterator error
this was the cause of  xo:backups:worker WARN possibly unhandled rejection
and the backup error code not triggerring on timeout and various
edge cases
2025-10-03 11:19:55 +02:00
Florent BEAUCHAMP
3b7f623f9e fix(fs): retry writing a file should not throw EEXIST on previous failed write
also removing the retry on outputfile, since outputfile call _writeFile which
is already retried
2025-10-03 11:19:55 +02:00
Florent BEAUCHAMP
16cdbc16fa fix(v2v): do not lock stopped VM (#9047)
imported VM are locked to prepare for a future resume
stopped VM are completly transferred and won't need a future resume
2025-10-03 10:23:26 +02:00
Mathieu
52510ed03c fix(xo-server/users): name property was not present in getAllUsers (#9019)
* fix(xo-server/users): name property was not present in getAllUsers

* Update packages/xo-server/src/xo-mixins/subjects.mjs

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>

---------

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2025-10-03 10:05:27 +02:00
Mathieu
bbec7d4532 feat(@xen-orchestra/rest-api): expose pools/:id/messages (#9022)
* feat(@xen-orchestra/rest-api): expose pools/:id/messages
2025-10-02 15:13:45 +02:00
Mathieu
cf0449c034 fix(@xen-orchestra/rest-api): fix messages tag (#9024) 2025-09-30 17:00:38 +02:00
Mathieu
9e93ba14ae feat(@xen-orchestra/rest-api): expose hosts/:id/messages (#9027) 2025-09-30 17:00:28 +02:00
Mathieu
bf330311a1 feat(@xen-orchestra/rest-api): expose srs/:id/messages (#9028) 2025-09-30 16:53:56 +02:00
Pierre Donias
481ef5e585 feat(lite): 0.15.0 (#9036) 2025-09-30 16:21:33 +02:00
Pierre Donias
a5debf3241 feat: release 5.111.0 (#9035) 2025-09-30 16:07:25 +02:00
Pierre Donias
d8c106fc5d feat: technical release (#9034) 2025-09-30 15:46:07 +02:00
Enée DI IORIO
8d2af27896 fix(xo-lite/settings): fix key-value alignment (#8981)
* feat(xo-lite/settings): fix key-value alignment

Refactor page for use of more regular components and match better to Figma

* review

* review
2025-09-30 15:40:21 +02:00
Joris K
893cdcbc86 feat(xo6): add alarms in host and vm dashboard, add patches in host dashboard (#8937)
* feat(xo6): add alarms in host and vm dashboard, add patches in host dashboard

* feedback
2025-09-30 15:00:33 +02:00
Pierre Donias
241c37b35c feat: technical release (#9033) 2025-09-30 13:54:47 +02:00
Thierry Goettelmann
57e1b94115 fix(web): failing build (#9030)
Introduced by #8966
2025-09-30 13:25:52 +02:00
1590 changed files with 89557 additions and 35948 deletions

View File

@@ -233,7 +233,29 @@ module.exports = {
'@intlify/vue-i18n/no-duplicate-keys-in-locale': 'error',
'@intlify/vue-i18n/no-missing-keys-in-other-locales': [
'error',
{ ignoreLocales: ['de', 'fa', 'cs', 'es', 'sv', 'it', 'ru', 'uk', 'nl', 'pt_BR'] },
{
ignoreLocales: [
'de',
'fa',
'cs',
'es',
'sv',
'it',
'ru',
'uk',
'nl',
'pt-BR',
'da',
'ko',
'nb-NO',
'ja',
'fi',
'pl',
'zh-Hans',
'sk',
'pt',
],
},
],
},
},
@@ -268,6 +290,17 @@ module.exports = {
'eslint-comments/no-unlimited-disable': 'off',
},
},
{
files: ['*.d.ts'],
parser: '@typescript-eslint/parser',
parserOptions: {
sourceType: 'module',
},
rules: {
'n/no-unsupported-features/es-syntax': 'off',
'no-unused-vars': 'off',
},
},
],
ignorePatterns: ['@vates/fatfs/'],

View File

@@ -25,7 +25,7 @@ jobs:
- name: Setup Node environment
uses: actions/setup-node@v4
with:
node-version: '20'
node-version: '24.14'
cache: 'yarn'
- name: Install project dependencies

5
@vates/async-each/index.d.ts vendored Normal file
View File

@@ -0,0 +1,5 @@
export function asyncEach<Item>(
iterable: Iterable<Item> | AsyncIterable<Item>,
iteratee: (item: Item, index: number, iterable: Iterable<Item> | AsyncIterable<Item>) => Promise<void>,
opts?: { concurrency?: number; signal?: AbortSignal; stopOnError?: boolean }
): Promise<void>

View File

@@ -73,20 +73,20 @@ exports.asyncEach = function asyncEach(iterable, iteratee, { concurrency = 10, s
}
nextIsRunning = true
if (running < concurrency) {
const cursor = await it.next()
if (cursor.done) {
next = () => {
if (running === 0) {
if (errors.length === 0) {
resolve()
} else {
reject(new AggregateError(errors))
try {
const cursor = await it.next()
if (cursor.done) {
next = () => {
if (running === 0) {
if (errors.length === 0) {
resolve()
} else {
reject(new AggregateError(errors))
}
}
}
}
} else {
++running
try {
} else {
++running
const result = iteratee.call(this, cursor.value, index++, iterable)
let then
if (result != null && typeof result === 'object' && typeof (then = result.then) === 'function') {
@@ -94,9 +94,9 @@ exports.asyncEach = function asyncEach(iterable, iteratee, { concurrency = 10, s
} else {
onFulfilled(result)
}
} catch (error) {
onRejected(error)
}
} catch (error) {
onRejected(error)
}
nextIsRunning = false
return next()

View File

@@ -67,20 +67,17 @@ describe('asyncEach', () => {
})
it('stops on first error when stopOnError is true', async () => {
const tracker = new assert.CallTracker()
const error = new Error()
const iteratee = tracker.calls((_, i) => {
const iteratee = spy((_, i) => {
if (i === 1) {
throw error
}
}, 2)
})
assert.deepStrictEqual(
await rejectionOf(asyncEach(iterable, iteratee, { concurrency: 1, stopOnError: true })),
error
)
tracker.verify()
assert.deepStrictEqual(iteratee.callCount, 2)
})
it('rejects AggregateError when stopOnError is false', async () => {
@@ -100,19 +97,30 @@ describe('asyncEach', () => {
})
it('can be interrupted with an AbortSignal', async () => {
const tracker = new assert.CallTracker()
const ac = new AbortController()
const iteratee = tracker.calls((_, i) => {
const iteratee = spy((_, i) => {
if (i === 1) {
ac.abort()
}
}, 2)
})
await assert.rejects(asyncEach(iterable, iteratee, { concurrency: 1, signal: ac.signal }), {
message: 'asyncEach aborted',
})
assert.deepStrictEqual(iteratee.callCount, 2)
})
tracker.verify()
it('handle error on iterable', async () => {
const iteratee = spy(async () => {})
const iterable = async function* () {
for (const value of [1, 2, 3]) {
if (value === 2) {
throw new Error('iterator error')
}
await randomDelay()
yield value
}
}
await assert.rejects(() => asyncEach(iterable(), iteratee, { concurrency: 1 }), { message: 'iterator error' })
})
})
)

View File

@@ -24,7 +24,8 @@
"url": "https://vates.fr"
},
"license": "ISC",
"version": "1.0.0",
"types": "index.d.ts",
"version": "1.0.3",
"engines": {
"node": ">=8.10"
},

View File

@@ -24,8 +24,8 @@
},
"dependencies": {
"@vates/multi-key-map": "^0.2.0",
"@xen-orchestra/async-map": "^0.1.2",
"@xen-orchestra/log": "^0.7.1",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/log": "^0.7.2",
"ensure-array": "^1.0.0"
},
"devDependencies": {

View File

@@ -62,7 +62,7 @@ exports.parseFlags = function (flags) {
case 'ax': info = {read:false, write:true, create:true, append:true, exclusive:true}; break;
case 'a+': info = {read:true, write:true, create:true, append:true}; break;
case 'ax+': info = {read:true, write:true, create:true, append:true, exclusive:true}; break;
default: throw Error("Uknown mode: "+flags); // TODO: throw as `S.err.INVAL`
default: throw Error("Unknown mode: "+flags); // TODO: throw as `S.err.INVAL`
}
if (info.sync) throw Error("Mode not implemented."); // TODO: what would this require of us?
if (_dir) info._openDir = true;

View File

@@ -1,7 +1,7 @@
{
"private": false,
"name": "@vates/fatfs",
"version": "0.11.0",
"version": "0.11.1",
"description": "fs implementation on top of raw FAT16/FAT32 block source",
"main": "index.js",
"scripts": {

View File

@@ -90,7 +90,7 @@ function startTests(vol, waitTime) {
assert(path === file, "Fixed fluffy path matches normal one.");
});
fs.realpath([BASE_DIR,"non","existent","path"].join('/'), function (e) {
assert(e, "Expected error calling fs.realpath on non-existent file.");
assert(e, "Expected error calling fs.realpath on nonexistent file.");
});
fs.readdir(BASE_DIR, function (e, arr) {

View File

@@ -19,12 +19,12 @@
},
"main": "./index.mjs",
"dependencies": {
"@xen-orchestra/fs": "^4.6.3",
"@xen-orchestra/log": "^0.7.1",
"@xen-orchestra/fs": "^4.8.0",
"@xen-orchestra/log": "^0.7.2",
"fuse-native": "^2.2.6",
"lru-cache": "^7.14.0",
"promise-toolbox": "^0.21.0",
"vhd-lib": "^4.14.1"
"vhd-lib": "^4.16.0"
},
"bin": {
"xo-fuse-vhd": "./cli.mjs"

View File

@@ -1,13 +1,13 @@
{
"name": "@vates/generator-toolbox",
"version": "1.0.4",
"version": "1.1.2",
"main": "dist/index.mjs",
"license": "MIT",
"private": false,
"type": "module",
"devDependencies": {
"@eslint/js": "^9.19.0",
"rimraf": "^5.0.1",
"rimraf": "^6.0.1",
"@types/node": "^20.6",
"typescript": "~5.6",
"typescript-eslint": "^8.23.0"

View File

@@ -144,14 +144,7 @@ class Forked<T, TReturn, TNext> implements AsyncGenerator<T, TReturn, TNext> {
return this.#parent.remove(this.#uid, e)
}
async *[Symbol.asyncIterator](): AsyncGenerator<T> {
while (true) {
const res = await this.next()
if (res.done) {
break
}
yield res.value
}
[Symbol.asyncIterator](): AsyncGenerator<T, TReturn, TNext> {
return this
}
}

View File

@@ -2,8 +2,7 @@ import assert from 'node:assert'
import { suite, test } from 'node:test'
import { Synchronized } from './synchronized.mjs'
// eslint-disable-next-line @typescript-eslint/no-unused-vars
async function* makeRangeGenerator(end = Infinity, progress = { yielded: 0 }, onYielded = (val: unknown) => {}) {
async function* makeRangeGenerator(end = Infinity, progress = { yielded: 0 }, onYielded = (_val: unknown) => {}) {
for (let i = 0; i < end; i++) {
await new Promise(resolve => setTimeout(resolve, 10))
yield i
@@ -17,7 +16,7 @@ async function consume(
iterable: AsyncGenerator,
delay = 2500,
// eslint-disable-next-line @typescript-eslint/no-unused-vars
onConsumed = (val: unknown, iterable: AsyncGenerator) => Promise.resolve(false)
onConsumed = (_val: unknown, _iterable: AsyncGenerator) => Promise.resolve(false)
) {
for await (const val of iterable) {
await new Promise(resolve => setTimeout(resolve, delay))
@@ -28,6 +27,30 @@ async function consume(
}
suite('success', () => {
test('for-await-of break stops the source generator', async () => {
let sourceClosed = false
const source = (async function* () {
try {
for (let i = 0; i < 10; i++) {
await new Promise(resolve => setTimeout(resolve, 10))
yield i
}
} finally {
sourceClosed = true
}
})()
const forker = new Synchronized(source)
const fork = forker.fork('first')
// for-await-of must call fork.return() on break, which propagates to the source
for await (const val of fork) {
if (val === 2) break
}
assert.strictEqual(sourceClosed, true)
})
test('if works with multiple consumer', async () => {
const progress = { yielded: 0 }
const generator = makeRangeGenerator(3, progress)

View File

@@ -19,7 +19,7 @@ export class Throttle {
} else {
speed = this.#bytesPerSecond
}
assert.ok(speed > 0, `speed must be greater than zero, ${speed} computed`)
assert.ok(speed >= 0, `speed must be positive, ${speed} computed`)
return speed
}
constructor(speed: number | (() => number)) {
@@ -29,7 +29,9 @@ export class Throttle {
getNextSlot(length: number): { timeout?: ReturnType<typeof setTimeout>; promise?: Promise<unknown> } {
assert.notStrictEqual(length, undefined, `throttled stream need to expose a length property }`)
assert.ok(length > 0, `throttled stream must expose a positive length property , ${length} given }`)
if (this.speed === 0) {
return {}
}
const previous = this.#previousSlot
const nextSlot = Math.round(previous + (length * 1000) / this.speed)
if (nextSlot < Date.now()) {

View File

@@ -1,4 +1,7 @@
import assert from 'node:assert'
const TIMEOUT = Symbol('timeout')
export class Timeout<T> implements AsyncGenerator {
#source: AsyncGenerator<T>
#timeout: number
@@ -8,21 +11,30 @@ export class Timeout<T> implements AsyncGenerator {
this.#timeout = timeout
}
async next(): Promise<IteratorResult<T>> {
let timeout: ReturnType<typeof setTimeout>
const promiseTimeout = new Promise((_, reject) => {
timeout = setTimeout(() => {
reject(new Error('Timeout reached '))
}, this.#timeout)
})
const promiseNext = new Promise((resolve, reject) => {
this.#source.next().then(res => {
// ensure timetout won't fire later
clearTimeout(timeout)
resolve(res)
}, reject)
})
// promiseTimeout will never resolve
return Promise.race([promiseNext, promiseTimeout]) as Promise<IteratorResult<T>>
let timeoutHandle: ReturnType<typeof setTimeout>
const sourceNext = this.#source.next()
const result = await Promise.race([
sourceNext.then(res => {
clearTimeout(timeoutHandle)
return res
}),
new Promise<typeof TIMEOUT>(resolve => {
timeoutHandle = setTimeout(() => resolve(TIMEOUT), this.#timeout)
}),
])
if (result === TIMEOUT) {
// stop the source once the in-flight next settles to avoid data loss on the next call
// we don't want to await for the end of timeouted code
sourceNext.then(
() => this.#source.return(undefined).catch(() => {}),
() => {} // source already errored, nothing to clean up
)
throw new Error('Timeout reached')
}
return result
}
return(): Promise<IteratorResult<T>> {
return this.#source.return(undefined)

View File

@@ -64,4 +64,24 @@ describe('Timeout class', () => {
await assert.rejects(timeout.throw(new Error('Test error')), 'Expected error was not thrown')
})
it('should close the source generator after a timeout', async () => {
let sourceClosed = false
const slowSource = (async function* () {
try {
// resolves after 100ms so the in-flight next() eventually settles
await new Promise(resolve => setTimeout(resolve, 100))
yield 1
} finally {
sourceClosed = true
}
})()
const timeout = new Timeout(slowSource, 10) // fires before source yields
await assert.rejects(() => timeout.next(), /Timeout reached/)
// wait for the in-flight next() to settle so source.return() can be called
await new Promise(resolve => setTimeout(resolve, 200))
assert.strictEqual(sourceClosed, true)
})
})

View File

@@ -2,6 +2,7 @@
"compilerOptions": {
"target": "es2023",
"module": "ESNext",
"types": ["node"], // Only use types from Node.js and packages that are explicitly imported
"moduleResolution": "bundler",
"strict": true,
"declaration": true, // Generates `.d.ts` files for type safety

View File

@@ -0,0 +1,65 @@
### `create([opts], [requestListener])`
Creates a new server instance. `opts` may contain `createServer` and `createSecureServer` factories (defaults to `http.createServer` and `https.createServer`). `requestListener` is registered as a `request` event listener.
```js
import { create } from '@vates/http-server-plus'
const server = create(function onRequest(req, res) {
res.end('Hello')
})
```
### `server.listen(opts)`
Starts listening. Returns a promise that resolves to a human-readable address string, or rejects with `error.niceAddress` set.
```js
// HTTP on a specific port
await server.listen({ port: 8080 })
// HTTPS — detected automatically when cert+key are provided
await server.listen({
port: 443,
cert: fs.readFileSync('certificate.pem'),
key: fs.readFileSync('private_key.pem'),
})
// Unix socket
await server.listen({ socket: '/var/run/http.sock' })
// File descriptor
await server.listen({ fd: 3 })
// Systemd socket activation
await server.listen({ systemdSocket: 0 })
```
> As a convenience, if `hostname` is `'localhost'`, it will listen on both `127.0.0.1` and `::1`, similar to what Node does when no hostname is provided.
Multiple endpoints can be started on the same instance:
```js
const [httpAddr, httpsAddr] = await Promise.all([server.listen({ port: 80 }), server.listen({ port: 443, cert, key })])
console.log('listening on', httpAddr, httpsAddr)
```
### HTTP/2
Pass a custom `createSecureServer` factory to enable HTTP/2:
```js
import { createSecureServer } from 'node:http2'
import { create } from '@vates/http-server-plus'
const server = create({ createSecureServer }, app)
await server.listen({ port: 443, cert, key })
```
### `server.close()`
Closes all underlying servers. Returns a promise that resolves once the `close` event is emitted.
```js
await server.close()
```

View File

@@ -0,0 +1 @@
../../scripts/npmignore

View File

@@ -0,0 +1,91 @@
<!-- DO NOT EDIT MANUALLY, THIS FILE HAS BEEN GENERATED -->
# @vates/http-server-plus
> Augmented &#x60;http.Server&#x60;, HTTP/HTTPS/HTTP2 and multiple ports on the same instance
## Usage
### `create([opts], [requestListener])`
Creates a new server instance. `opts` may contain `createServer` and `createSecureServer` factories (defaults to `http.createServer` and `https.createServer`). `requestListener` is registered as a `request` event listener.
```js
import { create } from '@vates/http-server-plus'
const server = create(function onRequest(req, res) {
res.end('Hello')
})
```
### `server.listen(opts)`
Starts listening. Returns a promise that resolves to a human-readable address string, or rejects with `error.niceAddress` set.
```js
// HTTP on a specific port
await server.listen({ port: 8080 })
// HTTPS — detected automatically when cert+key are provided
await server.listen({
port: 443,
cert: fs.readFileSync('certificate.pem'),
key: fs.readFileSync('private_key.pem'),
})
// Unix socket
await server.listen({ socket: '/var/run/http.sock' })
// File descriptor
await server.listen({ fd: 3 })
// Systemd socket activation
await server.listen({ systemdSocket: 0 })
```
> As a convenience, if `hostname` is `'localhost'`, it will listen on both `127.0.0.1` and `::1`, similar to what Node does when no hostname is provided.
Multiple endpoints can be started on the same instance:
```js
const [httpAddr, httpsAddr] = await Promise.all([
server.listen({ port: 80 }),
server.listen({ port: 443, cert, key }),
])
console.log('listening on', httpAddr, httpsAddr)
```
### HTTP/2
Pass a custom `createSecureServer` factory to enable HTTP/2:
```js
import { createSecureServer } from 'node:http2'
import { create } from '@vates/http-server-plus'
const server = create({ createSecureServer }, app)
await server.listen({ port: 443, cert, key })
```
### `server.close()`
Closes all underlying servers. Returns a promise that resolves once the `close` event is emitted.
```js
await server.close()
```
## Contributions
Contributions are _very_ welcomed, either on the documentation or on
the code.
You may:
- report any [issue](https://github.com/vatesfr/xen-orchestra/issues)
you've encountered;
- fork and create a pull request.
## License
[ISC](https://spdx.org/licenses/ISC) © [Vates SAS](https://vates.fr)

View File

@@ -0,0 +1,268 @@
'use strict'
// ===================================================================
const EventEmitter = require('events').EventEmitter
const formatUrl = require('url').format
const fromEvent = require('promise-toolbox/fromEvent')
const inherits = require('util').inherits
const resolvePath = require('path').resolve
// ===================================================================
const forwardedEvents = [
'checkContinue',
'checkExpectation',
'clientError',
'connect',
'connection',
'listening',
'request',
'secureConnection',
'upgrade',
]
function extractProperty(obj, prop) {
const value = obj[prop]
if (value !== undefined) {
delete obj[prop]
return value
}
}
function forOwn(obj, cb) {
const keys = Object.keys(obj)
for (let i = 0, n = keys.length; i < n; ++i) {
cb(obj[keys[i]])
}
}
function getSystemdFd(index) {
if (process.pid !== +process.env.LISTEN_PID) {
throw new Error('systemd sockets are not meant for us (LISTEN_PID)')
}
const listenFds = process.env.LISTEN_FDS
if (listenFds == null) {
throw new Error('no systemd sockets found (LISTEN_FDS)')
}
const max = listenFds - 1
if (index > max) {
throw new Error('no such systemd socket found')
}
return index + 3 // 3 = SD_LISTEN_FDS_START
}
const isEmpty = obj => Object.keys(obj).length === 0
// ===================================================================
function onNewListener(event, listener) {
if (forwardedEvents.includes(event)) {
forOwn(this._servers, server => {
server.on(event, listener)
})
}
}
function onRemoveListener(event, listener) {
if (forwardedEvents.includes(event)) {
forOwn(this._servers, server => {
server.off(event, listener)
})
}
}
function Server(opts) {
EventEmitter.call(this)
this._createServer = (opts != null && opts.createServer) || require('http').createServer
this._createSecureServer = (opts != null && opts.createSecureServer) || require('https').createServer
this._servers = Object.create(null)
this._niceAddresses = Object.create(null)
this.on('removeListener', onRemoveListener).on('newListener', onNewListener)
}
inherits(Server, EventEmitter)
const proto = Server.prototype
function getAddress(serverId) {
return this[serverId].address()
}
proto.addresses = function Server$addresses() {
const servers = this._servers
return Object.keys(servers).map(getAddress, servers)
}
function getNiceAddress(serverId) {
return this[serverId]
}
proto.niceAddresses = function Server$niceAddresses() {
const niceAddresses = this._niceAddresses
return Object.keys(niceAddresses).map(getNiceAddress, niceAddresses)
}
function close(server) {
server.close()
}
proto.close = function Server$close(callback) {
if (callback) {
this.once('close', callback)
}
// Emit the close event even if there are no registered servers.
if (isEmpty(this._servers)) {
const self = this
setImmediate(function () {
self.emit('close')
})
return Promise.resolve()
}
// Closes each servers.
forOwn(this._servers, close)
return fromEvent(this, 'close')
}
let nextId = 0
proto.listen = function Server$listen(opts) {
opts = Object.assign({}, opts)
if (opts.hostname === 'localhost') {
return Promise.all([
this.listen(Object.assign(opts, { hostname: '::1' })),
this.listen(Object.assign(opts, { hostname: '127.0.0.1' })),
]).then(addresses => addresses[0])
}
const hostname = extractProperty(opts, 'hostname')
let port = extractProperty(opts, 'port')
const systemdSocket = extractProperty(opts, 'systemdSocket')
let fd = extractProperty(opts, 'fd')
let socket = extractProperty(opts, 'socket')
const servers = this._servers
const niceAddresses = this._niceAddresses
let server, protocol
if (opts.pfx || opts.SNICallback || (opts.cert && opts.key)) {
server = this._createSecureServer(opts)
protocol = 'https'
} else {
server = this._createServer(opts)
protocol = 'http'
}
const id = nextId++
servers[id] = server
if (systemdSocket != null) {
fd = getSystemdFd(systemdSocket)
}
// Compute a temporary nice address to display in case of error.
let niceAddress
if (fd != null) {
server.listen({ fd })
niceAddress = protocol + '://<fd:' + fd + '>'
} else if (socket != null) {
socket = resolvePath(socket)
server.listen(socket)
niceAddress = protocol + '://' + socket
} else {
if (port == null) {
port = protocol === 'https' ? 443 : 80
}
server.listen(port, hostname)
niceAddress = formatUrl({
protocol,
// Hostname default to localhost.
hostname: hostname || 'localhost',
// No port means random, unknown for now.
port: port || '<unknown>',
})
}
const emit = this.emit.bind(this)
server.once('close', function onClose() {
delete servers[id]
delete niceAddresses[id]
if (isEmpty(servers)) {
emit('close')
}
})
server.on('error', function onError() {
delete servers[id]
// FIXME: Should it be forwarded and be fatal if there are no
// listeners?
})
forwardedEvents.forEach(event => {
this.listeners(event).forEach(listener => {
server.on(event, listener)
})
})
return fromEvent(server, 'listening').then(
function () {
const address = server.address()
const niceAddress =
typeof address === 'string'
? protocol + '://' + address
: formatUrl({
protocol,
hostname: address.address,
port: address.port,
})
niceAddresses[id] = niceAddress
return niceAddress
},
function (error) {
error.niceAddress = niceAddress
throw error
}
)
}
function ref(server) {
server.ref()
}
proto.ref = function Server$ref() {
forOwn(this._servers, ref)
}
function unref(server) {
server.unref()
}
proto.unref = function Server$unref() {
forOwn(this._servers, unref)
}
// ===================================================================
module.exports = exports = Server
exports.create = function create(opts, requestListener) {
if (typeof opts === 'function') {
requestListener = opts
opts = undefined
}
const server = new Server(opts)
if (requestListener !== undefined) {
server.on('request', requestListener)
}
return server
}

View File

@@ -0,0 +1,303 @@
'use strict'
// ===================================================================
const assert = require('assert/strict')
const { EventEmitter } = require('events')
const resolvePath = require('path').resolve
const { describe, it } = require('node:test')
const HttpServerPlus = require('./')
// ===================================================================
// Creates a minimal mock server that can be used as a drop-in for http/https.Server.
// `address` is the value returned by mock.address() — use an object for TCP, a string for sockets.
function createMock(address) {
const mock = new EventEmitter()
mock.listen = function () {
setImmediate(() => this.emit('listening'))
}
mock.address = () => address
mock.close = function () {
setImmediate(() => this.emit('close'))
}
return mock
}
// ===================================================================
describe('HttpServerPlus', function () {
describe('#create()', function () {
it('creates a new instance', function () {
assert(HttpServerPlus.create() instanceof HttpServerPlus)
})
it('can register a `request` listener', function () {
const listener = function () {}
const server = HttpServerPlus.create(listener)
assert.deepEqual(server.listeners('request'), [listener])
})
})
describe('.addresses()', function () {
it('returns the list of addresses the server is listening at', function () {
const addr = { address: '127.0.0.1', port: 8080, family: 'IPv4' }
const mock = createMock(addr)
const server = HttpServerPlus.create({ createServer: () => mock })
return server.listen({ port: 8080 }).then(function () {
assert.deepEqual(server.addresses(), [addr])
return server.close()
})
})
})
describe('.niceAddresses()', function () {
it('returns the list of human readable addresses the server is listening at', function () {
const mock = createMock({ address: '127.0.0.1', port: 8080 })
const server = HttpServerPlus.create({ createServer: () => mock })
return server.listen({ hostname: '127.0.0.1', port: 8080 }).then(function (address) {
assert.deepEqual(server.niceAddresses(), [address])
return server.close()
})
})
})
describe('secureConnection event forwarding', function () {
function createMockHttpsServer() {
const server = new EventEmitter()
server.listen = function () {
setImmediate(() => this.emit('listening'))
}
server.address = () => ({ address: '127.0.0.1', port: 8443 })
server.close = function () {
setImmediate(() => this.emit('close'))
}
return server
}
function createServerWithMock() {
let mockServer
const server = HttpServerPlus.create({
createSecureServer: () => {
mockServer = createMockHttpsServer()
return mockServer
},
})
return {
server,
getMock: () => mockServer,
}
}
it('forwards secureConnection to listeners registered before listen()', function () {
const { server, getMock } = createServerWithMock()
const received = []
server.on('secureConnection', socket => received.push(socket))
const fakeSocket = {}
return server.listen({ cert: 'fake-cert', key: 'fake-key' }).then(function () {
getMock().emit('secureConnection', fakeSocket)
assert.deepEqual(received, [fakeSocket])
return server.close()
})
})
it('forwards secureConnection to listeners registered after listen()', function () {
const { server, getMock } = createServerWithMock()
return server.listen({ cert: 'fake-cert', key: 'fake-key' }).then(function () {
const received = []
server.on('secureConnection', socket => received.push(socket))
const fakeSocket = {}
getMock().emit('secureConnection', fakeSocket)
assert.deepEqual(received, [fakeSocket])
return server.close()
})
})
it('stops forwarding secureConnection after listener is removed', function () {
const { server, getMock } = createServerWithMock()
const received = []
const listener = socket => received.push(socket)
server.on('secureConnection', listener)
return server.listen({ cert: 'fake-cert', key: 'fake-key' }).then(function () {
server.off('secureConnection', listener)
getMock().emit('secureConnection', {})
assert.deepEqual(received, [])
return server.close()
})
})
})
describe('.listen()', function () {
it('can use a host:port', function () {
let listenArgs
const mock = createMock({ address: '127.0.0.1', port: 8080 })
mock.listen = function (...args) {
listenArgs = args
setImmediate(() => this.emit('listening'))
}
const server = HttpServerPlus.create({ createServer: () => mock })
return server.listen({ hostname: '127.0.0.1', port: 8080 }).then(function () {
assert.deepEqual(listenArgs, [8080, '127.0.0.1'])
return server.close()
})
})
it('can use a systemd socket', function () {
const SD_LISTEN_FDS_START = 3
const SOCKET_INDEX = 0
let listenArgs
const mock = createMock('http://<fd:3>')
mock.listen = function (...args) {
listenArgs = args
setImmediate(() => this.emit('listening'))
}
const server = HttpServerPlus.create({ createServer: () => mock })
const env = process.env
env.LISTEN_FDS = String(SOCKET_INDEX + 1)
env.LISTEN_PID = String(process.pid)
return server.listen({ systemdSocket: SOCKET_INDEX }).then(function () {
delete env.LISTEN_FDS
delete env.LISTEN_PID
assert.deepEqual(listenArgs, [{ fd: SOCKET_INDEX + SD_LISTEN_FDS_START }])
return server.close()
})
})
it('can use a socket', function () {
const socketPath = '/tmp/http-server-plus-test.sock'
let listenArgs
const mock = createMock(socketPath)
mock.listen = function (...args) {
listenArgs = args
setImmediate(() => this.emit('listening'))
}
const server = HttpServerPlus.create({ createServer: () => mock })
return server.listen({ socket: socketPath }).then(function () {
assert.deepEqual(listenArgs, [resolvePath(socketPath)])
return server.close()
})
})
it('can use a HTTPS certificate', function () {
let capturedOpts
const mock = createMock({ address: '127.0.0.1', port: 8443 })
const server = HttpServerPlus.create({
createSecureServer: opts => {
capturedOpts = opts
return mock
},
})
return server.listen({ cert: 'my-cert', key: 'my-key', port: 8443 }).then(function (address) {
assert.equal(capturedOpts.cert, 'my-cert')
assert.equal(capturedOpts.key, 'my-key')
assert.ok(address.startsWith('https://'), `expected https URL, got: ${address}`)
return server.close()
})
})
it('returns a promise which will resolve once listening', function () {
const mock = createMock({ address: '127.0.0.1', port: 8080 })
const server = HttpServerPlus.create({ createServer: () => mock })
return server.listen({ port: 8080 }).then(function (address) {
assert.equal(typeof address, 'string')
assert.ok(address.startsWith('http://'), `expected http URL, got: ${address}`)
return server.close()
})
})
it('returns a promise which will reject if failure', function () {
const listenError = new Error('EADDRINUSE')
const mock = createMock(null)
mock.listen = function () {
setImmediate(() => this.emit('error', listenError))
}
const server = HttpServerPlus.create({ createServer: () => mock })
return server.listen({ port: 8080 }).then(
function () {
assert.fail('should have rejected')
},
function (err) {
assert.equal(err, listenError)
assert.equal(typeof err.niceAddress, 'string')
}
)
})
})
describe('.close()', function () {
it('closes all servers', function () {
const closed = []
let callCount = 0
const server = HttpServerPlus.create({
createServer: () => {
const id = callCount++
const mock = createMock({ address: '127.0.0.1', port: id })
mock.close = function () {
closed.push(id)
setImmediate(() => this.emit('close'))
}
return mock
},
})
return Promise.all([server.listen({ port: 0 }), server.listen({ port: 0 })])
.then(function () {
return server.close()
})
.then(function () {
assert.equal(closed.length, 2)
})
})
it('emit the `close` event when all servers are closed', function () {
const mock = createMock({ address: '127.0.0.1', port: 8080 })
const server = HttpServerPlus.create({ createServer: () => mock })
let closeEmitted = false
server.on('close', () => {
closeEmitted = true
})
return server
.listen({ port: 8080 })
.then(function () {
return server.close()
})
.then(function () {
assert.ok(closeEmitted)
})
})
it('emit the `close` event even if there are no servers', function () {
const server = HttpServerPlus.create()
// close() with no servers returns Promise.resolve() immediately (before emitting
// 'close' via setImmediate), so we must wait for the event itself, not the promise.
return new Promise(function (resolve) {
server.close(resolve)
})
})
})
})

View File

@@ -0,0 +1,37 @@
{
"name": "@vates/http-server-plus",
"version": "2.0.0",
"license": "ISC",
"description": "Augmented `http.Server`, HTTP/HTTPS/HTTP2 and multiple ports on the same instance",
"keywords": [
"http",
"https",
"server",
"ports"
],
"homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@vates/http-server-plus",
"bugs": "https://github.com/vatesfr/xen-orchestra/issues",
"repository": {
"directory": "@vates/http-server-plus",
"type": "git",
"url": "https://github.com/vatesfr/xen-orchestra.git"
},
"engines": {
"node": ">=20"
},
"dependencies": {
"promise-toolbox": "^0.21.0"
},
"devDependencies": {
"eslint-plugin-n": "^15.1.0"
},
"scripts": {
"pretest": "eslint .",
"test": "node --test index.spec.js"
},
"private": true,
"author": {
"name": "Vates SAS",
"url": "https://vates.fr"
}
}

View File

@@ -40,7 +40,17 @@ export class NbdDisk extends RandomAccessDisk {
if (!this.#nbdClient) {
throw new Error("can't readBlock before init")
}
const data = await this.#nbdClient.readBlock(index, this.getBlockSize())
let data = await this.#nbdClient.readBlock(index, this.getBlockSize())
if (data.length !== this.getBlockSize()) {
// this condition only work on non aligned disks
// we won't accept truncated block from unaligned disks
if (index === Math.ceil(this.getVirtualSize() / this.getBlockSize()) - 1) {
// last block add zeros at the end
data = Buffer.concat([data], this.getBlockSize())
} else {
throw new Error(`Block ${index} is not at the right size, expecting ${this.getBlockSize()}, got ${data.length}`)
}
}
return {
index,
data,
@@ -117,6 +127,6 @@ export class NbdDisk extends RandomAccessDisk {
}
const blockStart = index * this.getBlockSize()
const blockEnd = (index + 1) * this.getBlockSize()
return this.#dataMap.some(({ offset, length }) => offset + length > blockStart && blockEnd >= offset)
return this.#dataMap.some(({ offset, length }) => offset + length > blockStart && offset < blockEnd)
}
}

View File

@@ -353,6 +353,7 @@ export default class NbdClient {
async readBlock(index, size = NBD_DEFAULT_BLOCK_SIZE) {
return pRetry(() => this.#readBlock(index, size), {
tries: this.#readBlockRetries,
when: error => error.code !== 'ERR_ABORTED',
onRetry: async err => {
warn('will retry reading block ', index, err)
await this.reconnect()
@@ -368,9 +369,9 @@ export default class NbdClient {
* A promise that resolves to an array where each object represents a segment:
* - `offset` — The byte offset from the start.
* - `length` — The size of the segment in bytes.
* - `type` — A numeric code indicating the segment type ( 0 means no data).
* - `type` — A numeric code indicating the segment type ( 0 means data).
*/
/* async */ getMap() {
/* async */ getMap(signal) {
return new Promise((resolve, reject) => {
const process = spawn('nbdinfo', [
'--json',
@@ -381,8 +382,17 @@ export default class NbdClient {
let errText = ''
process.stdout.on('data', data => (text += data))
process.stderr.on('data', data => (errText += data))
process.on('error', reject)
const onAbort = () => {
process.kill()
reject(signal.reason)
}
signal?.addEventListener('abort', onAbort, { once: true })
process.on('error', err => {
signal?.removeEventListener('abort', onAbort)
reject(err)
})
process.on('close', code => {
signal?.removeEventListener('abort', onAbort)
if (code !== 0) {
return reject(new Error(`Error during getMap (code: ${code}): ${errText}`))
}

View File

@@ -138,8 +138,8 @@ export default class MultiNbdClient {
* - `length` — The size of the segment in bytes.
* - `type` — A numeric code indicating the segment type (0 means no data).
*/
async getMap() {
async getMap(signal) {
// ask the map from one of the connected client
return this.#clients[Math.floor(this.#clients.length * Math.random())].getMap()
return this.#clients[Math.floor(this.#clients.length * Math.random())].getMap(signal)
}
}

View File

@@ -13,19 +13,19 @@
"url": "https://vates.fr"
},
"license": "ISC",
"version": "3.2.1",
"version": "3.4.0",
"engines": {
"node": ">=14.0"
},
"main": "./index.mjs",
"dependencies": {
"@vates/async-each": "^1.0.0",
"@vates/async-each": "^1.0.3",
"@vates/read-chunk": "^1.2.1",
"@xen-orchestra/async-map": "^0.1.2",
"@xen-orchestra/disk-transform": "^1.0.1",
"@xen-orchestra/log": "^0.7.1",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/disk-transform": "^1.2.3",
"@xen-orchestra/log": "^0.7.2",
"promise-toolbox": "^0.21.0",
"xen-api": "^4.7.4"
"xen-api": "^4.7.7"
},
"devDependencies": {
"tap": "^18.7.0",

View File

@@ -1,6 +1,6 @@
{
"name": "@vates/node-vsphere-soap",
"version": "2.1.2",
"version": "2.1.3",
"description": "interface to vSphere SOAP/WSDL from node for interfacing with vCenter or ESXi, forked from node-vsphere-soap",
"main": "lib/client.mjs",
"author": "reedog117",
@@ -11,8 +11,8 @@
},
"bugs": "https://github.com/vatesfr/xen-orchestra/issues",
"dependencies": {
"@xen-orchestra/log": "^0.7.1",
"axios": "^1.8.0",
"@xen-orchestra/log": "^0.7.2",
"axios": "^1.15.0",
"soap": "^1.0.0",
"soap-cookie": "^0.10.1"
},

View File

@@ -23,7 +23,7 @@ exports.makeOnProgress = function ({ onRootTaskEnd = noop, onRootTaskStart = noo
Object.defineProperty(taskLog, '$root', { value: taskLog })
// start of a root task
onRootTaskStart(taskLog)
onRootTaskStart(taskLog, event)
} else {
// start of a subtask
const parent = taskLogs.get(parentId)
@@ -54,10 +54,10 @@ exports.makeOnProgress = function ({ onRootTaskEnd = noop, onRootTaskStart = noo
}
if (type === 'end' && taskLog.$root === taskLog) {
onRootTaskEnd(taskLog)
onRootTaskEnd(taskLog, event)
}
}
onTaskUpdate(taskLog)
onTaskUpdate(taskLog, event)
}
}

View File

@@ -7,6 +7,27 @@ function alreadyEnded() {
throw new Error('task has already ended')
}
const serializeErrors = errors => (Array.isArray(errors) ? errors.map(serializeError) : errors)
// Create a serializable object from an error.
//
// Otherwise some fields might be non-enumerable and missing from logs.
const serializeError = error => {
return error instanceof Error
? {
...error, // Copy enumerable properties.
cause: error.cause,
code: error.code,
errors: serializeErrors(error.errors), // supports AggregateError
message: error.message,
name: error.name,
stack: error.stack,
}
: error
}
exports.serializeError = serializeError
// define a read-only, non-enumerable, non-configurable property
function define(object, property, value) {
Object.defineProperty(object, property, { value })
@@ -72,7 +93,11 @@ class Task {
_onProgress
get id() {
return (this.id = Math.random().toString(36).slice(2))
// Use a compact, sortable, string representation of the creation date,
// and add a random part to differentiate tasks created at the same time
//
// Due to the padding, dates are sortable up to 5188-04-22T11:04:28.415Z
return (this.id = Date.now().toString(36).padStart(9, '0') + '-' + Math.random().toString(36).slice(2))
}
set id(value) {
define(this, 'id', value)
@@ -136,6 +161,10 @@ class Task {
assert.equal(this.#status, PENDING)
assert.equal(this.#running, false)
if (result instanceof Error && result.toJSON === undefined) {
result = serializeError(result)
}
this.#emit('end', { status, result })
this._onProgress = alreadyEnded
this.#status = status

View File

@@ -4,7 +4,7 @@ const assert = require('node:assert').strict
const { describe, it } = require('node:test')
const { makeOnProgress } = require('./combineEvents.js')
const { Task } = require('./index.js')
const { Task, serializeError } = require('./index.js')
const noop = Function.prototype
@@ -28,6 +28,36 @@ function createTask(opts) {
return task
}
describe('serializeError', function () {
it('serializes simple error', function () {
const err = new Error('test Error')
err.code = 404
const serialized = serializeError(err)
assert.equal(serialized.message, 'test Error')
assert.equal(serialized.name, 'Error')
assert.equal(serialized.code, 404)
assert.notEqual(serialized.stack, undefined)
})
it('serializes aggregate error', function () {
const err = new AggregateError([new Error('test error 1'), new Error('test error 2')], 'Error message', {
cause: 'broken',
})
const serialized = serializeError(err)
assert.equal(serialized.message, 'Error message')
assert.equal(serialized.name, 'AggregateError')
assert.equal(serialized.cause, 'broken')
assert.notEqual(serialized.stack, undefined)
assert.equal(serialized.errors.length, 2)
assert.equal(serialized.errors[0].name, 'Error')
assert.equal(serialized.errors[1].name, 'Error')
assert.equal(serialized.errors[0].message, 'test error 1')
assert.equal(serialized.errors[1].message, 'test error 2')
assert.notEqual(serialized.errors[0].stack, undefined)
assert.notEqual(serialized.errors[1].stack, undefined)
})
})
describe('Task', function () {
describe('constructor', function () {
it('data properties are passed to the start event', async function () {
@@ -106,7 +136,7 @@ describe('Task', function () {
assert.equal(task.$events.length, 3)
assertEvent(task, { type: 'start' }, 0)
assertEvent(task, { type: 'abortionRequested', reason }, 1)
assertEvent(task, { type: 'end', status: 'failure', result }, 2)
assertEvent(task, { type: 'end', status: 'failure', result: serializeError(result) }, 2)
})
it('does not abort if the task succeed', async function () {
@@ -176,7 +206,7 @@ describe('Task', function () {
assertEvent(task, {
status: 'failure',
result: error,
result: serializeError(error),
type: 'end',
})
})
@@ -475,7 +505,7 @@ describe('Task', function () {
assert.equal(task.status, 'failure')
assertEvent(task, {
status: 'failure',
result: e,
result: serializeError(e),
type: 'end',
})
})
@@ -511,7 +541,7 @@ describe('Task', function () {
assert.equal(task.status, 'failure')
assertEvent(task, {
status: 'failure',
result: e,
result: serializeError(e),
type: 'end',
})
})

View File

@@ -13,7 +13,7 @@
"url": "https://vates.fr"
},
"license": "ISC",
"version": "0.6.2",
"version": "0.7.0",
"engines": {
"node": ">=14"
},

View File

@@ -1,7 +1,7 @@
{
"name": "@vates/types",
"private": false,
"version": "1.12.0",
"version": "1.23.0",
"main": "./dist/index.mjs",
"exports": {
".": {
@@ -9,6 +9,7 @@
"types": "./dist/index.d.mts"
},
"./xo": "./dist/xo.d.mts",
"./xo-app": "./dist/xo-app.d.mts",
"./xen-api": "./dist/xen-api.d.mts",
"./common": {
"default": "./dist/common.mjs",

View File

@@ -1,5 +1,6 @@
export * from './common.mjs'
export * from './xen-api.mjs'
export * from './xo.mjs'
export * from './xo-app.mjs'
export * from './lib/xen-orchestra-xapi.mjs'
export * from './lib/complex-matcher.mjs'

View File

@@ -1,28 +1,42 @@
import {
WrappedXenApiRecord,
XenApiHost,
XenApiHostWrapped,
XenApiNetwork,
XenApiNetworkWrapped,
XenApiRecord,
XenApiSr,
XenApiTask,
XenApiVbd,
XenApiVdi,
XenApiVif,
XenApiVm,
XenApiVmWrapped,
XenApiVtpm,
} from '../xen-api.mjs'
import type {
BOND_MODE,
OPAQUE_REF_NULL,
SUPPORTED_VDI_FORMAT,
VBD_MODE,
VBD_TYPE,
VDI_TYPE,
VIF_LOCKING_MODE,
} from '../common.mjs'
import type { PassThrough, Readable } from 'node:stream'
import type {
XoGpuGroup,
XoVgpuType,
XoHost,
XoNetwork,
XoPif,
XoSr,
XoUser,
XoVdi,
XoVgpuType,
XoVm,
XoVmTemplate,
XoVif,
} from '../xo.mjs'
import type { SUPPORTED_VDI_FORMAT } from '../common.mjs'
export type XcpPatches = {
changelog?: {
@@ -57,11 +71,13 @@ export interface Xapi {
call: <ReturnType>(...args: unknown[]) => Promise<ReturnType>
callAsync: <ReturnType>(...args: unknown[]) => Promise<ReturnType>
barrier<T extends XenApiRecord>(ref: T['$ref']): Promise<Extract<WrappedXenApiRecord, { $ref: T['$ref'] }>>
getField<T extends XenApiRecord, K extends keyof T>(
type: Extract<WrappedXenApiRecord, T>['$type'],
ref: T['$ref'],
field: K
): Promise<T[K]>
connectVif(vifId: XoVif['id']): Promise<void>
createNetwork(
params:
| {
@@ -78,12 +94,58 @@ export interface Xapi {
vlan: number
}
): Promise<XenApiNetworkWrapped>
createBondedNetwork(params: {
bondMode: BOND_MODE
pifIds: XoPif['id'][]
name: string
description?: string
mtu?: number
}): Promise<XenApiNetworkWrapped>
deleteNetwork(id: XoNetwork['id']): Promise<void>
deleteVif(vifId: XoVif['id']): Promise<void>
disconnectVif(vifId: XoVif['id']): Promise<void>
exportVmOva(vmRef: XenApiVm['$ref']): Promise<PassThrough>
migrateVm(
vmId: XoVm['id'],
hostXapi: Xapi,
hostId: XoHost['id'],
opts?: {
force?: boolean
mapVdisSrs?: Record<XoVdi['id'], XoSr['id']>
mapVifsNetworks?: Record<XoVif['id'], XoNetwork['id']>
migrationNetworkId?: XoNetwork['id']
sr?: XoSr['id']
bypassAssert?: boolean
}
): Promise<void>
listMissingPatches(host: XoHost['id']): Promise<XcpPatches[] | XsPatches[]>
pool_emergencyShutdown(): Promise<void>
resumeVm(id: XoVm['id']): Promise<void>
unpauseVm(id: XoVm['id']): Promise<void>
cloneVm(
vmId: XoVm['id'],
opts?: {
nameLabel?: string
fast?: boolean
}
): Promise<XenApiVmWrapped>
copyVm(
vmId: XoVm['id'],
opts?: {
nameLabel?: string
srOrSrId?: XoSr['id'] | XoSr
}
): Promise<XenApiVmWrapped>
remoteCopyVm(
vmId: XoVm['id'],
targetXapi: Xapi,
targetSrId: XoSr['id'],
opts?: {
compress?: 'zstd' | 'gzip' | boolean
nameLabel?: string
}
): Promise<{ vm: XenApiVmWrapped }>
forgetSr(id: XoSr['id']): Promise<void>
SR_importVdi(
ref: XenApiSr['$ref'],
stream: Readable,
@@ -101,6 +163,7 @@ export interface Xapi {
xenstore_data?: Record<string, string>
}
): Promise<XenApiVdi['$ref']>
SR_reclaimSpace(ref: XenApiSr['$ref']): Promise<void>
startVm(
id: XoVm['id'],
opts?: {
@@ -130,6 +193,7 @@ export interface Xapi {
metadataVm: {
affinityHost?: XoHost['id']
autoPoweron?: boolean
cpus?: number
memory?: number
name_description?: string
name_label: string
@@ -171,17 +235,79 @@ export interface Xapi {
vgpuType?: XoVgpuType['id']
gpuGroup?: XoGpuGroup['id']
copyHostBiosStrings?: boolean
hvmBootFirmware?: 'uefi' | 'bios'
secureBoot?: boolean
},
checkLimits?: boolean,
creatorId?: XoUser['id'],
opts?: { destroyAllVifs: boolean }
): Promise<XenApiVmWrapped>
moveVdi(
vdiId: XenApiVdi['$ref'] | XoVdi['id'],
srId: XenApiSr['$ref'] | XoSr['id'],
opts?: { _failOnCbtError?: boolean }
): Promise<XenApiVdi>
VBD_create(params: {
bootable?: boolean
empty?: boolean
mode?: VBD_MODE
other_config?: Record<string, string>
qos_algorithm_params?: Record<string, string>
qos_algorithm_type?: string
type?: VBD_TYPE
unpluggable?: boolean
userdevice?: string
VDI: XenApiVdi['$ref'] | OPAQUE_REF_NULL
VM: XenApiVm['$ref']
}): Promise<XenApiVbd['$ref']>
VBD_destroy(vbdRef: XenApiVbd['$ref']): Promise<void>
VBD_unplug(vbdRef: XenApiVbd['$ref']): Promise<void>
VDI_create(
options: {
name_description?: XoVdi['name_description']
name_label?: XoVdi['name_label']
other_config?: XoVdi['other_config']
read_only?: boolean
sharable?: boolean
SR?: XenApiSr['$ref']
tags?: XoVdi['tags']
type?: VDI_TYPE
virtual_size: XoVdi['size']
xenstore_data?: Record<string, string>
},
extraOptions?: {
sm_config?: Record<string, string>
}
): Promise<XenApiVdi['$ref']>
VDI_destroy(vdiRef: XenApiVdi['$ref']): Promise<void>
VDI_destroyCloudInitConfig(vdiRef: XenApiVdi['$ref'], opts?: { timeLimit?: number }): Promise<void>
VDI_exportContent(
vdiRef: XenApiVdi['$ref'],
opts: { baseRef?: string; cancelToken?: unknown; format: SUPPORTED_VDI_FORMAT }
): Promise<Readable & { length?: number }>
VDI_importContent(
vdiRef: XenApiVdi['$ref'],
stream: Readable,
opts: { cancelToken?: unknown; format: SUPPORTED_VDI_FORMAT }
): Promise<void>
VIF_create(
options: {
currently_attached?: boolean
device?: string
ipv4_allowed?: string[]
ipv6_allowed?: string[]
locking_mode?: VIF_LOCKING_MODE
MTU?: number
network: XenApiNetwork['$ref']
other_config?: Record<string, string>
qos_algorithm_params?: Record<string, string>
qos_algorithm_type?: string
VM: XenApiVm['$ref']
},
extraOptions?: {
MAC?: string
}
): Promise<XenApiVif['$ref']>
VM_createCloudInitConfig(
vmRef: XenApiVm['$ref'],
cloudConfig: string,
@@ -200,5 +326,16 @@ export interface Xapi {
pathname: string,
params?: { host?: XenApiHost; query?: Record<string, unknown>; task?: boolean | XenApiTask['$ref'] }
): Promise<{ body: Readable }>
clearHost(host: Pick<XenApiHostWrapped, '$ref' | '$pool'>, force?: boolean): Promise<void>
disableHost(hostId: XoHost['id']): Promise<void>
enableHost(hostId: XoHost['id']): Promise<void>
getRecordByUuid<
Type extends WrappedXenApiRecord['$type'],
XenApiRecord extends WrappedXenApiRecord = Extract<WrappedXenApiRecord, { $type: Type }>,
>(
type: Type,
uuid: XenApiRecord['uuid']
): Promise<XenApiRecord>
isHyperThreadingEnabled(hostId: XoHost['id']): Promise<boolean | null>
VTPM_create(params: { VM: XenApiVm['$ref']; is_unique?: boolean; contents?: string }): Promise<XenApiVtpm['$ref']>
}

View File

@@ -70,6 +70,11 @@ import type { Xapi } from './lib/xen-orchestra-xapi.mjs'
// https://github.com/vatesfr/xen-orchestra/tree/wip-xapi-types-generator/%40xen-orchestra/xapi-generator
// All properties added after XenServer 7.0 (dundee) release have been marked as optional
type TagCallMethods = {
(method: 'add_tags', tag: string): Promise<void>
(method: 'remove_tags', tag: string): Promise<void>
}
/**
* Add properties injected by `xen-api`.
* $ref property is also injected by XOLite, so she is not present here.
@@ -77,6 +82,7 @@ import type { Xapi } from './lib/xen-orchestra-xapi.mjs'
type WrapperXenApi<T, Type extends string, Fn = { (): void }> = T & {
$call: Fn
$callAsync: Fn
$pool: XenApiPool
$type: Type
$snapshot(params: {
cancelToken?: unknown
@@ -162,6 +168,8 @@ export interface XenApiEvent {
timestamp?: string
}
type XenApiPoolCallMethods = TagCallMethods & {}
export interface XenApiPool {
$ref: Branded<'pool'>
allowed_operations: POOL_ALLOWED_OPERATIONS[]
@@ -224,7 +232,7 @@ export interface XenApiPool {
/** @deprecated */
wlb_verify_cert?: boolean
}
export type XenApiPoolWrapped = WrapperXenApi<XenApiPool, 'pool'>
export type XenApiPoolWrapped = WrapperXenApi<XenApiPool, 'pool', XenApiPoolCallMethods>
/** @deprecated */
export interface XenApiPoolPatch {
@@ -256,7 +264,7 @@ export interface XenApiPoolUpdate {
version?: string
}
type XenApiVmCallMethods = {
type XenApiVmCallMethods = TagCallMethods & {
(method: 'start', start_paused: boolean, force: boolean): Promise<void>
(method: 'clean_shutdown'): Promise<void>
(method: 'hard_shutdown'): Promise<void>
@@ -483,6 +491,7 @@ export interface XenApiDrTask {
uuid: string
}
type XenApiHostCallMethods = TagCallMethods & {}
export interface XenApiHost {
$ref: Branded<'host'>
address: string
@@ -559,7 +568,7 @@ export interface XenApiHost {
uuid: string
virtual_hardware_platform_versions: number[]
}
export type XenApiHostWrapped = WrapperXenApi<XenApiHost, 'host'>
export type XenApiHostWrapped = WrapperXenApi<XenApiHost, 'host', XenApiHostCallMethods>
export interface XenApiHostCrashdump {
$ref: Branded<'host_crashdump'>
@@ -614,6 +623,10 @@ export interface XenApiHostCpu {
vendor: string
}
type XenApiNetworkCallMethods = TagCallMethods & {
(method: 'add_purpose', purpose: NETWORK_PURPOSE): Promise<void>
}
export interface XenApiNetwork {
$ref: Branded<'network'>
allowed_operations: NETWORK_OPERATIONS[]
@@ -633,8 +646,7 @@ export interface XenApiNetwork {
uuid: string
VIFs: XenApiVif['$ref'][]
}
export interface XenApiNetworkWrapped extends WrapperXenApi<XenApiNetwork, 'network'> {}
export interface XenApiNetworkWrapped extends WrapperXenApi<XenApiNetwork, 'network', XenApiNetworkCallMethods> {}
export interface XenApiVif {
$ref: Branded<'VIF'>
allowed_operations: VIF_OPERATIONS[]
@@ -782,6 +794,7 @@ export interface XenApiSm {
}
export type XenApiSmWrapped = WrapperXenApi<XenApiSm, 'SM'>
type XenApiSrCallMethods = TagCallMethods & {}
export interface XenApiSr {
$ref: Branded<'SR'>
allowed_operations: STORAGE_OPERATIONS[]
@@ -806,7 +819,7 @@ export interface XenApiSr {
VDIs: XenApiVdi['$ref'][]
virtual_allocation: number
}
export type XenApiSrWrapped = WrapperXenApi<XenApiSr, 'SR'>
export type XenApiSrWrapped = WrapperXenApi<XenApiSr, 'SR', XenApiSrCallMethods>
export interface XenApiSrStat {
$ref: Branded<'sr_stat'>
@@ -832,6 +845,8 @@ export interface XenApiLvhd {
uuid: string
}
type XenApiVdiCallMethods = TagCallMethods & {}
export interface XenApiVdi {
$ref: Branded<'VDI'>
allow_caching: boolean
@@ -868,7 +883,7 @@ export interface XenApiVdi {
virtual_size: number
xenstore_data: Record<string, string>
}
export type XenApiVdiWrapped = WrapperXenApi<XenApiVdi, 'VDI'>
export type XenApiVdiWrapped = WrapperXenApi<XenApiVdi, 'VDI', XenApiVdiCallMethods>
export interface XenApiVbd {
$ref: Branded<'VBD'>
@@ -921,6 +936,7 @@ export interface XenApiPbd {
SR: XenApiSr['$ref']
uuid: string
}
export type XenApiPbdWrapped = WrapperXenApi<XenApiPbd, 'PBD'>
/** @deprecated */
export interface XenApiCrashdump {
@@ -995,6 +1011,7 @@ export interface XenApiMessage {
timestamp: string
uuid: string
}
export type XenApiMessageWrapped = WrapperXenApi<XenApiMessage, 'message'>
export interface XenApiSecret {
$ref: Branded<'secret'>
@@ -1357,6 +1374,7 @@ export type XenApiRecord =
export type WrappedXenApiRecord =
| XenApiHostWrapped
| XenApiNetworkWrapped
| XenApiPbdWrapped
| XenApiPifWrapped
| XenApiPoolWrapped
| XenApiSrWrapped

257
@vates/types/src/xo-app.mts Normal file
View File

@@ -0,0 +1,257 @@
import { EventEmitter } from 'node:stream'
import type {
AnyXoBackupJob,
AnyXoJob,
AnyXoLog,
XapiXoRecord,
XoAuthenticationToken,
XoBackupRepository,
XoConfigBackupArchive,
XoGroup,
XoHost,
XoPool,
XoPoolBackupArchive,
XoProxy,
XoSchedule,
XoServer,
XoTask,
XoUser,
XoVm,
XoVmBackupArchive,
} from './xo.mjs'
import { VatesTask } from './lib/vates-task.mjs'
import {
Xapi,
XapiHostStats,
XapiPoolStats,
XapiStatsGranularity,
XapiVmStats,
XenApiGpuGroupWrapped,
XenApiHostWrapped,
XenApiMessageWrapped,
XenApiNetworkWrapped,
XenApiPbdWrapped,
XenApiPciWrapped,
XenApiPgpuWrapped,
XenApiPifWrapped,
XenApiPoolWrapped,
XenApiSmWrapped,
XenApiSrWrapped,
XenApiVbdWrapped,
XenApiVdiWrapped,
XenApiVgpuTypeWrapped,
XenApiVgpuWrapped,
XenApiVifWrapped,
XenApiVmWrapped,
XenApiVtpmWrapped,
} from './index.mjs'
export type XapiConnection = Xapi & {
status: string
pool?: { uuid: string }
sessionId: string
_url?: { protocol: string; hostname: string; port?: string }
}
type XapiRecordByXapiXoRecord = {
gpuGroup: XenApiGpuGroupWrapped
host: XenApiHostWrapped
message: XenApiMessageWrapped
network: XenApiNetworkWrapped
PBD: XenApiPbdWrapped
PCI: XenApiPciWrapped
PGPU: XenApiPgpuWrapped
PIF: XenApiPifWrapped
pool: XenApiPoolWrapped
SR: XenApiSrWrapped
SM: XenApiSmWrapped
VBD: XenApiVbdWrapped
VDI: XenApiVdiWrapped
'VDI-snapshot': XenApiVdiWrapped
'VDI-unmanaged': XenApiVdiWrapped
vgpu: XenApiVgpuWrapped
vgpuType: XenApiVgpuTypeWrapped
VIF: XenApiVifWrapped
VM: XenApiVmWrapped
'VM-controller': XenApiVmWrapped
'VM-snapshot': XenApiVmWrapped
'VM-template': XenApiVmWrapped
VTPM: XenApiVtpmWrapped
}
export type XoApp = {
config: {
get<T = string>(path: string): T
getOptional(path: string): Record<string, string> | undefined
getOptionalDuration(path: string): number | undefined
getGuiRoutes(): Promise<{
default: {
url: string
path: string
}
v5?: {
url: string
path: string
}
v6?: {
url: string
path: string
}
[key: string]:
| undefined
| {
url: string
path: string
}
}>
}
objects: EventEmitter & {
allIndexes: {
type: {
getEventEmitterByType(type: XapiXoRecord['type']): EventEmitter
}
}
}
tasks: EventEmitter & {
abort(id: XoTask['id'], reason?: string): Promise<void>
clearLogs(): Promise<void>
create: (params: { name: string; objectId?: string; type?: string }) => VatesTask
deleteLog(id: XoTask['id']): Promise<void>
get(id: XoTask['id']): Promise<XoTask>
list(opts?: { filter?: string | ((obj: XoTask) => boolean); limit?: number }): AsyncGenerator<XoTask>
watch(id: XoTask['id'], cb: (task: XoTask) => void): Promise<() => void>
}
apiContext: {
user?: XoUser
permission?: XoUser['permission'] | 'none' | null
}
// methods ------------
addUserToGroup: (userId: XoUser['id'], groupId: XoGroup['id']) => Promise<void>
authenticateUser: (
credentials: { token?: string; username?: string; password?: string },
userData?: { ip?: string },
opts?: { bypassOtp?: boolean; bypassTaskCreation?: boolean }
) => Promise<{ bypassOtp: boolean; expiration: number; user: XoUser }>
/* Throw if no authorization */
checkFeatureAuthorization(featureCode: string): Promise<void>
/* connect a server (XCP-ng/XenServer) */
connectXenServer(id: XoServer['id']): Promise<void>
createAuthenticationToken(opts: {
client?: {
id?: string
[key: string]: unknown
}
description?: string
expiresIn?: string | number
userId: XoUser['id']
}): Promise<XoAuthenticationToken>
createUser(params: { name?: string; password?: string; [key: string]: unknown }): Promise<XoUser>
deleteGroup(id: XoGroup['id']): Promise<void>
deleteUser(id: XoUser['id']): Promise<void>
createGroup(params: { name: string; provider?: string; providerGroup?: string }): Promise<XoGroup>
/* disconnect a server (XCP-ng/XenServer) */
disconnectXenServer(id: XoServer['id']): Promise<void>
getAllGroups(): Promise<XoGroup[]>
getAllProxies(): Promise<XoProxy[]>
getAllJobs<T extends AnyXoBackupJob['type']>(type: T): Promise<Extract<AnyXoBackupJob, { type: T }>[]>
getAllJobs(type?: string): Promise<AnyXoJob[]>
getProxy(id: XoProxy['id']): Promise<XoProxy>
getRemote(id: XoBackupRepository['id']): Promise<XoBackupRepository>
getAllRemotes(): Promise<XoBackupRepository[]>
getAllRemotesInfo(): Promise<
Record<
XoBackupRepository['id'],
{
size?: number
used?: number
available?: number
encryption?: {
algorithm: string
isLegacy: boolean
recommanded: string
}
}
>
>
getAllSchedules(): Promise<XoSchedule[]>
getAllUsers(): Promise<XoUser[]>
getAllXenServers(): Promise<XoServer[]>
getAuthenticationTokensForUser(userId: XoUser['id']): Promise<XoAuthenticationToken[]>
getBackupNgLogs(): Promise<Record<string, AnyXoLog>>
getBackupNgLogs(id: AnyXoLog['id']): Promise<AnyXoLog>
getBackupNgLogsSorted(opts: {
after?: number
before?: number
filter: (log: AnyXoLog) => boolean
limit?: number
}): Promise<AnyXoLog[]>
getGroup(id: XoGroup['id']): Promise<XoGroup>
getHVSupportedVersions: undefined | (() => Promise<{ [key: XoHost['productBrand']]: string }>)
getJob<T extends AnyXoJob>(id: T['id']): Promise<T>
getObject: <T extends XapiXoRecord>(id: T['id'], type?: T['type'] | T['type'][]) => T
getObjectsByType: <T extends XapiXoRecord>(
type: T['type'],
opts?: { filter?: string | ((obj: T) => boolean); limit?: number }
) => Record<T['id'], T> | undefined
getTotalBackupSizeOnRemote(id: XoBackupRepository['id']): Promise<{ onDisk: number }>
getSchedule(id: XoSchedule['id']): Promise<XoSchedule>
getUser: (id: XoUser['id']) => Promise<XoUser>
getXapi(maybeId: XapiXoRecord['id'] | XapiXoRecord): Xapi
getXapiHostStats: (hostId: XoHost['id'], granularity?: XapiStatsGranularity) => Promise<XapiHostStats>
getXapiObject: <T extends XapiXoRecord>(
maybeId: T['id'] | T,
type: T['type'] | T['type'][]
) => XapiRecordByXapiXoRecord[T['type']]
getXapiPoolStats(poolId: XoPool['id'], granularity?: XapiStatsGranularity): Promise<XapiPoolStats>
getXapiVmStats: (vmId: XoVm['id'], granularity?: XapiStatsGranularity) => Promise<XapiVmStats>
getXenServer(id: XoServer['id']): Promise<XoServer>
hasFeatureAuthorization(featureCode: string): Promise<boolean>
hasObject<T extends XapiXoRecord>(id: T['id'], type: T['type']): boolean
listMetadataBackups(backupRepositoryIds: XoBackupRepository['id'][]): Promise<{
xo: Record<XoBackupRepository['id'], XoConfigBackupArchive[]>
pool: Record<XoBackupRepository['id'], Record<XoPool['id'], XoPoolBackupArchive[]>>
}>
listVmBackupsNg(
backupRepositoryIds: XoBackupRepository['id'][],
opts?: { _forceRefresh?: boolean; vmId: XoVm['id'] }
): Promise<Record<XoBackupRepository['id'], Record<XoVm['id'], XoVmBackupArchive[]>>>
/** Allow to add a new server in the DB (XCP-ng/XenServer) */
registerXenServer(
body: Pick<XoServer, 'host' | 'httpProxy' | 'label' | 'username'> & {
allowUnauthorized?: XoServer['allowUnauthorized']
password: string
readOnly?: XoServer['readOnly']
}
): Promise<XoServer>
rollingPoolReboot(pool: XoPool, opts?: { parentTask?: VatesTask }): Promise<void>
rollingPoolUpdate(pool: XoPool, opts?: { rebootVm?: boolean; parentTask?: VatesTask }): Promise<void>
removeUserFromGroup(userId: XoUser['id'], id: XoGroup['id']): Promise<void>
runJob(job: AnyXoJob, schedule: XoSchedule): void
runWithApiContext: (user: XoUser | undefined, fn: () => void) => Promise<unknown>
/** Remove a server from the DB (XCP-ng/XenServer) */
unregisterXenServer(id: XoServer['id']): Promise<void>
updateUser(
id: XoUser['id'],
updates: {
/**
* @deprecated
*/
email?: string
authProviders?: Record<string, string>
name?: string
password?: string
permission?: string
preferences?: Record<string, string>
}
): Promise<void>
updateGroup(
id: XoGroup['id'],
updates: {
name?: string
}
): void
getAllXapis(): Record<string, XapiConnection>
getObjects(opts?: { filter?: Record<string, unknown>; limit?: number }): Record<string, XapiXoRecord>
}

View File

@@ -19,6 +19,7 @@ import type {
VM_POWER_STATE,
} from './common.mjs'
import type * as CMType from './lib/complex-matcher.mjs'
import type { XenApiHost, XenApiPool } from './xen-api.mjs'
type BaseXapiXo = {
$pool: XoPool['id']
@@ -92,7 +93,7 @@ type BaseXoVm = BaseXapiXo & {
notes?: string
os_version: Record<string, string> | null
other: Record<string, string>
parent?: XoVm['id']
parent?: XoVmSnapshot['id']
power_state: VM_POWER_STATE
pvDriversDetected?: boolean
pvDriversUpToDate?: boolean
@@ -105,6 +106,10 @@ type BaseXoVm = BaseXapiXo & {
suspendSr?: XoSr['id']
tags: string[]
vga?: string
/**
* Value in MiB.
* See: https://wiki.xenproject.org/wiki/XCP_PV_templates_start
*/
videoram?: number
viridian: boolean
virtualizationMode: DOMAIN_TYPE
@@ -136,14 +141,63 @@ export type XoAlarm = Omit<XoMessage, '$object' | 'body'> & {
// TODO: to be typed when Bastien.N has finished working on the XO task
type BaseXoLog = {
id: Branded<'xo-log'>
infos?: { data: unknown; message: string }[]
[key: string]: unknown
}
export type XoBackupLog = BaseXoLog & {
message: 'backup' | 'metadata'
status: 'success' | 'skipped' | 'interrupted' | 'failure' | 'pending'
message: 'backup'
start: number
end?: number
tasks?: XoTask[]
jobId?: AnyXoBackupJob['id']
}
export type XoRestoreLog = BaseXoLog & {
message: 'restore'
}
export type XoVmBackupArchive = {
id: Branded<'vm-backup-archive'>
type: 'xo-vm-backup'
backupRepository: XoBackupRepository['id']
disks: { id: string; name: string; uuid: XoVdiSnapshot['uuid'] }[]
isImmutable?: boolean
jobId: XoVmBackupJob['id']
mode: XoVmBackupJob['mode']
scheduleId: XoSchedule['id']
size: number
timestamp: number
vm: {
uuid: XoVm['uuid']
name_description: string
name_label: string
tags: XoVm['tags']
}
differencingVhds?: number
dynamicVhds?: number
withMemory: boolean
}
type XoMetadataBackupArchive = {
id: Branded<'metadata-backup-archive'>
backupRepository: XoBackupRepository['id']
jobId: XoVmBackupJob['id']
jobName: XoVmBackupJob['name']
scheduleId: XoSchedule['id']
scheduleName: XoSchedule['name']
timestamp: number
}
export type XoConfigBackupArchive = XoMetadataBackupArchive & {
id: Branded<'config-backup-archive'>
data: string
type: 'xo-config-backup'
}
export type XoPoolBackupArchive = XoMetadataBackupArchive & {
id: Branded<'pool-backup-archive'>
pool: XenApiPool
poolMaster: XenApiHost
type: 'xo-pool-metadata-backup'
}
export type XoBackupRepository = {
benchmarks?: { readRate: number; timestamp: number; writeRate: number }[]
@@ -185,7 +239,7 @@ export type XoHost = BaseXapiXo & {
address: string
agentStartTime: null | number
bios_string: Record<string, string>
bios_strings: Record<string, string>
build: string
certificates?: {
fingerprint: string
@@ -194,7 +248,7 @@ export type XoHost = BaseXapiXo & {
chipset_info: {
iommu?: boolean
}
controlDomain?: XoVm['id']
controlDomain?: XoVmController['id']
cpus: {
cores?: number
sockets?: number
@@ -289,7 +343,19 @@ export type XoNetwork = BaseXapiXo & {
}
export type XoPbd = BaseXapiXo & {
attached: boolean
device_config:
| { device: string }
| { location: string }
| { path: string; location: string; legacy_mode: string }
| { provisioning: string; redundancy: string; 'group-name': string }
| { server: string; serverpath: string }
| { type: string; location: string }
| Record<string, string>
host: XoHost['id']
id: Branded<'PBD'>
otherConfig: Record<string, string>
SR: XoSr['id']
type: 'PBD'
}
@@ -330,7 +396,7 @@ export type XoPif = BaseXapiXo & {
attached: boolean
bondMaster?: XoPif['id']
bondSalves?: XoPif['id'][]
bondSlaves?: XoPif['id'][]
carrier: boolean
device: string
deviceName?: string
@@ -363,7 +429,7 @@ export type XoPool = BaseXapiXo & {
}
crashDumpSr?: XoSr['id']
current_operations: Record<string, POOL_ALLOWED_OPERATIONS>
defaultSr?: XoSr['id']
default_SR?: XoSr['id']
HA_enabled: boolean
haSrs: XoSr['id'][]
id: Branded<'pool'>
@@ -377,7 +443,6 @@ export type XoPool = BaseXapiXo & {
tags: string[]
type: 'pool'
vtpmSupported: boolean
xosanPackInstallationTime: number | null
zstdSupported: boolean
}
@@ -400,6 +465,7 @@ type BaseXoJob = {
type XoBackupJobGeneralSettings = {
backupReportTpl?: 'compactMjml' | 'mjml'
reportWhen?: 'always' | 'error' | 'failure' | 'never'
reportRecipients?: string[]
hideSuccessfulItems?: boolean
[key: string]: unknown
}
@@ -413,6 +479,8 @@ export type XoVmBackupJobGeneralSettings = XoBackupJobGeneralSettings & {
monthly?: { retention: number; settings: Record<string, unknown> }
yearly?: { retention: number; settings: Record<string, unknown> }
}
checkpointSnapshot?: boolean
offlineSnapshot?: boolean
maxExportRate?: number
nbdConcurrency?: number
nRetriesVmBackupFailures?: number
@@ -421,7 +489,6 @@ export type XoVmBackupJobGeneralSettings = XoBackupJobGeneralSettings & {
mergeBackupsSynchronously?: boolean
offlineBackup?: boolean
timeout?: number
[key: string]: unknown
}
export type XoVmBackupJobScheduleSettings = {
exportRetention?: number
@@ -469,11 +536,11 @@ export type XoMetadataBackupJob = BaseXoJob & {
export type XoMirrorBackupGeneralSettings = XoBackupJobGeneralSettings & {
concurrency?: number
nRetriesVmBackupFailures?: number
mergeBackupsSynchronously?: boolean
timeout?: number
maxExportRate?: number
backupReportTpl?: 'compactMjml'
reportWhen: 'failure'
[key: string]: unknown
}
export type XoMirrorBackupScheduleSettings = {
exportRetention?: number
@@ -486,13 +553,16 @@ export type XoMirrorBackupJob = BaseXoJob & {
mode: 'full' | 'delta'
sourceRemote: XoBackupRepository['id']
remotes: CMType.IdOr<XoBackupRepository['id']>
proxy?: XoProxy['id']
settings: {
'': XoMirrorBackupGeneralSettings
[scheduleId: XoSchedule['id']]: XoMirrorBackupScheduleSettings | undefined
}
}
export type XoJob = BaseXoJob & {}
export type XoJob = BaseXoJob & {
type: 'call'
}
export type XoSchedule = {
cron: string
@@ -527,7 +597,7 @@ export type XoSr = BaseXapiXo & {
VDIs: AnyXoVdi['id'][]
allocationStrategy: 'thin' | 'thick' | 'unknown'
allocationStrategy?: 'thin' | 'thick' | 'unknown'
content_type: string
current_operations: Record<string, STORAGE_OPERATIONS>
id: Branded<'SR'>
@@ -562,14 +632,32 @@ export type XoSm = BaseXapiXo & {
export type XoTask = {
abortionRequestedAt?: number
data?: Record<string, string>
end?: number
id: Branded<'task'>
infos?: { data: unknown; message: string }[]
progress?: number
properties: {
method?: string
name?: string
objectId?: string
objectType?:
| XapiXoRecord['type']
| 'backup'
| 'backup-archive'
| 'backup-job'
| 'backup-log'
| 'backup-repository'
| 'group'
| 'proxy'
| 'restore'
| 'restore-log'
| 'schedule'
| 'server'
| 'task'
| 'user'
params?: Record<string, unknown>
progress?: number
type?: string
userId?: string
[key: string]: unknown | undefined
@@ -641,6 +729,7 @@ type BaseXoVdi = BaseXapiXo & {
export type XoVdi = BaseXoVdi & {
id: Branded<'VDI'>
chainPhysicalUsage: number
type: 'VDI'
}
@@ -741,6 +830,7 @@ export type XapiXoRecord =
| XoHost
| XoMessage
| XoNetwork
| XoPbd
| XoPci
| XoPgpu
| XoPif
@@ -761,10 +851,14 @@ export type XapiXoRecord =
| XoSm
export type NonXapiXoRecord =
| AnyXoBackupArchive
| AnyXoJob
| AnyXoLog
| XoGroup
| XoProxy
| XoGroup
| XoProxy
| XoJob
| XoBackupRepository
| XoSchedule
| XoServer
@@ -782,3 +876,5 @@ export type AnyXoJob = XoJob | AnyXoBackupJob
export type AnyXoBackupJob = XoVmBackupJob | XoMetadataBackupJob | XoMirrorBackupJob
export type AnyXoLog = XoBackupLog | XoRestoreLog
export type AnyXoBackupArchive = XoVmBackupArchive | XoConfigBackupArchive | XoPoolBackupArchive

View File

@@ -5,7 +5,8 @@
"strict": true,
"module": "ESNext",
"target": "ES2022",
"skipLibCheck": true
"skipLibCheck": true,
"types": ["node"] // Only use types from Node.js and packages that are explicitly imported
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]

View File

@@ -1,7 +1,7 @@
{
"private": false,
"name": "@xen-orchestra/async-map",
"version": "0.1.2",
"version": "0.1.3",
"license": "ISC",
"description": "Promise.all + map for all iterables",
"keywords": [
@@ -28,7 +28,7 @@
"node": ">=6"
},
"dependencies": {
"lodash": "^4.17.4"
"lodash": "^4.18.0"
},
"scripts": {
"postversion": "npm publish",

View File

@@ -17,7 +17,7 @@
},
"dependencies": {
"@vates/decorate-with": "^2.1.0",
"@xen-orchestra/log": "^0.7.1",
"@xen-orchestra/log": "^0.7.2",
"golike-defer": "^0.5.1",
"object-hash": "^2.0.1"
},

View File

@@ -6,13 +6,13 @@
"preferGlobal": true,
"bugs": "https://github.com/vatesfr/xen-orchestra/issues",
"dependencies": {
"@vates/async-each": "^1.0.0",
"@xen-orchestra/async-map": "^0.1.2",
"@xen-orchestra/backups": "^0.64.3",
"@xen-orchestra/fs": "^4.6.3",
"@vates/async-each": "^1.0.3",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/backups": "^0.72.0",
"@xen-orchestra/fs": "^4.8.0",
"filenamify": "^6.0.0",
"getopts": "^2.2.5",
"lodash": "^4.17.15",
"lodash": "^4.18.0",
"promise-toolbox": "^0.21.0"
},
"engines": {
@@ -28,7 +28,7 @@
"scripts": {
"postversion": "npm publish --access public"
},
"version": "1.1.3",
"version": "1.1.11",
"license": "AGPL-3.0-or-later",
"author": {
"name": "Vates SAS",

View File

@@ -1,4 +1,4 @@
import { Task } from './Task.mjs'
import { Task } from '@vates/task'
export class HealthCheckVmBackup {
#restoredVm
@@ -14,7 +14,7 @@ export class HealthCheckVmBackup {
async run() {
return Task.run(
{
name: 'vmstart',
properties: { name: 'vmstart' },
},
async () => {
let restoredVm = this.#restoredVm

View File

@@ -2,15 +2,16 @@ import assert from 'node:assert'
import { formatFilenameDate } from './_filenameDate.mjs'
import { importIncrementalVm } from './_incrementalVm.mjs'
import { Task } from './Task.mjs'
import { watchStreamSize } from './_watchStreamSize.mjs'
import { decorateClass } from '@vates/decorate-with'
import { Task } from '@vates/task'
import { createLogger } from '@xen-orchestra/log'
import { dirname, join } from 'node:path'
import pickBy from 'lodash/pickBy.js'
import { defer } from 'golike-defer'
import { NegativeDisk } from '@xen-orchestra/disk-transform'
import { openDiskChain } from './disks/openDiskChain.mjs'
import { resetVmOtherConfig } from './_otherConfig.mjs'
const { debug, info, warn } = createLogger('xo:backups:importVmBackup')
async function resolveUuid(xapi, cache, uuid, type) {
@@ -29,10 +30,10 @@ export class ImportVmBackup {
metadata,
srUuid,
xapi,
settings: { additionnalVmTag, newMacAddresses, mapVdisSrs = {}, useDifferentialRestore = false } = {},
settings: { additionalVmTag, newMacAddresses, mapVdisSrs = {}, useDifferentialRestore = false } = {},
}) {
this._adapter = adapter
this._importIncrementalVmSettings = { additionnalVmTag, newMacAddresses, mapVdisSrs, useDifferentialRestore }
this._importIncrementalVmSettings = { additionalVmTag, newMacAddresses, mapVdisSrs, useDifferentialRestore }
this._metadata = metadata
this._srUuid = srUuid
this._xapi = xapi
@@ -61,11 +62,11 @@ export class ImportVmBackup {
const { mapVdisSrs } = this._importIncrementalVmSettings
const { vbds, vhds, vifs, vm, vmSnapshot, vtpms } = metadata
const disks = {}
const metdataDir = dirname(metadata._filename)
const metadataDir = dirname(metadata._filename)
const vdis = ignoredVdis === undefined ? metadata.vdis : pickBy(metadata.vdis, vdi => !ignoredVdis.has(vdi.uuid))
for (const [vdiRef, vdi] of Object.entries(vdis)) {
const vhdPath = join(metdataDir, vhds[vdiRef])
const vhdPath = join(metadataDir, vhds[vdiRef])
let xapiDisk
try {
@@ -77,7 +78,7 @@ export class ImportVmBackup {
let snapshotCandidate, backupCandidate
if (xapiDisk !== undefined) {
debug('found disks, wlll search its snapshots', { snapshots: xapiDisk.snapshots })
debug('found disks, will search its snapshots', { snapshots: xapiDisk.snapshots })
for (const snapshotRef of xapiDisk.snapshots) {
const snapshot = await this._xapi.getRecord('VDI', snapshotRef)
@@ -96,7 +97,7 @@ export class ImportVmBackup {
// have a corresponding backup more recent than metadata ?
const pathToSnapshotData = await this.#getPathOfVdiSnapshot(snapshot.uuid)
if (pathToSnapshotData === undefined) {
debug('no backup linked to this snaphot')
debug('no backup linked to this snapshot')
continue
}
if (snapshot.$SR.uuid !== (mapVdisSrs[vdi.$snapshot_of$uuid] ?? this._srUuid)) {
@@ -112,7 +113,7 @@ export class ImportVmBackup {
}
let disk
const backupWithSnapshotPath = join(metdataDir, backupCandidate ?? '')
const backupWithSnapshotPath = join(metadataDir, backupCandidate ?? '')
if (vhdPath === backupWithSnapshotPath) {
// all the data are already on the host
debug('direct reuse of a snapshot')
@@ -194,7 +195,7 @@ export class ImportVmBackup {
}
async #decorateIncrementalVmMetadata() {
const { additionnalVmTag, mapVdisSrs, useDifferentialRestore } = this._importIncrementalVmSettings
const { additionalVmTag, mapVdisSrs, useDifferentialRestore } = this._importIncrementalVmSettings
const ignoredVdis = new Set(
Object.entries(mapVdisSrs)
@@ -211,8 +212,8 @@ export class ImportVmBackup {
const cache = new Map()
const mapVdisSrRefs = {}
if (additionnalVmTag !== undefined) {
backup.vm.tags.push(additionnalVmTag)
if (additionalVmTag !== undefined) {
backup.vm.tags.push(additionalVmTag)
}
for (const [vdiUuid, srUuid] of Object.entries(mapVdisSrs)) {
mapVdisSrRefs[vdiUuid] = await resolveUuid(xapi, cache, srUuid, 'SR')
@@ -243,7 +244,7 @@ export class ImportVmBackup {
return Task.run(
{
name: 'transfer',
properties: { name: 'transfer' },
},
async () => {
const xapi = this._xapi
@@ -254,6 +255,14 @@ export class ImportVmBackup {
: await importIncrementalVm(backup, await xapi.getRecord('SR', srRef), {
newMacAddresses,
})
let size = 0
if (isFull) {
size = sizeContainer.size
} else {
for (const disk of Object.values(backup.disks)) {
size += disk.getNbGeneratedBlock() * disk.getBlockSize()
}
}
const remoteName = adapter._handler._remote.name
let desc = `Restored on ${formatFilenameDate(+new Date())}`
if (remoteName !== undefined) {
@@ -270,10 +279,11 @@ export class ImportVmBackup {
`${metadata.vm.name_label} (${formatFilenameDate(metadata.timestamp)})`
),
xapi.call('VM.set_name_description', vmRef, desc),
resetVmOtherConfig(xapi, vmRef),
])
return {
size: sizeContainer.size,
size,
id: await xapi.getField('VM', vmRef, 'uuid'),
}
}

View File

@@ -18,7 +18,7 @@ import groupBy from 'lodash/groupBy.js'
import pDefer from 'promise-toolbox/defer'
import pickBy from 'lodash/pickBy.js'
import reduce from 'lodash/reduce.js'
import tar from 'tar'
import * as tar from 'tar'
import zlib from 'zlib'
import { BACKUP_DIR } from './_getVmBackupDir.mjs'
@@ -27,11 +27,11 @@ import { formatFilenameDate } from './_filenameDate.mjs'
import { getTmpDir } from './_getTmpDir.mjs'
import { isMetadataFile } from './_backupType.mjs'
import { isValidXva } from './_isValidXva.mjs'
import { listPartitions, LVM_PARTITION_TYPE } from './_listPartitions.mjs'
import { listPartitions, LVM_PARTITION_TYPE_MBR, LVM_PARTITION_TYPE_GPT } from './_listPartitions.mjs'
import { lvs, pvs } from './_lvm.mjs'
import { watchStreamSize } from './_watchStreamSize.mjs'
import { RemoteVhd } from './disks/RemoteVhd.mjs'
import { RemoteVhdDisk } from './disks/RemoteVhdDisk.mjs'
import { openDiskChain } from './disks/openDiskChain.mjs'
import { toVhdStream, writeToVhdDirectory } from 'vhd-lib/disk-consumer/index.mjs'
import { ReadAhead } from '@xen-orchestra/disk-transform'
@@ -40,11 +40,11 @@ export const DIR_XO_CONFIG_BACKUPS = 'xo-config-backups'
export const DIR_XO_POOL_METADATA_BACKUPS = 'xo-pool-metadata-backups'
const IMMUTABILTY_METADATA_FILENAME = '/immutability.json'
const IMMUTABILITY_METADATA_FILENAME = '/immutability.json'
const { debug, warn } = createLogger('xo:backups:RemoteAdapter')
const compareTimestamp = (a, b) => a.timestamp - b.timestamp
export const compareTimestamp = (a, b) => a.timestamp - b.timestamp
const noop = Function.prototype
@@ -312,8 +312,8 @@ export class RemoteAdapter {
}
async deleteVmBackups(files) {
const metadatas = await asyncMap(files, file => this.readVmBackupMetadata(file))
const { delta, full, ...others } = groupBy(metadatas, 'mode')
const metadata = await asyncMap(files, file => this.readVmBackupMetadata(file))
const { delta, full, ...others } = groupBy(metadata, 'mode')
const unsupportedModes = Object.keys(others)
if (unsupportedModes.length !== 0) {
@@ -498,7 +498,7 @@ export class RemoteAdapter {
const results = []
await asyncMapSettled(partitions, partition =>
partition.type === LVM_PARTITION_TYPE
partition.type === LVM_PARTITION_TYPE_MBR || partition.type === LVM_PARTITION_TYPE_GPT
? this._listLvmLogicalVolumes(devicePath, partition, results)
: results.push(partition)
)
@@ -575,7 +575,7 @@ export class RemoteAdapter {
}
}
async #getCachabledDataListVmBackups(dir) {
async #getCacheableDataListVmBackups(dir) {
debug('generating cache', { path: dir })
const handler = this._handler
@@ -613,6 +613,11 @@ export class RemoteAdapter {
// if cache is missing or broken => regenerate it and return
async _readCacheListVmBackups(vmUuid) {
// immutable remote can't use any caching
// since the cache file may be non modifiable
if (this._handler.isImmutable()) {
return this.#getCacheableDataListVmBackups(`${BACKUP_DIR}/${vmUuid}`)
}
const path = this.#getVmBackupsCache(vmUuid)
const cache = await this._readCache(path)
@@ -622,7 +627,7 @@ export class RemoteAdapter {
}
// nothing cached, or cache unreadable => regenerate it
const backups = await this.#getCachabledDataListVmBackups(`${BACKUP_DIR}/${vmUuid}`)
const backups = await this.#getCacheableDataListVmBackups(`${BACKUP_DIR}/${vmUuid}`)
if (backups === undefined) {
return
}
@@ -709,7 +714,7 @@ export class RemoteAdapter {
})
} else {
const stream = await toVhdStream(disk)
await this.outputStream(path, stream, { validator })
await this.outputStream(path, stream, { validator, checksum: false })
await validator(path)
}
}
@@ -739,7 +744,7 @@ export class RemoteAdapter {
if (useChain) {
disk = await openDiskChain({ handler, path })
} else {
disk = new RemoteVhd({ handler, path })
disk = new RemoteVhdDisk({ handler, path })
await disk.init()
}
disk = new ReadAhead(disk)
@@ -779,7 +784,7 @@ export class RemoteAdapter {
// if the remote is immutable, check if this metadata is also immutable
try {
// this file is not encrypted
await this._handler._readFile(IMMUTABILTY_METADATA_FILENAME)
await this._handler._readFile(IMMUTABILITY_METADATA_FILENAME)
remoteIsImmutable = true
} catch (error) {
if (error.code !== 'ENOENT') {

View File

@@ -1,155 +0,0 @@
import CancelToken from 'promise-toolbox/CancelToken'
import Zone from 'node-zone'
const logAfterEnd = log => {
const error = new Error('task has already ended')
error.log = log
throw error
}
const noop = Function.prototype
const serializeErrors = errors => (Array.isArray(errors) ? errors.map(serializeError) : errors)
// Create a serializable object from an error.
//
// Otherwise some fields might be non-enumerable and missing from logs.
const serializeError = error =>
error instanceof Error
? {
...error, // Copy enumerable properties.
code: error.code,
errors: serializeErrors(error.errors), // supports AggregateError
message: error.message,
name: error.name,
stack: error.stack,
}
: error
const $$task = Symbol('@xen-orchestra/backups/Task')
export class Task {
static get cancelToken() {
const task = Zone.current.data[$$task]
return task !== undefined ? task.#cancelToken : CancelToken.none
}
static run(opts, fn) {
return new this(opts).run(fn, true)
}
static wrapFn(opts, fn) {
// compatibility with @decorateWith
if (typeof fn !== 'function') {
;[fn, opts] = [opts, fn]
}
return function () {
return Task.run(typeof opts === 'function' ? opts.apply(this, arguments) : opts, () => fn.apply(this, arguments))
}
}
#cancelToken
#id = Math.random().toString(36).slice(2)
#onLog
#zone
constructor({ name, data, onLog }) {
let parentCancelToken, parentId
if (onLog === undefined) {
const parent = Zone.current.data[$$task]
if (parent === undefined) {
onLog = noop
} else {
onLog = log => parent.#onLog(log)
parentCancelToken = parent.#cancelToken
parentId = parent.#id
}
}
const zone = Zone.current.fork('@xen-orchestra/backups/Task')
zone.data[$$task] = this
this.#zone = zone
const { cancel, token } = CancelToken.source(parentCancelToken && [parentCancelToken])
this.#cancelToken = token
this.cancel = cancel
this.#onLog = onLog
this.#log('start', {
data,
message: name,
parentId,
})
}
failure(error) {
this.#end('failure', serializeError(error))
}
info(message, data) {
this.#log('info', { data, message })
}
/**
* Run a function in the context of this task
*
* In case of error, the task will be failed.
*
* @typedef Result
* @param {() => Result} fn
* @param {boolean} last - Whether the task should succeed if there is no error
* @returns Result
*/
run(fn, last = false) {
return this.#zone.run(() => {
try {
const result = fn()
let then
if (result != null && typeof (then = result.then) === 'function') {
then.call(result, last && (value => this.success(value)), error => this.failure(error))
} else if (last) {
this.success(result)
}
return result
} catch (error) {
this.failure(error)
throw error
}
})
}
success(value) {
this.#end('success', value)
}
warning(message, data) {
this.#log('warning', { data, message })
}
wrapFn(fn, last) {
const task = this
return function () {
return task.run(() => fn.apply(this, arguments), last)
}
}
#end(status, result) {
this.#log('end', { result, status })
this.#onLog = logAfterEnd
}
#log(event, props) {
this.#onLog({
...props,
event,
taskId: this.#id,
timestamp: Date.now(),
})
}
}
for (const method of ['info', 'warning']) {
Task[method] = (...args) => Zone.current.data[$$task]?.[method](...args)
}

View File

@@ -1,4 +1,5 @@
export const isMetadataFile = filename => filename.endsWith('.json')
export const isVhdFile = filename => filename.endsWith('.vhd')
export const isVhdSumFile = filename => filename.endsWith('.vhd.checksum')
export const isXvaFile = filename => filename.endsWith('.xva')
export const isXvaSumFile = filename => filename.endsWith('.xva.checksum')

View File

@@ -14,10 +14,10 @@ import { decorateMethodsWith } from '@vates/decorate-with'
import { deduped } from '@vates/disposable/deduped.js'
import { getHandler } from '@xen-orchestra/fs'
import { parseDuration } from '@vates/parse-duration'
import { Task } from '@vates/task'
import { Xapi } from '@xen-orchestra/xapi'
import { RemoteAdapter } from './RemoteAdapter.mjs'
import { Task } from './Task.mjs'
createCachedLookup().patchGlobal()
@@ -178,8 +178,8 @@ process.on('message', async message => {
const result = message.runWithLogs
? await Task.run(
{
name: 'backup run',
onLog: data =>
properties: { name: 'backup run', ...message.data.jobData },
onProgress: data =>
emitMessage({
data,
type: 'log',

View File

@@ -83,12 +83,12 @@ test('It remove broken vhd', async () => {
await handler.writeFile(`${basePath}/notReallyAVhd.vhd`, 'I AM NOT A VHD')
assert.equal((await handler.list(basePath)).length, 1)
let loggued = ''
let logged = ''
const logInfo = message => {
loggued += message
logged += message
}
await adapter.cleanVm(rootPath, { remove: false, logInfo, logWarn: logInfo, lock: false })
assert.equal(loggued, `VHD check error`)
assert.equal(logged, `VHD check error`)
// not removed
assert.deepEqual(await handler.list(basePath), ['notReallyAVhd.vhd'])
// really remove it
@@ -98,7 +98,7 @@ test('It remove broken vhd', async () => {
test('it remove vhd with missing or multiple ancestors', async () => {
// one with a broken parent, should be deleted
await generateVhd(`${basePath}/abandonned.vhd`, {
await generateVhd(`${basePath}/abandoned.vhd`, {
header: {
parentUnicodeName: 'gone.vhd',
parentUuid: uniqueIdBuffer(),
@@ -118,18 +118,18 @@ test('it remove vhd with missing or multiple ancestors', async () => {
`metadata.json`,
JSON.stringify({
mode: 'delta',
vhds: [`${basePath}/child.vhd`, `${basePath}/abandonned.vhd`],
vhds: [`${basePath}/child.vhd`, `${basePath}/abandoned.vhd`],
}),
{ flags: 'w' }
)
// clean
let loggued = ''
let logged = ''
const logInfo = message => {
loggued += message + '\n'
logged += message + '\n'
}
await adapter.cleanVm(rootPath, { remove: true, logInfo, logWarn: logInfo, lock: false })
const deletedOrphanVhd = loggued.match(/deleting orphan VHD/g) || []
const deletedOrphanVhd = logged.match(/deleting orphan VHD/g) || []
assert.equal(deletedOrphanVhd.length, 1) // only one vhd should have been deleted
// we don't test the file on disk, since they will all be marker as unused and deleted without a metadata.json file
@@ -144,12 +144,12 @@ test('it remove backup meta data referencing a missing vhd in delta backup', asy
vhds: [
`${relativePath}/orphan.vhd`,
`${relativePath}/child.vhd`,
// abandonned.json is not here
// abandoned.json is not here
],
})
)
await generateVhd(`${basePath}/abandonned.vhd`)
await generateVhd(`${basePath}/abandoned.vhd`)
// one orphan, which is a full vhd, no parent
const orphan = await generateVhd(`${basePath}/orphan.vhd`)
@@ -162,12 +162,12 @@ test('it remove backup meta data referencing a missing vhd in delta backup', asy
},
})
let loggued = ''
let logged = ''
const logInfo = message => {
loggued += message + '\n'
logged += message + '\n'
}
await adapter.cleanVm(rootPath, { remove: true, logInfo, logWarn: logInfo, lock: false })
let matched = loggued.match(/deleting unused VHD/g) || []
let matched = logged.match(/deleting unused VHD/g) || []
assert.equal(matched.length, 1) // only one vhd should have been deleted
// a missing vhd cause clean to remove all vhds
@@ -179,14 +179,14 @@ test('it remove backup meta data referencing a missing vhd in delta backup', asy
`deleted.vhd`, // in metadata but not in vhds
`orphan.vhd`,
`child.vhd`,
// abandonned.vhd is not here anymore
// abandoned.vhd is not here anymore
],
}),
{ flags: 'w' }
)
loggued = ''
logged = ''
await adapter.cleanVm(rootPath, { remove: true, logInfo, logWarn: () => {}, lock: false })
matched = loggued.match(/deleting unused VHD/g) || []
matched = logged.match(/deleting unused VHD/g) || []
assert.equal(matched.length, 2) // all vhds (orphan and child ) should have been deleted
})
@@ -212,6 +212,7 @@ test('it merges delta of non destroyed chain', async () => {
parentUnicodeName: 'orphan.vhd',
parentUuid: orphan.footer.uuid,
},
blocks: [0, 1],
})
// a grand child
await generateVhd(`${basePath}/grandchild.vhd`, {
@@ -221,21 +222,22 @@ test('it merges delta of non destroyed chain', async () => {
},
})
let loggued = []
let logged = []
const logInfo = message => {
loggued.push(message)
logged.push(message)
}
await adapter.cleanVm(rootPath, { remove: true, logInfo, logWarn: logInfo, lock: false })
assert.equal(loggued[0], `unexpected number of entries in backup cache`)
assert.equal(logged[0], `unexpected number of entries in backup cache`)
loggued = []
await adapter.cleanVm(rootPath, { remove: true, merge: true, logInfo, logWarn: () => {}, lock: false })
const [merging] = loggued
logged = []
const result = await adapter.cleanVm(rootPath, { remove: true, merge: true, logInfo, logWarn: () => {}, lock: false })
assert.ok(result.size > 0, `no merged size`)
const [merging] = logged
assert.equal(merging, `merging VHD chain`)
const metadata = JSON.parse(await handler.readFile(`${rootPath}/metadata.json`))
// size should be the size of children + grand children after the merge
assert.equal(metadata.size, 104960)
assert.equal(metadata.size, 4299776)
// merging is already tested in vhd-lib, don't retest it here (and theses vhd are as empty as my stomach at 12h12)
// only check deletion
@@ -286,6 +288,54 @@ test('it finish unterminated merge ', async () => {
assert.equal(remainingVhds.includes('child.vhd'), true)
})
test('it finishes unterminated merge when child is a disk chain', async () => {
// This test covers the case where a merge was interrupted and the child disk is a disk chain.
await handler.writeFile(
`${rootPath}/metadata.json`,
JSON.stringify({
mode: 'delta',
size: 209920,
vhds: [`${relativePath}/child.vhd`],
})
)
const ancestor = await generateVhd(`${basePath}/ancestor.vhd`)
const intermediate = await generateVhd(`${basePath}/intermediate.vhd`, {
header: {
parentUnicodeName: 'ancestor.vhd',
parentUuid: ancestor.footer.uuid,
},
})
await generateVhd(`${basePath}/child.vhd`, {
header: {
parentUnicodeName: 'intermediate.vhd',
parentUuid: intermediate.footer.uuid,
},
})
// merge state file with chain field
await handler.writeFile(
`${basePath}/.ancestor.vhd.merge.json`,
JSON.stringify({
parent: { uuid: '0' },
child: { uuid: '0' },
chain: ['ancestor.vhd', 'intermediate.vhd', 'child.vhd'],
currentBlock: 0,
mergedDataSize: 0,
step: 'mergeBlocks',
diskSize: 0,
})
)
await adapter.cleanVm(rootPath, { remove: true, merge: true, logWarn: () => {}, lock: false })
// ancestor should be renamed to child.vhd (replacing it), intermediate.vhd should be deleted
const remainingVhds = await handler.list(basePath)
assert.equal(remainingVhds.length, 1)
assert.equal(remainingVhds[0], 'child.vhd')
})
// each of the vhd can be a file, a directory, an alias to a file or an alias to a directory
// the message and resulting files should be identical to the output with vhd files which is tested independently
describe('tests multiple combination ', { concurrency: 1 }, () => {
@@ -311,7 +361,7 @@ describe('tests multiple combination ', { concurrency: 1 }, () => {
// a vhd non referenced in metadata
await generateVhd(`${basePath}/nonreference.vhd`, { useAlias, mode: vhdMode })
// an abandoned delta vhd without its parent
await generateVhd(`${basePath}/abandonned.vhd`, {
await generateVhd(`${basePath}/abandoned.vhd`, {
useAlias,
mode: vhdMode,
header: {
@@ -402,13 +452,12 @@ describe('tests multiple combination ', { concurrency: 1 }, () => {
const metadata = JSON.parse(await handler.readFile(`${rootPath}/metadata.json`))
// size should be the size of children + grand children + clean after the merge
assert.deepEqual(metadata.size, vhdMode === 'file' ? 6502400 : 6501888)
assert.deepEqual(metadata.size, 4404224)
// broken vhd, non referenced, abandoned should be deleted ( alias and data)
// ancestor and child should be merged
// grand child and clean vhd should not have changed
const survivors = await handler.list(basePath)
// console.log(survivors)
if (useAlias) {
const dataSurvivors = await handler.list(basePath + '/data')
// the goal of the alias : do not move a full folder

View File

@@ -1,40 +1,20 @@
import * as UUID from 'uuid'
import sum from 'lodash/sum.js'
import { asyncMap } from '@xen-orchestra/async-map'
import { Constants, openVhd, VhdAbstract, VhdFile } from 'vhd-lib'
import { Constants, openVhd, VhdAbstract } from 'vhd-lib'
import { isVhdAlias, resolveVhdAlias } from 'vhd-lib/aliases.js'
import { basename, dirname, resolve } from 'node:path'
import { isMetadataFile, isVhdFile, isXvaFile, isXvaSumFile } from './_backupType.mjs'
import { isMetadataFile, isVhdFile, isVhdSumFile, isXvaFile, isXvaSumFile } from './_backupType.mjs'
import { limitConcurrency } from 'limit-concurrency-decorator'
import { mergeVhdChain } from 'vhd-lib/merge.js'
import { RemoteVhdDisk } from './disks/RemoteVhdDisk.mjs'
import { RemoteVhdDiskChain } from './disks/RemoteVhdDiskChain.mjs'
import { MergeRemoteDisk } from './disks/MergeRemoteDisk.mjs'
import { Task } from './Task.mjs'
import { Disposable } from 'promise-toolbox'
import { Task } from '@vates/task'
import handlerPath from '@xen-orchestra/fs/path'
const { DISK_TYPES } = Constants
// checking the size of a vhd directory is costly
// 1 Http Query per 1000 blocks
// we only check size of all the vhd are VhdFiles
function shouldComputeVhdsSize(handler, vhds) {
if (handler.isEncrypted) {
return false
}
return vhds.every(vhd => vhd instanceof VhdFile)
}
const computeVhdsSize = (handler, vhdPaths) =>
Disposable.use(
vhdPaths.map(vhdPath => openVhd(handler, vhdPath)),
async vhds => {
if (shouldComputeVhdsSize(handler, vhds)) {
const sizes = await asyncMap(vhds, vhd => vhd.getSize())
return sum(sizes)
}
}
)
// chain is [ ancestor, child_1, ..., child_n ]
async function _mergeVhdChain(handler, chain, { logInfo, remove, mergeBlockConcurrency }) {
logInfo(`merging VHD chain`, { chain })
@@ -51,7 +31,15 @@ async function _mergeVhdChain(handler, chain, { logInfo, remove, mergeBlockConcu
}
}, 10e3)
try {
return await mergeVhdChain(handler, chain, {
const parentDisk = new RemoteVhdDisk({ handler, path: chain.shift() })
const childDisks = []
for (const path of chain) {
childDisks.push(new RemoteVhdDisk({ handler, path }))
}
const childDiskChain = new RemoteVhdDiskChain({ disks: childDisks })
const mergeRemoteDisk = new MergeRemoteDisk(handler, {
logInfo,
mergeBlockConcurrency,
onProgress({ done: d, total: t }) {
@@ -60,6 +48,14 @@ async function _mergeVhdChain(handler, chain, { logInfo, remove, mergeBlockConcu
},
removeUnused: remove,
})
const isResumingMerge = await mergeRemoteDisk.isResuming(parentDisk)
await parentDisk.init({ force: isResumingMerge })
await childDiskChain.init({ force: isResumingMerge })
const result = await mergeRemoteDisk.merge(parentDisk, childDiskChain)
return result
} finally {
clearInterval(handle)
}
@@ -68,10 +64,12 @@ async function _mergeVhdChain(handler, chain, { logInfo, remove, mergeBlockConcu
const noop = Function.prototype
const INTERRUPTED_VHDS_REG = /^\.(.+)\.merge.json$/
const listVhds = async (handler, vmDir, logWarn) => {
const vhds = new Set()
const aliases = {}
const interruptedVhds = new Map()
const checksums = new Set()
await asyncMap(
await handler.list(`${vmDir}/vdis`, {
@@ -85,11 +83,15 @@ const listVhds = async (handler, vmDir, logWarn) => {
}),
async vdiDir => {
const list = await handler.list(vdiDir, {
filter: file => isVhdFile(file) || INTERRUPTED_VHDS_REG.test(file),
filter: file => isVhdFile(file) || INTERRUPTED_VHDS_REG.test(file) || isVhdSumFile(file),
})
aliases[vdiDir] = list.filter(vhd => isVhdAlias(vhd)).map(file => `${vdiDir}/${file}`)
await asyncMap(list, async file => {
if (isVhdSumFile(file)) {
checksums.add(`${vdiDir}/${file}`)
return
}
const res = INTERRUPTED_VHDS_REG.exec(file)
if (res === null) {
vhds.add(`${vdiDir}/${file}`)
@@ -111,7 +113,7 @@ const listVhds = async (handler, vmDir, logWarn) => {
)
)
return { vhds, interruptedVhds, aliases }
return { vhds, interruptedVhds, aliases, checksums }
}
export async function checkAliases(
@@ -228,7 +230,16 @@ export async function cleanVm(
const vhdParents = { __proto__: null }
const vhdChildren = { __proto__: null }
const { vhds, interruptedVhds, aliases } = await listVhds(handler, vmDir, logWarn)
const { vhds, interruptedVhds, aliases, checksums } = await listVhds(handler, vmDir, logWarn)
// from 5.110 to 5.113 we computed checksum for vhd file
// but never used nor removed them
await asyncMap(checksums, async path => {
if (remove) {
logInfo('deleting checksum file ', { path })
return handler.unlink(path)
}
})
// remove broken VHDs
await asyncMap(vhds, async path => {
@@ -503,23 +514,27 @@ export async function cleanVm(
})
}
let totalMergedDataSize = 0
const metadataWithMergedVhd = {}
const doMerge = async () => {
await asyncMap(toMerge, async chain => {
const { finalVhdSize } = await limitedMergeVhdChain(handler, chain, {
const { finalDiskSize, mergedDataSize } = await limitedMergeVhdChain(handler, chain, {
logInfo,
logWarn,
remove,
mergeBlockConcurrency,
})
totalMergedDataSize += mergedDataSize
const metadataPath = vhdsToJSons[chain[chain.length - 1]] // all the chain should have the same metadata file
metadataWithMergedVhd[metadataPath] = (metadataWithMergedVhd[metadataPath] ?? 0) + finalVhdSize
metadataWithMergedVhd[metadataPath] = (metadataWithMergedVhd[metadataPath] ?? 0) + finalDiskSize
})
return { size: totalMergedDataSize }
}
await Promise.all([
...unusedVhdsDeletion,
toMerge.length !== 0 && (merge ? Task.run({ name: 'merge' }, doMerge) : () => Promise.resolve()),
toMerge.length !== 0 && (merge ? Task.run({ properties: { name: 'merge' } }, doMerge) : () => Promise.resolve()),
asyncMap(unusedXvas, path => {
logWarn('unused XVA', { path })
if (remove) {
@@ -547,7 +562,7 @@ export async function cleanVm(
let fileSystemSize
const mergedSize = metadataWithMergedVhd[metadataPath]
const { mode, size, vhds, xva } = metadata
const { mode, size, xva } = metadata
try {
if (mode === 'full') {
@@ -563,21 +578,7 @@ export async function cleanVm(
})
}
} catch (error) {
// can fail with encrypted remote
}
} else if (mode === 'delta') {
// don't warn if the size has changed after a merge
if (mergedSize === undefined) {
const linkedVhds = Object.keys(vhds).map(key => resolve('/', vmDir, vhds[key]))
fileSystemSize = await computeVhdsSize(handler, linkedVhds)
// the size is not computed in some cases (e.g. VhdDirectory)
if (fileSystemSize !== undefined && fileSystemSize !== size) {
logWarn('cleanVm: incorrect backup size in metadata', {
path: metadataPath,
actual: size ?? 'none',
expected: fileSystemSize,
})
}
// will fail with encrypted remote
}
}
} catch (error) {
@@ -622,5 +623,6 @@ export async function cleanVm(
return {
// boolean whether some VHDs were merged (or should be merged)
merge: toMerge.length !== 0,
size: totalMergedDataSize,
}
}

View File

@@ -1,5 +1,58 @@
// @ts-check
import moment from 'moment-timezone'
import assert from 'node:assert'
/**
* @typedef {import('moment-timezone').Moment} Moment
*/
/**
* @typedef {Object} LTRSettings
* @property {number} [firstHourOfTheDay] - First hour of the day (for daily retention)
* @property {number} [firstDayOfWeek] - First day of the week (for weekly retention)
* @property {number} [firstDayOfMonth] - First day of the month (for monthly retention)
* @property {number} [firstDayOfYear] - First day of the year (for yearly retention)
*/
/**
* @typedef {Object} LTRConfig
* @property {number} retention - Number of time buckets to retain
* @property {LTRSettings} [settings] - Optional settings for the retention period
*/
/**
* @typedef {Object} LongTermRetention
* @property {LTRConfig} [daily] - Daily retention configuration
* @property {LTRConfig} [weekly] - Weekly retention configuration
* @property {LTRConfig} [monthly] - Monthly retention configuration
* @property {LTRConfig} [yearly] - Yearly retention configuration
*/
/**
* @typedef {Object} Entry
* @property {number | undefined} timestamp - Unix timestamp of the entry
* @property {string} id
* @property {*} [key] - Any other properties
*/
/**
* @typedef {Object} DateBucket
* @property {number} remaining - Number of remaining buckets to fill
* @property {string|null} lastMatchingBucket - Last bucket key that was matched
* @property {(date: Moment) => string} formatter - Function to format date into bucket key
* @property {Object.<string, Entry>} entries - Map of bucket keys to entries
*/
/**
* @typedef {Object} LTRDefinition
* @property {(options: {firstHourOfTheDay?: number, firstDayOfWeek?: number, firstDayOfMonth?: number, firstDayOfYear?: number, dateCreator: (date: Moment) => Moment}) => (date: Moment) => string} makeDateFormatter
* @property {string} [ancestor] - Name of the parent retention period
*/
/**
* Creates a function that converts dates to moment objects in a specific timezone
* @param {string|undefined} timezone - IANA timezone string (e.g., 'America/New_York')
* @returns {(date: Moment | number) => Moment} Function that creates timezoned moment objects
*/
function instantiateTimezonedDateCreator(timezone) {
return date => {
@@ -9,9 +62,12 @@ function instantiateTimezonedDateCreator(timezone) {
}
}
/**
* @type {Object.<string, LTRDefinition>}
*/
const LTR_DEFINITIONS = {
daily: {
makeDateFormatter: ({ firstHourOfTheDay = 0, dateCreator } = {}) => {
makeDateFormatter: ({ firstHourOfTheDay = 0, dateCreator }) => {
return date => {
const copy = dateCreator(date)
copy.hour(copy.hour() - firstHourOfTheDay)
@@ -20,11 +76,11 @@ const LTR_DEFINITIONS = {
},
},
weekly: {
makeDateFormatter: ({ firstDayOfWeek = 0 /* relative to timezone week start */, dateCreator } = {}) => {
makeDateFormatter: ({ firstDayOfWeek = 0 /* relative to timezone week start */, dateCreator }) => {
return date => {
const copy = dateCreator(date)
copy.date(date.date() - firstDayOfWeek)
copy.date(copy.date() - firstDayOfWeek)
// warning, the year in term of week may different from YYYY
// since the computation of the first week of a year is timezone dependent
return copy.format('gggg-ww')
@@ -33,7 +89,7 @@ const LTR_DEFINITIONS = {
ancestor: 'daily',
},
monthly: {
makeDateFormatter: ({ firstDayOfMonth = 0, dateCreator } = {}) => {
makeDateFormatter: ({ firstDayOfMonth = 0, dateCreator }) => {
return date => {
const copy = dateCreator(date)
copy.date(copy.date() - firstDayOfMonth)
@@ -43,7 +99,7 @@ const LTR_DEFINITIONS = {
ancestor: 'weekly',
},
yearly: {
makeDateFormatter: ({ firstDayOfYear = 0, dateCreator } = {}) => {
makeDateFormatter: ({ firstDayOfYear = 0, dateCreator }) => {
return date => {
const copy = dateCreator(date)
copy.date(copy.date() - firstDayOfYear)
@@ -53,27 +109,23 @@ const LTR_DEFINITIONS = {
ancestor: 'monthly',
},
}
// returns all entries but the last retention-th
/**
* return the entries too old to be kept
* if multiple entries are i the same time bucket : keep only the most recent one
* if an entry is valid in any of the bucket OR the minRetentionCount : keep it
* if a bucket is completely empty : it does not count as one, thus it may extend the retention
* @returns Array<Backup>
* Groups entries into date buckets based on long-term retention configuration
* @template {Entry} T
* @param {Array<T>} entries - Array of entries sorted in descending order by timestamp
* @param {LongTermRetention} longTermRetention - Configuration for retention periods
* @param {string | undefined} timezone - IANA timezone string
* @returns {Object.<string, DateBucket>} Map of retention period names to their date buckets
*/
export function getOldEntries(minRetentionCount, entries, { longTermRetention = {}, timezone } = {}) {
assert.strictEqual(
typeof minRetentionCount,
'number',
`minRetentionCount must be a number, got ${JSON.stringify(minRetentionCount)}`
)
assert.strictEqual(
minRetentionCount >= 0,
true,
`minRetentionCount must be a positive number, got ${JSON.stringify(minRetentionCount)}`
)
export function getLtrEntries(entries, longTermRetention, timezone) {
/**
* @type {Object.<string, DateBucket>}
*/
const dateBuckets = {}
if (!longTermRetention || Object.keys(longTermRetention).length === 0) {
return {}
}
const dateCreator = instantiateTimezonedDateCreator(timezone)
// only check buckets that have a retention set
for (const [duration, { retention, settings }] of Object.entries(longTermRetention)) {
@@ -81,7 +133,7 @@ export function getOldEntries(minRetentionCount, entries, { longTermRetention =
assert.notStrictEqual(
timezone,
undefined,
`timezone must defined for ltr, got ${JSON.stringify({ minRetentionCount, longTermRetention, timezone })}`
`timezone must defined for ltr, got ${JSON.stringify({ longTermRetention, timezone })}`
)
assert.strictEqual(
typeof retention,
@@ -97,40 +149,106 @@ export function getOldEntries(minRetentionCount, entries, { longTermRetention =
remaining: retention,
lastMatchingBucket: null,
formatter: LTR_DEFINITIONS[duration].makeDateFormatter({ ...settings, dateCreator }),
entries: {}, // bucket => entry id
}
}
const nb = entries.length
const oldEntries = []
let previousTimestamp = -1
for (let i = nb - 1; i >= 0; i--) {
const entry = entries[i]
// force it so the compiler is sure we have a timestamp
if (entry.timestamp === undefined) {
assert.notStrictEqual(
entry.timestamp,
undefined,
`Can't compute long term retention if entries don't have a timestamp`
)
continue
}
// we go through the entries from the last (most recent) to the first (oldest)
assert.ok(
previousTimestamp === -1 || entry.timestamp < previousTimestamp,
`entries must be sorted in desc order ${new Date(entry.timestamp)} , previous : > ${new Date(previousTimestamp)} `
)
previousTimestamp = entry.timestamp
const entryDate = dateCreator(entry.timestamp)
let shouldBeKept = false
for (const [duration, { remaining, lastMatchingBucket, formatter }] of Object.entries(dateBuckets)) {
if (remaining === 0) {
continue
}
const bucket = formatter(entryDate)
if (lastMatchingBucket !== bucket) {
if (lastMatchingBucket !== null) {
assert.strictEqual(
lastMatchingBucket > bucket,
true,
`entries must be sorted in asc order ${lastMatchingBucket} ${bucket}`
)
if (remaining === 0) {
continue
}
shouldBeKept = true
dateBuckets[duration].remaining -= 1
dateBuckets[duration].lastMatchingBucket = bucket
dateBuckets[duration].remaining -= 1
}
}
if (i >= nb - minRetentionCount) {
shouldBeKept = true
}
if (!shouldBeKept) {
oldEntries.push(entry)
dateBuckets[duration].entries[bucket] = entry
}
}
// we expect the entries to be in the right order
return oldEntries.reverse()
return dateBuckets
}
/**
* return the status of an entry
* @param {Array<Entry>} entries
* @param {Entry} entry
* @param {LongTermRetention} longTermRetention - Configuration for retention periods
* @param {string | undefined} timezone - IANA timezone string
* @returns {Array<{duration: string, dateBucket:string}>}
*/
export function getEntryStatus(entries, entry, longTermRetention, timezone) {
const dateBuckets = getLtrEntries(entries, longTermRetention, timezone)
const entryBuckets = []
for (const [duration, { entries }] of Object.entries(dateBuckets)) {
for (const [dateBucket, bucketEntry] of Object.entries(entries)) {
if (entry.id === bucketEntry.id) {
entryBuckets.push({ duration, dateBucket })
}
}
}
return entryBuckets
}
/**
* return the entries too old to be kept
* if multiple entries are in the same time bucket: keep only the oldest one
* if an entry is valid in any of the bucket OR the minRetentionCount: keep it
* if a bucket is completely empty: it does not count as one, thus it may extend the retention
* if an entry is within the most recents minRetentionCount entries : keep it
*
* @param {number} minRetentionCount - Minimum number of most recent entries to always keep
* @param {Array<Entry>} entries - Array of entries sorted in descending order by timestamp
* @param {Object} options - Additional options
* @param {LongTermRetention} [options.longTermRetention={}] - Configuration for retention periods
* @param {string | undefined} [options.timezone] - IANA timezone string
* @returns {Array<Entry>} Array of entries that can be removed
*/
export function getOldEntries(minRetentionCount, entries, { longTermRetention = {}, timezone } = {}) {
assert.strictEqual(
typeof minRetentionCount,
'number',
`minRetentionCount must be a number, got ${JSON.stringify(minRetentionCount)}`
)
assert.ok(
minRetentionCount >= 0,
`minRetentionCount must be a positive number, got ${JSON.stringify(minRetentionCount)}`
)
const dateBuckets = getLtrEntries(entries, longTermRetention, timezone)
const kept = new Set(entries.filter((_, index) => index >= entries.length - minRetentionCount))
/**
* @type {Set<Entry>}
*/
for (const { entries } of Object.values(dateBuckets)) {
for (const entry of Object.values(entries)) {
kept.add(entry)
}
}
// ensure order is the same as the source
const oldEntries = entries.filter((entry, index) => {
return !kept.has(entry)
})
return oldEntries
}

View File

@@ -10,24 +10,28 @@ describe('_getOldEntries() should succeed', () => {
1,
[
{ timestamp: 1, id: 1 },
{ timestamp: 3, id: 2 },
{ timestamp: 2, id: 3 },
{ timestamp: 2, id: 2 },
{ timestamp: 3, id: 3 },
],
],
expectedIds: [1, 2],
testLabel: 'should handle number based retention ',
},
{
args: [3, [{ id: 1 }, 4, new Date(), 'whatever']],
expectedIds: [1],
testLabel: 'should handle number based retention without timestamp ',
},
{
args: [
0,
[
{ timestamp: +new Date('2024-09-01 00:01:00'), id: 1 }, // too old
{ timestamp: +new Date('2024-09-01 00:00:00'), id: 2 }, // too old
{ timestamp: +new Date('2024-09-02 00:09:00'), id: 3 }, // too old in same day
{ timestamp: +new Date('2024-09-01 00:00:00'), id: 1 }, // too old
{ timestamp: +new Date('2024-09-01 00:01:00'), id: 2 }, // too old
{ timestamp: +new Date('2024-09-02 00:09:00'), id: 3 }, // oldest in same day
{ timestamp: +new Date('2024-09-02 00:10:00'), id: 4 },
{ timestamp: +new Date('2024-09-03 00:09:00'), id: 5 },
{ timestamp: +new Date('2024-09-04 00:09:00'), id: 6 }, // too old in same day
{ timestamp: +new Date('2024-09-04 00:09:00'), id: 6 }, // oldest in same day
{ timestamp: +new Date('2024-09-04 00:10:00'), id: 7 },
],
{
@@ -37,22 +41,22 @@ describe('_getOldEntries() should succeed', () => {
timezone: 'Europe/Paris',
},
],
expectedIds: [1, 2, 3, 6],
expectedIds: [1, 2, 4, 7],
testLabel: 'should handle day based retention ',
},
{
args: [
0,
[
{ timestamp: +new Date('2024-09-01 00:01:00'), id: 1 }, // week n-3 too old
{ timestamp: +new Date('2024-09-02 00:00:00'), id: 2 }, // week n-3 too old
{ timestamp: +new Date('2024-09-03 00:09:00'), id: 3 }, // week n-2
{ timestamp: +new Date('2024-08-30 00:01:00'), id: 1 }, // week n-3 too old
{ timestamp: +new Date('2024-08-31 00:00:00'), id: 2 }, // week n-3 too old
{ timestamp: +new Date('2024-09-03 00:09:00'), id: 3 }, // week n-2, oldest
{ timestamp: +new Date('2024-09-04 00:09:00'), id: 4 }, // week n-2
{ timestamp: +new Date('2024-09-05 00:09:00'), id: 5 }, // week n-2
{ timestamp: +new Date('2024-09-06 00:09:00'), id: 6 }, // week n-2
{ timestamp: +new Date('2024-09-07 00:09:00'), id: 7 }, // week n-2, most recent kept
{ timestamp: +new Date('2024-09-09 00:09:00'), id: 8 }, // week n-1, too old
{ timestamp: +new Date('2024-09-15 00:09:00'), id: 9 }, // week n-1 kept
{ timestamp: +new Date('2024-09-07 00:09:00'), id: 7 }, // week n-2
{ timestamp: +new Date('2024-09-15 00:09:00'), id: 8 }, // week n-1, oldest
{ timestamp: +new Date('2024-09-21 00:09:00'), id: 9 }, // week n-1 most recent
{ timestamp: +new Date('2024-09-22 00:09:00'), id: 10 }, // week n kept
],
{
@@ -62,7 +66,7 @@ describe('_getOldEntries() should succeed', () => {
timezone: 'Europe/Paris',
},
],
expectedIds: [1, 2, 3, 4, 5, 6, 8],
expectedIds: [1, 2, 4, 5, 6, 7, 9],
testLabel: 'should handle week based retention ',
},
{
@@ -85,26 +89,26 @@ describe('_getOldEntries() should succeed', () => {
timezone: 'Europe/Paris',
},
],
expectedIds: [1, 3, 4, 6, 7],
expectedIds: [1, 4, 5, 7, 8],
testLabel: 'should handle month based retention',
},
{
args: [
0,
2,
[
{ timestamp: +new Date('2023-05-18 00:09:00'), id: 1 }, // too old
{ timestamp: +new Date('2024-06-15 00:09:00'), id: 2 }, // too old in same year
{ timestamp: +new Date('2024-07-04 00:09:00'), id: 3 }, // too old
{ timestamp: +new Date('2024-08-12 00:09:00'), id: 4 }, // too old
{ timestamp: +new Date('2024-09-05 00:09:00'), id: 5 }, // too old
{ timestamp: +new Date('2024-10-02 00:09:00'), id: 6 }, // new month,
{ timestamp: +new Date('2024-11-01 00:09:00'), id: 7 }, // new month , week reached retention
{ timestamp: +new Date('2024-12-17 00:09:00'), id: 8 }, // new week
{ timestamp: +new Date('2024-12-24 00:09:00'), id: 9 }, // new week/month / year daily reach retention
{ timestamp: +new Date('2025-01-01 00:09:00'), id: 10 }, // same day/week/month/year
{ timestamp: +new Date('2025-01-01 00:10:00'), id: 11 }, // new day / week / month
{ timestamp: +new Date('2025-12-31 00:09:00'), id: 12 }, // same day/week/month/year
{ timestamp: +new Date('2025-12-31 00:09:00'), id: 13 }, // new month /year
{ timestamp: +new Date('2023-05-18 00:09:00'), id: 1 }, //
{ timestamp: +new Date('2024-06-15 00:09:00'), id: 2 }, // year n -1
{ timestamp: +new Date('2024-07-04 00:09:00'), id: 3 }, //
{ timestamp: +new Date('2024-08-12 00:09:00'), id: 4 }, //
{ timestamp: +new Date('2024-09-05 00:09:00'), id: 5 }, //
{ timestamp: +new Date('2024-10-02 00:09:00'), id: 6 }, // month n-4
{ timestamp: +new Date('2024-11-01 00:09:00'), id: 7 }, // month n-3
{ timestamp: +new Date('2024-12-17 00:09:00'), id: 8 }, // week n-3
{ timestamp: +new Date('2024-12-24 00:09:00'), id: 9 }, // week n-2, month n-2,
{ timestamp: +new Date('2025-01-01 00:09:00'), id: 10 }, // week n-1, year n
{ timestamp: +new Date('2025-01-02 00:10:00'), id: 11 }, // day n-1, month n-1,
{ timestamp: +new Date('2025-12-31 00:09:00'), id: 12 }, // day n , week n , month n
{ timestamp: +new Date('2025-12-31 10:09:00'), id: 13 }, // min retention
],
{
longTermRetention: {
@@ -116,29 +120,29 @@ describe('_getOldEntries() should succeed', () => {
timezone: 'Europe/Paris', // use a time zone here because week definition is timezone bound
},
],
expectedIds: [1, 2, 3, 4, 5, 10, 12],
expectedIds: [1, 3, 4, 5],
testLabel: 'complete test ',
},
{
args: [
0,
[
{ timestamp: +new Date('2024-09-01 00:01:00'), id: 1 }, // thrid day too old
{ timestamp: +new Date('2024-09-01 00:10:00'), id: 2 }, // thrid day
{ timestamp: +new Date('2024-09-01 00:20:00'), id: 3 }, // second day, too old
{ timestamp: +new Date('2024-09-02 00:22:00'), id: 4 }, // second day too old
{ timestamp: +new Date('2024-09-03 00:20:00'), id: 5 }, // second day in NZ
{ timestamp: +new Date('2024-09-04 00:09:00'), id: 6 }, // same day in NZ
{ timestamp: +new Date('2024-09-04 00:10:00'), id: 7 }, // most recent
{ timestamp: +new Date('2024-08-31T09:01:00Z'), id: 1 }, // 4th day too old
{ timestamp: +new Date('2024-09-01T00:10:00Z'), id: 2 }, // third day
{ timestamp: +new Date('2024-09-01T14:20:00Z'), id: 3 }, // second day, oldest
{ timestamp: +new Date('2024-09-02T00:22:00Z'), id: 4 }, // second day
{ timestamp: +new Date('2024-09-03T14:00:00Z'), id: 5 }, // same day oldest
{ timestamp: +new Date('2024-09-04T00:09:00Z'), id: 6 }, // same day in NZ
{ timestamp: +new Date('2024-09-04T00:10:00Z'), id: 7 }, // most recent
],
{
longTermRetention: {
daily: { retention: 3 },
},
timezone: 'Pacific/Auckland', // GMT +6
timezone: 'Pacific/Auckland', // GMT +12 (NZST in September)
},
],
expectedIds: [1, 2, 3, 6],
expectedIds: [1, 4, 6, 7],
testLabel: 'should handle timezone ',
},
]
@@ -149,13 +153,20 @@ describe('_getOldEntries() should succeed', () => {
assert.strictEqual(
oldEntries.length,
expectedIds.length,
`different length , ${JSON.stringify({ oldEntries, expectedIds })}`
`different length , ${JSON.stringify({ expectedIds, oldEntries: oldEntries.map(({ id }) => id) })}`
)
for (let i = 0; i < expectedIds.length; i++) {
if (i > 0) {
assert.strictEqual(
expectedIds[i - 1] < expectedIds[i],
true,
`expect ids in ascending order ${expectedIds[i - 1]} ${expectedIds[i]}`
)
}
assert.strictEqual(
oldEntries[i].id,
expectedIds[i],
`different id , ${JSON.stringify({ i, expectedIds, oldEntries })}`
`different id , ${JSON.stringify({ i, expectedIds, oldEntries: oldEntries.map(({ id }) => id) })}`
)
}
})
@@ -272,6 +283,19 @@ describe('_getOldEntries() should fail when called incorrectly', () => {
],
testLabel: 'broken date ',
},
{
args: [
1,
[{ obj: 1 }, 'toto', Date.now()],
{
longTermRetention: {
daily: { retention: 5 },
},
timezone: 'Europe/Paris',
},
],
testLabel: 'should fail if LTR without timestamp',
},
]
for (const { args, testLabel } of tests) {

View File

@@ -4,16 +4,15 @@ import { asyncMap } from '@xen-orchestra/async-map'
import { CancelToken } from 'promise-toolbox'
import { compareVersions } from 'compare-versions'
import { defer } from 'golike-defer'
import { Task } from '@vates/task'
import { cancelableMap } from './_cancelableMap.mjs'
import { Task } from './Task.mjs'
import pick from 'lodash/pick.js'
import { BASE_DELTA_VDI, COPY_OF, VM_UUID } from './_otherConfig.mjs'
import { BASE_DELTA_VDI, CONTENT_KEY, COPY_OF, VM_UUID } from './_otherConfig.mjs'
import { XapiDiskSource } from '@xen-orchestra/xapi'
import { VHD_MAX_SIZE, XapiDiskSource } from '@xen-orchestra/xapi'
import { toVhdStream } from 'vhd-lib/disk-consumer/index.mjs'
import { toQcow2Stream } from '@xen-orchestra/qcow2'
import { VHD_MAX_SIZE } from '@xen-orchestra/xapi/disks/Xapi.mjs'
const ensureArray = value => (value === undefined ? [] : Array.isArray(value) ? value : [value])
@@ -125,7 +124,7 @@ export const importIncrementalVm = defer(async function importIncrementalVm(
$defer,
incrementalVm,
sr,
{ cancelToken = CancelToken.none, newMacAddresses = false } = {}
{ cancelToken = CancelToken.none, newMacAddresses = false, targetRef = undefined } = {}
) {
const { version } = incrementalVm
if (compareVersions(version, '1.0.0') < 0) {
@@ -137,44 +136,106 @@ export const importIncrementalVm = defer(async function importIncrementalVm(
const vdiRecords = incrementalVm.vdis
// 0. Create suspend_VDI
// When targetRef is provided, update the existing VM instead of creating a new one.
const targetVm = targetRef !== undefined ? xapi.getObjectByRef(targetRef, undefined) : undefined
const isUpdate = targetVm !== undefined && !targetVm.is_a_snapshot && !targetVm.is_a_template
// 0. Create suspend_VDI (only when creating a new VM).
let suspendVdi
if (vmRecord.suspend_VDI !== undefined && vmRecord.suspend_VDI !== 'OpaqueRef:NULL') {
const vdi = vdiRecords[vmRecord.suspend_VDI]
if (vdi === undefined) {
Task.warning('Suspend VDI not available for this suspended VM', {
vm: pick(vmRecord, 'uuid', 'name_label', 'suspend_VDI'),
})
} else {
suspendVdi = await xapi.getRecord('VDI', await xapi.VDI_create(vdi))
$defer.onFailure(() => suspendVdi.$destroy())
if (!isUpdate) {
if (vmRecord.suspend_VDI !== undefined && vmRecord.suspend_VDI !== 'OpaqueRef:NULL') {
const vdi = vdiRecords[vmRecord.suspend_VDI]
if (vdi === undefined) {
Task.warning('Suspend VDI not available for this suspended VM', {
vm: pick(vmRecord, 'uuid', 'name_label', 'suspend_VDI'),
})
} else {
suspendVdi = await xapi.getRecord('VDI', await xapi.VDI_create(vdi))
$defer.onFailure(() => suspendVdi.$destroy())
}
}
}
// 1. Create the VM.
const vmRef = await xapi.VM_create(
{
...vmRecord,
affinity: undefined,
blocked_operations: {
...vmRecord.blocked_operations,
// 1. Create the VM or update the existing one.
let vmRef
if (isUpdate) {
vmRef = targetRef
await Promise.all([
xapi.setFields('VM', vmRef, {
actions_after_crash: vmRecord.actions_after_crash,
actions_after_reboot: vmRecord.actions_after_reboot,
actions_after_shutdown: vmRecord.actions_after_shutdown,
domain_type: vmRecord.domain_type,
ha_restart_priority: vmRecord.ha_restart_priority,
has_vendor_device: vmRecord.has_vendor_device,
HVM_boot_params: vmRecord.HVM_boot_params,
HVM_boot_policy: vmRecord.HVM_boot_policy,
HVM_shadow_multiplier: vmRecord.HVM_shadow_multiplier,
memory_dynamic_max: vmRecord.memory_dynamic_max,
memory_dynamic_min: vmRecord.memory_dynamic_min,
memory_static_max: vmRecord.memory_static_max,
memory_static_min: vmRecord.memory_static_min,
name_label: vmRecord.name_label,
name_description: vmRecord.name_description,
order: vmRecord.order,
other_config: vmRecord.other_config,
platform: vmRecord.platform,
PV_args: vmRecord.PV_args,
PV_bootloader: vmRecord.PV_bootloader,
PV_bootloader_args: vmRecord.PV_bootloader_args,
PV_kernel: vmRecord.PV_kernel,
PV_legacy_args: vmRecord.PV_legacy_args,
PV_ramdisk: vmRecord.PV_ramdisk,
recommendations: vmRecord.recommendations,
shutdown_delay: vmRecord.shutdown_delay,
start_delay: vmRecord.start_delay,
suspend_SR: vmRecord.suspend_SR,
tags: vmRecord.tags,
user_version: vmRecord.user_version,
VCPUs_at_startup: vmRecord.VCPUs_at_startup,
VCPUs_max: vmRecord.VCPUs_max,
VCPUs_params: vmRecord.VCPUs_params,
xenstore_data: vmRecord.xenstore_data,
}),
targetVm.update_blocked_operations({
start: 'Importing…',
start_on: 'Importing…',
}),
])
} else {
vmRef = await xapi.VM_create(
{
...vmRecord,
affinity: undefined,
blocked_operations: {
...vmRecord.blocked_operations,
start: 'Importing…',
start_on: 'Importing…',
},
ha_always_run: false,
is_a_template: false,
name_label: '[Importing…] ' + vmRecord.name_label,
},
ha_always_run: false,
is_a_template: false,
name_label: '[Importing…] ' + vmRecord.name_label,
},
{
bios_strings: vmRecord.bios_strings,
generateMacSeed: newMacAddresses,
suspend_VDI: suspendVdi?.$ref,
}
)
$defer.onFailure.call(xapi, 'VM_destroy', vmRef)
{
bios_strings: vmRecord.bios_strings,
generateMacSeed: newMacAddresses,
suspend_VDI: suspendVdi?.$ref,
}
)
$defer.onFailure.call(xapi, 'VM_destroy', vmRef)
}
// 2. Delete all VBDs which may have been created by the import.
await asyncMap(await xapi.getField('VM', vmRef, 'VBDs'), ref => ignoreErrors.call(xapi.call('VBD.destroy', ref)))
// In update mode, also collect the VDI refs so orphaned VDIs can be destroyed after new ones are created.
const existingVbdRefs = await xapi.getField('VM', vmRef, 'VBDs')
const oldVdiRefs = new Set(
isUpdate
? (await asyncMap(existingVbdRefs, ref => xapi.getField('VBD', ref, 'VDI'))).filter(
ref => ref !== undefined && ref !== 'OpaqueRef:NULL'
)
: []
)
await asyncMap(existingVbdRefs, ref => ignoreErrors.call(xapi.call('VBD.destroy', ref)))
// 3. Create VDIs & VBDs.
const vbdRecords = incrementalVm.vbds
@@ -185,10 +246,18 @@ export const importIncrementalVm = defer(async function importIncrementalVm(
let newVdi
if (vdi.baseVdi?.$ref !== undefined) {
newVdi = await xapi.getRecord('VDI', await xapi.VDI_clone(vdi.baseVdi.$ref))
$defer.onFailure(() => newVdi.$destroy())
if (isUpdate) {
// In update mode, reuse the existing target VDI directly — no clone needed.
newVdi = vdi.baseVdi
oldVdiRefs.delete(newVdi.$ref)
} else {
newVdi = await xapi.getRecord('VDI', await xapi.VDI_clone(vdi.baseVdi.$ref))
$defer.onFailure(() => newVdi.$destroy())
}
await newVdi.update_other_config(COPY_OF, vdi.uuid)
if (vdi.other_config[CONTENT_KEY] !== undefined) {
await newVdi.update_other_config(CONTENT_KEY, vdi.other_config[CONTENT_KEY])
}
if (vdi.virtual_size > newVdi.virtual_size) {
await newVdi.$callAsync('resize', vdi.virtual_size)
}
@@ -214,6 +283,16 @@ export const importIncrementalVm = defer(async function importIncrementalVm(
newVdis[vdiRef] = newVdi
})
// 3.5. Destroy old VDIs that are no longer attached to the VM.
// Uses ignoreErrors because some storage backends refuse to destroy a VDI that still has snapshot children;
// those VDIs will become truly orphaned once the old snapshot VMs are cleaned up by _deleteOldEntries.
await asyncMap([...oldVdiRefs], ref => ignoreErrors.call(xapi.call('VDI.destroy', ref)))
// 4. For updates, destroy existing VIFs before recreating them.
if (isUpdate) {
await asyncMap(await xapi.getField('VM', vmRef, 'VIFs'), ref => ignoreErrors.call(xapi.call('VIF.destroy', ref)))
}
const networksByNameLabelByVlan = {}
let defaultNetwork
Object.values(xapi.objects.all).forEach(object => {
@@ -286,16 +365,26 @@ export const importIncrementalVm = defer(async function importIncrementalVm(
}
}),
])
// recreate VTPMs
await Promise.all(
(incrementalVm.vtpms ?? []).map(async contents => {
await xapi.VTPM_create({ VM: vmRef, contents })
})
)
// recreate vtpm (note there is normally only one VTPM per VM at most)
const existingVtpmRefs = await xapi.getField('VM', vmRef, 'VTPMs')
for (const vtpmRef of existingVtpmRefs ?? []) {
await xapi.call('VTPM_destroy', vtpmRef)
}
for (const contents of incrementalVm.vtpms ?? []) {
await xapi.VTPM_create({ VM: vmRef, contents })
}
const vm = await xapi.getRecord('VM', vmRef)
await Promise.all([
incrementalVm.vm.ha_always_run && xapi.setField('VM', vmRef, 'ha_always_run', true),
xapi.setField('VM', vmRef, 'name_label', incrementalVm.vm.name_label),
vmRecord.ha_always_run && xapi.setField('VM', vmRef, 'ha_always_run', true),
xapi.setField('VM', vmRef, 'name_label', vmRecord.name_label),
// correctly unlock the VM and reapply the target blocked operations
vm.update_blocked_operations({
start: null,
start_on: null,
...vmRecord.blocked_operations,
}),
])
return vmRef

View File

@@ -20,13 +20,31 @@ const IGNORED_PARTITION_TYPES = {
0xd5: true,
0x82: true, // swap
// GPT Linux swap GUID
'0657fd6d-a4ab-43c4-84e5-0933c84b4f4f': true,
}
export const LVM_PARTITION_TYPE = 0x8e
// MBR LVM type
export const LVM_PARTITION_TYPE_MBR = 0x8e
// GPT LVM type
export const LVM_PARTITION_TYPE_GPT = 'e6d6d379-f507-44c2-a23c-238f2a3df928'
const parsePartxLine = createParser({
keyTransform: key => (key === 'UUID' ? 'id' : key.toLowerCase()),
valueTransform: (value, key) => (key === 'start' || key === 'size' || key === 'type' ? +value : value),
valueTransform: (value, key) => {
if (key === 'start' || key === 'size') {
return +value
}
// For GPT partitions, type is a UUID string
if (key === 'type' && !value.startsWith('0x')) {
return value.toLowerCase()
}
// For MBR partitions, type is a hex number as string (e.g., "0x8e")
if (key === 'type' && value.startsWith('0x')) {
return parseInt(value, 16)
}
return value
},
})
// returns an empty array in case of a non-partitioned disk

View File

@@ -28,6 +28,11 @@ export const EXPORTED_SUCCESSFULLY = 'xo:backup:exported'
// the VM ( not the snapshot) uuid
export const VM_UUID = 'xo:backup:vm'
// in `other_config` of source snapshot VDIs and replicated target VDIs
// contains the UUID of the source snapshot VDI whose content this VDI represents
// allows direction-agnostic discovery of common base VDIs (e.g. for failback/reverse replication)
export const CONTENT_KEY = 'xo:backup:contentKey'
async function listVdiRefs(xapi, vmRef) {
return xapi.VM_getDisks(vmRef)
}
@@ -70,7 +75,8 @@ export async function getVmDeltaChainLength(xapi, vmRef) {
/**
*
* Reset the other_config field of a VM and its VDIs
* Reset the other_config field related to backups of a VM and its VDIs
*
*
* @param {Xapi} xapi
* @param {String} vmRef
@@ -79,6 +85,7 @@ export async function getVmDeltaChainLength(xapi, vmRef) {
export function resetVmOtherConfig(xapi, vmRef) {
return applyToVmAndVdis(xapi, vmRef, (type, ref) => {
return xapi.setFieldEntries(type, ref, 'other_config', {
[COPY_OF]: null,
[DATETIME]: null,
[DELTA_CHAIN_LENGTH]: null,
[EXPORTED_SUCCESSFULLY]: null,
@@ -90,39 +97,6 @@ export function resetVmOtherConfig(xapi, vmRef) {
})
}
/**
*
* used to ensure compatibility with the previous snapshots that were having the config stored only into VM
*
* @param {Xapi} xapi
* @param {String} vmRef
* @returns {Promise}
*/
export async function populateVdisOtherConfig(xapi, vmRef) {
const otherConfig = await xapi.getField('VM', vmRef, 'other_config')
const {
[DATETIME]: datetime,
[DELTA_CHAIN_LENGTH]: chainLength,
[EXPORTED_SUCCESSFULLY]: successfully,
[JOB_ID]: jobId,
[REPLICATED_TO_SR_UUID]: replicatedTo,
[SCHEDULE_ID]: scheduleId,
[VM_UUID]: vmUuid,
} = otherConfig
return applyToVmAndVdis(xapi, vmRef, (type, ref) =>
xapi.setFieldEntries(type, ref, 'other_config', {
[DATETIME]: datetime,
[DELTA_CHAIN_LENGTH]: chainLength,
[EXPORTED_SUCCESSFULLY]: successfully,
[JOB_ID]: jobId,
[REPLICATED_TO_SR_UUID]: replicatedTo,
[SCHEDULE_ID]: scheduleId,
[VM_UUID]: vmUuid,
})
)
}
/**
*
* set the other_config key related to a backup of a VM and its associated VDIs
@@ -163,3 +137,24 @@ export async function markExportSuccessfull(xapi, vmRef) {
xapi.setFieldEntry(type, ref, 'other_config', EXPORTED_SUCCESSFULLY, 'true')
)
}
/**
* Set CONTENT_KEY on each VDI of a VM snapshot to its own UUID.
*
* This marks each snapshot VDI with a unique content identifier that can be
* propagated to replicated copies for direction-agnostic base VDI discovery
* (e.g. for failback / reverse replication).
*
* @param {Xapi} xapi
* @param {String} snapshotRef
* @returns {Promise}
*/
export async function setVmSnapshotContentKeys(xapi, snapshotRef) {
return applyToVmAndVdis(xapi, snapshotRef, async (type, ref) => {
if (type !== 'VDI') {
return
}
const uuid = await xapi.getField(type, ref, 'uuid')
return xapi.setFieldEntry(type, ref, 'other_config', CONTENT_KEY, uuid)
})
}

View File

@@ -1,4 +1,5 @@
import { asyncMap } from '@xen-orchestra/async-map'
import { Task } from '@vates/task'
import Disposable from 'promise-toolbox/Disposable'
import ignoreErrors from 'promise-toolbox/ignoreErrors'
@@ -6,9 +7,10 @@ import { extractIdsFromSimplePattern } from '../extractIdsFromSimplePattern.mjs'
import { PoolMetadataBackup } from './_PoolMetadataBackup.mjs'
import { XoMetadataBackup } from './_XoMetadataBackup.mjs'
import { DEFAULT_SETTINGS, Abstract } from './_Abstract.mjs'
import { runTask } from './_runTask.mjs'
import { getAdaptersByRemote } from './_getAdaptersByRemote.mjs'
const noop = Function.prototype
const DEFAULT_METADATA_SETTINGS = {
retentionPoolMetadata: 0,
retentionXoMetadata: 0,
@@ -55,13 +57,16 @@ export const Metadata = class MetadataBackupRunner extends Abstract {
poolIds.map(id =>
this._getRecord('pool', id).catch(error => {
// See https://github.com/vatesfr/xen-orchestra/commit/6aa6cfba8ec939c0288f0fa740f6dfad98c43cbb
runTask(
Task.run(
{
name: 'get pool record',
data: { type: 'pool', id },
properties: {
id,
name: 'get pool record',
type: 'pool',
},
},
() => Promise.reject(error)
)
).catch(noop)
})
)
),
@@ -81,11 +86,11 @@ export const Metadata = class MetadataBackupRunner extends Abstract {
if (pools.length !== 0 && settings.retentionPoolMetadata !== 0) {
promises.push(
asyncMap(pools, async pool =>
runTask(
Task.run(
{
name: `Starting metadata backup for the pool (${pool.$id}). (${job.id})`,
data: {
properties: {
id: pool.$id,
name: `Starting metadata backup for the pool (${pool.$id}). (${job.id})`,
pool,
poolMaster: await ignoreErrors.call(pool.$xapi.getRecord('host', pool.master)),
type: 'pool',
@@ -100,17 +105,17 @@ export const Metadata = class MetadataBackupRunner extends Abstract {
schedule,
settings,
}).run()
)
).catch(noop)
)
)
}
if (job.xoMetadata !== undefined && settings.retentionXoMetadata !== 0) {
promises.push(
runTask(
Task.run(
{
name: `Starting XO metadata backup. (${job.id})`,
data: {
properties: {
name: `Starting XO metadata backup. (${job.id})`,
type: 'xo',
},
},
@@ -122,7 +127,7 @@ export const Metadata = class MetadataBackupRunner extends Abstract {
schedule,
settings,
}).run()
)
).catch(noop)
)
}
await Promise.all(promises)

View File

@@ -1,14 +1,15 @@
import { asyncMapSettled } from '@xen-orchestra/async-map'
import Disposable from 'promise-toolbox/Disposable'
import { limitConcurrency } from 'limit-concurrency-decorator'
import { Task } from '@vates/task'
import { extractIdsFromSimplePattern } from '../extractIdsFromSimplePattern.mjs'
import { Task } from '../Task.mjs'
import { DEFAULT_SETTINGS, Abstract } from './_Abstract.mjs'
import { getAdaptersByRemote } from './_getAdaptersByRemote.mjs'
import { FullRemote } from './_vmRunners/FullRemote.mjs'
import { IncrementalRemote } from './_vmRunners/IncrementalRemote.mjs'
import { Throttle } from '@vates/generator-toolbox'
import createStreamThrottle from './_createStreamThrottle.mjs'
const noop = Function.prototype
@@ -41,7 +42,8 @@ export const VmsRemote = class RemoteVmsBackupRunner extends Abstract {
const schedule = this._schedule
const settings = this._settings
const throttleGenerator = new Throttle()
const throttleGenerator = new Throttle(settings.maxExportRate)
const throttleStream = createStreamThrottle(settings.maxExportRate)
const config = this._config
@@ -76,7 +78,7 @@ export const VmsRemote = class RemoteVmsBackupRunner extends Abstract {
}
nTriesByVmId[vmUuid]++
const taskStart = { name: 'backup VM', data: { type: 'VM', id: vmUuid } }
const taskStart = { properties: { id: vmUuid, name: 'backup VM', type: 'VM' } }
const vmSettings = { ...settings, ...allSettings[vmUuid] }
const isLastRun = nTriesByVmId[vmUuid] === vmSettings.nRetriesVmBackupFailures + 1
@@ -89,7 +91,8 @@ export const VmsRemote = class RemoteVmsBackupRunner extends Abstract {
schedule,
settings: vmSettings,
sourceRemoteAdapter,
throttleGenerator,
throttleGenerator, // for incrementals
throttleStream, // for full
vmUuid,
}
let vmBackup
@@ -110,23 +113,27 @@ export const VmsRemote = class RemoteVmsBackupRunner extends Abstract {
taskByVmId[vmUuid] = new Task(taskStart)
}
const task = taskByVmId[vmUuid]
// error has to be caught in the task to prevent its failure, but handled outside the task to execute another task.run()
let taskError
return task
.run(async () => {
try {
const result = await vmBackup.run()
task.success(result)
return result
} catch (error) {
if (isLastRun) {
throw error
} else {
Task.warning(`Retry the VM mirror backup due to an error`, {
attempt: nTriesByVmId[vmUuid],
error: error.message,
})
queue.add(vmUuid)
}
.runInside(async () =>
vmBackup.run().catch(error => {
taskError = error
})
)
.then(result => {
if (taskError === undefined) {
return task.success(result)
}
if (isLastRun) {
return task.failure(taskError)
}
// don't end the task
task.warning(`Retry the VM mirror backup due to an error`, {
attempt: nTriesByVmId[vmUuid],
error: taskError.message,
})
queue.add(vmUuid)
})
.catch(noop)
}

View File

@@ -1,15 +1,15 @@
import { asyncMapSettled } from '@xen-orchestra/async-map'
import Disposable from 'promise-toolbox/Disposable'
import { limitConcurrency } from 'limit-concurrency-decorator'
import { Task } from '@vates/task'
import { extractIdsFromSimplePattern } from '../extractIdsFromSimplePattern.mjs'
import { Task } from '../Task.mjs'
import { DEFAULT_SETTINGS, Abstract } from './_Abstract.mjs'
import { runTask } from './_runTask.mjs'
import { getAdaptersByRemote } from './_getAdaptersByRemote.mjs'
import { IncrementalXapi } from './_vmRunners/IncrementalXapi.mjs'
import { FullXapi } from './_vmRunners/FullXapi.mjs'
import { Throttle } from '@vates/generator-toolbox'
import createStreamThrottle from './_createStreamThrottle.mjs'
const noop = Function.prototype
@@ -55,7 +55,8 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
const schedule = this._schedule
const settings = this._settings
const throttleGenerator = new Throttle()
const throttleGenerator = new Throttle(settings.maxExportRate)
const throttleStream = createStreamThrottle(settings.maxExportRate)
const config = this._config
@@ -63,13 +64,12 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
Disposable.all(
extractIdsFromSimplePattern(job.srs).map(id =>
this._getRecord('SR', id).catch(error => {
runTask(
Task.run(
{
name: 'get SR record',
data: { type: 'SR', id },
properties: { id, name: 'get SR record', type: 'SR' },
},
() => Promise.reject(error)
)
).catch(noop)
})
)
),
@@ -106,11 +106,12 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
}
return taskByVmId[vmUuid]
}
const vmBackupFailed = error => {
const vmBackupFailed = async (error, task) => {
if (isLastRun) {
throw error
return task.failure(error)
} else {
Task.warning(`Retry the VM backup due to an error`, {
// don't end the task
task.warning(`Retry the VM backup due to an error`, {
attempt: nTriesByVmId[vmUuid],
error: error.message,
})
@@ -124,19 +125,21 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
nTriesByVmId[vmUuid]++
const vmSettings = { ...settings, ...allSettings[vmUuid] }
const taskStart = { name: 'backup VM', data: { type: 'VM', id: vmUuid } }
const taskStart = { properties: { id: vmUuid, name: 'backup VM', type: 'VM' } }
const isLastRun = nTriesByVmId[vmUuid] === vmSettings.nRetriesVmBackupFailures + 1
return this._getRecord('VM', vmUuid).then(
disposableVm =>
Disposable.use(disposableVm, async vm => {
if (taskStart.data.name_label === undefined) {
taskStart.data.name_label = vm.name_label
if (taskStart.properties.name_label === undefined) {
taskStart.properties.name_label = vm.name_label
}
const task = getVmTask()
// error has to be caught in the task to prevent its failure, but handled outside the task to execute another task.run()
let taskError
return task
.run(async () => {
.runInside(async () => {
const opts = {
baseSettings,
config,
@@ -148,6 +151,7 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
settings: vmSettings,
srs,
throttleGenerator,
throttleStream,
vm,
}
@@ -161,21 +165,21 @@ export const VmsXapi = class VmsXapiBackupRunner extends Abstract {
throw new Error(`Job mode ${job.mode} not implemented`)
}
}
try {
const result = await vmBackup.run()
return vmBackup.run().catch(error => {
taskError = error
})
})
.then(result => {
if (taskError === undefined) {
task.success(result)
return result
} catch (error) {
vmBackupFailed(error)
} else {
// ending the task with error or not ending the task
vmBackupFailed(taskError, task)
}
})
.catch(noop) // errors are handled by logs
}),
error =>
getVmTask().run(() => {
vmBackupFailed(error)
})
error => vmBackupFailed(error, getVmTask())
)
}
const { concurrency } = settings

View File

@@ -1,8 +1,10 @@
import Disposable from 'promise-toolbox/Disposable'
import pTimeout from 'promise-toolbox/timeout'
import { compileTemplate } from '@xen-orchestra/template'
import { runTask } from './_runTask.mjs'
import { RemoteTimeoutError } from './_RemoteTimeoutError.mjs'
import { Task } from '@vates/task'
const noop = Function.prototype
export const DEFAULT_SETTINGS = {
getRemoteTimeout: 300e3,
@@ -36,13 +38,16 @@ export const Abstract = class AbstractRunner {
})
} catch (error) {
// See https://github.com/vatesfr/xen-orchestra/commit/6aa6cfba8ec939c0288f0fa740f6dfad98c43cbb
runTask(
Task.run(
{
name: 'get remote adapter',
data: { type: 'remote', id: remoteId },
properties: {
id: remoteId,
name: 'get remote adapter',
type: 'remote',
},
},
() => Promise.reject(error)
)
).catch(noop)
}
}
}

View File

@@ -1,9 +1,9 @@
import { asyncMap } from '@xen-orchestra/async-map'
import { Task } from '@vates/task'
import { DIR_XO_POOL_METADATA_BACKUPS } from '../RemoteAdapter.mjs'
import { forkStreamUnpipe } from './_forkStreamUnpipe.mjs'
import { formatFilenameDate } from '../_filenameDate.mjs'
import { Task } from '../Task.mjs'
export const PATH_DB_DUMP = '/pool/xmldbdump'
@@ -54,8 +54,8 @@ export class PoolMetadataBackup {
([remoteId, adapter]) =>
Task.run(
{
name: `Starting metadata backup for the pool (${pool.$id}) for the remote (${remoteId}). (${job.id})`,
data: {
properties: {
name: `Starting metadata backup for the pool (${pool.$id}) for the remote (${remoteId}). (${job.id})`,
id: remoteId,
type: 'remote',
},

View File

@@ -1,9 +1,9 @@
import { asyncMap } from '@xen-orchestra/async-map'
import { join } from '@xen-orchestra/fs/path'
import { Task } from '@vates/task'
import { DIR_XO_CONFIG_BACKUPS } from '../RemoteAdapter.mjs'
import { formatFilenameDate } from '../_filenameDate.mjs'
import { Task } from '../Task.mjs'
export class XoMetadataBackup {
constructor({ config, job, remoteAdapters, schedule, settings }) {
@@ -51,8 +51,8 @@ export class XoMetadataBackup {
([remoteId, adapter]) =>
Task.run(
{
name: `Starting XO metadata backup for the remote (${remoteId}). (${job.id})`,
data: {
properties: {
name: `Starting XO metadata backup for the remote (${remoteId}). (${job.id})`,
id: remoteId,
type: 'remote',
},

View File

@@ -1,5 +0,0 @@
import { Task } from '../Task.mjs'
const noop = Function.prototype
export const runTask = (...args) => Task.run(...args).catch(noop) // errors are handled by logs

View File

@@ -2,10 +2,11 @@ import { AbstractRemote } from './_AbstractRemote.mjs'
import { FullRemoteWriter } from '../_writers/FullRemoteWriter.mjs'
import { forkStreamUnpipe } from '../_forkStreamUnpipe.mjs'
import { watchStreamSize } from '../../_watchStreamSize.mjs'
import { AggregatedFullRemoteWriter } from '../_writers/AggregatedFullRemoteWriter.mjs'
export const FullRemote = class FullRemoteVmBackupRunner extends AbstractRemote {
_getRemoteWriter() {
return FullRemoteWriter
_getRemoteWriters() {
return [FullRemoteWriter, AggregatedFullRemoteWriter]
}
_filterTransferList(transferList) {
@@ -16,7 +17,7 @@ export const FullRemote = class FullRemoteVmBackupRunner extends AbstractRemote
const transferList = await this._computeTransferList(({ mode }) => mode === 'full')
for (const metadata of transferList) {
const stream = await this._sourceRemoteAdapter.readFullVmBackup(metadata)
const stream = this._throttleStream(await this._sourceRemoteAdapter.readFullVmBackup(metadata))
const sizeContainer = watchStreamSize(stream)
// @todo shouldn't transfer backup if it will be deleted by retention policy (higher retention on source than destination)

View File

@@ -5,12 +5,14 @@ import { FullRemoteWriter } from '../_writers/FullRemoteWriter.mjs'
import { FullXapiWriter } from '../_writers/FullXapiWriter.mjs'
import { watchStreamSize } from '../../_watchStreamSize.mjs'
import { AbstractXapi } from './_AbstractXapi.mjs'
import { AggregatedFullRemoteWriter } from '../_writers/AggregatedFullRemoteWriter.mjs'
import { AggregatedFullXapiWriter } from '../_writers/AggregatedFullXapiWriter.mjs'
const { debug } = createLogger('xo:backups:FullXapiVmBackup')
export const FullXapi = class FullXapiVmBackupRunner extends AbstractXapi {
_getWriters() {
return [FullRemoteWriter, FullXapiWriter]
return [FullRemoteWriter, FullXapiWriter, AggregatedFullRemoteWriter, AggregatedFullXapiWriter]
}
async _mustDoSnapshot() {
@@ -49,16 +51,14 @@ export const FullXapi = class FullXapiVmBackupRunner extends AbstractXapi {
const { compression } = this.job
const vm = this._vm
const exportedVm = this._exportedVm
// @todo put back throttle for full backup/Replication
const stream =
/* this._throttleStream( */
const stream = this._throttleStream(
(
await this._xapi.VM_export(exportedVm.$ref, {
compress: Boolean(compression) && (compression === 'native' ? 'gzip' : 'zstd'),
useSnapshot: false,
})
).body
/* ) */
)
const vdis = await exportedVm.$getDisks()
let maxStreamLength = 1024 * 1024 // Ovf file and tar headers are a few KB, let's stay safe
@@ -66,9 +66,16 @@ export const FullXapi = class FullXapiVmBackupRunner extends AbstractXapi {
const vdi = await this._xapi.getRecord('VDI', vdiRef)
// the size a of fully allocated vdi will be virtual_size exactly, it's a gross over evaluation
// of the real stream size in general, since a disk is never completely full
// of the real stream size in general, since a disk is rarely completely full
// vdi.physical_size seems to underevaluate a lot the real disk usage of a VDI, as of 2023-10-30
maxStreamLength += vdi.virtual_size
// xva files are tar archive with data cut in 1MB blocks
// each block is signed with a hash (in xxhash format)
// each of these blocks + hash add one tar header entry in the file
const XVA_BLOCK_SIZE = 1024 * 1024
const nbBlocks = Math.ceil(vdi.virtual_size / XVA_BLOCK_SIZE)
const headersSize = nbBlocks * (512 /* file header */ + 512 /* xxhash header */ + 512) /* xhxhash size */
maxStreamLength += XVA_BLOCK_SIZE * nbBlocks + headersSize
}
const sizeContainer = watchStreamSize(stream)

View File

@@ -10,12 +10,13 @@ import { IncrementalRemoteWriter } from '../_writers/IncrementalRemoteWriter.mjs
import { Disposable } from 'promise-toolbox'
import { openVhd } from 'vhd-lib'
import { getVmBackupDir } from '../../_getVmBackupDir.mjs'
import { SynchronizedDisk } from '@xen-orchestra/disk-transform'
import { SynchronizedDisk, ThrottledDisk } from '@xen-orchestra/disk-transform'
import { AggregatedIncrementalRemoteWriter } from '../_writers/AggregatedIncrementalRemoteWriter.mjs'
const { warn } = createLogger('xo:backups:Incrementalremote')
class IncrementalRemoteVmBackupRunner extends AbstractRemote {
_getRemoteWriter() {
return IncrementalRemoteWriter
_getRemoteWriters() {
return [IncrementalRemoteWriter, AggregatedIncrementalRemoteWriter]
}
// we'll transfer the full list if at least one backup should be transferred
@@ -71,8 +72,9 @@ class IncrementalRemoteVmBackupRunner extends AbstractRemote {
const isVhdDifferencing = {}
for (const key in incrementalExport.disks) {
const disk = incrementalExport.disks[key]
let disk = incrementalExport.disks[key]
isVhdDifferencing[key] = disk.isDifferencing()
disk = new ThrottledDisk(disk, this._throttleGenerator)
incrementalExport.disks[key] = new SynchronizedDisk(disk)
}

View File

@@ -0,0 +1,45 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { IncrementalRemote } from './IncrementalRemote.mjs'
const makeAdapter = () => ({
listVmBackups: async () => [],
_handler: { getInfo: async () => ({ available: 0 }), _remote: { id: 'remote-id' } },
})
const makeRunner = ({ distributeBackups = false, schedule } = {}) => {
const remoteId = 'remote-id'
return new IncrementalRemote({
config: {},
job: { id: 'job-id', mode: 'delta', remotes: { id: remoteId }, settings: {}, filter: undefined },
healthCheckSr: undefined,
remoteAdapters: { [remoteId]: makeAdapter() },
schedule,
settings: { distributeBackups, exportRetention: 1, deleteFirst: false, skipDeleteOldEntries: false },
sourceRemoteAdapter: {},
throttleGenerator: {},
throttleStream: {},
vmUuid: 'vm-uuid',
})
}
describe('IncrementalRemote prepare()', () => {
it('prepare() with IncrementalRemoteWriter (distributeBackups: false)', async () => {
const schedule = { id: 'schedule-id' }
const runner = makeRunner({ distributeBackups: false, schedule })
for (const writer of runner._writers) {
await writer.prepare({ isFull: false })
assert.equal(writer._schedule, schedule)
}
})
it('prepare() AggregatedIncrementalRemoteWriter (distributeBackups: true)', async () => {
const schedule = { id: 'schedule-id' }
const runner = makeRunner({ distributeBackups: true, schedule })
const [aggWriter] = runner._writers
await aggWriter.setupWriters()
await aggWriter.prepare({ isFull: false })
assert.equal(aggWriter.props.schedule, schedule)
})
})

View File

@@ -1,4 +1,5 @@
import { createLogger } from '@xen-orchestra/log'
import { Task } from '@vates/task'
import keyBy from 'lodash/keyBy.js'
import { AbstractXapi } from './_AbstractXapi.mjs'
@@ -6,7 +7,6 @@ import { forkDeltaExport } from './_forkDeltaExport.mjs'
import { exportIncrementalVm } from '../../_incrementalVm.mjs'
import { IncrementalRemoteWriter } from '../_writers/IncrementalRemoteWriter.mjs'
import { IncrementalXapiWriter } from '../_writers/IncrementalXapiWriter.mjs'
import { Task } from '../../Task.mjs'
import {
DATETIME,
DELTA_CHAIN_LENGTH,
@@ -14,13 +14,20 @@ import {
setVmDeltaChainLength,
markExportSuccessfull,
} from '../../_otherConfig.mjs'
import { SynchronizedDisk } from '@xen-orchestra/disk-transform'
import { ThrottledDisk, SynchronizedDisk } from '@xen-orchestra/disk-transform'
import { AggregatedIncrementalRemoteWriter } from '../_writers/AggregatedIncrementalRemoteWriter.mjs'
import { AggregatedIncrementalXapiWriter } from '../_writers/AggregatedIncrementalXapiWriter.mjs'
const { debug } = createLogger('xo:backups:IncrementalXapiVmBackup')
export const IncrementalXapi = class IncrementalXapiVmBackupRunner extends AbstractXapi {
_getWriters() {
return [IncrementalRemoteWriter, IncrementalXapiWriter]
return [
IncrementalRemoteWriter,
IncrementalXapiWriter,
AggregatedIncrementalRemoteWriter,
AggregatedIncrementalXapiWriter,
]
}
async _mustDoSnapshot() {
@@ -43,13 +50,14 @@ export const IncrementalXapi = class IncrementalXapiVmBackupRunner extends Abstr
const isVhdDifferencing = {}
let useNbd = false
for (const key in deltaExport.disks) {
const disk = deltaExport.disks[key]
let disk = deltaExport.disks[key]
isVhdDifferencing[key] = disk.isDifferencing()
if (!isFull && !isVhdDifferencing[key] && key !== exportedVm.$suspend_VDI?.$ref) {
Task.warning('Backup fell back to a full')
}
deltaExport.disks[key] = new SynchronizedDisk(disk)
useNbd = useNbd || disk.useNbd()
disk = new ThrottledDisk(disk, this._throttleGenerator)
deltaExport.disks[key] = new SynchronizedDisk(disk)
}
if (useNbd) {
Task.info('Transfer data using NBD')
@@ -127,16 +135,6 @@ export const IncrementalXapi = class IncrementalXapiVmBackupRunner extends Abstr
const deltaChainLength = Math.max(
...lastExportedVdis.map(({ other_config }) => Number(other_config[DELTA_CHAIN_LENGTH] ?? 0))
)
const fullInterval = this._settings.fullInterval
if (fullInterval !== 0 && fullInterval <= deltaChainLength + 1) {
debug('not using base VM because fullInterval reached', {
fullInterval,
deltaChainLength,
eq: fullInterval < deltaChainLength + 1,
dc1: deltaChainLength + 1,
})
return
}
const srcVdis = keyBy(await xapi.getRecords('VDI', await this._vm.$getDisks()), '$ref')
@@ -165,6 +163,20 @@ export const IncrementalXapi = class IncrementalXapiVmBackupRunner extends Abstr
debug('no base VM found')
return
}
// we do tafter checkbasevdis because we want the writer to know the target VM
// especially on replication when we alwayr update the same VM
const fullInterval = this._settings.fullInterval
if (fullInterval !== 0 && fullInterval <= deltaChainLength + 1) {
debug('not using base VM because fullInterval reached', {
fullInterval,
deltaChainLength,
eq: fullInterval < deltaChainLength + 1,
dc1: deltaChainLength + 1,
})
return
}
baseUuidToSrcVdiUuid.forEach((srcVdiUuid, baseUuid) => {
if (presentBaseVdis.has(baseUuid)) {
debug('found base VDI', {

View File

@@ -1,6 +1,6 @@
import { asyncMap } from '@xen-orchestra/async-map'
import { createLogger } from '@xen-orchestra/log'
import { Task } from '../../Task.mjs'
import { Task } from '@vates/task'
const { debug, warn } = createLogger('xo:backups:AbstractVmRunner')
@@ -83,7 +83,7 @@ export const Abstract = class AbstractVmBackupRunner {
// create a task to have an info in the logs and reports
return Task.run(
{
name: 'health check',
properties: { name: 'health check' },
},
() => {
Task.info(`This VM doesn't match the health check's tags for this schedule`)

View File

@@ -4,12 +4,12 @@ import { decorateMethodsWith } from '@vates/decorate-with'
import { defer } from 'golike-defer'
import { Disposable } from 'promise-toolbox'
import { createPredicate } from 'value-matcher'
import { Task } from '@vates/task'
import { getVmBackupDir } from '../../_getVmBackupDir.mjs'
import { Abstract } from './_Abstract.mjs'
import { extractIdsFromSimplePattern } from '../../extractIdsFromSimplePattern.mjs'
import { Task } from '../../Task.mjs'
export const AbstractRemote = class AbstractRemoteVmBackupRunner extends Abstract {
_filterPredicate
@@ -24,6 +24,7 @@ export const AbstractRemote = class AbstractRemoteVmBackupRunner extends Abstrac
settings,
sourceRemoteAdapter,
throttleGenerator,
throttleStream,
vmUuid,
}) {
super()
@@ -37,32 +38,48 @@ export const AbstractRemote = class AbstractRemoteVmBackupRunner extends Abstrac
this._healthCheckSr = healthCheckSr
this._sourceRemoteAdapter = sourceRemoteAdapter
this._throttleGenerator = throttleGenerator
this._throttleStream = throttleStream
this._vmUuid = vmUuid
const allSettings = job.settings
const writers = new Set()
this._writers = writers
const RemoteWriter = this._getRemoteWriter()
extractIdsFromSimplePattern(job.remotes).forEach(remoteId => {
const adapter = remoteAdapters[remoteId]
const targetSettings = {
...settings,
...allSettings[remoteId],
}
const [BackupWriter, AggregratedBackupWriter] = this._getRemoteWriters()
if (settings.distributeBackups) {
writers.add(
new RemoteWriter({
adapter,
new AggregratedBackupWriter({
adapters: remoteAdapters,
BackupWriter,
config,
healthCheckSr,
job,
scheduleId: schedule.id,
schedule,
vmUuid,
remoteId,
settings: targetSettings,
settings,
})
)
})
} else {
extractIdsFromSimplePattern(job.remotes).forEach(remoteId => {
const adapter = remoteAdapters[remoteId]
const targetSettings = {
...settings,
...allSettings[remoteId],
}
writers.add(
new BackupWriter({
adapter,
config,
healthCheckSr,
job,
schedule,
vmUuid,
remoteId,
settings: targetSettings,
})
)
})
}
const { filter } = job
if (filter === undefined) {
this._filterPredicate = () => true
@@ -72,28 +89,28 @@ export const AbstractRemote = class AbstractRemoteVmBackupRunner extends Abstrac
}
async #computeTransferListPerJob(sourceBackups, remotesBackups) {
const localMetada = new Map()
const localMetadata = new Map()
sourceBackups.forEach(metadata => {
const timestamp = metadata.timestamp
localMetada.set(timestamp, metadata)
localMetadata.set(timestamp, metadata)
})
const nbRemotes = remotesBackups.length
const remoteMetadatas = {}
const remoteMetadata = {}
remotesBackups.forEach(async remoteBackups => {
remoteBackups.forEach(metadata => {
const timestamp = metadata.timestamp
remoteMetadatas[timestamp] = (remoteMetadatas[timestamp] ?? 0) + 1
remoteMetadata[timestamp] = (remoteMetadata[timestamp] ?? 0) + 1
})
})
let transferList = []
const timestamps = [...localMetada.keys()]
const timestamps = [...localMetadata.keys()]
timestamps.sort()
for (const timestamp of timestamps) {
if (remoteMetadatas[timestamp] !== nbRemotes) {
if (remoteMetadata[timestamp] !== nbRemotes) {
// this backup is not present in all the remote
// should be retransferred if not found later
transferList.push(localMetada.get(timestamp))
transferList.push(localMetadata.get(timestamp))
} else {
// backup is present in local and remote : the chain has already been transferred
transferList = []
@@ -101,17 +118,10 @@ export const AbstractRemote = class AbstractRemoteVmBackupRunner extends Abstrac
}
if (transferList.length > 0) {
const filteredTransferList = this._filterTransferList(transferList)
if (filteredTransferList.length > 0) {
return filteredTransferList
} else {
Task.info('This VM is excluded by the job filter')
return []
}
return { transferList: filteredTransferList, filtered: true }
} else {
Task.info('No new data to upload for this VM')
return { transferList, filtered: false }
}
return []
}
/**
@@ -125,12 +135,21 @@ export const AbstractRemote = class AbstractRemoteVmBackupRunner extends Abstrac
Object.values(this.remoteAdapters).map(remoteAdapter => remoteAdapter.listVmBackups(this._vmUuid, vmPredicate))
)
const sourceBackupByJobId = groupBy(sourceBackups, 'jobId')
const transferByJobs = await Promise.all(
const transferObjectsList = await Promise.all(
Object.values(sourceBackupByJobId).map(vmBackupsByJob =>
this.#computeTransferListPerJob(vmBackupsByJob, remotesBackups)
)
)
return transferByJobs.flat(1)
const transferList = transferObjectsList.map(transferObject => transferObject.transferList).flat(1)
if (transferList.length === 0 && Object.values(sourceBackupByJobId).length > 0) {
if (transferObjectsList.some(transferObject => transferObject.filtered)) {
// Happens if any job transfer list wasn't empty before filter and is empty after filter.
Task.info('This VM is excluded by the job filter')
} else {
Task.info('No new data to upload for this VM')
}
}
return transferList
}
async run($defer) {

View File

@@ -1,22 +1,30 @@
import assert from 'node:assert'
import groupBy from 'lodash/groupBy.js'
import { createLogger } from '@xen-orchestra/log'
import ignoreErrors from 'promise-toolbox/ignoreErrors'
import { asyncMap } from '@xen-orchestra/async-map'
import { asyncEach } from '@vates/async-each'
import { decorateMethodsWith } from '@vates/decorate-with'
import { defer } from 'golike-defer'
import { Task } from '@vates/task'
import { getOldEntries } from '../../_getOldEntries.mjs'
import { Task } from '../../Task.mjs'
import { Abstract } from './_Abstract.mjs'
import {
COPY_OF,
DATETIME,
JOB_ID,
SCHEDULE_ID,
populateVdisOtherConfig,
VM_UUID,
resetVmOtherConfig,
setVmOtherConfig,
setVmSnapshotContentKeys,
} from '../../_otherConfig.mjs'
const { warn, info } = createLogger('xo:backups:AbstractXapi')
const TEMP_SNAPSHOT_NAME = 'xo-backup-temp-snapshot-name'
export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
constructor({
config,
@@ -29,6 +37,7 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
settings,
srs,
throttleGenerator,
throttleStream,
vm,
}) {
super()
@@ -63,6 +72,7 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
this._jobId = job.id
this._jobSnapshotVdis = undefined
this._throttleGenerator = throttleGenerator
this._throttleStream = throttleStream
this._xapi = vm.$xapi
// Base VM for the export
@@ -81,58 +91,92 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
const writers = new Set()
this._writers = writers
const [BackupWriter, ReplicationWriter] = this._getWriters()
const [BackupWriter, ReplicationWriter, AggregratedBackupWriter, AggregratedReplicationWriter] =
this._getWriters()
const allSettings = job.settings
Object.entries(remoteAdapters).forEach(([remoteId, adapter]) => {
const targetSettings = {
...settings,
...allSettings[remoteId],
}
if (targetSettings.exportRetention !== 0) {
if (settings.exportRetention > 0) {
if (settings.distributeBackups) {
writers.add(
new BackupWriter({
adapter,
new AggregratedBackupWriter({
adapters: remoteAdapters,
BackupWriter,
config,
healthCheckSr,
job,
scheduleId: schedule.id,
schedule,
vmUuid: vm.uuid,
remoteId,
settings: targetSettings,
settings,
})
)
} else {
Object.entries(remoteAdapters).forEach(([remoteId, adapter]) => {
const targetSettings = {
...settings,
...allSettings[remoteId],
}
if (targetSettings.exportRetention !== 0) {
writers.add(
new BackupWriter({
adapter,
config,
healthCheckSr,
job,
schedule,
vmUuid: vm.uuid,
remoteId,
settings: targetSettings,
})
)
}
})
}
})
srs.forEach(sr => {
const targetSettings = {
...settings,
...allSettings[sr.uuid],
}
if (targetSettings.copyRetention !== 0) {
}
if (settings.copyRetention) {
if (settings.distributeReplications) {
writers.add(
new ReplicationWriter({
new AggregratedReplicationWriter({
config,
healthCheckSr,
job,
scheduleId: schedule.id,
ReplicationWriter,
schedule,
vmUuid: vm.uuid,
sr,
settings: targetSettings,
srs,
settings,
})
)
} else {
srs.forEach(sr => {
const targetSettings = {
...settings,
...allSettings[sr.uuid],
}
if (targetSettings.copyRetention !== 0) {
writers.add(
new ReplicationWriter({
config,
healthCheckSr,
job,
schedule,
vmUuid: vm.uuid,
sr,
settings: targetSettings,
})
)
}
})
}
})
}
}
}
// ensure the VM itself does not have any backup metadata which would be
// copied on manual snapshots and interfere with the backup jobs
async _cleanMetadata() {
const vm = this._vm
if (JOB_ID in vm.other_config) {
await resetVmOtherConfig(this._xapi, vm.$ref)
}
await resetVmOtherConfig(this._xapi, this._vm.$ref)
}
async _snapshot() {
@@ -142,7 +186,7 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
const settings = this._settings
if (await this._mustDoSnapshot()) {
await Task.run({ name: 'snapshot' }, async () => {
await Task.run({ properties: { name: 'snapshot' } }, async () => {
if (!settings.bypassVdiChainsCheck) {
await vm.$assertHealthyVdiChains()
}
@@ -158,7 +202,7 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
const snapshotRef = await vm[settings.checkpointSnapshot ? '$checkpoint' : '$snapshot']({
ignoredVdisTag: '[NOBAK]',
name_label: this._getSnapshotNameLabel(vm),
name_label: TEMP_SNAPSHOT_NAME,
unplugVusbs: true,
})
this.timestamp = Date.now()
@@ -168,6 +212,10 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
scheduleId: this.scheduleId,
vmUuid: vm.uuid,
})
await setVmSnapshotContentKeys(xapi, snapshotRef)
const snapshot = await xapi.getRecord('VM', snapshotRef)
await snapshot.set_name_label(this._getSnapshotNameLabel(vm))
// reload data to ensure it is up to date with the new name label
this._exportedVm = await xapi.getRecord('VM', snapshotRef)
return this._exportedVm.uuid
})
@@ -177,43 +225,119 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
}
}
// handle snapshot by VDI since snapshot of cbtDestroySnapshotData jobs
// aren't attached to any VM
// look at all the vdi snapshots ( snapshot_of not empty )
// with the same vm_uuid and job_uuid
// for cbt_metadata list them unconditionnaly to remove older one
// for other: only list them if they are attached to a VM snapshot
// and if this vm snapshot is also part of the backup
// ensure they are attached to only one vm snapshot
// ensure any VM-snapshot harvested by this has all its disk harvested (no mix of vdi snapshot from this job and not)
async _fetchJobSnapshots() {
const jobId = this._jobId
const vmRef = this._vm.$ref
const xapi = this._xapi
// to ensure compatibility with snapshots older than CBT implementation
// update vdi data to ensure the vdi are correctly fetched in _jobSnapshotVdis
// remove by then end of 2024
const vmSnapshotsRef = await xapi.getField('VM', vmRef, 'snapshots')
const vmSnapshotsOtherConfig = await asyncMap(vmSnapshotsRef, ref => xapi.getField('VM', ref, 'other_config'))
const vdiCandidates = {}
const vdiUuids = this._vm.$VBDs.map(({ VDI }) => VDI)
Object.values(xapi.objects.indexes.type.VDI)
.filter(_ => !!_) // filter nullish
.filter(({ other_config, snapshot_of }) => {
return (
vdiUuids.includes(snapshot_of) &&
other_config[JOB_ID] === jobId &&
other_config[VM_UUID] === this._vm.uuid &&
other_config[COPY_OF] === undefined
)
})
.forEach(vdi => {
vdiCandidates[vdi.uuid] = vdi
})
const vmSnapshots = []
vmSnapshotsOtherConfig.forEach((other_config, i) => {
if (other_config[JOB_ID] === jobId) {
vmSnapshots.push({ other_config, $ref: vmSnapshotsRef[i] })
}
})
await Promise.all(vmSnapshots.map(snapshot => populateVdisOtherConfig(xapi, snapshot.$ref)))
// end of compatibility handling
// check that user snapshots are clean
// handle snapshot by VDI
this._jobSnapshotVdis = []
const srcVdis = await xapi.getRecords('VDI', await this._vm.$getDisks())
for (const srcVdi of srcVdis) {
const snapshots = await xapi.getRecords('VDI', srcVdi.snapshots)
for (const snapshot of snapshots) {
// only keep the snapshot related to this backup job
// and only if the job is still using purge snapshot data or if the disk
// is not a cbt metadata disk ( expect a type: user for normal disks)
if (
snapshot.other_config[JOB_ID] === jobId &&
(this._settings.cbtDestroySnapshotData || snapshot.type !== 'cbt_metadata')
) {
this._jobSnapshotVdis.push(snapshot)
}
for (const vdi of Object.values(vdiCandidates)) {
// cbt metadata are always considered linked to a backup job
// if they have the right other_config
if (vdi.type === 'cbt_metadata') {
continue
}
const vbds = vdi.$VBDs
.filter(({ $VM }) => !!$VM) // filter empty VMs
.filter(({ $VM }) => $VM.is_control_domain === false) // do not handle control domain
if (vbds.length === 0) {
// orphan vdi snapshot
info(
`disk snapshot ${vdi.name_label} is orphan or attached only to control domain,
it will be removed at the end of a successful backup run`,
{ vdi, attachedto: vdi.$VBDs.map(vbd => vbd?.$VM) }
)
continue
}
const userVms = vbds.map(({ $VM }) => $VM)
if (vbds.length > 1) {
warn(
`vdi ${vdi.name_label} (${vdi.uuid}) is linked to multipe vms : ${userVms.map(({ name_label, uuid }) => `${name_label} ${uuid}`).join(', ')}.
This disk snapshot will be excluded from the backup cleaning`,
{ vdi, userVms }
)
delete vdiCandidates[vdi.uuid]
continue
}
const vm = vbds[0].$VM
// xapi has some issue regarding vdi snapshot attached to non snapshot VM
// it can also be some user action forcibely linking a vdi snapshot to a vm
// => we exclude these from the backup processing
if (vm.$snapshot_of === undefined) {
warn(
`vdi ${vdi.name_label} (${vdi.uuid}) is a snapshot linked to a non snapshot vm ${vm.name_label} ${vm.uuid}.
This disk snapshot will be excluded from the backup cleaning`,
{ vdi, vm }
)
delete vdiCandidates[vdi.uuid]
continue
}
// vdi is attached only to a snapshot that is not a backup snapshot
// we don't check scheduleId since we are looking for all the snapshot of this job
// => excludes from the list to be cleared
if (
vm.other_config[DATETIME] !== vdi.other_config[DATETIME] ||
vm.other_config[JOB_ID] !== vdi.other_config[JOB_ID] ||
vm.other_config[VM_UUID] !== vdi.other_config[VM_UUID]
) {
warn(
`vdi ${vdi.name_label} (${vdi.uuid}) is a snapshot linked to a snapshot vm ${vm.name_label} ${vm.uuid} out of this backup job scope.
This disk snapshot will be excluded from the backup cleaning`,
{ vdi, vm }
)
delete vdiCandidates[vdi.uuid]
continue
}
// check if all the disks of these VM snapshot have been harvested
// if not => remove it from the list to ensure we won't half destroy VM later
vm.$VBDs
.filter(({ $VDI }) => !!$VDI) // filter missing keys
.filter(({ $VDI }) => $VDI.$snapshot_of !== undefined) // skip non-snapshot VDIs (e.g., ISOs/CD-ROMs)
.filter(({ $VDI }) => $VDI && vdiCandidates[$VDI.uuid] === undefined)
.forEach(({ $VDI: outOfSnapshotsVdi, ...other }) => {
warn(
`vdi ${vdi.name_label} ${vdi.uuid} is recognized as a snapshot of the backup job,
linked to vm ${vm.name_label} ${vm.uuid} but vdi ${outOfSnapshotsVdi.name_label} ${outOfSnapshotsVdi.uuid}
is not linked to the job. This disk snapshot will be excluded from the backup cleaning`,
{ vdi, vm, vbds, outOfSnapshotsVdi }
)
// this will be called multiple time but it is not really an issue
delete vdiCandidates[vdi.uuid]
})
}
this._jobSnapshotVdis = Object.values(vdiCandidates)
}
async _removeUnusedSnapshots() {
@@ -253,28 +377,47 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
return
}
const vdis = snapshotPerDatetime[datetime]
let vmRef
let vm
// if there is an attached VM => destroy the VM (Non CBT backups)
for (const vdi of vdis) {
const vbds = vdi.$VBDs.filter(({ $VM }) => $VM.is_control_domain === false)
if (vbds.length > 0) {
// only one VM linked to this vdi
// this will throw error for VDI still attached to control domain
// since we won't be able to remove an attached VDI
assert.strictEqual(vbds.length, 1, 'VDI must be free or attached to exactly one VM')
const vm = vbds[0].$VM
assert.strictEqual(vm.is_a_snapshot, true, `VM must be a snapshot`) // don't delete a VM (especially a control domain)
const vmRefVdi = vm.$ref
// same vm than other vdi of the same batch
assert.ok(
vmRef === undefined || vmRef === vmRefVdi,
'_removeUnusedSnapshots don t handle vdi related to multiple VMs '
)
vmRef = vmRefVdi
const vdiVm = vbds[0].$VM
if (vdiVm.$snapshot_of === undefined) {
// don't delete a VM (especially a control domain)
warn(
`VM ${vdiVm.uuid} (${vdiVm.name_label}) linked to VDI ${vdi.uuid} (${vdi.name_label}) should be a snapshot`
)
throw new Error(`VM must be a snapshot`)
}
if (vm !== undefined && vm.$ref !== vdiVm.$ref) {
// this VDI is attached to another VM than the other vdi of
// this batch
// in doubt, do not delete anything
warn("_removeUnusedSnapshots don't handle vdi related to multiple VMs ", {
vm1: {
label: vm.name_label,
id: vm.$id,
},
vm2: {
label: vdiVm.name_label,
id: vdiVm.$id,
},
vdis: vdis.map(({ name_label, $id }) => ({ name_label, $id })),
})
throw new Error(
`_removeUnusedSnapshots don't handle vdi related to multiple VMs ${vm.name_label} and ${vdiVm.name_label}`
)
}
vm = vdiVm
}
}
if (vmRef !== undefined) {
return xapi.VM_destroy(vmRef)
if (vm?.$ref !== undefined) {
return xapi.VM_destroy(vm.$ref)
} else {
return asyncMap(
vdis.map(async ({ $ref }) => {
@@ -284,6 +427,11 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
}
})
})
// list and remove the snapshot were the jobs failed between
// makesnapshot and update_other_config
const snapshots = this._vm.$snapshots.filter(_ => !!_).filter(({ name_label }) => name_label === TEMP_SNAPSHOT_NAME)
await asyncEach(snapshots, snapshot => snapshot.$destroy())
}
async _removeSnapshotData() {
@@ -291,8 +439,9 @@ export const AbstractXapi = class AbstractXapiVmBackupRunner extends Abstract {
// going back to a previous version of XO not supporting CBT will create a full backup
// this will only do something after snapshot and transfer
if (
this._exportedVm !== undefined &&
// don't modify the VM
this._exportedVm?.is_a_snapshot &&
this._exportedVm.$snapshot_of !== undefined &&
// user don't want to keep the snapshot data
this._settings.snapshotRetention === 0 &&
// preferNbd is not a guarantee that the backup used NBD, depending on the network configuration,

View File

@@ -0,0 +1,38 @@
import { AbstractAggregatedRemoteWriter } from './_AbstractAggregatedRemoteWriter.mjs'
export class AggregatedFullRemoteWriter extends AbstractAggregatedRemoteWriter {
async deleteOldBackupsOnAdapter(adapter, backups) {
await adapter.deleteFullVmBackups(backups)
}
// take the lock on every adapters with this VM
// cleanup if deleteFirst is selected
async beforeBackup() {
await this.setupWriters()
await Promise.all(this.writers.map(writer => writer.beforeBackup()))
await this.setOldBackups()
if (this.props.settings.deleteFirst) {
await this.deleteOldBackups()
}
}
async run(props) {
const mainAdapter = await this.computeAdapterCandidate()
this.mainWriter = this.getWriterByAdapter(mainAdapter)
return this.mainWriter.run(props)
}
// cleanup if deleteFirst is NOT selected
// release the lock on every adapters with this VM
async afterBackup() {
if (!this.props.settings.deleteFirst) {
await this.deleteOldBackups()
}
await Promise.all(this.writers.map(writer => writer.afterBackup()))
}
healthCheck() {
return this.mainWriter.healthCheck()
}
}

View File

@@ -0,0 +1,31 @@
import { AbstractAggregatedXapiWriter } from './_AbstractAggregatedXapiWriter.mjs'
export class AggregatedFullXapiWriter extends AbstractAggregatedXapiWriter {
// delete replica if deleteFirst is selected
async beforeBackup() {
await this.setupWriters()
await Promise.all(this.writers.map(writer => writer.beforeBackup()))
this.setOldReplicaList()
if (this.props.settings.deleteFirst) {
await this.deleteOldReplicas()
}
}
async run(props) {
const mainSr = await this.computeSRCandidate()
this.mainWriter = this.getWriterBySr(mainSr)
await this.mainWriter.run(props)
}
// delete replica if not using deleteFirst
async afterBackup() {
if (!this.props.settings.deleteFirst) {
await this.deleteOldReplicas()
}
await Promise.all(this.writers.map(writer => writer.afterBackup()))
}
healthCheck() {
return this.mainWriter.healthCheck()
}
}

View File

@@ -0,0 +1,78 @@
import { AbstractAggregatedRemoteWriter } from './_AbstractAggregatedRemoteWriter.mjs'
export class AggregatedIncrementalRemoteWriter extends AbstractAggregatedRemoteWriter {
async deleteOldBackupsOnAdapter(adapter, backups) {
await adapter.deleteDeltaVmBackups(backups)
}
/**
*
* @param {Map<string,string>} baseUuidToSrcVdi
*/
async checkBaseVdis(baseUuidToSrcVdi) {
let selectedCopy = new Map()
for (const writer of this.writers) {
const copy = new Map(baseUuidToSrcVdi)
await writer.checkBaseVdis(copy)
if (copy.size > 0) {
// there can be multiple candidates
// we use the last one in this case
this.mainWriter = writer
selectedCopy = copy
}
}
// update the parameter with our best candidate
// that can be an empty map
baseUuidToSrcVdi.forEach(key => {
if (!selectedCopy.has(key)) {
baseUuidToSrcVdi.delete(key)
}
})
}
// take the lock on every adapters with this VM
// cleanup if deleteFirst is selected
async beforeBackup() {
await this.setupWriters()
await Promise.all(this.writers.map(writer => writer.beforeBackup()))
await this.setOldBackups()
if (this.props.settings.deleteFirst) {
await this.deleteOldBackups()
}
}
async prepare({ isFull }) {
if (this.mainWriter === undefined) {
// use the remote with the most empty space for starting a new chain
const mainAdapter = await this.computeAdapterCandidate()
this.mainWriter = this.getWriterByAdapter(mainAdapter)
}
await Promise.all(this.writers.map(writer => writer.prepare({ isFull })))
}
// write only on the main writer
transfer({ isVhdDifferencing, timestamp, deltaExport, vm, vmSnapshot }) {
return this.mainWriter.transfer({ isVhdDifferencing, timestamp, deltaExport, vm, vmSnapshot })
}
// chain only on the main writer
updateUuidAndChain({ isVhdDifferencing, vdis }) {
return this.mainWriter.updateUuidAndChain({ isVhdDifferencing, vdis })
}
// remove the backups and remove the entries
async cleanup() {
if (!this.props.settings.deleteFirst) {
await this.deleteOldBackups()
}
await Promise.all(this.writers.map(writer => writer.cleanup()))
}
// cleanVM on all, since we may want to delete a chani
// that is not on the main writer
async afterBackup() {
await Promise.all(this.writers.map(writer => writer.afterBackup()))
}
healthCheck() {
return this.mainWriter.healthCheck()
}
}

View File

@@ -0,0 +1,85 @@
import { AbstractAggregatedXapiWriter } from './_AbstractAggregatedXapiWriter.mjs'
import { createLogger } from '@xen-orchestra/log'
const { debug } = createLogger('xo:backups:AggregatedIncrementalXapiWriter')
export class AggregatedIncrementalXapiWriter extends AbstractAggregatedXapiWriter {
/**
*
* @param {Map<string,string>} baseUuidToSrcVdi
*/
async checkBaseVdis(baseUuidToSrcVdi) {
debug('checkBaseVdis', { baseUuidToSrcVdi })
let selectedCopy = new Map()
for (const writer of this.writers) {
const copy = new Map(baseUuidToSrcVdi)
await writer.checkBaseVdis(copy)
if (copy.size > 0) {
debug('checkBaseVdis found a mainwriter candidate ', writer._sr.name_label)
// there can be multiple candidates
// we use the last one in this case
this.mainWriter = writer
selectedCopy = copy
} else {
debug(writer._sr.name_label, 'is not a delta candidate')
}
}
// update the parameter with our best candidate
// that can be an empty map
baseUuidToSrcVdi.forEach(key => {
if (!selectedCopy.has(key)) {
baseUuidToSrcVdi.delete(key)
}
})
debug('checkBaseVdis', { baseUuidToSrcVdi })
}
// take the lock on every adapters with this VM
// cleanup if deleteFirst is selected
async beforeBackup() {
await this.setupWriters()
await Promise.all(this.writers.map(writer => writer.beforeBackup()))
this.setOldReplicaList()
if (this.props.settings.deleteFirst) {
await this.deleteOldReplicas()
}
}
async prepare(args) {
if (this.mainWriter === undefined) {
debug('no mainwriter found, fallback to a new SR')
// use the SR with the most empty space for starting a new chain
const mainSr = await this.computeSRCandidate()
this.mainWriter = this.getWriterBySr(mainSr)
}
await Promise.all(this.writers.map(writer => writer.prepare(args)))
}
// write only on the main writer
transfer(args) {
return this.mainWriter.transfer(args)
}
// chain only on the main writer
updateUuidAndChain(args) {
return this.mainWriter.updateUuidAndChain(args)
}
// remove the backups and remove the entries
async cleanup() {
debug('cleanup')
if (!this.props.settings.deleteFirst) {
await this.deleteOldReplicas()
}
await Promise.all(this.writers.map(writer => writer.cleanup()))
}
// cleanVM on all, since we may want to delete a chani
// that is not on the main writer
async afterBackup() {
await Promise.all(this.writers.map(writer => writer.afterBackup()))
}
healthCheck() {
return this.mainWriter.healthCheck()
}
}

View File

@@ -1,6 +1,7 @@
import { Task } from '@vates/task'
import { formatFilenameDate } from '../../_filenameDate.mjs'
import { getOldEntries } from '../../_getOldEntries.mjs'
import { Task } from '../../Task.mjs'
import { MixinRemoteWriter } from './_MixinRemoteWriter.mjs'
import { AbstractFullWriter } from './_AbstractFullWriter.mjs'
@@ -9,11 +10,11 @@ export class FullRemoteWriter extends MixinRemoteWriter(AbstractFullWriter) {
constructor(props) {
super(props)
this.run = Task.wrapFn(
this.run = Task.wrap(
{
name: 'export',
data: {
properties: {
id: props.remoteId,
name: 'export',
type: 'remote',
// necessary?
@@ -27,13 +28,13 @@ export class FullRemoteWriter extends MixinRemoteWriter(AbstractFullWriter) {
async _run({ maxStreamLength, timestamp, sizeContainer, stream, streamLength, vm, vmSnapshot }) {
const settings = this._settings
const job = this._job
const scheduleId = this._scheduleId
const scheduleId = this._schedule.id
const adapter = this._adapter
let metadata = await this._isAlreadyTransferred(timestamp)
if (metadata !== undefined) {
// @todo : should skip backup while being vigilant to not stuck the forked stream
Task.info('This backup has already been transfered')
Task.info('This backup has already been transferred')
}
const oldBackups = getOldEntries(
@@ -59,12 +60,12 @@ export class FullRemoteWriter extends MixinRemoteWriter(AbstractFullWriter) {
xva: './' + dataBasename,
}
const { deleteFirst } = settings
if (deleteFirst) {
const { skipDeleteOldEntries, deleteFirst } = settings
if (!skipDeleteOldEntries && deleteFirst) {
await deleteOldBackups()
}
await Task.run({ name: 'transfer' }, async () => {
await Task.run({ properties: { name: 'transfer' } }, async () => {
await adapter.outputStream(dataFilename, stream, {
maxStreamLength,
streamLength,
@@ -73,9 +74,10 @@ export class FullRemoteWriter extends MixinRemoteWriter(AbstractFullWriter) {
return { size: sizeContainer.size }
})
metadata.size = sizeContainer.size
metadata.tags = await this.getLongTermRetentionTags(metadata)
this._metadataFileName = await adapter.writeVmBackupMetadata(vm.uuid, metadata)
if (!deleteFirst) {
if (!skipDeleteOldEntries && !deleteFirst) {
await deleteOldBackups()
}

View File

@@ -1,9 +1,9 @@
import ignoreErrors from 'promise-toolbox/ignoreErrors'
import { asyncMap, asyncMapSettled } from '@xen-orchestra/async-map'
import { Task } from '@vates/task'
import { formatFilenameDate } from '../../_filenameDate.mjs'
import { getOldEntries } from '../../_getOldEntries.mjs'
import { Task } from '../../Task.mjs'
import { AbstractFullWriter } from './_AbstractFullWriter.mjs'
import { MixinXapiWriter } from './_MixinXapiWriter.mjs'
@@ -14,11 +14,11 @@ export class FullXapiWriter extends MixinXapiWriter(AbstractFullWriter) {
constructor(props) {
super(props)
this.run = Task.wrapFn(
this.run = Task.wrap(
{
name: 'export',
data: {
properties: {
id: props.sr.uuid,
name: 'export',
name_label: this._sr.name_label,
type: 'SR',
@@ -34,7 +34,7 @@ export class FullXapiWriter extends MixinXapiWriter(AbstractFullWriter) {
const sr = this._sr
const settings = this._settings
const job = this._job
const scheduleId = this._scheduleId
const scheduleId = this._schedule.id
const { uuid: srUuid, $xapi: xapi } = sr
@@ -46,13 +46,13 @@ export class FullXapiWriter extends MixinXapiWriter(AbstractFullWriter) {
const oldVms = getOldEntries(settings.copyRetention - 1, listReplicatedVms(xapi, scheduleId, srUuid, vm.uuid))
const deleteOldBackups = () => asyncMapSettled(oldVms, vm => xapi.VM_destroy(vm.$ref))
const { deleteFirst, _warmMigration } = settings
if (deleteFirst) {
const { deleteFirst, skipDeleteOldEntries, _warmMigration } = settings
if (!skipDeleteOldEntries && deleteFirst) {
await deleteOldBackups()
}
let targetVmRef
await Task.run({ name: 'transfer' }, async () => {
await Task.run({ properties: { name: 'transfer' } }, async () => {
targetVmRef = await xapi.VM_import(stream, sr.$ref, vm =>
Promise.all([
!_warmMigration && vm.add_tags('Disaster Recovery'),
@@ -85,7 +85,7 @@ export class FullXapiWriter extends MixinXapiWriter(AbstractFullWriter) {
}),
])
if (!deleteFirst) {
if (!skipDeleteOldEntries && !deleteFirst) {
await deleteOldBackups()
}
}

View File

@@ -7,10 +7,10 @@ import { createLogger } from '@xen-orchestra/log'
import { decorateClass } from '@vates/decorate-with'
import { defer } from 'golike-defer'
import { dirname, basename } from 'node:path'
import { Task } from '@vates/task'
import { formatFilenameDate } from '../../_filenameDate.mjs'
import { getOldEntries } from '../../_getOldEntries.mjs'
import { Task } from '../../Task.mjs'
import { MixinRemoteWriter } from './_MixinRemoteWriter.mjs'
import { AbstractIncrementalWriter } from './_AbstractIncrementalWriter.mjs'
@@ -72,25 +72,26 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
prepare({ isFull }) {
// create the task related to this export and ensure all methods are called in this context
const task = new Task({
name: 'export',
data: {
properties: {
id: this._remoteId,
isFull,
name: 'export',
type: 'remote',
},
})
this.transfer = task.wrapFn(this.transfer)
this.healthCheck = task.wrapFn(this.healthCheck)
this.cleanup = task.wrapFn(this.cleanup)
this.afterBackup = task.wrapFn(this.afterBackup, true)
this._prepare = task.wrapInside(this._prepare)
this.transfer = task.wrapInside(this.transfer)
this.healthCheck = task.wrapInside(this.healthCheck)
this.cleanup = task.wrapInside(this.cleanup)
this.afterBackup = task.wrap(this.afterBackup)
return task.run(() => this._prepare())
return this._prepare()
}
async _prepare() {
const adapter = this._adapter
const settings = this._settings
const scheduleId = this._scheduleId
const scheduleId = this._schedule.id
const vmUuid = this._vmUuid
const oldEntries = getOldEntries(
@@ -114,13 +115,13 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
oldEntries.length = maxMergedDeltasPerRun
}
if (settings.deleteFirst) {
if (settings.deleteFirst && !settings.skipDeleteOldEntries) {
await this._deleteOldEntries()
}
}
async cleanup() {
if (!this._settings.deleteFirst) {
if (!this._settings.deleteFirst && !this._settings.skipDeleteOldEntries) {
await this._deleteOldEntries()
}
}
@@ -180,7 +181,7 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
async _transfer($defer, { isVhdDifferencing, timestamp, deltaExport, vm, vmSnapshot }) {
const adapter = this._adapter
const job = this._job
const scheduleId = this._scheduleId
const scheduleId = this._schedule.id
const settings = this._settings
const jobId = job.id
const handler = adapter.handler
@@ -223,7 +224,7 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
vtpms: deltaExport.vtpms,
}
let size = 0
await Task.run({ name: 'transfer' }, async () => {
await Task.run({ properties: { name: 'transfer' } }, async () => {
await asyncEach(
Object.entries(deltaExport.disks),
async ([diskRef, disk]) => {
@@ -244,6 +245,7 @@ export class IncrementalRemoteWriter extends MixinRemoteWriter(AbstractIncrement
return { size }
})
metadataContent.tags = await this.getLongTermRetentionTags(metadataContent)
metadataContent.size = size // @todo return exactly the size written by this writer
this._metadataFileName = await adapter.writeVmBackupMetadata(vm.uuid, metadataContent)

View File

@@ -1,79 +1,227 @@
import { asyncMap, asyncMapSettled } from '@xen-orchestra/async-map'
import humanFormat from 'human-format'
import { asyncMapSettled } from '@xen-orchestra/async-map'
import { Task } from '@vates/task'
import ignoreErrors from 'promise-toolbox/ignoreErrors'
import { formatFilenameDate } from '../../_filenameDate.mjs'
import { getOldEntries } from '../../_getOldEntries.mjs'
import { importIncrementalVm } from '../../_incrementalVm.mjs'
import { Task } from '../../Task.mjs'
import { AbstractIncrementalWriter } from './_AbstractIncrementalWriter.mjs'
import { MixinXapiWriter } from './_MixinXapiWriter.mjs'
import { listReplicatedVms } from './_listReplicatedVms.mjs'
import { COPY_OF, setVmOtherConfig, BASE_DELTA_VDI } from '../../_otherConfig.mjs'
import { compareReplicatedVmDatetime, listReplicatedVms } from './_listReplicatedVms.mjs'
import {
COPY_OF,
setVmOtherConfig,
BASE_DELTA_VDI,
JOB_ID,
SCHEDULE_ID,
REPLICATED_TO_SR_UUID,
DATETIME,
VM_UUID,
} from '../../_otherConfig.mjs'
import { formatFilenameDate } from '../../_filenameDate.mjs'
import { XapiDiskSource } from '@xen-orchestra/xapi'
import { asyncEach } from '@vates/async-each'
import { createLogger } from '@xen-orchestra/log'
const { debug } = createLogger('xo:backups:IncrementalXapiWriter')
export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWriter) {
// Map of source VDI UUID (COPY_OF) → validated active VDI on the target SR.
// Built by checkBaseVdis, consumed by #decorateVmMetadata to set baseVdi.
#baseVdisBySourceUuid = new Map()
async checkBaseVdis(baseUuidToSrcVdi) {
const sr = this._sr
this.#baseVdisBySourceUuid = new Map()
if (baseUuidToSrcVdi.size === 0) {
// searching for the vdis is expensive
// don't do it if there is nothing to find
return
}
// @todo use an index if possible
// @todo : this seems similar to decorateVmMetadata
const replicatedVdis = sr.$VDIs
.filter(vdi => {
// REPLICATED_TO_SR_UUID is not used here since we are already filtering from sr.$VDIs
return (
vdi?.managed &&
!vdi?.is_a_snapshot /* only look for real vdi */ &&
baseUuidToSrcVdi.has(vdi?.other_config[COPY_OF])
)
// look for the same snapshot
// ensure there are no data between the snapshot and the active disk
const snapshotCandidates = sr.$VDIs.filter(vdi => {
return (
vdi?.managed &&
vdi?.is_a_snapshot &&
vdi.other_config[JOB_ID] === this._job.id &&
vdi.other_config[VM_UUID] === this._vmUuid &&
baseUuidToSrcVdi.has(vdi?.other_config[COPY_OF])
)
})
debug('checkBaseVdis, got snapshot candidates,', snapshotCandidates.length)
if (snapshotCandidates.length > 0) {
// reset before searching for candidates
this.#baseVdisBySourceUuid = new Map()
this._targetVmRef = undefined
const { baseVdisBySourceUuid, targetVmRef } = await this.#validateSnapshotCandidates(snapshotCandidates)
for (const [sourceUuid, vdi] of baseVdisBySourceUuid) {
this.#baseVdisBySourceUuid.set(sourceUuid, vdi)
}
if (targetVmRef !== undefined) {
this._targetVmRef = targetVmRef
}
} else {
// Legacy fallback (upgrade from pre-6.3): no target snapshots exist yet,
// look for active (non-snapshot) VDIs with matching COPY_OF, like the old code did.
debug('checkBaseVdis, no snapshot candidates, falling back to legacy active VDI lookup')
const legacyVdis = sr.$VDIs.filter(vdi => {
return vdi?.managed && !vdi?.is_a_snapshot && baseUuidToSrcVdi.has(vdi?.other_config[COPY_OF])
})
.map(({ other_config }) => other_config?.[COPY_OF])
.filter(_ => !!_)
for (const vdi of legacyVdis) {
const sourceUuid = vdi.other_config[COPY_OF]
if (sourceUuid) {
this.#baseVdisBySourceUuid.set(sourceUuid, vdi)
}
}
}
for (const uuid of baseUuidToSrcVdi.keys()) {
if (!replicatedVdis.includes(uuid)) {
if (!this.#baseVdisBySourceUuid.has(uuid)) {
baseUuidToSrcVdi.delete(uuid)
}
}
}
/**
* 6.3+ snapshot-based validation: for each snapshot candidate, check whether
* the active VDI has diverged from the snapshot. Returns a baseVdisBySourceUuid
* map and, when all disks are clean, the targetVmRef to reuse.
*/
async #validateSnapshotCandidates(snapshotCandidates) {
const sr = this._sr
const baseVdisBySourceUuid = new Map()
let targetVmRef
let canChainToTargetVm = true
await asyncEach(
snapshotCandidates,
async snapshot => {
let diffDisk
let activeVdi
try {
activeVdi = sr.$xapi.getObject(snapshot.$snapshot_of)
const userVbds = activeVdi.$VBDs?.filter(vbd => vbd.$VM && !vbd.$VM.is_control_domain) ?? []
if (userVbds.length !== 1) {
debug('checkBaseVdis, shared vbd ', { ref: snapshot.$ref, userVbds })
// shared vdi ignore
return
}
const vm = userVbds[0].$VM
if (!('start' in vm.blocked_operations)) {
debug('checkBaseVdis, vm not blocked', { vmRef: vm.$ref })
// vm start unlocked
// not really an issue since we have check the delta
// but it indicates the users played with the blocked operations
return
}
diffDisk = new XapiDiskSource({
xapi: sr.$xapi,
vdiRef: activeVdi.$ref,
baseRef: snapshot.$ref,
onlyListChangedBlocks: true,
})
await diffDisk.init()
const sourceUuid = snapshot.other_config?.[COPY_OF]
if (diffDisk.getBlockIndexes().length === 0) {
// no block modification since the common snapshot, we can chain VM and disk
if (sourceUuid) {
baseVdisBySourceUuid.set(sourceUuid, activeVdi)
}
// Track the target VM (the replicated VM to update on the next transfer).
targetVmRef = vm.$ref
} else {
if (sourceUuid) {
baseVdisBySourceUuid.set(sourceUuid, snapshot)
}
// there are changed block since the snapshot
// we can reuse it to transfer a delta, but we will
// create a new VM
canChainToTargetVm = false
debug('checkBaseVdis, data between snapshot and active disk', {
vdiRef: snapshot.$ref,
nbBlocks: diffDisk.getBlockIndexes().length,
})
}
} catch (error) {
debug('checkBaseVdis, skipping snapshot', { ref: snapshot.$ref, error })
return
} finally {
await diffDisk?.close().catch(error => debug('checkBaseVdis, error closing', error))
await sr.$xapi.VDI_disconnectFromControlDomain(snapshot.$ref)
if (activeVdi !== undefined) {
await sr.$xapi.VDI_disconnectFromControlDomain(activeVdi.$ref)
}
}
},
{
concurrency: 4,
}
)
if (!canChainToTargetVm) {
// if at least one disk has new data, create a new VM
// instead of updating it
targetVmRef = undefined
} else if (targetVmRef !== undefined) {
debug('checkBaseVdis,got a valid vm target', targetVmRef)
}
return { baseVdisBySourceUuid, targetVmRef }
}
updateUuidAndChain() {
// nothing to do, the chaining is not modified in this case
}
prepare({ isFull }) {
// create the task related to this export and ensure all methods are called in this context
const task = new Task({
name: 'export',
data: {
properties: {
id: this._sr.uuid,
isFull,
name: 'export',
name_label: this._sr.name_label,
type: 'SR',
},
})
const hasHealthCheckSr = this._healthCheckSr !== undefined
this.transfer = task.wrapFn(this.transfer)
this.cleanup = task.wrapFn(this.cleanup, !hasHealthCheckSr)
this.healthCheck = task.wrapFn(this.healthCheck, hasHealthCheckSr)
this._prepare = task.wrapInside(this._prepare)
this.transfer = task.wrapInside(this.transfer)
if (this._healthCheckSr !== undefined) {
this.cleanup = task.wrapInside(this.cleanup)
this.healthCheck = task.wrap(this.healthCheck)
} else {
this.cleanup = task.wrap(this.cleanup)
}
return task.run(() => this._prepare(isFull))
return this._prepare(isFull)
}
async _prepare(isFull) {
const settings = this._settings
const { uuid: srUuid, $xapi: xapi } = this._sr
const vmUuid = this._vmUuid
const scheduleId = this._scheduleId
const scheduleId = this._schedule.id
// delete previous interrupted copies
ignoreErrors.call(asyncMapSettled(listReplicatedVms(xapi, scheduleId, undefined, vmUuid), vm => vm.$destroy))
this._oldEntries = getOldEntries(settings.copyRetention - 1, listReplicatedVms(xapi, scheduleId, srUuid, vmUuid))
const allEntries = listReplicatedVms(xapi, scheduleId, srUuid, vmUuid)
if (settings.deleteFirst) {
// In the snapshot-based flow a non-snapshot VM (the live target) coexists with its
// snapshots (one per transfer). That VM must not be subject to retention — only its
// snapshots are. Build the set of VM refs that already have snapshots in the list so
// we can exclude them, while keeping old-style non-snapshot VMs (no snapshots).
const vmRefsWithSnapshots = new Set(allEntries.filter(e => e.is_a_snapshot).map(e => e.snapshot_of))
const retentionEntries = allEntries.filter(e => e.is_a_snapshot || !vmRefsWithSnapshots.has(e.$ref))
retentionEntries.sort(compareReplicatedVmDatetime)
this._oldEntries = getOldEntries(settings.copyRetention - 1, retentionEntries)
if (settings.deleteFirst && settings.skipDeleteOldEntries) {
// we want to keep the baseVM when copying a delta
// even if we want to keep only one after
let mostRecentEntry
@@ -86,45 +234,53 @@ export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWr
}
async cleanup() {
await this._deleteOldEntries()
if (!this._settings.skipDeleteOldEntries) {
await this._deleteOldEntries()
}
}
async _deleteOldEntries() {
return asyncMapSettled(this._oldEntries, vm => vm.$destroy())
return asyncMapSettled(this._oldEntries, vm => vm.$destroy({ bypassBlockedOperation: true }))
}
#decorateVmMetadata(backup) {
#decorateVmMetadata(backup, timestamp) {
const { _warmMigration } = this._settings
const sr = this._sr
const vm = backup.vm
const job = this._job
const scheduleId = this._schedule.id
// update other_config data as soon as possible to ensure the next job
// will be able to detect any partial transfer and lean them
vm.other_config[COPY_OF] = vm.uuid
vm.other_config[JOB_ID] = job.id
vm.other_config[SCHEDULE_ID] = scheduleId
vm.other_config[REPLICATED_TO_SR_UUID] = sr.uuid
// set the timestamp in the past to ensure any incomplete VM will be deleted on next run
vm.other_config[DATETIME] = formatFilenameDate(0)
vm.blocked_operations = {
start: 'Start operation for this vm is blocked, clone it if you want to use it.',
start_on: 'Start operation for this vm is blocked, clone it if you want to use it.',
}
if (!_warmMigration) {
vm.tags.push('Continuous Replication')
}
// extracting the uuid of each delta vdi on the source
// get all in one pass, since there is a lot of objects
const sourceVdiUuids = Object.values(backup.vdis)
.map(({ other_config }) => other_config[BASE_DELTA_VDI])
// full vdi don't have a base
.filter(_ => !!_)
// @todo use index ?
const replicatedVdis = sr.$VDIs.filter(vdi => {
// REPLICATED_TO_SR_UUID is not used here since we are already filtering from sr.$VDIs
return sourceVdiUuids.includes(vdi?.other_config[COPY_OF])
})
Object.values(backup.vdis).forEach(vdi => {
vdi.other_config[COPY_OF] = vdi.uuid
if (sourceVdiUuids.length > 0) {
const baseReplicatedTo = replicatedVdis.find(
replicatedVdi => replicatedVdi.other_config[COPY_OF] === vdi.other_config[BASE_DELTA_VDI]
)
// baseReplicatedTo can be undefined if a new disk is added and other are already replicated
vdi.baseVdi = baseReplicatedTo
vdi.other_config[JOB_ID] = job.id
vdi.other_config[SCHEDULE_ID] = scheduleId
vdi.other_config[REPLICATED_TO_SR_UUID] = sr.uuid
vdi.other_config[VM_UUID] = vm.uuid
const baseDeltaVdiUuid = vdi.other_config[BASE_DELTA_VDI]
if (baseDeltaVdiUuid !== undefined) {
// reuse the validated mapping built by checkBaseVdis
vdi.baseVdi = this.#baseVdisBySourceUuid.get(baseDeltaVdiUuid)
} else {
// first replication of this disk
// first replication of this disk (full, no base)
vdi.baseVdi = undefined
}
// ensure the VDI are created on the target SR
@@ -138,19 +294,43 @@ export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWr
const { _warmMigration } = this._settings
const sr = this._sr
const job = this._job
const scheduleId = this._scheduleId
const schedule = this._schedule
const scheduleId = schedule.id
const { uuid: srUuid, $xapi: xapi } = sr
let targetVmRef
await Task.run({ name: 'transfer' }, async () => {
targetVmRef = await importIncrementalVm(this.#decorateVmMetadata(deltaExport), sr)
// size is mandatory to ensure the task have the right data
await Task.run({ properties: { name: 'transfer' } }, async () => {
targetVmRef = await importIncrementalVm(this.#decorateVmMetadata(deltaExport, timestamp), sr, {
targetRef: this._targetVmRef,
})
// this also ensure the data are up to date on the snapshot
await setVmOtherConfig(xapi, targetVmRef, {
timestamp, // updated at the end to mark the transfer as complete
jobId: job.id,
scheduleId,
vmUuid: vm.uuid,
srUuid,
})
const size = Object.values(deltaExport.disks).reduce(
(sum, disk) => sum + disk.getNbGeneratedBlock() * disk.getBlockSize(),
0
)
await xapi.setField(
'VM',
targetVmRef,
'name_description',
deltaExport.vm.name_description +
` -- last replication: ${formatFilenameDate(timestamp)} ${humanFormat.bytes(size)} read`
)
// take a snapshot to ensure these data are not modified until next snapshot
await Task.run({ properties: { name: 'target snapshot' } }, async () => {
await xapi.VM_snapshot(targetVmRef, {
name_label: `${vm.name_label} - ${job.name} / ${schedule.name} ${formatFilenameDate(timestamp)}`,
})
})
return {
size: Object.values(deltaExport.disks).reduce(
(sum, disk) => sum + disk.getNbGeneratedBlock() * disk.getBlockSize(),
0
),
size,
}
})
this._targetVmRef = targetVmRef
@@ -161,20 +341,6 @@ export class IncrementalXapiWriter extends MixinXapiWriter(AbstractIncrementalWr
!_warmMigration &&
targetVm.ha_restart_priority !== '' &&
Promise.all([targetVm.set_ha_restart_priority(''), targetVm.add_tags('HA disabled')]),
targetVm.set_name_label(`${vm.name_label} - ${job.name} - (${formatFilenameDate(timestamp)})`),
asyncMap(['start', 'start_on'], op =>
targetVm.update_blocked_operations(
op,
'Start operation for this vm is blocked, clone it if you want to use it.'
)
),
setVmOtherConfig(xapi, targetVmRef, {
timestamp,
jobId: job.id,
scheduleId,
vmUuid: vm.uuid,
srUuid,
}),
])
}
}

View File

@@ -0,0 +1,157 @@
import { asyncEach } from '@vates/async-each'
import { getOldEntries } from '../../_getOldEntries.mjs'
import { compareTimestamp } from '../../RemoteAdapter.mjs'
import { createLogger } from '@xen-orchestra/log'
const { debug } = createLogger('xo:backups:AbstractAggregatedRemoteWriter')
export const DEFAULT_AGGREGATED_REMOTE_DELETE_CONCURRENCY = 2
export class AbstractAggregatedRemoteWriter {
#BackupWriter
#adapters
get adapters() {
return this.#adapters
}
mainWriter
#props
get props() {
return this.#props
}
#oldBackups
#writers = []
get writers() {
return this.#writers
}
constructor({ adapters, BackupWriter, ...props }) {
debug('instantiate AggregatedFullRemoteWriter')
if (Object.keys(adapters).length < 1) {
throw new Error('AggregatedRemoteWriter need at least one backup repository to work ')
}
this.#props = props
this.#adapters = adapters
this.#BackupWriter = BackupWriter
}
getWriterByAdapter(adapter) {
for (const writer of this.#writers) {
if (writer._adapter === adapter) {
return writer
}
}
throw new Error("Can't find writer for adapter")
}
/**
* select the best adapter candidate to write this backup
* if all the adapter infos expose an available info use the one with the more free space
* if none of the adapter exposes this info : choose at random
* if mixed : throw an error
* @returns {RemoteAdapter}
*/
async computeAdapterCandidate() {
const adapters = Object.values(this.#adapters)
const infos = await Promise.all(
adapters.map(async adapter => {
const infos = await adapter._handler.getInfo()
return { adapter, infos }
})
)
const withLimit = infos.filter(({ infos: { available } }) => available > 0).length
if (withLimit !== 0 && withLimit !== infos.length) {
throw new Error(`an Aggregate remote can't use a mix of limited and unlimited storages`)
}
let candidates
if (withLimit === 0) {
debug('no storage limit , choose at random')
candidates = Object.values(this.adapters)
} else {
debug('storage limit , choose at random between those with the max space avilable')
let maxSpace = 0
infos.forEach(({ infos: { available } }) => {
if (available > maxSpace) {
maxSpace = available
}
})
// candidates are the adapter with the same free space , equal to the max free space
candidates = infos.filter(({ infos }) => infos.available === maxSpace).map(({ adapter }) => adapter)
}
const adapterCandidate = candidates[Math.floor(Math.random() * candidates.length)]
debug('adapter candidate is selected ', adapterCandidate._handler._remote.id)
return adapterCandidate
}
async setupWriters() {
debug('Setup writers ')
const settings = this.#props.settings
const allSettings = this.#props.job.settings
Object.entries(this.#adapters).forEach(([remoteId, adapter]) => {
const targetSettings = {
...settings,
...allSettings[remoteId],
skipDeleteOldEntries: true, // delete is handled globally
}
if (targetSettings.exportRetention !== 0) {
const writer = new this.#BackupWriter({
...this.#props,
adapter,
remoteId,
settings: targetSettings,
})
this.#writers.push(writer)
}
})
}
async getEntriesPerAdapter(adapter) {
const scheduleId = this.#props.schedule.id
const vmUuid = this.#props.vmUuid
return (await adapter.listVmBackups(vmUuid, _ => _.scheduleId === scheduleId)).map(entry => ({
...entry,
adapter,
}))
}
async setOldBackups() {
const entriesPerAdapter = await Promise.all(
Object.values(this.#adapters).map(adapter => this.getEntriesPerAdapter(adapter))
)
const entries = entriesPerAdapter
.flat(1)
.filter(_ => !!_)
.sort(compareTimestamp)
const settings = this.#props.settings
this.#oldBackups = getOldEntries(settings.exportRetention - 1, entries, {
longTermRetention: settings.longTermRetention,
timezone: settings.timezone,
})
debug(`got ${this.#oldBackups.length} old backup(s) to delete `)
}
async deleteOldBackupsOnAdapter(adapter, backup) {
throw new Error('not implemented')
}
async deleteOldBackups() {
const byAdapters = new Map()
this.#oldBackups.forEach(({ adapter, ...backup }) => {
const current = byAdapters.has(adapter) ? byAdapters.get(adapter) : []
current.push(backup)
byAdapters.set(adapter, current)
})
await asyncEach(
byAdapters.entries(),
async ([adapter, backups]) => {
debug(`delete ${backups.length} old backup(s) to delete from ${adapter._handler._remote.id}`)
await this.deleteOldBackupsOnAdapter(adapter, backups)
},
{
concurrency: this.#props.deleteConcurrency ?? DEFAULT_AGGREGATED_REMOTE_DELETE_CONCURRENCY,
}
)
}
}

View File

@@ -0,0 +1,114 @@
import { getOldEntries } from '../../_getOldEntries.mjs'
import { createLogger } from '@xen-orchestra/log'
import { compareReplicatedVmDatetime, listReplicatedVms } from './_listReplicatedVms.mjs'
import { asyncMapSettled } from '@xen-orchestra/async-map'
const { debug } = createLogger('xo:backups:AbstractAggregatedXapiWriter')
export class AbstractAggregatedXapiWriter {
#ReplicationWriter
#storageRepositories
#writers = []
get writers() {
return this.#writers
}
mainWriter
#props
get props() {
return this.#props
}
#oldVmReplicaList = []
constructor({ srs, ReplicationWriter, ...props }) {
debug('instantiate AbstractAggregatedXapiWriter')
if (srs.length < 1) {
throw new Error('AggregatedXapiWriter need at least one storage repository to work ')
}
this.#props = props
this.#ReplicationWriter = ReplicationWriter
this.#storageRepositories = srs
}
getWriterBySr(storageRepository) {
for (const writer of this.#writers) {
if (writer._sr === storageRepository) {
return writer
}
}
throw new Error("Can't find writer for storage repository")
}
async computeSRCandidate() {
let maxSpace = 0
this.#storageRepositories.forEach(({ physical_utilisation, physical_size }) => {
if (physical_size - physical_utilisation > maxSpace) {
maxSpace = physical_size - physical_utilisation
}
})
debug(`computeSRCandidate found the max free space ${maxSpace} for vm${this.#props.vmUuid}`)
const srCandidates = this.#storageRepositories.filter(
({ physical_utilisation, physical_size }) => maxSpace === physical_size - physical_utilisation
)
// one at random from the sr with the most free space
const candidate = srCandidates[Math.floor(Math.random() * srCandidates.length)]
debug(
`computeSRCandidate found ${srCandidates.length} sr with ${maxSpace} free size, and chose ${candidate.name_label} for vm${this.#props.vmUuid}`
)
return candidate
}
setOldReplicaList() {
debug(`setOldReplicaList for vm ${this.#props.vmUuid}`)
const scheduleId = this.#props.scheduleId
const vmUuid = this.#props.vmUuid
const settings = this.#props.settings
const replicatedVms = this.#storageRepositories
.map(sr => {
return listReplicatedVms(sr.$xapi, scheduleId, sr.uuid, vmUuid)
})
.flat(1)
.filter(_ => !!_)
.sort(compareReplicatedVmDatetime)
this.#oldVmReplicaList = getOldEntries(settings.copyRetention - 1, replicatedVms)
debug(
`setOldReplicaList found ${replicatedVms.length} replica, and will delete ${this.#oldVmReplicaList.length} (retention ${settings.copyRetention}) for vm ${this.#props.vmUuid}`
)
}
async deleteOldReplicas() {
debug(`deleteOldReplicas will delete ${this.#oldVmReplicaList.length} replicated vms for vm ${this.#props.vmUuid}`)
// destroy the VM on the right xapi
await asyncMapSettled(this.#oldVmReplicaList, async vm => {
debug('will delete old replica', vm.name_label)
await vm.$xapi.VM_destroy(vm.$ref)
})
debug(`deleteOldReplicas deleted ${this.#oldVmReplicaList.length} replicated vms for vm ${this.#props.vmUuid}`)
}
async setupWriters() {
debug('Setup writers ')
const settings = this.#props.settings
const allSettings = this.#props.job.settings
this.#storageRepositories.forEach(storageRepository => {
const targetSettings = {
...settings,
...allSettings[storageRepository.uuid],
skipDeleteOldEntries: true, // delete is handled globally
}
if (targetSettings.copyRetention !== 0) {
const writer = new this.#ReplicationWriter({
...this.#props,
sr: storageRepository,
settings: targetSettings,
})
this.#writers.push(writer)
}
})
}
}

View File

@@ -1,12 +1,9 @@
import { formatFilenameDate } from '../../_filenameDate.mjs'
import { getVmBackupDir } from '../../_getVmBackupDir.mjs'
export class AbstractWriter {
constructor({ config, healthCheckSr, job, vmUuid, scheduleId, settings }) {
constructor({ config, healthCheckSr, job, vmUuid, schedule, settings }) {
this._config = config
this._healthCheckSr = healthCheckSr
this._job = job
this._scheduleId = scheduleId
this._schedule = schedule
this._settings = settings
this._vmUuid = vmUuid
}
@@ -16,14 +13,4 @@ export class AbstractWriter {
afterBackup() {}
healthCheck(sr) {}
_isAlreadyTransferred(timestamp) {
const vmUuid = this._vmUuid
const adapter = this._adapter
const backupDir = getVmBackupDir(vmUuid)
try {
const actualMetadata = JSON.parse(adapter._handler.readFile(`${backupDir}/${formatFilenameDate(timestamp)}.json`))
return actualMetadata
} catch (error) {}
}
}

View File

@@ -1,14 +1,15 @@
import { createLogger } from '@xen-orchestra/log'
import { join } from 'node:path'
import { Task } from '@vates/task'
import assert from 'node:assert'
import { formatFilenameDate } from '../../_filenameDate.mjs'
import { getVmBackupDir } from '../../_getVmBackupDir.mjs'
import { HealthCheckVmBackup } from '../../HealthCheckVmBackup.mjs'
import { ImportVmBackup } from '../../ImportVmBackup.mjs'
import { Task } from '../../Task.mjs'
import * as MergeWorker from '../../merge-worker/index.mjs'
import ms from 'ms'
import { getEntryStatus } from '../../_getOldEntries.mjs'
const { info, warn } = createLogger('xo:backups:MixinBackupWriter')
@@ -27,7 +28,7 @@ export const MixinRemoteWriter = (BaseClass = Object) =>
async _cleanVm(options) {
try {
return await Task.run({ name: 'clean-vm' }, () => {
return await Task.run({ properties: { name: 'clean-vm' } }, () => {
return this._adapter.cleanVm(this._vmBackupDir, {
...options,
fixMetadata: true,
@@ -47,6 +48,19 @@ export const MixinRemoteWriter = (BaseClass = Object) =>
}
}
async getLongTermRetentionTags(currentEntry) {
const settings = this._settings
const scheduleId = this._schedule.id
const vmUuid = this._vmUuid
const adapter = this._adapter
const entries = await adapter.listVmBackups(vmUuid, _ => _.scheduleId === scheduleId)
entries.push(currentEntry)
const buckets = getEntryStatus(entries, currentEntry, settings.longTermRetention, settings.timezone)
return buckets.map(({ duration, dateBucket }) => `${duration}=${dateBucket}`)
}
async beforeBackup() {
const { handler } = this._adapter
const vmBackupDir = this._vmBackupDir
@@ -86,7 +100,7 @@ export const MixinRemoteWriter = (BaseClass = Object) =>
}
return Task.run(
{
name: 'health check',
properties: { name: 'health check' },
},
async () => {
const xapi = sr.$xapi
@@ -99,7 +113,7 @@ export const MixinRemoteWriter = (BaseClass = Object) =>
srUuid,
xapi,
settings: {
additionnalVmTag: 'xo:no-bak=Health Check',
additionalVmTag: 'xo:no-bak=Health Check',
},
}).run()
let restoredVm
@@ -116,7 +130,7 @@ export const MixinRemoteWriter = (BaseClass = Object) =>
xapi,
}).run()
} finally {
await xapi.VM_destroy(restoredVm.$ref)
await xapi.VM_destroy(restoredVm.$ref, { bypassBlockedOperation: true })
}
}
)

View File

@@ -1,7 +1,7 @@
import assert from 'node:assert/strict'
import { Task } from '@vates/task'
import { HealthCheckVmBackup } from '../../HealthCheckVmBackup.mjs'
import { Task } from '../../Task.mjs'
import ms from 'ms'
export const MixinXapiWriter = (BaseClass = Object) =>
@@ -29,11 +29,11 @@ export const MixinXapiWriter = (BaseClass = Object) =>
// the SR that the VM has been replicated on
const sr = this._sr
assert.notStrictEqual(sr, undefined, 'SR should be defined before making a health check')
assert.notEqual(this._targetVmRef, undefined, 'A vm should have been transfered to be health checked')
assert.notEqual(this._targetVmRef, undefined, 'A vm should have been transferred to be health checked')
// copy VM
return Task.run(
{
name: 'health check',
properties: { name: 'health check' },
},
async () => {
const { $xapi: xapi } = sr
@@ -47,12 +47,12 @@ export const MixinXapiWriter = (BaseClass = Object) =>
}
if (await this.#isAlreadyOnHealthCheckSr(baseVm)) {
healthCheckVmRef = await Task.run(
{ name: 'cloning-vm' },
{ properties: { name: 'cloning-vm' } },
async () => await xapi.callAsync('VM.clone', this._targetVmRef, `Health Check - ${baseVm.name_label}`)
)
} else {
healthCheckVmRef = await Task.run(
{ name: 'copying-vm' },
{ properties: { name: 'copying-vm' } },
async () =>
await xapi.callAsync(
'VM.copy',
@@ -72,7 +72,7 @@ export const MixinXapiWriter = (BaseClass = Object) =>
xapi,
}).run()
} finally {
healthCheckVmRef && (await xapi.VM_destroy(healthCheckVmRef))
healthCheckVmRef && (await xapi.VM_destroy(healthCheckVmRef, { bypassBlockedOperation: true }))
}
}
)

View File

@@ -5,24 +5,31 @@ const getReplicatedVmDatetime = vm => {
return datetime
}
const compareReplicatedVmDatetime = (a, b) => (getReplicatedVmDatetime(a) < getReplicatedVmDatetime(b) ? -1 : 1)
export const compareReplicatedVmDatetime = (a, b) => (getReplicatedVmDatetime(a) < getReplicatedVmDatetime(b) ? -1 : 1)
/**
*
* @param {Xapi} xapi
* @param {string} scheduleOrJobId
* @param {string} srUuid
* @param {string} vmUuid
* @returns {Array<XoVm>}
*/
export function listReplicatedVms(xapi, scheduleOrJobId, srUuid, vmUuid) {
const { all } = xapi.objects
const all = xapi.objects.indexes.type.VM
const vms = {}
for (const key in all) {
const object = all[key]
const oc = object.other_config
if (
object.$type === 'VM' &&
!object.is_a_snapshot &&
!object.is_a_template &&
'start' in object.blocked_operations &&
(oc[JOB_ID] === scheduleOrJobId || oc[SCHEDULE_ID] === scheduleOrJobId) &&
oc[REPLICATED_TO_SR_UUID] === srUuid &&
(oc[VM_UUID] === vmUuid ||
// 2018-03-28, JFT: to catch VMs replicated before this fix
oc[VM_UUID] === undefined)
oc[VM_UUID] === vmUuid &&
// Old-style replication: one VM per transfer (non-snapshot, start blocked)
// New-style replication: snapshots of the target VM represent each transfer
(!object.is_a_snapshot ? 'start' in object.blocked_operations : true)
) {
vms[object.$id] = object
}

View File

@@ -0,0 +1,325 @@
// @ts-check
/**
* @typedef {import('./RemoteDisk.mjs').RemoteDisk} RemoteDisk
* @typedef {import('@xen-orchestra/disk-transform').FileAccessor} FileAccessor
*/
import assert from 'assert'
import { createLogger } from '@xen-orchestra/log'
import { basename, dirname } from 'path'
import { asyncEach } from '@vates/async-each'
import { relativeFromFile } from '@xen-orchestra/fs/path'
// @ts-ignore
const { warn } = createLogger('remote-disk:merge')
/**
* @typedef {Object} MergeState
* @property {{ uuid: string }} child
* @property {{ uuid: string }} parent
* @property { string[] | undefined} chain
* @property {number} currentBlock
* @property {number} mergedDataSize
* @property {'mergeBlocks' | 'cleanup'} step
* @property {number} diskSize
* @typedef {(message: string, data?: Record<string, unknown>) => void} Logger
*/
export class MergeRemoteDisk {
/**
* @type {MergeState}
*/
#state = {
child: { uuid: '0' },
parent: { uuid: '0' },
chain: undefined,
currentBlock: 0,
mergedDataSize: 0,
step: 'mergeBlocks',
diskSize: 0,
}
/**
* @type {string}
*/
#statePath = ''
/**
* @type {boolean}
*/
#isResuming
/**
* @type {Logger | Function}
*/
#logInfo
/**
* @type {number}
*/
#mergeBlockConcurrency
/**
* @type {Function}
*/
#onProgress
/**
* @type {boolean}
*/
#removeUnused
/**
* @type {number}
*/
#writeStateDelay
/**
* @type {number}
*/
#lastStateWrittenAt
/**
* @type {FileAccessor}
*/
#handler
/**
* @param {FileAccessor} handler
* @param {Object} params
* @param {Function} [params.onProgress]
* @param {Logger | Function} [params.logInfo]
* @param {boolean} [params.removeUnused]
* @param {number} [params.mergeBlockConcurrency]
* @param {number} [params.writeStateDelay]
*/
constructor(
handler,
{
onProgress = () => {},
logInfo = () => {},
removeUnused = false,
mergeBlockConcurrency = 2,
writeStateDelay = 10e3,
}
) {
this.#handler = handler
this.#logInfo = logInfo
this.#onProgress = onProgress
this.#removeUnused = removeUnused
this.#mergeBlockConcurrency = mergeBlockConcurrency
this.#writeStateDelay = writeStateDelay
this.#isResuming = false
this.#lastStateWrittenAt = 0
}
/**
* @param {RemoteDisk} parentDisk
* @returns {Promise<boolean>} isResuming
*/
async isResuming(parentDisk) {
try {
await this.#handler.readFile(
dirname(parentDisk.getPath()) + '/.' + basename(parentDisk.getPath()) + '.merge.json'
)
return true
} catch (error) {
return false
}
}
/**
* @param {RemoteDisk} parentDisk
* @param {RemoteDisk} childDisk
*/
async merge(parentDisk, childDisk) {
this.#statePath = dirname(parentDisk.getPath()) + '/.' + basename(parentDisk.getPath()) + '.merge.json'
try {
const mergeStateContent = await this.#handler.readFile(this.#statePath)
this.#state = JSON.parse(mergeStateContent)
// work-around a bug introduce in 97d94b795
//
// currentBlock could be `null` due to the JSON.stringify of a `NaN` value
if (this.#state?.currentBlock === null) this.#state.currentBlock = 0
this.#isResuming = true
} catch (error) {
// @ts-ignore
if (error.code !== 'ENOENT') {
warn('problem while checking the merge state', { error })
}
}
try {
/* eslint-disable no-fallthrough */
switch (this.#state?.step ?? 'mergeBlocks') {
case 'mergeBlocks':
await this.#step_mergeBlocks(parentDisk, childDisk)
case 'cleanup':
await this.#step_cleanup(parentDisk, childDisk)
return this.#cleanup(parentDisk, childDisk, true)
default:
warn(`Step ${this.#state?.step} is unknown`, { state: this.#state })
}
/* eslint-enable no-fallthrough */
} catch (error) {
await this.#cleanup(parentDisk, childDisk, false)
throw error
}
}
async #writeState() {
try {
await this.#handler.writeFile(this.#statePath, JSON.stringify(this.#state), { flags: 'w' })
} catch (err) {
warn('failed to write merge state', { error: err })
}
}
async #writeStateThrottled() {
const now = Date.now()
if (now - this.#lastStateWrittenAt > this.#writeStateDelay) {
this.#lastStateWrittenAt = now
await this.#writeState()
}
}
/**
* @param {RemoteDisk} parentDisk
* @param {RemoteDisk} childDisk
*/
async #step_mergeBlocks(parentDisk, childDisk) {
const getMaxBlockCount = childDisk.getMaxBlockCount()
await parentDisk.resize(getMaxBlockCount)
if (this.#isResuming) {
const alreadyMergedBlocks = []
for (let blockId = 0; blockId < this.#state.currentBlock; blockId++) {
if (childDisk.hasBlock(blockId)) {
alreadyMergedBlocks.push(blockId)
}
}
parentDisk.setAllocatedBlocks(alreadyMergedBlocks)
} else {
this.#state.child = { uuid: childDisk.getUuid() ?? undefined }
this.#state.parent = { uuid: parentDisk.getUuid() ?? undefined }
this.#state.chain = [parentDisk.getPath(), ...childDisk.getPaths()].map(path =>
relativeFromFile(this.#statePath, path)
)
// Finds first allocated block for the 2 following loops
while (this.#state.currentBlock < getMaxBlockCount && !childDisk.hasBlock(this.#state.currentBlock)) {
++this.#state.currentBlock
}
await this.#writeState()
}
await this.#mergeBlocks(parentDisk, childDisk)
await parentDisk.flushMetadata(childDisk)
await parentDisk.mergeMetadata(childDisk)
}
/**
* @param {RemoteDisk} parentDisk
* @param {RemoteDisk} childDisk
*/
async #mergeBlocks(parentDisk, childDisk) {
this.#mergeBlockConcurrency =
(await parentDisk.canMergeConcurently()) && (await childDisk.canMergeConcurently())
? this.#mergeBlockConcurrency
: 1
const toMerge = []
for (let block = this.#state.currentBlock; block < childDisk.getMaxBlockCount(); block++) {
if (childDisk.hasBlock(block)) {
toMerge.push(block)
}
}
const nBlocks = toMerge.length
this.#onProgress({ total: nBlocks, done: 0 })
const merging = new Set()
let counter = 0
await asyncEach(
toMerge,
async blockId => {
merging.add(blockId)
const blockSize = await parentDisk.mergeBlock(childDisk, blockId, this.#isResuming)
this.#state.mergedDataSize += blockSize
this.#state.currentBlock = Math.min(...merging) - 1
merging.delete(blockId)
this.#onProgress({ total: nBlocks, done: counter + 1 })
counter++
await this.#writeStateThrottled()
},
{ concurrency: this.#mergeBlockConcurrency }
)
await this.#writeState()
this.#state.diskSize = childDisk.getSizeOnDisk()
this.#onProgress({ total: nBlocks, done: nBlocks })
}
/**
* @param {RemoteDisk} parentDisk
* @param {RemoteDisk} childDisk
*/
async #step_cleanup(parentDisk, childDisk) {
assert.notEqual(this.#state, undefined)
this.#state.step = 'cleanup'
await this.#writeState()
const mergeTargetPath = childDisk.getPath()
// delete intermediate children if needed
if (this.#removeUnused) {
await childDisk.unlink()
}
try {
await parentDisk.rename(mergeTargetPath)
} catch (error) {
// @ts-ignore
if (error.code === 'ENOENT' && this.#isResuming) {
// @ts-ignore
this.#logInfo(`the parent disk was already renamed`, {
parent: parentDisk.getPath(),
mergeTarget: mergeTargetPath,
})
} else {
throw error
}
}
}
/**
* @param {RemoteDisk} parentDisk
* @param {RemoteDisk} childDisk
* @param {boolean} cleanStateFile
*
* @returns {Promise<{mergedDataSize: number, finalDiskSize: number}>} result
*/
async #cleanup(parentDisk, childDisk, cleanStateFile) {
const finalDiskSize = this.#state?.diskSize ?? 0
const mergedDataSize = this.#state?.mergedDataSize ?? 0
await parentDisk.close().catch(warn)
await childDisk.close().catch(warn)
if (cleanStateFile) {
await this.#handler.unlink(this.#statePath).catch(warn)
}
return { mergedDataSize, finalDiskSize }
}
}

View File

@@ -0,0 +1,223 @@
// @ts-check
/**
* @typedef {import('@xen-orchestra/disk-transform').DiskBlock} DiskBlock
*/
import { RandomAccessDisk } from '@xen-orchestra/disk-transform'
export class RemoteDisk extends RandomAccessDisk {
/**
* Abstract
* @param {Object} options
* @param {boolean} options.force
* @returns {Promise<void>}
*/
async init(options = { force: false }) {
throw new Error(`init must be implemented`)
}
/**
* Abstract
* @returns {Promise<void>}
*/
async close() {
throw new Error(`close must be implemented`)
}
/**
* Abstract
* @returns {number}
*/
getVirtualSize() {
throw new Error(`getVirtualSize must be implemented`)
}
/**
* Abstract
* @returns {number} size
*/
getSizeOnDisk() {
throw new Error(`getSizeOnDisk must be implemented`)
}
/**
* @returns {number}
*/
getBlockSize() {
throw new Error(`getBlockSize must be implemented`)
}
/**
* Abstract
* @returns {string}
*/
getPath() {
throw new Error(`getPath must be implemented`)
}
/**
* Abstract
* Returns an array of disk paths.
*
* @returns {string[]}
*/
getPaths() {
throw new Error(`getPaths must be implemented`)
}
/**
* Abstract
* @returns {string}
*/
getUuid() {
throw new Error(`getUuid must be implemented`)
}
/**
* Abstract
* @returns {Promise<boolean>} canMergeConcurently
*/
async canMergeConcurently() {
throw new Error(`canMergeConcurently must be implemented`)
}
/**
* @returns {number} getMaxBlockCount
*/
getMaxBlockCount() {
return Math.ceil(this.getVirtualSize() / this.getBlockSize())
}
/**
* Checks if the VHD contains a specific block.
* @param {number} index
* @returns {boolean}
*/
hasBlock(index) {
throw new Error(`hasBlock must be implemented`)
}
/**
* Abstract
* Gets the indexes of all blocks in the VHD.
* @returns {Array<number>}
*/
getBlockIndexes() {
throw new Error(`getBlockIndexes must be implemented`)
}
/**
* Abstract
* Returns the parent non inizialized instance
* @returns {RemoteDisk}
*/
instantiateParent() {
throw new Error(`instantiateParent must be implemented`)
}
/**
* Abstract
* Writes a full block.
* @param {DiskBlock} diskBlock
* @return {Promise<number>} blockSize
*/
async writeBlock(diskBlock) {
throw new Error(`writeBlock must be implemented`)
}
/**
* Abstract
* Reads a specific block from the VHD.
* @param {number} index
* @returns {Promise<DiskBlock>} diskBlock
*/
async readBlock(index) {
throw new Error(`readBlock must be implemented`)
}
/**
* Abstract
* Reads a specific block from the child disk to copy/move it to this disk.
* @param {RemoteDisk} childDisk
* @param {number} index
* @param {boolean} isResumingMerge
* @returns {Promise<number>} blockSize
*/
async mergeBlock(childDisk, index, isResumingMerge) {
throw new Error(`mergeBlock must be implemented`)
}
/**
* Abstract
* Manually set the disk allocated blocks.
* @param {Array<number>} blockIds
* @returns {Promise<void>}
*/
async setAllocatedBlocks(blockIds) {
throw new Error(`setAllocatedBlocks must be implemented`)
}
/**
* Abstract
* @param {number} blockCount
* @returns {Promise<void>}
*/
async resize(blockCount) {
throw new Error(`resize must be implemented`)
}
/**
* Abstract
* @param {RemoteDisk} childDisk
* @returns {Promise<void>}
*/
async flushMetadata(childDisk) {
throw new Error(`flushMetadata must be implemented`)
}
/**
* Abstract
* @param {RemoteDisk} childDisk
* @returns {Promise<void>}
*/
mergeMetadata(childDisk) {
throw new Error(`mergeMetadata must be implemented`)
}
/**
* Abstract
* Checks if the VHD is a differencing disk.
* @returns {boolean}
*/
isDifferencing() {
throw new Error(`isDifferencing must be implemented`)
}
/**
* Abstract
* Rename alias/disk
* @param {string} newPath
*/
async rename(newPath) {
throw new Error(`rename must be implemented`)
}
/**
* Abstract
* Deletes alias/disk/disks
*/
async unlink() {
throw new Error(`unlink must be implemented`)
}
/**
* @returns {Promise<RemoteDisk>}
*/
async openParent() {
const parent = await super.openParent()
if (!(parent instanceof RemoteDisk)) {
throw new Error('parent of a RemoteDisk must be also a RemoteDisk')
}
return parent
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,171 +0,0 @@
// @ts-check
/**
* @typedef {import('@xen-orchestra/disk-transform').FileAccessor} FileAccessor
* @typedef {import('@xen-orchestra/disk-transform').DiskBlock} DiskBlock
* @typedef {import('@xen-orchestra/disk-transform').Disk} Disk
* @typedef {import('vhd-lib/Vhd/VhdDirectory.js').VhdDirectory} VhdDirectory
* @typedef {import('vhd-lib/Vhd/VhdFile.js').VhdFile} VhdFile
*/
import { openVhd } from 'vhd-lib'
import { DISK_TYPES } from 'vhd-lib/_constants.js'
import { dirname, join } from 'node:path'
import { RandomAccessDisk } from '@xen-orchestra/disk-transform'
/**
* Represents a remote VHD (Virtual Hard Disk) that extends RandomAccessDisk.
*/
export class RemoteVhd extends RandomAccessDisk {
/**
* @type {string}
*/
#path
/**
* @type {FileAccessor}
*/
#handler
/**
* @type {VhdFile | VhdDirectory | undefined}
*/
#vhd
/**
* @type {boolean | undefined}
*/
#isDifferencing
/**
* @type {() => any}
*/
#dispose = () => {}
/**
* @returns {string}
*/
get path() {
return this.#path
}
/**
* @param {Object} params
* @param {FileAccessor} params.handler
* @param {string} params.path
*/
constructor({ handler, path }) {
super()
// @todo : ensure this is the full path from the root of the remote
this.#path = path
this.#handler = handler
}
/**
* @returns {number}
*/
getVirtualSize() {
if (this.#vhd === undefined) {
throw new Error(`can't call getvirtualsize of a RemoteVhd before init`)
}
return this.#vhd.footer.currentSize
}
/**
* @returns {number}
*/
getBlockSize() {
return 2 * 1024 * 1024
}
/**
* Initializes the VHD.
* @returns {Promise<void>}
*/
async init() {
const { value, dispose } = await openVhd(this.#handler, this.#path)
this.#vhd = value
this.#dispose = dispose
await this.#vhd.readBlockAllocationTable()
this.#isDifferencing = value.footer.diskType === DISK_TYPES.DIFFERENCING
}
/**
* Closes the VHD.
* @returns {Promise<void>}
*/
async close() {
await this.#dispose()
}
/**
* Checks if the VHD contains a specific block.
* @param {number} index
* @returns {boolean}
*/
hasBlock(index) {
if (this.#vhd === undefined) {
throw new Error(`can't call hasblock of a RemoteVhd before init`)
}
return this.#vhd.containsBlock(index)
}
/**
* Gets the indexes of all blocks in the VHD.
* @returns {Array<number>}
*/
getBlockIndexes() {
if (this.#vhd === undefined) {
throw new Error(`can't call getBlockIndexes of a RemoteVhd before init`)
}
const index = []
for (let blockIndex = 0; blockIndex < this.#vhd.header.maxTableEntries; blockIndex++) {
if (this.hasBlock(blockIndex)) {
index.push(blockIndex)
}
}
return index
}
/**
* Reads a specific block from the VHD.
* @param {number} index
* @returns {Promise<DiskBlock>}
*/
async readBlock(index) {
if (this.#vhd === undefined) {
throw new Error(`can't call readBlock of a RemoteVhd before init`)
}
const { data } = await this.#vhd.readBlock(index)
return {
index,
data,
}
}
/**
*
* @returns {RandomAccessDisk}
*/
instantiateParent() {
if (this.#vhd === undefined) {
throw new Error(`can't call openParent of a RemoteVhd before init`)
}
const parentPath = this.#vhd.header.parentUnicodeName
const fullParentPath = join(dirname(this.#path), parentPath)
if (!parentPath) {
throw new Error(`Disk ${this.#path} doesn't have parents`)
}
const parent = new RemoteVhd({ handler: this.#handler, path: fullParentPath })
return parent
}
/**
* Checks if the VHD is a differencing disk.
* @returns {boolean}
*/
isDifferencing() {
if (this.#isDifferencing === undefined) {
throw new Error(`can't call isDifferencing of a RemoteVhd before init`)
}
return this.#isDifferencing
}
}

View File

@@ -0,0 +1,480 @@
// @ts-check
/**
* @typedef {import('vhd-lib/Vhd/VhdDirectory.js').VhdDirectory} VhdDirectory
* @typedef {import('vhd-lib/Vhd/VhdFile.js').VhdFile} VhdFile
* @typedef {import('vhd-lib/_createFooterHeader').VhdFooter} VhdFooter
* @typedef {import('@xen-orchestra/disk-transform').DiskBlock} DiskBlock
* @typedef {import('@xen-orchestra/disk-transform').FileAccessor} FileAccessor
*
*/
import { openVhd, VhdAbstract, VhdDirectory } from 'vhd-lib'
import { RemoteDisk } from './RemoteDisk.mjs'
import { DISK_TYPES } from 'vhd-lib/_constants.js'
import { isVhdAlias, resolveVhdAlias } from 'vhd-lib/aliases.js'
import { stringify } from 'uuid'
import { dirname, join } from 'node:path'
import { RemoteVhdDiskChain } from './RemoteVhdDiskChain.mjs'
export class RemoteVhdDisk extends RemoteDisk {
/**
* @type {string}
*/
#path
/**
* @type {FileAccessor}
*/
#handler
/**
* @type {VhdFile | VhdDirectory | undefined}
*/
#vhd
/**
* @type {boolean | undefined}
*/
#isDifferencing
/**
* @type {number}
*/
#blockSize = 2 * 1024 * 1024
/**
* @type {number}
*/
#bitmapSize = 512
/**
* @type {() => any}
*/
#dispose = () => {}
/**
* @param {Object} params
* @param {FileAccessor} params.handler
* @param {string} params.path
*/
constructor({ handler, path }) {
super()
// @todo : ensure this is the full path from the root of the remote
this.#path = path
this.#handler = handler
}
/**
* @param {Object} [options]
* @param {boolean} [options.force=false]
* @returns {Promise<void>}
*/
async init(options = {}) {
if (this.#vhd === undefined) {
try {
const { value, dispose } = await openVhd(this.#handler, await resolveVhdAlias(this.#handler, this.#path), {
checkSecondFooter: !options.force,
})
this.#vhd = value
if ((await this.isDirectory()) && !isVhdAlias(this.#path)) {
this.#vhd = undefined
throw Object.assign(new Error("Can't init vhd directory without using alias"), { code: 'NOT_SUPPORTED' })
}
this.#dispose = dispose
await this.#vhd.readBlockAllocationTable()
this.#isDifferencing = value.footer.diskType === DISK_TYPES.DIFFERENCING
} catch (error) {
await this.close()
throw error
}
}
}
/**
* Closes the VHD.
* We replace the dispose function call so the disk can only be closed once.
*
* @returns {Promise<void>}
*/
async close() {
const dispose = this.#dispose
this.#dispose = () => Promise.resolve()
await dispose()
}
/**
* @returns {number}
*/
getVirtualSize() {
if (this.#vhd === undefined) {
throw new Error(`can't call getVirtualSize of a RemoteVhdDisk before init`)
}
return this.#vhd.footer.currentSize
}
/**
* @returns {number} size
*/
getSizeOnDisk() {
if (this.#vhd === undefined) {
throw new Error(`can't call getVirtualSize of a RemoteVhdDisk before init`)
}
return this.#vhd.streamSize()
}
/**
* @returns {number}
*/
getBlockSize() {
return this.#blockSize
}
/**
* @returns {string}
*/
getPath() {
return this.#path
}
/**
* Returns the disk path in an array.
*
* @returns {string[]}
*/
getPaths() {
return [this.getPath()]
}
/**
* @returns {string}
*/
getUuid() {
if (this.#vhd === undefined) {
throw new Error(`can't call getUid of a RemoteVhdDisk before init`)
}
return stringify(this.#vhd.footer.uuid)
}
/**
* @returns {string}
*/
getParentUuid() {
if (this.#vhd === undefined) {
throw new Error(`can't call getParentUid of a RemoteVhdDisk before init`)
}
return stringify(this.#vhd.header.parentUuid)
}
/**
* @returns {Promise<boolean>} canMergeConcurently
*/
async canMergeConcurently() {
return await this.isDirectory()
}
/**
* Checks if the VHD contains a specific block.
* @param {number} index
* @returns {boolean}
*/
hasBlock(index) {
if (this.#vhd === undefined) {
throw new Error(`can't call hasblock of a RemoteVhdDisk before init`)
}
return this.#vhd.containsBlock(index)
}
/**
* Gets the indexes of all blocks in the VHD.
* @returns {Array<number>}
*/
getBlockIndexes() {
if (this.#vhd === undefined) {
throw new Error(`can't call getBlockIndexes of a RemoteVhdDisk before init`)
}
const indexes = []
for (let blockIndex = 0; blockIndex < this.#vhd.header.maxTableEntries; blockIndex++) {
if (this.hasBlock(blockIndex)) {
indexes.push(blockIndex)
}
}
return indexes
}
/**
* Returns the parent non inizialized instance
* @returns {RemoteDisk}
*/
instantiateParent() {
if (this.#vhd === undefined) {
throw new Error(`can't call instantiateParent of a RemoteVhdDisk before init`)
}
const parentPath = this.#vhd.header.parentUnicodeName
const fullParentPath = join(dirname(this.#path), parentPath)
if (!parentPath) {
throw new Error(`disk ${this.#path} doesn't have parents`)
}
const parent = new RemoteVhdDisk({ handler: this.#handler, path: fullParentPath })
return parent
}
/**
* Writes a full block into this VHD.
* @param {DiskBlock} diskBlock
* @return {Promise<number>} blockSize
*/
async writeBlock(diskBlock) {
if (this.#vhd === undefined) {
throw new Error(`can't call readBlock of a RemoteVhdDisk before init`)
}
await this.#vhd.writeEntireBlock({
id: diskBlock.index,
buffer: Buffer.concat([Buffer.alloc(this.#bitmapSize, 255), diskBlock.data]),
})
return this.getBlockSize()
}
/**
* Reads a specific block from the VHD.
* @param {number} index
* @returns {Promise<DiskBlock>} diskBlock
*/
async readBlock(index) {
if (this.#vhd === undefined) {
throw new Error(`can't call readBlock of a RemoteVhdDisk before init`)
}
const { data } = await this.#vhd.readBlock(index)
return {
index,
data,
}
}
/**
* Reads a specific block from the child disk to copy/move it to this disk.
* @param {RemoteDisk} childDisk
* @param {number} index
* @param {boolean} isResumingMerge
* @returns {Promise<number>} blockSize
*/
async mergeBlock(childDisk, index, isResumingMerge) {
if (
(childDisk instanceof RemoteVhdDisk || childDisk instanceof RemoteVhdDiskChain) &&
(await this.isDirectory()) &&
(await childDisk.isDirectory())
) {
try {
await this.#handler.rename(childDisk.getBlockPath(index), this.getBlockPath(index))
this.setAllocatedBlocks([index])
} catch (error) {
// @ts-ignore
if (error.code === 'ENOENT' && isResumingMerge === true) {
// when resuming, the blocks moved since the last merge state write are
// not in the child anymore but it should be ok
// it will throw an error if block is missing in parent
// won't detect if the block was already in parent and is broken/missing in child
// since we can't know the initial size, this will create a discrepancy
// on the size
const { data } = await this.readBlock(index)
if (data.length !== this.getBlockSize()) {
throw error
} else {
this.setAllocatedBlocks([index])
}
} else {
throw error
}
}
return this.getBlockSize()
} else {
return this.writeBlock(await childDisk.readBlock(index))
}
}
/**
* Gets a specific block path from the VHD directory.
* @param {number} index
* @returns {string} blockPath
*/
getBlockPath(index) {
if (this.#vhd === undefined) {
throw new Error(`can't call readBlock of a RemoteVhdDisk before init`)
}
if (this.#vhd instanceof VhdDirectory) {
return this.#vhd.getFullBlockPath(index)
} else {
throw new Error(`can't call getBlockPath of non directory VHD`)
}
}
/**
* Manually set the disk allocated blocks.
* @param {Array<number>} blockIds
* @returns {Promise<void>}
*/
async setAllocatedBlocks(blockIds) {
if (this.#vhd instanceof VhdDirectory) {
this.#vhd.setAllocatedBlocks(blockIds)
}
}
/**
* Ensure that the disk can handle at least the new block count.
* @param {number} blockCount
* @returns {Promise<void>}
*/
async resize(blockCount) {
if (this.#vhd === undefined) {
throw new Error(`can't call resize of a RemoteVhdDisk before init`)
}
// Checks that the BAT is at least as big as the provided block count, if not, increases it and shift the blocks position
await this.#vhd.ensureBatSize(blockCount)
}
/**
* Writes Block Allocation Table
* @param {RemoteDisk} childDisk
* @returns {Promise<void>}
*/
async flushMetadata(childDisk) {
if (this.#vhd === undefined) {
throw new Error(`can't call flushMetadata of a RemoteVhdDisk before init`)
}
await this.#vhd.writeBlockAllocationTable()
}
/**
* @returns {VhdFooter}
*/
getMetadata() {
if (this.#vhd === undefined) {
throw new Error(`can't call getMetadata of a RemoteVhdDisk before init`)
}
return this.#vhd.footer
}
/**
* @param {RemoteVhdDisk} childDisk
* @returns {Promise<void>}
*/
async mergeMetadata(childDisk) {
const childDiskMetadata = childDisk.getMetadata()
if (this.#vhd === undefined) {
throw new Error(`can't call mergeMetadata of a RemoteVhdDisk before init`)
}
// @ts-ignore
this.#vhd.footer.currentSize = childDiskMetadata.currentSize
// @ts-ignore
this.#vhd.footer.diskGeometry = { ...childDiskMetadata.diskGeometry }
// @ts-ignore
this.#vhd.footer.originalSize = childDiskMetadata.originalSize
// @ts-ignore
this.#vhd.footer.timestamp = childDiskMetadata.timestamp
// @ts-ignore
this.#vhd.footer.uuid = childDiskMetadata.uuid
await this.#vhd.writeFooter()
}
/**
* Checks if the VHD is a differencing disk.
* @returns {boolean}
*/
isDifferencing() {
if (this.#isDifferencing === undefined) {
throw new Error(`can't call isDifferencing of a RemoteVhdDisk before init`)
}
return this.#isDifferencing
}
/**
* Rename alias/disk
* @param {string} newPath
*/
async rename(newPath) {
if (isVhdAlias(newPath)) {
const dataPath = await resolveVhdAlias(this.#handler, this.#path)
await this.#handler.unlink(this.#path)
await this.#handler.unlink(newPath)
await VhdAbstract.createAlias(this.#handler, newPath, dataPath)
this.#path = newPath
} else {
try {
await this.#handler.unlink(newPath)
} catch (err) {
if (err && typeof err === 'object' && 'code' in err && err.code === 'EISDIR') {
await this.#handler.rmtree(newPath).catch(() => {})
}
}
await this.#handler.rename(this.#path, newPath)
this.#path = newPath
}
}
/**
* Deletes disk
*/
async unlink() {
if (this.#vhd === undefined) {
throw new Error(`can't call unlink of a RemoteVhdDisk before init`)
}
await this.close()
if (isVhdAlias(this.#path)) {
try {
await this.#handler.unlink(await resolveVhdAlias(this.#handler, this.#path))
} catch (err) {
if (err && typeof err === 'object' && 'code' in err && err.code === 'EISDIR') {
await this.#handler.rmtree(await resolveVhdAlias(this.#handler, this.#path)).catch(() => {})
}
}
}
try {
await this.#handler.unlink(this.#path)
} catch (err) {
if (err && typeof err === 'object' && 'code' in err && err.code === 'EISDIR') {
await this.#handler.rmtree(this.#path).catch(() => {})
}
}
}
/**
* Check if the disk is a VHD directory.
* @returns {Promise<boolean>}
*/
async isDirectory() {
if (this.#vhd === undefined) {
throw new Error(`can't call isDirectory of a RemoteVhdDisk before init`)
}
return this.#vhd instanceof VhdDirectory
}
}

View File

@@ -0,0 +1,304 @@
// @ts-check
/**
* @typedef {import('./RemoteVhdDisk.mjs').VhdFooter} VhdFooter
* @typedef {import('./RemoteVhdDisk.mjs').RemoteVhdDisk} RemoteVhdDisk
* @typedef {import('@xen-orchestra/disk-transform').DiskBlock} DiskBlock
* @typedef {import('@xen-orchestra/disk-transform').FileAccessor} FileAccessor
*/
import { RemoteDisk } from './RemoteDisk.mjs'
export class RemoteVhdDiskChain extends RemoteDisk {
/**
* @type {RemoteVhdDisk[]}
*/
#disks
/**
* @type {number}
*/
#blockSize = 2 * 1024 * 1024
/**
* @type {number}
*/
#headerSize = 1024
/**
* @type {number}
*/
#footerSize = 512
/**
* @type {number}
*/
#bitmapSize = 512
/**
* @param {Object} params
* @param {FileAccessor} params.handler
* @param {RemoteVhdDisk[]} params.disks
*/
constructor({ disks }) {
super()
this.#disks = disks
}
/**
* Initializes the VHD chain
* @param {Object} options
* @param {boolean} options.force
* @returns {Promise<void>}
*/
async init(options) {
try {
await Promise.all(this.#disks.map(disk => disk.init(options)))
let parentUuid = ''
for (const [index, disk] of this.#disks.entries()) {
if (index !== 0) {
if (!disk.isDifferencing()) {
throw Object.assign(new Error("Can't init vhd directory with non differencing child disks"), {
code: 'NOT_SUPPORTED',
})
}
if (disk.getParentUuid() !== parentUuid) {
throw Object.assign(new Error("Can't init vhd directory with incorrect parentage"), {
code: 'NOT_SUPPORTED',
})
}
}
parentUuid = disk.getUuid()
}
} catch (error) {
await this.close()
throw error
}
}
/**
* Closes the VHD.
* @returns {Promise<void>}
*/
async close() {
await Promise.all(this.#disks.map(disk => disk.close()))
}
/**
* @returns {number}
*/
getVirtualSize() {
return this.#disks[this.#disks.length - 1].getVirtualSize()
}
/**
* @returns {number} size
*/
getSizeOnDisk() {
const batEntrySize = 4
const sectorSize = 512
const batSize = Math.ceil((this.getMaxBlockCount() * batEntrySize) / sectorSize) * sectorSize
return (
this.#footerSize +
this.#headerSize +
batSize +
this.getBlockIndexes().length * (this.#blockSize + this.#bitmapSize) +
this.#footerSize
)
}
/**
* @returns {number}
*/
getBlockSize() {
return this.#disks[this.#disks.length - 1].getBlockSize()
}
/**
* @returns {string}
*/
getPath() {
return this.#disks[this.#disks.length - 1].getPath()
}
/**
* Disk chains return an array of disk paths.
*
* @returns {string[]}
*/
getPaths() {
return this.#disks.map(disk => disk.getPath())
}
/**
* @returns {string}
*/
getUuid() {
return this.#disks[this.#disks.length - 1].getUuid()
}
/**
* @returns {Promise<boolean>} canMergeConcurently
*/
async canMergeConcurently() {
return this.isDirectory()
}
/**
* @returns {number} getMaxBlockCount
*/
getMaxBlockCount() {
return this.#disks[this.#disks.length - 1].getMaxBlockCount()
}
/**
* Checks if the VHD contains a specific block.
* @param {number} index
* @returns {boolean}
*/
hasBlock(index) {
for (const disk of this.#disks) {
if (disk.hasBlock(index)) {
return true
}
}
return false
}
/**
* Gets the indexes of all blocks in the VHD.
* @returns {Array<number>}
*/
getBlockIndexes() {
const indexes = new Set()
for (const disk of this.#disks) {
for (const index of disk.getBlockIndexes()) {
indexes.add(index)
}
}
return [...indexes]
}
/**
* Returns the parent non inizialized instance
* @returns {RemoteDisk}
*/
instantiateParent() {
return this.#disks[0].instantiateParent()
}
/**
* Writes a full block into this VHD.
* @param {DiskBlock} diskBlock
* @return {Promise<number>} blockSize
*/
async writeBlock(diskBlock) {
throw new Error(`Can't write blocks into a disk chain`)
}
/**
* Reads a specific block from the VHD.
* @param {number} index
* @returns {Promise<DiskBlock>} diskBlock
*/
async readBlock(index) {
const reversedDisks = this.#disks.slice().reverse()
for (const disk of reversedDisks) {
if (disk.hasBlock(index)) {
return disk.readBlock(index)
}
}
throw new Error(`Block ${index} not found in chain `)
}
/**
* Reads a specific block from the child disk to copy/move it to this disk.
* @param {RemoteDisk} childDisk
* @param {number} index
* @param {boolean} isResumingMerge
* @returns {Promise<number>} blockSize
*/
async mergeBlock(childDisk, index, isResumingMerge) {
throw new Error(`Can't merge block into a disk chain`)
}
/**
* Gets a specific block path from the VHD directory disk.
* @param {number} index
* @returns {string} blockPath
*/
getBlockPath(index) {
for (const disk of [...this.#disks].reverse()) {
if (disk.hasBlock(index)) {
return disk.getBlockPath(index)
}
}
throw new Error(`Block ${index} not found in chain`)
}
/**
* @returns {VhdFooter}
*/
getMetadata() {
return this.#disks[this.#disks.length - 1].getMetadata()
}
/**
* @param {RemoteVhdDisk} childDisk
* @returns {Promise<void>}
*/
async flushMetadata(childDisk) {
throw new Error(`Can't flush metadata on a disk chain`)
}
/**
* @param {RemoteVhdDisk} childDisk
* @returns {Promise<void>}
*/
mergeMetadata(childDisk) {
throw new Error(`Can't merge metadata on a disk chain`)
}
/**
* Checks if the VHD is a differencing disk.
* @returns {boolean}
*/
isDifferencing() {
return this.#disks[0].isDifferencing()
}
/**
* Abstract
* Rename alias/disk
* @param {string} newPath
*/
async rename(newPath) {
throw new Error(`Can't rename a disk chain`)
}
/**
* Deletes all the disks
*/
async unlink() {
for (const disk of this.#disks) {
await disk.unlink()
}
}
/**
* Check if all the disks in the chain are VHD directories.
* @returns {Promise<boolean>}
*/
async isDirectory() {
for (const disk of this.#disks) {
if (!(await disk.isDirectory())) {
return false
}
}
return true
}
}

View File

@@ -0,0 +1,49 @@
import { RemoteVhdDisk } from './RemoteVhdDisk.mjs'
export { RemoteDisk } from './RemoteDisk.mjs'
export { openDiskChain } from './openDiskChain.mjs'
/**
* @typedef {import('../../disk-transform/src/FileAccessor.mjs').FileAccessor} FileAccessor
* @typedef {import('./RemoteDisk.mjs').RemoteDisk} RemoteDisk
*/
/**
* @param {Object} params
* @param {FileAccessor} params.handler
* @param {string} params.path
* @returns {Promise<RemoteDisk>}
*/
export async function openDisk({ handler, path }) {
const disk = new RemoteVhdDisk({ handler, path })
await disk.init()
return disk
}
/**
*
* @param {Object} params
* @param {FileAccessor} params.handler
* @param {string} params.path
* @returns {Promise<Disposable<RemoteDisk>>}
*/
export async function openDisposableDisk({ handler, path }) {
const disk = new RemoteVhdDisk({ handler, path })
await disk.init()
return {
value: disk,
dispose: () => disk.close(),
}
}
const DISK_EXTENSIONS = ['.vhd']
/**
* Returns true if the path points to a supported disk format.
*
* @param {FileAccessor} _handler - Remote file handler (reserved for future use)
* @param {string} path - Path to check
* @returns {boolean}
*/
export function isDisk(_handler, path) {
return DISK_EXTENSIONS.some(ext => path.endsWith(ext))
}

View File

@@ -2,12 +2,14 @@
/**
*
* @typedef {import('../../disk-transform/src/FileAccessor.mjs').FileAccessor} FileAccessor
* @typedef {import('./RemoteDisk.mjs').RemoteDisk} RemoteDisk
*/
import { DiskChain } from '@xen-orchestra/disk-transform'
import { RemoteVhd } from './RemoteVhd.mjs'
import { RemoteVhdDisk } from './RemoteVhdDisk.mjs'
import { defer } from 'golike-defer'
/**
* @param {any} $defer
* @param {Object} params
* @param {FileAccessor} params.handler
* @param {string} params.path
@@ -15,15 +17,18 @@ import { defer } from 'golike-defer'
*/
async function _openDiskChain($defer, { handler, path, until }) {
let disk
/**
* @type {Array<RemoteDisk>}
*/
const disks = []
$defer.onFailure(() => Promise.all(disks.map(disk => disk.close())))
disk = new RemoteVhd({ handler, path })
disk = new RemoteVhdDisk({ handler, path })
await disk.init()
disks.push(disk)
while (disk.isDifferencing()) {
disk = await disk.openParent()
if (disk.path === until) {
if (disk.getPath() === until) {
break
}
disks.unshift(disk)

View File

@@ -0,0 +1,116 @@
# Incremental Replication (CR) - Target Selection
This document explains how incremental replication (Continuous Replication to a XAPI SR) selects its target VM and VDIs, and the resulting transfer modes.
## Overview
The goal is to maintain a replica VM on a target SR. On each run, the system decides:
1. **Target VM**: should we update an existing replicated VM, or create a new one?
2. **Target VDIs**: for each disk, should we transfer a delta (incremental) or a full copy?
These two decisions are independent: we can create a new VM but still chain disks to existing VDIs from a previous replica, reducing transfer size.
## Key metadata (`other_config`)
Replicated VDI snapshots on the target SR are tagged with:
| Key | Purpose |
| -------------------- | -------------------------------------------------- |
| `xo:copy_of` | UUID of the source VDI this is a copy of |
| `xo:backup:job` | Job ID that created it |
| `xo:backup:vm` | Source VM UUID |
| `xo:backup:schedule` | Schedule ID |
| `xo:backup:sr` | Target SR UUID |
| `xo:base_delta_vdi` | UUID of the source VDI used as base for this delta |
## Step-by-step flow
### 1. `_selectBaseVm` (runner)
Finds the last successfully exported snapshot VDIs on the source. Builds `baseUuidToSrcVdiUuid`: a map of `snapshotVdiUuid -> currentSrcVdiUuid` for each disk. Then calls `checkBaseVdis` on each writer with a shared copy of this map (`presentBaseVdis`).
Each writer removes entries from `presentBaseVdis` for disks it cannot handle as delta. Since writers are called sequentially on the same map, the result is an **intersection**: only disks that are valid across all writers survive. If any single writer (e.g. a remote writer) lacks a base for a disk, that disk is removed for everyone, and the runner will export it as full.
### 2. `checkBaseVdis` (writer)
Searches the target SR for VDI snapshots that match the source snapshot by `COPY_OF`, scoped to the current `JOB_ID` and `VM_UUID`. For each candidate, it validates:
1. **Active VDI exists**: the snapshot's `$snapshot_of` must resolve to a live VDI.
2. **Single user VM**: the active VDI must be attached to exactly one non-control-domain VM (shared VDIs are skipped).
3. **VM is blocked**: the VM must have `start` in `blocked_operations` (confirms it's a managed replica).
4. **No data written since snapshot**: a diff between the active VDI and the snapshot must be empty (no blocks changed).
This produces two results stored as instance state:
- **`#baseVdisBySourceUuid`**: a `Map<sourceVdiUuid, activeVdi>` of validated disk links, reused later by `#decorateVmMetadata`.
- **`_targetVmRef`**: the ref of the replicated VM to update, set only if **all** disks can chain to the same VM (`canChainToTargetVm`).
Any source VDI UUID not present in `#baseVdisBySourceUuid` is removed from `baseUuidToSrcVdi`, signaling the runner that no valid base exists for that disk.
### 3. `#decorateVmMetadata` (writer, called during transfer)
Sets `vdi.baseVdi` for each disk in the export by looking up `vdi.other_config[BASE_DELTA_VDI]` in `#baseVdisBySourceUuid`. This directly reuses the validated mapping from `checkBaseVdis` instead of re-querying the SR.
**Important**: a disk may arrive as a full export (no `BASE_DELTA_VDI`) even though this writer's `checkBaseVdis` had found a valid candidate for it. This happens when another writer in the job could not handle that disk as delta, causing the runner to remove it from the shared map and export it as full for all writers. In that case `BASE_DELTA_VDI` is absent, `#baseVdisBySourceUuid` is not consulted, and the disk is transferred as full.
### 4. `_transfer` (writer)
Calls `importIncrementalVm` with:
- `targetRef = this._targetVmRef` (may be `undefined` if no valid target VM)
- Each VDI's `baseVdi` set by `#decorateVmMetadata` (may be `undefined` per-disk)
## Decision matrix
### Target VM selection
| All VDI diffs empty | VM blocked | `_targetVmRef` set | Result |
| ---------------------- | ---------- | ------------------ | -------------------------------------------------------- |
| Yes | Yes | Yes | **Update existing VM** (incremental into the same VM) |
| No (some disk changed) | - | No | **Create new VM** (but may still chain individual disks) |
| - | No | No | **Create new VM** (target VM was tampered with) |
| No candidates found | - | No | **Create new VM** (first replication or target deleted) |
### Per-disk transfer mode
| `baseVdi` set | `BASE_DELTA_VDI` present | Result |
| ------------- | ------------------------ | ---------------------------------------------------------------------------- |
| Yes | Yes | **Delta transfer**: only changed blocks since the base are sent |
| No | Yes | **Full transfer for this disk**: base was not found or not valid on target |
| - | No | **Full transfer**: first replication of this disk (no base exists on source) |
### Combined scenarios
#### Scenario 1: Normal incremental run
All disks have valid bases on the target, diffs are empty. `_targetVmRef` is set. Transfer is fully incremental into the existing VM.
#### Scenario 2: One disk was modified on the target
Disk A's diff is empty, disk B has blocks written. `canChainToTargetVm = false`, so a new VM is created. But disk A still has a valid entry in `#baseVdisBySourceUuid`, so its transfer is incremental (chained to the existing active VDI). Disk B falls back to full.
#### Scenario 3: New disk added to source VM
Existing disks have valid bases. The new disk has no `BASE_DELTA_VDI`. Existing disks transfer as delta; the new disk transfers as full. If the existing disks' VM is valid, `_targetVmRef` is set and the new disk is added to that VM.
#### Scenario 4: Target VM deleted or first run
No snapshot candidates found. `#baseVdisBySourceUuid` is empty. All disks are full. A new VM is created.
#### Scenario 5: Disk valid on one writer but not another
A job replicates to both SR-A (XAPI writer) and Remote-B (remote writer). Disk X has a valid base on SR-A but the VHD chain is broken on Remote-B. Remote-B removes disk X from the shared `presentBaseVdis` map. The runner exports disk X as full. SR-A's `#decorateVmMetadata` receives a full disk (no `BASE_DELTA_VDI`), so `baseVdi` is `undefined` despite SR-A having a valid candidate locally. Both writers get a full transfer for that disk.
#### Scenario 6: Multiple jobs replicate the same VM to the same SR
`checkBaseVdis` scopes candidates by `JOB_ID` + `VM_UUID`, so each job only sees its own VDI snapshots. `#decorateVmMetadata` reuses this scoped mapping, avoiding cross-job conflicts.
## After transfer
Once the transfer completes:
1. `other_config` is updated on the target VM with the final timestamp.
2. A snapshot of the target VM is taken (to serve as the base for the next delta).
3. `other_config` is reset on the target VM (the snapshot retains the metadata).
4. Old entries beyond `copyRetention` are destroyed during cleanup.

View File

@@ -18,6 +18,8 @@ function formatVmBackup(backup) {
}
}
return {
type: 'xo-vm-backup',
backupRepository: backup.backupRepositoryId,
disks:
backup.vhds === undefined
? []
@@ -38,17 +40,24 @@ function formatVmBackup(backup) {
size: backup.size,
timestamp: backup.timestamp,
vm: {
uuid: backup.vm.uuid,
name_description: backup.vm.name_description,
name_label: backup.vm.name_label,
tags: backup.vm.tags,
},
tags: backup.tags ?? [],
differencingVhds,
dynamicVhds,
withMemory,
}
}
// format all backups as returned by RemoteAdapter#listAllVmBackups()
export function formatVmBackups(backupsByVM) {
return mapValues(backupsByVM, backups => backups.map(formatVmBackup))
/**
* format all backups as returned by RemoteAdapter#listAllVmBackups()
* @param {Record<string, object[]>} backupsByVM
* @param {string} backupRepositoryId
* @returns {Record<string, object[]>}
*/
export function formatVmBackups(backupsByVM, backupRepositoryId) {
return mapValues(backupsByVM, backups => backups.map(backup => formatVmBackup({ ...backup, backupRepositoryId })))
}

View File

@@ -8,32 +8,34 @@
"type": "git",
"url": "https://github.com/vatesfr/xen-orchestra.git"
},
"version": "0.64.3",
"version": "0.72.0",
"engines": {
"node": ">=14.18"
},
"scripts": {
"postversion": "npm publish --access public",
"test": "node --test",
"test-integration": "node --test *.integ.mjs"
},
"dependencies": {
"@iarna/toml": "^2.2.5",
"@kldzj/stream-throttle": "^1.1.1",
"@vates/async-each": "^1.0.0",
"@vates/async-each": "^1.0.3",
"@vates/cached-dns.lookup": "^1.0.0",
"@vates/compose": "^2.1.0",
"@vates/decorate-with": "^2.1.0",
"@vates/disposable": "^0.1.6",
"@vates/fuse-vhd": "^2.1.2",
"@vates/generator-toolbox": "^1.0.4",
"@vates/nbd-client": "^3.2.1",
"@vates/generator-toolbox": "^1.1.2",
"@vates/nbd-client": "^3.4.0",
"@vates/parse-duration": "^0.1.1",
"@xen-orchestra/async-map": "^0.1.2",
"@xen-orchestra/disk-transform": "^1.1.0",
"@xen-orchestra/fs": "^4.6.3",
"@xen-orchestra/log": "^0.7.1",
"@xen-orchestra/qcow2": "^1.0.0",
"@xen-orchestra/template": "^0.1.0",
"@vates/task": "^0.7.0",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/disk-transform": "^1.2.3",
"@xen-orchestra/fs": "^4.8.0",
"@xen-orchestra/log": "^0.7.2",
"@xen-orchestra/qcow2": "^1.3.0",
"@xen-orchestra/template": "^0.1.1",
"app-conf": "^3.0.0",
"compare-versions": "^6.0.0",
"d3-time-format": "^4.1.0",
@@ -41,32 +43,36 @@
"golike-defer": "^0.5.1",
"human-format": "^1.2.0",
"limit-concurrency-decorator": "^0.6.0",
"lodash": "^4.17.20",
"lodash": "^4.18.0",
"moment-timezone": "^0.5.46",
"ms": "^2.1.3",
"node-zone": "^0.4.0",
"parse-pairs": "^2.0.0",
"promise-toolbox": "^0.21.0",
"proper-lockfile": "^4.1.2",
"tar": "^6.1.15",
"tar": "^7.5.3",
"uuid": "^9.0.0",
"value-matcher": "^0.2.0",
"vhd-lib": "^4.14.1",
"xen-api": "^4.7.4",
"vhd-lib": "^4.16.0",
"xen-api": "^4.7.7",
"yazl": "^2.5.1"
},
"devDependencies": {
"fs-extra": "^11.1.0",
"rimraf": "^5.0.1",
"rimraf": "^6.0.1",
"sinon": "^18.0.0",
"tmp": "^0.2.1"
},
"peerDependencies": {
"@xen-orchestra/xapi": "^8.4.1"
"@xen-orchestra/xapi": "^8.7.2"
},
"license": "AGPL-3.0-or-later",
"author": {
"name": "Vates SAS",
"url": "https://vates.fr"
},
"exports": {
"./disks": "./disks/index.mjs",
"./*": "./*"
}
}

View File

@@ -18,7 +18,7 @@
"preferGlobal": true,
"dependencies": {
"golike-defer": "^0.5.1",
"xen-api": "^4.7.4"
"xen-api": "^4.7.7"
},
"scripts": {
"postversion": "npm publish"

View File

@@ -1,7 +1,7 @@
{
"private": false,
"name": "@xen-orchestra/cron",
"version": "1.0.6",
"version": "1.0.7",
"license": "ISC",
"description": "Focused, well maintained, cron parser/scheduler",
"keywords": [
@@ -34,7 +34,7 @@
"node": ">=8.3"
},
"dependencies": {
"lodash": "^4.17.4",
"lodash": "^4.18.0",
"moment-timezone": "^0.5.14"
},
"scripts": {

View File

@@ -0,0 +1,79 @@
```
$ xo-disk-cli --help
Usage: xo-disk-cli <command> <handler-url> <path> [options]
Commands:
info Show disk info (virtual size, uid, parent uid, block size)
list List all disks at a path and display their properties in a table
transform Convert a disk to another format and write to stdout (raw | vhd | qcow2)
```
## Commands
### `info`
Display metadata for a single disk file.
```
xo-disk-cli info <handler-url> <disk-path>
```
Output includes: UID, parent UID (for differencing disks), virtual size, and block size.
```
$ xo-disk-cli info file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd
Disk info: /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd
UID: <uuid>
Parent UID: <parent-uuid>
Virtual size: 8.00 GiB (8589934592 bytes)
Block size: 2.00 MiB (2097152 bytes)
```
### `list`
List all disk files found at a directory path and display their properties in a table. Disks are sorted so that each child appears directly after its parent. When a disk's parent is the row immediately above, the parent UID column shows `↑` for quick chain health assessment.
```
xo-disk-cli list <handler-url> <dir-path>
```
```
$ xo-disk-cli list file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/
┌──────────────┬──────────────────────────────────────┬─────────────┬──────────────┬─────────────┬──────────────────────────────────────┐
│ File │ UID │ Size on disk│ Virtual size │ Differencing │ Parent UID │
├──────────────┼──────────────────────────────────────┼─────────────┼──────────────┼─────────────┼──────────────────────────────────────┤
│ base.vhd │ xxxxxxxx-... │ 1.20 GiB │ 8.00 GiB │ no │ (none) │
│ snapshot.vhd │ yyyyyyyy-... │ 128.00 MiB │ 8.00 GiB │ yes │ ↑ │
└──────────────┴──────────────────────────────────────┴─────────────┴──────────────┴─────────────┴──────────────────────────────────────┘
```
### `transform`
Convert a disk (or its full parent chain if differencing) to the specified format and write the result to stdout. Redirect to a file to save the output.
```
xo-disk-cli transform <handler-url> <disk-path> <format>
```
Supported formats:
- `raw` — flat binary image; absent blocks are written as zeros
- `vhd` — VHD fixed/dynamic image
- `qcow2` — QCOW2 image
```
$ xo-disk-cli transform file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd raw > disk.img
$ xo-disk-cli transform file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd qcow2 > disk.qcow2
$ xo-disk-cli transform file:///mnt/backups /xo-vm-backups/<vm-uuid>/vdis/<vdi-uuid>/snapshot.vhd vhd > disk.vhd
```
## Handler URLs
The `<handler-url>` argument identifies the remote storage backend. Examples:
| Backend | URL format |
| ---------------- | ----------------------- |
| Local filesystem | `file:///absolute/path` |
| NFS | `nfs://host/export` |
| SMB | `smb://host/share` |
| S3 | `s3://bucket/prefix` |

View File

@@ -0,0 +1 @@
../../scripts/npmignore

Some files were not shown because too many files have changed in this diff Show More