* refactor(mixins,proxy,xo-server): pass the XAPI names as one xapiLabels object
The SR name was a lone `nameLabel` threaded through xo-server, xo-proxy and
the mixin; the VDI names are about to join it. No change in behavior: the
proxy API still takes `nameLabel`.
Also export OC_MOUNT from _utils.mjs, the cache module will need it.
* feat(iscsi): FileBlockDevice can serve a block device node
With `size`, the path must be a block device (e.g. a VDI hot-plugged onto the
appliance, /dev/xvdX): its inode reports a size of 0, so the capacity comes
from the caller, and a short read is an I/O error instead of a sparse tail.
Also add unit tests for FileBlockDevice, which had none, and integration
tests against a loop device.
* feat(iscsi): advertise read-only LUNs as write protected
A read-only BlockDevice now sets the Write Protect bit of MODE SENSE, so the
guest mounts the disk read-only instead of failing on its first write. This is
the case of a live mounted disk, served by a read-only DiskBlockDevice.
* feat(live-mount): name the SR and VDI after the backed up VM and restore point
The SR is now `[XO live mount] <VM> (<backup date>)` and the VDI keeps the
name of the backed up disk, so a live mounted disk reads like the disk the
user knows. The VDI is also tagged with `xo:live-mount`, like the SR.
The proxy API takes them as `xapiLabels`; `nameLabel` is still accepted from
older XOs.
* feat(live-mount): optional local VDI cache, making the mount read/write
Without a cache, a live mounted disk is read-only and every read goes to the
backup repository. With `cacheSrId` (xo-server) / `cacheSr` (proxy API), the
disk is materialized block by block into a VDI of that SR, hot-plugged onto
the appliance serving the mount (XOA or the proxy):
- a block is read from the backup repository at most once;
- the LUN is read/write, writes land in the cache VDI and the backup is never
modified;
- the VDI is created with the mount and destroyed with it: what was written
is lost on unmount.
The appliance must be a VM of the target pool, and the SR plugged on the host
running it. On a local SR, the appliance must not be migrated while the mount
lasts, since the cache VDI cannot follow it: this is documented, not enforced.
The cache VDI is tagged `[NOBAK] [NOSNAP]` to stay out of the appliance's own
backups, and the SR of a cached mount is named `[XO live restore] …`. The
cache device is opened with retries, since its node may lag behind the
hot-plug.
The `backup-archive` SSE collection (#10472) only announced a change when a client listed the
repository: XO never read one on its own, so a subscriber learnt of a backup created, merged or
deleted at the next listing by any client, and after a restart heard nothing from a repository
until a client listed it.
XO now reads the repositories itself:
- every enabled repository is listed once XO has started, one after another, and a repository
enabled later as soon as `updateRemote()` has stored it;
- the changes this XO makes are pushed at once, for a repository whose journal can be replayed: at
the end of each VM export to it (worker, no-worker mode and proxy alike), at the end of the job
and after a deletion. A read in progress when `refresh()` is called reads again, since it may
have missed the change, at most 3 times and never when that means listing the repository in full;
- the other changes (the size the merge worker rewrites, another XO sharing the repository) are
read by two timers, 2 repositories at a time and through the listing backoff: every minute from
the journal, every 10 minutes by a full listing for a repository whose journal cannot be read,
e.g. behind a proxy which does not expose it;
- a repository is listed in full again an hour after its entry was built, rather than when it
crosses a UTC day, which bounds the drift from what the journal does not record. The
`fullListingInterval` parameter of its URL sets this delay, in whole minutes; the XO 6 and XO 5
repository forms now keep it when the repository is edited.
The archives are registered through `registerCollection` like any other collection, which drops
the REST API special case for them. Their events now carry an array of archives (`add`, `update`)
or of ids (`remove`); SSE subscribers still receive one event per archive.
* fix(@vates/iscsi): only replace the session once the newcomer has logged in
A new TCP connection used to tear down the active session as soon as it
was accepted, before sending anything. Any peer able to reach the port,
a port scan, a failed CHAP login or a Discovery session, interrupted the
disk in use.
New connections are now kept aside until they establish a Normal session
(CHAP included when configured), and only then replace the active one.
That keeps the ability to recover from a stale connection, which is why
connections are replaced rather than refused.
Connections without a session are bounded in time (`loginTimeoutMs`,
15 s by default, also closing lingering Discovery sessions) and in number
(`maxPendingConnections`, 8 by default, oldest dropped first).
* chore: add PR number to changelog
* fix(@vates/iscsi): refuse extra connections and guard the login race
Review follow-up:
- let net.Server refuse connections past maxPendingConnections (+1 for
the one serving the LUN) instead of evicting the oldest pending one
- a connection destroyed while its final Login Response is being written
must not go on and replace the current session
* feat(@vates/iscsi): bound what a peer can make the target do
- allowedAddresses: close connections from other initiators as soon as
they are accepted, so that they cannot take the pending connection
slots and block a legitimate login
- reject a PDU whose data segment exceeds 8 KiB during login, or the
declared MaxRecvDataSegmentLength afterwards, before reading it
- maxTransferLength (8 MiB by default): advertised in the block limits VPD
page, longer READ/WRITE refused, which also bounds the WRITE buffer
- BlockDevice.readOnly: report the LUN write-protected and refuse WRITE
with DATA PROTECT before soliciting any data; DiskBlockDevice sets it
* chore: add PR number to changelog
* feat(@vates/iscsi): rename allowedInitiatorAddresses, report refused connections
`allowedAddresses` did not tell whose addresses: these are the source
addresses of the initiators. `onConnectionRefused` reports each refused
one, e.g. so that a caller can tell that an initiator connects through a
NAT.
---------
Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
The CHAP interop test failed on CI with ENXIO on /dev/sdb (master 641f98878,
and #10549), right after waitForDevice() returned the disk.
waitForDevice() took a nonzero /sys/class/block/<name>/size as proof the disk
was ready, but the sd driver sets the size before device_add_disk(), which
creates the sysfs entry, then the /dev node, then registers the block device:
open() fails with ENOENT or ENXIO until then. Polling sysfs in a tight loop
after login hit that window in 15 of 100 logins.
isDeviceReady() now opens the device, and treats ENXIO and ENOENT as not ready
yet.
The cache now announces what a listing finds and what a journal replay
changes, as `add`, `update` and `remove` events carrying the archive and its
previous value, on an emitter the app exposes as `app.vmBackupArchives`. What
was last announced for a repository outlives its entry, so a repository read
again from scratch only announces what really changed. A repository which is
removed or disabled goes through the new `forgetVmBackupRepository()`, which
announces all its archives as removed. A listener which throws is logged and no
longer fails the listing.
The REST API adds `backup-archive` to the SSE collections. It is bound to
`app.vmBackupArchives` the way `task` is, and filtered by `backup-archive:read`
through the same path as every other non-XAPI collection.
Nothing polls the repositories: changes are pushed whenever any client lists a
repository. The first listing after a restart sends every archive it already
holds as `add`.
XO-2992
XO-2993
the live mount feature create a lot of SR , each with 1 disk
this PR introduce an automatic cleanup that will forget a SR as soon as the disk is removed, thus removing any user action needed to cleanup thins in the nominal case
crashing XO can leave SR pluggued
* feat(xo-server,rest-api,xo-web/rpu): force-close an incomplete rolling pool update record
A pool whose previous rolling pool update left a recovery record refuses a
new run until that record is dealt with. Once the operator has reviewed the
pool, `pool.finalizeRollingUpdate` and
`POST /rest/v0/pools/{id}/actions/finalize_rolling_update` close it.
Without `force`, the finalization is refused (`incorrectState`, property
`unrestoredItems`) while the record says the run changed something the pool
still shows changed: HA, auto power-on, WLB, load balancer, backup
schedules, hosts left disabled, running VMs away from their home host,
pinned VMs shut down and not started again. A record that cannot be read
is refused too, with an unknown list.
With `force`, the record is deleted, the abandoned items are written to the
task and the trace, nothing is restored nor changed in the pool.
To tell what the run changed from what the operator did, the run now
persists, before each change, the settings it touches (strict write) and
whether each host was enabled before the run.
XO 5 shows a "Finalize" action on the incomplete update banner, which is
now also shown for a run that succeeded but whose record could not be
removed. Forcing asks for a second confirmation listing the items.
implementation of an iscsi target and initiator in typescript
this will serve as a basis of 2 projects:
connecting directly to 3rd party san
exposing a disk from a backup archive to the host, effectively enabling direct mount to a running VM. Final target is to be able to boot completely the VM
@vates/iscsi/VALIDATION.md file is a journal of the additional test I run on my lab
@vates/iscsi/PROTOCOL.md describe the subset of the protocol implemented . For example we only support 1 LUN per target, not multiple connection , only one type of auth and keep alive
* feat(xo-server,rest-api,xo-web/rpu): safe start on a partially updated pool
A rolling pool update used to overwrite the recovery record left by the
previous run and to look only at the master to decide whether there was
anything to update. Both hid a pool left half updated: an interrupted run,
or updates applied by hand, could be silently completed or forgotten.
- `startRpuRecoveryRun` now refuses a new run while the pool still has a
record, whatever its status, with an `incorrectState` error (property
`rollingUpdateRecovery`). The RPU guard held by the caller makes the
check-then-write safe. A run which fails before any host was handled
drops its record instead of marking it failed: the orchestrator restores
what it changed on its way out, and a refused precondition (a guidance to
accept, a pinned VM to shut down) must not block the retry the operator
just consented to.
- The update refuses a pool whose master is already up to date while
another host is not (`incorrectState`, property `partiallyUpdatedPool`,
`actual` listing the outdated hosts) unless the new
`acceptCurrentStateAsBaseline` option is set. The option is exposed on
`pool.rollingUpdate` and on `POST /rest/v0/pools/{id}/actions/rolling_update`.
- XO5 enables the rolling pool update button as soon as any host of the
pool has missing patches, not only the master, disables it while a
recovery record exists, and asks for confirmation before accepting the
current state as the baseline.
Until the force-close operation ships, a leftover record can only be
removed by hand: it must land in the same release.
* feat(xo-server,rest-api,xo-web/rpu): changelog entry for #10394
* fix(xo-server/rpu): getRollingUpdateRecovery returns null on no record
* fix(xo-server/rpu): persist acceptCurrentStateAsBaseline, mark record succeeded on delete failure
* fix(xo-server/rpu): drop an empty record after the pool is restored, update LINSTOR only past the guards
* feat(xo-server,rest-api,xo-web/rpu): persist a recovery record across restarts
A rolling pool update now writes a `rpuRecovery` record (LevelDB, one per
pool) before its first side effect and keeps it updated during the run:
operator options, update variant, host order, initial VM placement, per-host
step statuses, last filtered error, and the pinned VMs shut down for their
host's reboot (previously only tracked in memory, so an xo-server crash left
them halted with nothing to restart them). A record left in a live status at
xo-server boot is flipped to `interrupted`; a successful run deletes its
record; an unreadable or unknown-version record is reported as `blocked`
without touching the stored value.
The status is exposed through `pool.getRollingUpdateRecovery` and
`GET /rest/v0/pools/{id}/rolling_update_recovery` (filtered view: never the
raw intent, reusing the existing pool permissions), and the pool Patches tab
in XO5 shows a warning with per-host progress, the last error and the pinned
VMs still halted.
First card of the RPU resume-after-failure spike: the record only reports
for now, resuming an interrupted run comes separately.
* docs(changelog): entry for the RPU recovery record (#10331)
* fix(xo-server,rest-api,xo-web/rpu): updates after review
* test(xo-server/rpu): cover the recovery record during the migrate back phase
The migrate back phase now retries the rejected migrations before giving
up, so a host is only recorded as failed once the retry passes are over,
and a pool which opted out of the phase must still close its `restoreVms`
steps, otherwise a finished run reads as still running in the record.
Both are asserted through a recorder spying on the step transitions.
* fix(xo-server/rpu): keep per-key reads in the recovery reconciliation
* test(xo-server/rpu): stub the recovery recorder in the rolling pool update tests
Since #10313, XenServers.rollingPoolUpdate runs the backup guard itself and
its tests drive it with a fake app. With the recovery record, the run also
calls app.startRpuRecoveryRun, which the fake app did not have.
The stub records its call, so the tests also check that a refusal from the
backup guard leaves no record behind.
* fix(xo-server/rpu): do not fail a successful run when its record cannot be deleted
This branch is a reliability overhaul of the ESXi/vSphere source side of V2V, plus a scope reduction of the package. The SOAP plumbing is rewritten:
VimClient wraps @vates/node-vsphere-soap in a promise API with one emitter per call (no more cross-talk between concurrent calls, no listener leak) and a pTimeout;
vim25 faults are now parsed (_parseFault.mjs) so callers get a real fault type instead of task execution failed;
the library no longer throws from async callbacks (which killed xo-server) and its beforeExit logout hook is per-client instead of removeAllListeners.
Request specs are built in explicit schema order (soap/specs.mjs) with xsi:type, and property values are normalized (soap/normalize.mjs) so search() and the new #retrieveProperty share one shape.
search() now paginates, cancels the retrieval and destroys the container view.
Datastore HTTP downloads get session-cookie reuse, retry with jittered backoff, a headers-only timeout, and range-honouring validation.
Task waiting polls with backoff and per-operation timeouts (6 h for snapshot consolidation).
nbdkit servers are memoized by a key that includes the export options, the readiness wait races the process death instead of sleeping 2 s
the password file is 0600 and unlinked
the thumbprint is read over TLS in-process instead of two openssl spawns.
The old VHD-over-datastore readers (VhdEsxi*, Datastore*, openDeltaVmdkAsVhd) are deleted, hence the majo, rleaving the vddk/NBD path with a COWD grain-directory parser as data-map fallback.
~1.7k lines of unit tests are added.
* fix(xo-server,rest-api): apply the backup guard to rolling pool operations started over REST
The JSON-RPC methods pool.rollingUpdate and pool.rollingReboot refused to run
while a backup job targets the pool, but the REST actions
POST /pools/:id/actions/rolling_update and rolling_reboot called the
orchestrators directly and skipped that check.
The guard now lives in XenServers.rollingPoolUpdate and Pools.rollingPoolReboot,
so every entry point (JSON-RPC, REST, scheduled jobs) applies the same rule.
The bypass moves into backupGuard(poolId, { bypassBackupCheck, operation }),
which logs a single warning with the operation and the user, and the other
callers (host.restart, host.restartAgent, host.stop, sr.reclaimSpace and their
REST counterparts) use it instead of their own if/else and log.
The REST rolling actions accept bypassBackupCheck in their body and record the
body in the task params, like the host actions do.
* fix(xo-server,rest-api): pass the targeted object ID to the backup guard
* fix(openmetrics): keep the Prometheus secret across xo-server restarts
The `secret` property of `configurationSchema` used a random `default`. That
expression is re-evaluated every time the module is loaded, and xo-server never
persists the values ajv fills in from schema defaults, so each restart handed
the metrics endpoint a brand new bearer token and Prometheus started getting
401s.
Drop the default and generate the secret in `load()` instead, saving it through
`xo.configurePlugin()` so it survives a restart.
Introduced by #9323
See https://xcp-ng.org/forum/topic/12415
* chore(changelog): add openmetrics secret entry
* fix(xo-server-openmetrics): generate a 256-bit Prometheus secret
Buffer.from(getRandomValues(new Uint32Array(8))) coerces each 32-bit
element to a single byte, so the generated secret held 64 bits of
entropy instead of 256. Use randomBytes(32) instead.
Introduced by fe29bd0d4
* fix(backups): timeout was not used correctly for NBD transfers
* fix(nbd-client): evict a dead client in multi instead
actual code was retrying 5 times, each time with a 60s timeout
multi also have its own retry on conneciton
since the client used is determnistic for a block, any stalled client
will be hammerred for a lot of time before giving up
this PR evict a nbd client that is failing to read a block (with its
retry logic)
* fix(nbd-client/multi): better spread read accross client
the actual code derivate the clientId from the block index
we have no information on the block distribution, only hopes that
it's random enough
this PR introduce a mechanism to use a round robin to select
the next client
* fix: correct fall back to full with /without nbd on qcow2
