feat(xen-api): reconnect to a surviving pool member when the master is unreachable (#10016)

* feat(xen-api): reconnect to a surviving pool member when the master is unreachable XO follows a pool master change only through HOST_IS_SLAVE redirection, which needs the contacted host to be up and answering. That covers a graceful master hand-over (the old master stays up as a slave and redirects us), but not the case where the master host itself becomes unreachable while another host takes over as master: HA automatic failover, or a manual emergency transition. The stored URL then points at a host that no longer responds, login fails with a transport error instead of HOST_IS_SLAVE, the existing retry never redirects, and the connection stays stuck on the dead master. Maintain a set of pool-member addresses (seeded from the configured target and optional opts.candidateHostnames, refreshed from host.get_all_records on every session open) and, when the current target is unreachable at the transport level, probe the other members in parallel. The first to answer wins; if it is a slave, its login redirects us to the current master through the existing HOST_IS_SLAVE retry. XapiError responses (including auth failures) are not failed over, so wrong credentials still fail fast. * feat(xo-server): persist pool-member addresses so master failover survives a restart The xen-api failover relies on an in-memory set of pool-member addresses, which is lost when xo-server restarts: it then only knows the configured host. That is exactly the address that is dead when XO runs on the very pool whose master just failed and was HA-restarted onto a survivor, leaving XO unable to reconnect. Persist the member addresses on the server record (Redis, JSON-encoded like the existing error field), refresh them whenever the pool object is seen via the xen-api candidateHostnames getter, and feed them back as opts.candidateHostnames when (re)building the connection. After a restart the connection can then probe the surviving members and reconnect to the new master. * chore(changelog): reconnect to a surviving pool member on master failure Add the user-facing changelog entry and list xen-api in the packages to release for the master-failover reconnect feature. * fix(xen-api): prune ejected hosts from pool-member failover candidates host.get_all_records is authoritative current membership, so replace the candidate set on refresh instead of unioning into it. An ejected host is typically still alive as its own standalone master with the same credentials, so leaving it in the set could make a failover probe silently connect to the wrong pool. The currently-connected target is preserved since host.address may differ from the address XO used to reach it. * fix(xen-api): harden surviving-member failover per review - reject failover probes on sessionId-based connections (no reusable credentials to open a session on another host) - bracket bare IPv6 addresses (host.address / HOST_IS_SLAVE) before using them as URL hostnames - log the probe session out even when the host answers after the probe timeout, to avoid leaking a session - abort the mid-session failover loop when the connection was torn down or replaced while waiting (e.g. user disconnected the server) - xo-server: drop persisted pool-member addresses when the connection identity (host/credentials) changes, so a stale slave list can't send a failover to the wrong pool See #10016
2026-11-09 17:06:13 -06:00 · 2026-07-15 11:04:16 +02:00
parent 913148ec2f
commit b078d3d42d
7 changed files with 488 additions and 4 deletions
--- a/CHANGELOG.unreleased.md
+++ b/CHANGELOG.unreleased.md
@@ -17,6 +17,7 @@
  - `/pools/:id/actions/management-reconfigure` (PR [#9891](https://github.com/vatesfr/xen-orchestra/pull/9891))
 - [REST API] Possibility of sending `autoEnable` in the body of the `/hosts/:id/actions/disable` endpoint (PR [#10040](https://github.com/vatesfr/xen-orchestra/pull/10040))
 - [REST API] `PATCH /rest/v0/vdis/{id}` to update a VDI's name, description and size (PR [#9945](https://github.com/vatesfr/xen-orchestra/pull/9945))
+- [Pool] XO now reconnects to a surviving pool member when the master becomes unreachable (e.g. HA promoted a new master after the old one died), instead of staying stuck on the dead master, including after an XO restart (PR [#10016](https://github.com/vatesfr/xen-orchestra/pull/10016))

 ### Bug fixes

@@ -45,6 +46,7 @@
 - @xen-orchestra/rest-api minor
 - @xen-orchestra/web patch
 - vhd-lib patch
+- xen-api minor
 - xo-server minor

 <!--packages-end-->
--- a/packages/xen-api/README.md
+++ b/packages/xen-api/README.md
@@ -53,6 +53,8 @@ Options:
 - `readOnly = false`: if true, no methods with side-effects can be called
 - `callTimeout`: number of milliseconds after which a call is considered failed (can also be a map of timeouts by methods)
 - `httpProxy`: URL of the HTTP/HTTPS proxy used to reach the host, can include credentials
+- `candidateHostnames`: addresses of other pool members to fall back on when the configured target becomes unreachable (e.g. the master died and another host took over). Refreshed automatically from the pool on each connection and exposed back through the `candidateHostnames` getter, so a caller can persist them and feed them back across restarts
+- `probeTimeout = 20e3`: number of milliseconds after which an unreachable member is given up on while probing for a survivor

 ```js
 // Force connection.
--- a/packages/xen-api/failover.spec.mjs
+++ b/packages/xen-api/failover.spec.mjs
@@ -0,0 +1,248 @@
+import t from 'tap'
+
+import { Xapi } from './index.mjs'
+import XapiError from './_XapiError.mjs'
+
+// Build a client without starting the event loop or opening any connection.
+// The failover methods under test only rely on `_sessionOpen`/`_probeHost`,
+// which the tests stub, so no XAPI is contacted.
+const MASTER = '192.168.1.1'
+const makeXapi = opts =>
+  new Xapi({
+    url: `https://user:password@${MASTER}`,
+    watchEvents: false,
+    ...opts,
+  })
+
+// A transport-level failure (no XAPI answered), as opposed to a XapiError.
+const connError = code => Object.assign(new Error(code), { code })
+
+t.test('candidateHostnames', t => {
+  t.test('is seeded with the configured target', t => {
+    t.same(makeXapi().candidateHostnames, [MASTER])
+    t.end()
+  })
+
+  t.test('includes the addresses passed through opts', t => {
+    const xapi = makeXapi({ candidateHostnames: ['192.168.1.2', '192.168.1.3'] })
+    t.same(new Set(xapi.candidateHostnames), new Set([MASTER, '192.168.1.2', '192.168.1.3']))
+    t.end()
+  })
+
+  t.test('_registerCandidateHostnames dedups and ignores non-string/empty values', t => {
+    const xapi = makeXapi()
+    xapi._registerCandidateHostnames(['192.168.1.2', MASTER, '', null, undefined, 42])
+    t.same(new Set(xapi.candidateHostnames), new Set([MASTER, '192.168.1.2']))
+    t.end()
+  })
+
+  t.test('_refreshCandidateHostnames prunes an ejected-but-alive host', t => {
+    // Seeded with a member that is then ejected from the pool. An ejected host
+    // typically stays powered on as its own standalone master with the same
+    // credentials, so it would answer a failover probe and silently connect us
+    // to the wrong pool — it must not linger in the candidate set once the live
+    // membership no longer lists it.
+    const ejected = '192.168.1.2'
+    const xapi = makeXapi({ candidateHostnames: [ejected] })
+    t.ok(xapi.candidateHostnames.includes(ejected), 'ejected host starts in the set')
+
+    // current membership reported by host.get_all_records after the eject
+    xapi._refreshCandidateHostnames([MASTER, '192.168.1.3'])
+
+    t.same(new Set(xapi.candidateHostnames), new Set([MASTER, '192.168.1.3']))
+    t.notOk(xapi.candidateHostnames.includes(ejected), 'ejected host was pruned')
+    t.end()
+  })
+
+  t.test('_refreshCandidateHostnames preserves the current target even when not reported', t => {
+    // `host.address` can differ from the address XO used to reach the host
+    // (FQDN, management IP, NAT…), so the connected target must survive a
+    // refresh that does not list it.
+    const xapi = makeXapi()
+    xapi._refreshCandidateHostnames(['192.168.1.3'])
+    t.same(new Set(xapi.candidateHostnames), new Set([MASTER, '192.168.1.3']))
+    t.end()
+  })
+
+  t.test('brackets bare IPv6 addresses so they are usable as URL hostnames', t => {
+    // `host.address` is a bare IPv6 (no brackets), but URL hostnames require
+    // brackets.
+    const xapi = makeXapi()
+    xapi._registerCandidateHostnames(['fe80::1', '192.168.1.3'])
+    t.ok(xapi.candidateHostnames.includes('[fe80::1]'), 'bare IPv6 is bracketed')
+    t.ok(xapi.candidateHostnames.includes('192.168.1.3'), 'IPv4 is left untouched')
+    t.end()
+  })
+
+  t.end()
+})
+
+t.test('_probeHost', t => {
+  // A resolved/rejected transport stub: `login` drives the login result,
+  // `onLogout` observes the fire-and-forget logout.
+  const stubTransport = (xapi, { login, onLogout = () => {} } = {}) => {
+    xapi._createTransport = ({ url }) => {
+      xapi._probedUrlHostname = url.hostname
+      return (method, args) => {
+        if (method === 'session.login_with_password') {
+          return login ?? Promise.resolve('probe-session')
+        }
+        if (method === 'session.logout') {
+          onLogout(args[0])
+          return Promise.resolve()
+        }
+      }
+    }
+  }
+
+  t.test('rejects when the connection is sessionId-based (no credentials to reuse)', async t => {
+    const xapi = makeXapi()
+    xapi._auth = { sessionId: 'opaque' } // no user/password
+    xapi._createTransport = () => t.fail('should not open a transport without credentials')
+    await t.rejects(xapi._probeHost('192.168.1.2'), /credentials/)
+  })
+
+  t.test('brackets a bare IPv6 candidate for the probe URL and return value', async t => {
+    const xapi = makeXapi()
+    stubTransport(xapi)
+    const target = await xapi._probeHost('fe80::1')
+    t.equal(xapi._probedUrlHostname, '[fe80::1]', 'IPv6 hostname is bracketed in the URL')
+    t.equal(target, '[fe80::1]', 'returns the bracketed hostname')
+  })
+
+  t.test('follows HOST_IS_SLAVE to the (bracketed) master', async t => {
+    const xapi = makeXapi()
+    stubTransport(xapi, {
+      login: Promise.reject(Object.assign(new Error('HOST_IS_SLAVE'), { code: 'HOST_IS_SLAVE', params: ['fe80::2'] })),
+    })
+    const target = await xapi._probeHost('192.168.1.2')
+    t.equal(target, '[fe80::2]', 'redirected to the bracketed master address')
+  })
+
+  t.test('logs the probe session out once the host answers', async t => {
+    const xapi = makeXapi()
+    let loggedOut
+    stubTransport(xapi, { onLogout: sessionId => (loggedOut = sessionId) })
+    await xapi._probeHost('192.168.1.2')
+    await new Promise(resolve => setImmediate(resolve)) // let the fire-and-forget logout run
+    t.equal(loggedOut, 'probe-session', 'logged the probe session out')
+  })
+
+  t.end()
+})
+
+t.test('_sessionOpenWithFailover', t => {
+  t.test('does not probe when the current target answers', async t => {
+    const xapi = makeXapi({ candidateHostnames: ['192.168.1.2'] })
+    let opens = 0
+    xapi._sessionOpen = async () => {
+      opens++
+    }
+    xapi._probeHost = async () => t.fail('should not probe when the target answers')
+
+    await xapi._sessionOpenWithFailover()
+    t.equal(opens, 1)
+    t.equal(xapi._url.hostname, MASTER, 'target is unchanged')
+  })
+
+  t.test('fails over to a surviving member when the target is unreachable', async t => {
+    const xapi = makeXapi({ candidateHostnames: ['192.168.1.2'] })
+    let opens = 0
+    xapi._sessionOpen = async () => {
+      // first open is against the dead master, the retry against the survivor
+      if (++opens === 1) {
+        throw connError('ECONNRESET')
+      }
+    }
+    xapi._probeHost = async hostname => {
+      if (hostname === '192.168.1.2') {
+        return hostname
+      }
+      throw connError('EHOSTUNREACH')
+    }
+
+    await xapi._sessionOpenWithFailover()
+    t.equal(opens, 2, 'reopened the session after failover')
+    t.equal(xapi._url.hostname, '192.168.1.2', 'switched to the survivor')
+  })
+
+  t.test('does not fail over on a XapiError (e.g. bad credentials)', async t => {
+    const xapi = makeXapi({ candidateHostnames: ['192.168.1.2'] })
+    xapi._sessionOpen = async () => {
+      throw new XapiError('SESSION_AUTHENTICATION_FAILED', [])
+    }
+    xapi._probeHost = async () => t.fail('should not probe on a XapiError')
+
+    await t.rejects(xapi._sessionOpenWithFailover(), XapiError)
+    t.equal(xapi._url.hostname, MASTER, 'target is unchanged')
+  })
+
+  t.test('rethrows the original error when every member is unreachable', async t => {
+    const xapi = makeXapi({ candidateHostnames: ['192.168.1.2'] })
+    const original = connError('ECONNRESET')
+    xapi._sessionOpen = async () => {
+      throw original
+    }
+    xapi._probeHost = async () => {
+      throw connError('EHOSTUNREACH')
+    }
+
+    await t.rejects(xapi._sessionOpenWithFailover(), original, 'surfaces the original error, not the probe failures')
+  })
+
+  t.end()
+})
+
+t.test('_failoverToSurvivor', t => {
+  t.test('probes every member and reopens against the one that answers', async t => {
+    const xapi = makeXapi({ candidateHostnames: ['192.168.1.2'] })
+    let opens = 0
+    xapi._sessionOpen = async () => {
+      opens++
+    }
+    xapi._probeHost = async hostname => {
+      if (hostname === '192.168.1.2') {
+        return hostname
+      }
+      throw connError('EHOSTUNREACH')
+    }
+
+    await xapi._failoverToSurvivor()
+    t.equal(xapi._url.hostname, '192.168.1.2', 'switched to the survivor')
+    t.equal(opens, 1, 'reopened the session')
+  })
+
+  t.test('keeps the current target when it is the one that recovers', async t => {
+    const xapi = makeXapi({ candidateHostnames: ['192.168.1.2'] })
+    let opens = 0
+    xapi._sessionOpen = async () => {
+      opens++
+    }
+    xapi._probeHost = async hostname => {
+      if (hostname === MASTER) {
+        return hostname
+      }
+      throw connError('EHOSTUNREACH')
+    }
+
+    await xapi._failoverToSurvivor()
+    t.equal(xapi._url.hostname, MASTER, 'target is unchanged')
+    t.equal(opens, 1, 'reopened the session')
+  })
+
+  t.test('throws without reopening when no member answers', async t => {
+    const xapi = makeXapi({ candidateHostnames: ['192.168.1.2'] })
+    let opens = 0
+    xapi._sessionOpen = async () => {
+      opens++
+    }
+    xapi._probeHost = async () => {
+      throw connError('EHOSTUNREACH')
+    }
+
+    await t.rejects(xapi._failoverToSurvivor())
+    t.equal(opens, 0, 'did not try to reopen a session')
+  })
+
+  t.end()
+})
--- a/packages/xen-api/index.mjs
+++ b/packages/xen-api/index.mjs
@@ -25,6 +25,7 @@ import makeCallSetting from './_makeCallSetting.mjs'
 import parseUrl from './_parseUrl.mjs'
 import Ref from './_Ref.mjs'
 import transports from './transports/index.mjs'
+import XapiError from './_XapiError.mjs'
 import { noSuchObject } from 'xo-common/api-errors.js'

 const { debug } = createLogger('xen-api')
@@ -56,6 +57,13 @@ function getPool() {
  return this.$xapi.pool
 }

+// Wrap a bare IPv6 address in brackets so it is usable as a URL hostname.
+// Addresses coming from XAPI (`host.address`, `HOST_IS_SLAVE`) are bare, while
+// `URL`/`_parseUrl` expect IPv6 to be bracketed.
+function formatHostname(hostname) {
+  return typeof hostname === 'string' && hostname.includes(':') && hostname[0] !== '[' ? `[${hostname}]` : hostname
+}
+
 // -------------------------------------------------------------------

 const CONNECTED = 'connected'
@@ -175,6 +183,19 @@ export class Xapi extends EventEmitter {
    }
    this._setUrl(url)

+    // Addresses of known pool members, used to fail over to a surviving host
+    // when the master becomes unreachable (e.g. HA promoted a new master after
+    // the old one died). Seeded with the configured target and any addresses
+    // passed by the caller (xo-server can persist these across restarts), then
+    // refreshed from `host.get_all_records` on every successful session open.
+    this._candidateHostnames = new Set([this._url.hostname])
+    if (Array.isArray(opts.candidateHostnames)) {
+      this._registerCandidateHostnames(opts.candidateHostnames)
+    }
+    // A dead host must fail the probe quickly rather than hang for the full
+    // call timeout (which defaults to an hour).
+    this._probeTimeout = opts.probeTimeout ?? 20e3
+
    this._connected = new Promise(resolve => {
      this._resolveConnected = resolve
    })
@@ -231,6 +252,14 @@ export class Xapi extends EventEmitter {
    return this._pool
  }

+  // Addresses of the known pool members (current target + every host seen on
+  // the pool). Callers can persist this and feed it back as
+  // `opts.candidateHostnames` so master failover still works after a restart,
+  // when the only otherwise-known address may be a dead master.
+  get candidateHostnames() {
+    return [...this._candidateHostnames]
+  }
+
  get sessionId() {
    assert.strictEqual(this._status, CONNECTED)
    return this._sessionId
@@ -257,7 +286,7 @@ export class Xapi extends EventEmitter {
    })

    try {
-      await this._sessionOpen()
+      await this._sessionOpenWithFailover()

      debug(this._humanId + ': connected')
      this._status = CONNECTED
@@ -919,7 +948,7 @@ export class Xapi extends EventEmitter {
    tries: 2,
    when: { code: 'HOST_IS_SLAVE' },
    onRetry: error => {
-      this._setUrl({ ...this._url, hostname: error.params[0] })
+      this._setUrl({ ...this._url, hostname: formatHostname(error.params[0]) })
    },
  }
  _sessionOpen = coalesceCalls(this._sessionOpen)
@@ -948,6 +977,15 @@ export class Xapi extends EventEmitter {
    const poolRef = Object.keys(pools)[0]
    this._pool = this._wrapRecord('pool', poolRef, pools[poolRef])

+    // Refresh the set of pool-member addresses so a later master failure can
+    // fail over to a survivor. Best-effort: a failure here must not break the
+    // (otherwise successful) session open.
+    await ignoreErrors.call(
+      this._call('host.get_all_records', [this._sessionId]).then(hosts =>
+        this._refreshCandidateHostnames(map(hosts, host => host.address))
+      )
+    )
+
    this.emit('sessionId', this._sessionId)

    // if the pool ref has changed, it means that the XAPI has been restarted or
@@ -968,6 +1006,119 @@ export class Xapi extends EventEmitter {
    }
  }

+  _registerCandidateHostnames(hostnames) {
+    for (const hostname of hostnames) {
+      if (typeof hostname === 'string' && hostname !== '') {
+        this._candidateHostnames.add(formatHostname(hostname))
+      }
+    }
+  }
+
+  // Replace the candidate set with the authoritative current pool membership
+  // (from `host.get_all_records`) rather than union into it. Replacing prunes
+  // ejected hosts, which matters because an ejected host is typically still
+  // alive as its own standalone master and would answer a failover probe with
+  // the same credentials, silently connecting us to the wrong pool. The
+  // currently-connected target is preserved because `host.address` (the
+  // pool-network address XAPI reports) is not guaranteed to be the address XO
+  // actually used to reach the host (FQDN, management IP, NAT…).
+  _refreshCandidateHostnames(hostnames) {
+    this._candidateHostnames = new Set([this._url.hostname])
+    this._registerCandidateHostnames(hostnames)
+  }
+
+  // Probe a single candidate address. Resolves with the hostname to connect to
+  // (the candidate itself when it answers, or the master it points to when it
+  // is a slave); rejects when the host cannot be reached or refuses the login.
+  async _probeHost(hostname) {
+    const { user, password } = this._auth
+    // A probe must open a brand new session on another host, which requires
+    // credentials. A sessionId-based connection has no reusable credentials (a
+    // sessionId is bound to the host that issued it), so there is nothing to
+    // fail over with.
+    if (user === undefined || password === undefined) {
+      throw new Error('cannot fail over to another pool member without user/password credentials')
+    }
+
+    const url = { ...this._url, hostname: formatHostname(hostname) }
+    const transport = this._createTransport({ dispatcher: this._undiciDispatcher, url })
+    try {
+      const login = transport('session.login_with_password', [user, password])
+      // Always log out once the host answers, even if that happens after the
+      // probe already timed out: otherwise a slow-but-alive host leaks a
+      // session. Chained to `login` (not to the timed-out result) so a late
+      // success is still cleaned up.
+      login.then(sessionId => ignoreErrors.call(transport('session.logout', [sessionId])), noop)
+      await pTimeout.call(login, this._probeTimeout)
+      // Reachable and willing to log us in: this host is (or has become) the
+      // master, or a standalone host.
+      return url.hostname
+    } catch (error) {
+      // A slave answers too: the pool is alive and it tells us where the
+      // current master is. That is a successful probe.
+      if (error?.code === 'HOST_IS_SLAVE') {
+        return formatHostname(error.params[0])
+      }
+      throw error
+    }
+  }
+
+  // Open a session, failing over to a surviving pool member when the current
+  // target is unreachable at the transport level (typically: the master died
+  // and HA promoted a new one, so the stored URL no longer answers).
+  async _sessionOpenWithFailover() {
+    try {
+      return await this._sessionOpen()
+    } catch (error) {
+      // Only fail over on transport/connection failures. A XapiError means a
+      // XAPI did answer: HOST_IS_SLAVE is already followed inside _sessionOpen,
+      // and an auth failure (SESSION_AUTHENTICATION_FAILED) must fail fast
+      // rather than be retried against every member.
+      if (error instanceof XapiError) {
+        throw error
+      }
+
+      // The current target just failed, so only probe the other members; if
+      // there are none there is nothing to fail over to.
+      const candidates = [...this._candidateHostnames].filter(hostname => hostname !== this._url.hostname)
+      if (candidates.length === 0) {
+        throw error
+      }
+
+      console.warn(`${this._humanId}: target unreachable, trying ${candidates.length} other pool member(s)`, {
+        candidates,
+        error,
+      })
+
+      // On total failure, surface the original connection error rather than the
+      // aggregate of probe failures.
+      return this._failoverToSurvivor(candidates, error)
+    }
+  }
+
+  // Probe pool members in parallel and reopen the session against whichever
+  // answers first; a slave redirects us to the current master via the
+  // HOST_IS_SLAVE handling in `_probeHost`. Each probe is bounded by
+  // `_probeTimeout`, so dead members fail fast while a live survivor wins the
+  // race immediately. By default every known member is probed (including the
+  // current target, in case the failure was a transient toolstack restart
+  // rather than a dead host), which is what the event loop uses to recover
+  // mid-session. Throws `fallbackError` when provided, otherwise the aggregate
+  // probe failure, when no member answers.
+  async _failoverToSurvivor(candidates = [...this._candidateHostnames], fallbackError) {
+    let target
+    try {
+      target = await Promise.any(candidates.map(hostname => this._probeHost(hostname)))
+    } catch (error) {
+      throw fallbackError ?? error
+    }
+    if (target !== this._url.hostname) {
+      console.warn(`${this._humanId}: failing over to ${target}`)
+      this._setUrl({ ...this._url, hostname: target })
+    }
+    return this._sessionOpen()
+  }
+
  async _getHostBackupAddress(host) {
    if (host === undefined) {
      host = await this.getRecord('host', this._pool.master)
@@ -1267,6 +1418,40 @@ export class Xapi extends EventEmitter {
          this.emit('eventFetchingError', error)
          this._watchEventsError = error
          console.warn('_watchEvents', error)
+
+          // A connection-level failure (no XAPI answered, e.g. ECONNRESET)
+          // means the current target became unreachable mid-session, typically
+          // the master died and HA promoted a survivor. A XapiError, by
+          // contrast, means a XAPI did answer, so it is not a reachability
+          // problem and must not trigger a failover. Probe the known pool
+          // members, reconnect to whichever answers, then restart the event
+          // loop from the top (re-injecting and refetching against the new
+          // master). Retry indefinitely until a survivor appears.
+          if (!(error instanceof XapiError)) {
+            while (true) {
+              // If the connection was torn down (the user disconnected the
+              // server) or already replaced (reconnected through another path)
+              // while we waited, stop failing over and let the main loop
+              // re-evaluate the current state instead of forcing a reconnection
+              // the user may no longer want.
+              const sessionIdAtFailure = this._sessionId
+              await this._connected
+              if (this._sessionId !== sessionIdAtFailure) {
+                // eslint-disable-next-line no-labels
+                continue mainLoop
+              }
+              try {
+                await this._failoverToSurvivor()
+                // eslint-disable-next-line no-labels
+                continue mainLoop
+              } catch (failoverError) {
+                this._watchEventsError = failoverError
+                console.warn('_watchEvents: no surviving pool member reachable yet, retrying', failoverError)
+                await pDelay(this._eventPollDelay)
+              }
+            }
+          }
+
          await pDelay(this._eventPollDelay)
          continue
        }
--- a/packages/xen-api/package.json
+++ b/packages/xen-api/package.json
@@ -73,6 +73,6 @@
  "scripts": {
    "plot": "gnuplot -p memory-test.gnu",
    "postversion": "npm publish",
-    "test": "tap"
+    "test": "tap --allow-incomplete-coverage"
  }
 }
--- a/packages/xo-server/src/models/server.mjs
+++ b/packages/xo-server/src/models/server.mjs
@@ -13,6 +13,10 @@ export class Servers extends Collection {
    const { error } = server
    server.error = error != null ? JSON.stringify(serializeError(error)) : undefined
    server.readOnly = server.readOnly ? 'true' : undefined
+    server.poolMembersAddresses =
+      Array.isArray(server.poolMembersAddresses) && server.poolMembersAddresses.length > 0
+        ? JSON.stringify(server.poolMembersAddresses)
+        : undefined
  }

  _unserialize(server) {
@@ -25,6 +29,12 @@ export class Servers extends Collection {
    }
    server.readOnly = server.readOnly === 'true'

+    if (server.poolMembersAddresses) {
+      server.poolMembersAddresses = parseProp('server', server, 'poolMembersAddresses', [])
+    } else {
+      delete server.poolMembersAddresses
+    }
+
    // see https://github.com/vatesfr/xen-orchestra/issues/6656
    if (server.httpProxy === '') {
      delete server.httpProxy
--- a/packages/xo-server/src/xo-mixins/xen-servers.mjs
+++ b/packages/xo-server/src/xo-mixins/xen-servers.mjs
@@ -152,6 +152,12 @@ export default class XenServers {

    let hasChanged = false

+    // Changing any of these invalidates the persisted pool-member addresses:
+    // they may point at a different pool now, so a stale slave list must not
+    // survive the change (a live connection repopulates it within seconds).
+    const CONNECTION_IDENTITY_KEYS = new Set(['allowUnauthorized', 'host', 'httpProxy', 'password', 'username'])
+    let connectionIdentityChanged = false
+
    for (const key of [
      'allowUnauthorized',
      'enabled',
@@ -173,10 +179,24 @@ export default class XenServers {
        if (value !== server[key]) {
          server[key] = value
          hasChanged = true
+          if (CONNECTION_IDENTITY_KEYS.has(key)) {
+            connectionIdentityChanged = true
+          }
        }
      }
    }

+    // Drop the stale pool-member addresses on a connection-identity change,
+    // unless this very update is the internal refresh that carries a fresh list.
+    if (
+      connectionIdentityChanged &&
+      properties.poolMembersAddresses === undefined &&
+      server.poolMembersAddresses !== undefined
+    ) {
+      server.poolMembersAddresses = undefined
+      hasChanged = true
+    }
+
    // special handling for readOnly
    const { readOnly } = properties
    if (readOnly !== undefined && readOnly !== server.readOnly) {
@@ -187,6 +207,16 @@ export default class XenServers {
      hasChanged = true
    }

+    // special handling for poolMembersAddresses (an array, compared by content)
+    const { poolMembersAddresses } = properties
+    if (
+      poolMembersAddresses !== undefined &&
+      JSON.stringify(poolMembersAddresses) !== JSON.stringify(server.poolMembersAddresses)
+    ) {
+      server.poolMembersAddresses = poolMembersAddresses.length > 0 ? poolMembersAddresses : undefined
+      hasChanged = true
+    }
+
    if (hasChanged) {
      await this._servers.update(server)
    }
@@ -232,12 +262,14 @@ export default class XenServers {
        serverIdsByPool[xapiObject.$id] = conId
      }

-      // save pool name and description in server properties
+      // save pool name and description, and the pool-member addresses (so
+      // master failover survives an XO restart), in the server properties
      if (xapiObject.$type === 'pool') {
        self
          .updateXenServer(serverIdsByPool[xapiId], {
            poolNameDescription: xapiObject.name_description,
            poolNameLabel: xapiObject.name_label,
+            poolMembersAddresses: self._xapis[conId]?.candidateHostnames,
          })
          ::ignoreErrors()
      }
@@ -354,6 +386,11 @@ export default class XenServers {
        user: server.username,
        password: server.password,
      },
+      // Persisted pool-member addresses so that, after an XO restart, the
+      // connection can still fail over to a surviving master even when the
+      // configured `host` is the dead one (e.g. XO is HA-restarted on the very
+      // pool whose master just died).
+      candidateHostnames: server.poolMembersAddresses,
      url: server.host,
      watchEvents: false,
    }))