Compare commits

..

479 Commits

Author SHA1 Message Date
Olivier Lambert
857b71078f fix(web): Firewall: the unenforced hosts' alert says what is true, where nothing else says it
The alert said its VMs were started or moved there outside XO and that
their traffic is not filtered. The usual case is VMs already running
when the agent stopped: the flows it installed stay. And it repeated
what the pool overview (agent gaps, Exposed list) and the VM's Firewall
tab (Exposed) already said, three alerts for one event.

- Where it shows: the host's dashboard; the Firewall pages, XO-wide and
  of a pool, but their overviews, which list exposed VMs already (a
  pool's overview keeps it when the pool has no policy, where it has no
  Exposed list: a quarantine may still go unenforced). No longer on the
  VM's Firewall tab, whose status says Exposed with its host.
- What it says, per host, as the existing Exposed texts: no agent,
  nothing is filtered for the VMs there; an agent not reporting (stopped
  or stuck), they keep the flows it last applied, but a policy published
  since, or a VM started or arrived since, isn't enforced; with its last
  report. Nothing on how they got there. The title counts VMs, singular
  for one.
2026-11-13 23:10:32 +00:00
Olivier Lambert
a7e3b66ba3 fix(web): Firewall: a VM out of the firewall by choice is never listed as exposed
The pool overview's exposed VMs counted a protected or quarantined VM
whose interfaces are all on vdf:unprotected networks (or that has none)
when its host has no working agent, though nothing would filter it on
any host. Its donut already left such VMs out (outside), and so does the
plugin's GET /unenforced (vmProtections).

isOutsideFirewall, the donut's test, now decides both, so the overview's
"Exposed" list and the unenforced hosts' alert name the same VMs.
2026-11-13 23:08:21 +00:00
Olivier Lambert
3a304358bb fix(xo-server-vdf): GET /unenforced lists running VMs, as XO 6 counts exposed ones
It listed paused VMs too, which XO 6's exposed VMs (pool overview, VM
tab) leave out: a paused VM exchanges nothing. The alert and the
"Exposed" notices now count the same VMs.
2026-11-13 23:07:26 +00:00
Olivier Lambert
e6f4d77a80 fix(xo-server-vdf): GET /unenforced's description says what it lists
Its OpenAPI description still counted an agent rejecting the published
policy among those that can't enforce (it keeps enforcing its last valid
one, and isn't listed since), and said the VMs got there outside XO or
with bypassFirewallCheck, while the usual case is VMs already running
when the agent stopped. It now gives the states as they are: missing
(nothing filtered there), stopped or stale (the flows it installed stay,
what came since isn't enforced), and that pools without VDF are left
out.
2026-11-13 23:07:08 +00:00
Olivier Lambert
aa6e86b326 fix(xo-server): a guard's refusal stands when deciding who may override it fails
When a guard's check refused and its mayBypass then threw, core went on
with the next guard: a placement the guard knows to be wrong went ahead,
even with no override asked.

Now the failure is still logged as a warning (its message says the
refusal stands), the override is taken as not allowed, and core goes on
as for a user the guard doesn't allow: placementRefused without
bypassFirewallCheck, forbiddenOperation with it. Only a failing check
lets the operation go on.
2026-11-13 22:45:52 +00:00
Olivier Lambert
033fd914be fix(xo-server-vdf): a pool without VDF refuses nothing to its own VMs
A pool where no host's agent ever wrote a status and nothing is
published has no VDF: nothing was ever filtered there, yet the
placement guard refused to start a vdf:quarantine VM in it (no agent on
any host), and GET /unenforced listed its hosts.

managesVdf reads a pool as XO 6's useVdfAvailability does (a host
carries vdf:status, or a vdf:policy key is on the pool). In a pool
without VDF the guard now refuses nothing to a start, a resume or a
migration within the pool, quarantine included, and the alert leaves
the pool out. A VM migrating into it from another pool is decided as
before: quarantined, refused; protected only by a group, allowed, as no
policy applies to it there (the vdf: tag guard decides about its group
tags).
2026-11-13 22:40:26 +00:00
Olivier Lambert
b9c63fd57d fix(xo-server-vdf): the placement guard reads a malformed document without throwing
Since core lets the operation go on when a guard throws, every throw on
the guard's path is a protection lost: none must come from an ordinary
case. vmProtections read a published document as well-formed: one that
parses but whose policies, or a group's tags or networks, aren't lists,
or holds a null policy (published outside XO, read as it is) threw.
Such parts now count as none, as agents read them; a quarantine still
counts.

The other ordinary cases already threw nothing, now tested through the
guard and GET /unenforced: a VM without VIFs (not protected), a VIF
whose network XO lost (an untagged network: the quarantine holds), a
pool with nothing published, a global configuration without document.
VM.get_possible_hosts failing already falls back to every enabled host.
2026-11-13 22:39:30 +00:00
Olivier Lambert
051532710a fix(xo-server): a placement guard that fails doesn't stop the operation
checkPlacement awaited each guard's check and mayBypass as they were: a
guard that threw (a plugin bug, a XAPI call failing, a pool
unreachable) failed vm.start, vm.resume, vm.recoveryStart and
vm.migrate, and their REST actions, with its own error. A firewall
plugin bug would stop people from starting VMs.

A guard that throws, in check or in mayBypass, or answers a refusal
without a reason, now refuses nothing: core logs a warning with the
guard's name, the VM, the host and the error, and goes on with the next
guard, then the operation. A VM whose pool XO doesn't know asks no
guard (logged too). A real refusal is unchanged, whatever another guard
does.
2026-11-13 22:38:36 +00:00
Olivier Lambert
d91215f40e fix(xo-server-vdf): an agent that rejected the published policy still enforces
An agent that rejects a publish keeps filtering with its last valid
policy: a forced publish (vdf#2) left some hosts rejecting it, and the
placement guard then refused every protected VM's start or migration
there, and the alert listed them, while XO 6 (hostEnforces) shows their
VMs as protected, outdated.

hostEnforcement now reads hosts as XO 6's hostState and hostEnforces
do: rejected enforces, whatever document it rejected; only missing,
stopped and stale agents don't. Neither the guard nor GET /unenforced
count it any more; the web's type of the alert loses the state, and
each definition names the other.
2026-11-13 22:37:52 +00:00
Olivier Lambert
9b5e3ed088 feat(web): a VM start or migration the firewall refuses says why, and who may goes on anyway
xo-server refuses to start, resume or migrate a VM the firewall protects
onto a host whose firewall agent can't enforce (vates/vdf#141 follow-up):
403, placementRefused's data with the reason and whether the user may
override it. XO 6's start, start on, resume and migrate showed nothing
of it (the start jobs only log their failures).

Now, refused by the firewall (useVdfPlacementOverride):
- a user allowed to override it (XO administrators, the firewall publish
  privilege on the pool) is asked "Start anyway, without firewall"
  (resume, migrate), with the reason; confirmed, the action is sent
  again with bypassFirewallCheck, which xo-server logs;
- the others get the reason and who may go on.
One question at a time when several VMs are started at once; any other
error is left as it was.
2026-11-13 22:31:50 +00:00
Olivier Lambert
03ab632b6e feat(web): Firewall: hosts that can't enforce while protected VMs run there
XO refuses to start or move a protected VM onto a host whose firewall
agent can't enforce, but xe, another XO, HA or an evacuation can, and
the VM then runs unfiltered (vates/vdf#141 follow-up). The plugin lists
those hosts (GET plugins/vdf/unenforced, polled every 30 s): an alert
says so, with each host's reason and its VMs,

- on every Firewall page, XO-wide (every pool) and of a pool;
- on the host's dashboard, with the VMs it runs that way (XO 6 has no
  host Firewall tab);
- on the VM's Firewall tab, naming its host.

For users who may read the pool's firewall (useVdfFirewallRights; the
plugin lists only their pools), while the plugin is loaded.
2026-11-13 22:28:33 +00:00
Olivier Lambert
24951270ef feat(xo-server-vdf): a protected VM doesn't start on, or move to, a host that can't enforce
A VM the firewall protects (vdf:quarantine, or an enforced policy of
the destination pool applied to one of its interfaces: the published
document and this XO's global configuration where it applies) runs
unfiltered on a host whose agent is missing or not working.

On load, the plugin registers its placement guard with xo-server, and
removes it on unload. Before a start, resume or migration through XO:
- a host named (start on, migration within or across pools): refused
  unless its agent enforces (hostEnforcement), the reason naming it;
- no host (XAPI picks one): refused if any enabled, running host XAPI
  could start it on (VM.get_possible_hosts, a read; every enabled
  running host of the pool if XAPI can't tell) can't enforce, naming
  them and those that can;
- a migration to another pool decides on the networks its mapping
  gives the VIFs there (vdf:unprotected ones left out), unknown
  networks counting as untagged.
What XO can't read (a pool it isn't connected to, a halted host XAPI
refuses anyway) is let through. The override (bypassFirewallCheck) is
for XO administrators and users with the firewall publish privilege on
the pool.

Where XO can't prevent it (xe, another XO, HA, an evacuation), it says
so: GET /plugins/vdf/unenforced lists, per pool the user may read the
firewall of, the running hosts that can't enforce while protected VMs
run there (paused ones too), with why and those VMs.
2026-11-13 22:24:54 +00:00
Olivier Lambert
113e16e748 feat(xo-server-vdf): whether a host can enforce, and whether a VM is protected
The two decisions the placement guard and its alert share (vdf#141
follow-up), next to the agent status reading they build on:

hostEnforcement(host, status, publishedDocId): a host's agent enforces
when it reported recently (isFresh: twice its heartbeat period, no stop
status, a minute of clock skew, the thresholds of vdf-agent status and
XO 6) and enforces the document published in its pool or is converging
on it (behind: not applied yet; error: applied, not everywhere). It
doesn't when it has no status (missing), stopped, is stale, or rejected
the document published now; the reason names the host and, when known,
since when.

vmProtections({ tags, ifaces, documents }): vdf:quarantine, and the
enforced policies (not monitored) applied to a group selecting one of
the VM's interfaces, in the pool's documents (published, global
configuration). Nothing when every interface is on a vdf:unprotected
network, or there is none.
2026-11-13 22:19:32 +00:00
Olivier Lambert
4d3b3ac1e5 feat(rest-api): VM start, resume and migrate go through the placement guards
POST /vms/{id}/actions/start, resume and migrate call xo-server's
checkPlacement before creating their task, so that a refusal is an HTTP
error with its reason rather than a failed task: placementRefused
answers 403, its data naming the VM, the host, the reason and the
override, as forbiddenOperation does. The migration passes its VIF
mapping.

Each takes bypassFirewallCheck in its body (resume now takes a JSON
body for it): the override, for the users the refusing guard allows.
2026-11-13 22:18:16 +00:00
Olivier Lambert
46b9a3e253 feat(types): XoApp's checkPlacement and registerPlacementGuard 2026-11-13 22:16:50 +00:00
Olivier Lambert
71b20b57bb feat(xo-server): placement guards: a plugin may refuse the host a VM starts on or moves to
A VM the firewall protects can be started on, or migrated to, a host
whose agent is missing or not working: it runs there unfiltered, and
nothing says so (vates/vdf#141 follow-up).

Core gets a small, generic registration point, as for the reserved tags:
registerPlacementGuard({ name, bypassRight, check, mayBypass }) returns
its removal, and checkPlacement({ operation, user, vm, host, bypass,
vifNetworks }) asks every guard before an operation that puts the VM
onto a host: start (halted or suspended VM), resume (suspended), migrate
(running or paused, within the pool or to another one). `host` is
undefined when XAPI picks one: the guard decides about the hosts of the
pool the VM could land on. The first refusal is a placementRefused error
naming the VM, the host, the reason and the override.

The override is its own boolean, bypassFirewallCheck, not the `force`
of start and migrate: who may use it is the refusing guard's decision
(mayBypass), anyone else gets forbiddenOperation; a bypass is logged as
a warning with the user, the VM, the host and the reason. Without a
guard registered, nothing changes.

JSON-RPC vm.start, vm.resume, vm.recoveryStart and vm.migrate (with its
VIF mapping) call it before XAPI is asked anything.
2026-11-13 22:16:35 +00:00
Olivier Lambert
6601ea2523 feat(xo-common): placementRefused API error
Code 29: a placement guard refused to start, resume or migrate a VM onto
a host. Its data names the operation, the VM, the host (none when XAPI
picks one), the guard, its reason, and the override: the parameter that
overrides it, whether the refused user may set it, and who may. The
message says all of it, the override included.
2026-11-13 22:16:35 +00:00
Olivier Lambert
5966a5e2e9 fix(xo-server-vdf): GET /global for other users while XO has lost a pool
A pool XO lost (its server kept, its objects gone) is no XO object any
more, and XO core throws no such object when asked for an ACL or a
privilege on it: GET /global answered 404 to every non-administrator,
legacy ACLs and RBAC privileges alike, as long as XO had lost a pool. A
lost pool is now shown to administrators only.
2026-11-13 17:29:11 +00:00
Olivier Lambert
43cfb152da feat(web): quarantine and group membership controls follow their firewall rights
Quarantining a VM takes `quarantine:add`, lifting its quarantine
`quarantine:remove`, wherever it's offered (the VM's menu, its Firewall tab,
the rule log and Live rows): useVdfQuarantine now takes the VMs' pool and
disables the action with the missing right as its hint or tooltip. The
quarantine-lifted alert's Acknowledge, which the plugin takes as lifting
the quarantine, decides itself from `quarantine:remove` instead of the
pool's publish right.

Joining or leaving a group takes `membership`: the VM Firewall tab's "Add
to group" and group chips, the groups board's "Add VMs" and the group
panel's add and remove. A Firewall operator quarantines and releases VMs
but sees group membership locked.
2026-11-13 10:11:52 +00:00
Olivier Lambert
8aa6f270ab feat(web): publishing and network tags follow the firewall publish right
Publish (the draft bar's "Review and publish", its read-only notice) asks
the rights helper for `publish` instead of reading canEdit. Network tags
take `publish` too (xo-server's rule): the network panel's "Leave this
network out of the firewall" switch is disabled with the reason, and the
group drawer's networks (scope, networks, network tag) are locked in a pool
where the user may not publish. The network panel's firewall card shows
with `read`.

The read-only notice names the right instead of the pool's administrators.
2026-11-13 10:11:43 +00:00
Olivier Lambert
0e38759347 feat(web): the firewall pages and tabs take the firewall read right
A user who may not read a pool's firewall (the policy route refuses them)
doesn't get the pool's Firewall tab, the VM's Firewall tab, nor the
firewall's statuses on the traffic rules pages; a Firewall page opened
anyway says which right it takes. The availability (plugin loaded, agent
on the pool) is decided as before, the right by the rights helper.
2026-11-13 10:11:43 +00:00
Olivier Lambert
9b54785f30 feat(web): VM tag editor: each firewall tag follows its firewall right
A `vdf:` tag is locked per tag and per direction, from the user's firewall
rights on the VM's pool: a group tag without `membership`, adding
vdf:quarantine without `quarantine:add`, removing it without
`quarantine:remove`. The lock's tooltip names the missing right; the
suggestions offer only the tags the user may add. A Firewall operator
quarantines a VM and lifts its quarantine from the tag editor, and sees
group tags locked.
2026-11-13 10:02:32 +00:00
Olivier Lambert
01af55b2a2 feat(web): one place for the current user's firewall rights on a pool
useVdfFirewallRights(pool) says what the current user may do with a pool's
firewall, per action of the RBAC `firewall` resource (read, publish,
membership, quarantine:add, quarantine:remove), from the plugin's GET
policy: its `rights`, which xo-server decides (XO administrators have all
of them); none when the route refuses the user (403); from `canEdit` with
an older plugin. While they aren't known, nothing is locked and the server
decides. A control asks can(action), and shows lockReason(action), which
names the missing right in plain words.

firewallActionsForTag mirrors xo-server's: which rights adding or removing
a `vdf:` tag takes.
2026-11-13 10:02:24 +00:00
Olivier Lambert
477fb00f6d feat(web-core): VtsTagSelect: a tag can be locked one way only
`tagLock` is given the change it would be: adding the tag when the item
doesn't have it, removing it when it does. A tag that may be added but not
removed (or the other way round) is locked only that way. Select all no
longer selects a tag that can't be added.
2026-11-13 09:58:29 +00:00
Olivier Lambert
a560e0743d feat(xo-server): the Administrator template role has the firewall privileges
Read only reads every resource, so it reads every pool's firewall
(`firewall read`); Administrator, every action of Read only's resources,
gets every `firewall` action.

Template roles are brought up to date at each start; the roles copied from
them are not, and keep the privileges they were copied with.
2026-11-13 09:58:14 +00:00
Olivier Lambert
bb99af5d6b feat(web-core): remote resources keep the HTTP status of a failed fetch
A fetch answered with an error status sets `lastError` to a
RemoteResourceFetchError carrying the response's `status`, so that a caller
can tell a refusal (403) from a server or network failure. Its message is
unchanged.
2026-11-13 09:55:59 +00:00
Olivier Lambert
01385e27c4 fix(xo-server): vdf: tags on networks take the firewall publish privilege
A network's tags decide where the policies apply: a group scoped to
networks applies on the networks carrying its tags (`vdf:net=dmz`), and
`vdf:unprotected` takes a whole network out of the firewall. Changing
them is a publish, not a group membership change.

firewallActionsForTag decides from the object and the tag: any `vdf:` tag
on a network takes `publish`; on a VM, `vdf:quarantine` takes
`quarantine:add` / `quarantine:remove`, a group tag `membership`, and a
network tag both `membership` and `publish`: it means nothing to the
firewall there, but a group may select VMs by any tag, so it can't be a
way around `membership`. The refusal names the first missing right.
2026-11-13 09:45:34 +00:00
Olivier Lambert
6e7a1c0679 feat(types): XoApp's hasFirewallRight
The decision on a pool's firewall rights that xo-server's reserved tag guard makes, for the firewall plugin's routes and any other caller.
2026-11-13 09:41:08 +00:00
Olivier Lambert
8c683fae6f feat(xo-server-vdf): pool routes take the firewall privileges
Each pool route asks xo-server's hasFirewallRight, the decision the vdf:
tag guard makes: `read` for the policy, history, trace, stats, rule log,
monitor and SDN controller, `publish` for check_policy and publish_policy,
`quarantine:remove` to acknowledge a quarantine lifted outside XO. Legacy
ACLs still count (view for read, administrate for every action), and
GET /global lists the pools whose firewall the user may read. The global
configuration stays for administrators.

The REST API's acl middleware can't declare these: it ignores legacy ACLs
and refuses every non-administrator where RBAC isn't licensed, so the
routes check in the handler.

GET policy adds `rights`, what the user may do with the pool's firewall per
action, beside `canEdit` (publish).
2026-11-13 09:39:39 +00:00
Olivier Lambert
c6e3d9bd9d feat(xo-server): Firewall administrator and Firewall operator template roles
Firewall administrator: every `firewall` action. Firewall operator:
`firewall` read, quarantine:add and quarantine:remove. Both read what the
firewall pages load (pools, hosts, networks, VMs, VIFs) and hold the first
key of a firewall tag, `update:tags` on VMs (and on networks for the
administrator), which also lets them change the objects' other tags.
2026-11-13 09:37:00 +00:00
Olivier Lambert
0682b0c91f feat(xo-server): vdf: tags take a firewall privilege on the pool
The reserved tag guard decides with one function, hasFirewallRight(user,
poolId, action), which the firewall plugin's routes can use too: an XO
administrator, the legacy ACL that counts as the action on the pool
(administrate for every action, view for read), or the RBAC privilege
`firewall <action>` evaluated against the pool.

A reserved tag takes `membership` (group and network tags), or
`quarantine:add` / `quarantine:remove` for vdf:quarantine. A new VM keeps
its source's group tags with `membership` on its pool, and a VM with group
tags migrates to another pool with `membership` there. The refusal names
the missing right.

Where the RBAC feature isn't licensed, only legacy ACLs count; from the
sources RBAC is always available, and a user without a role has no
privilege.
2026-11-13 09:36:11 +00:00
Olivier Lambert
1ae0e81889 feat(acl): firewall resource
A pool's firewall (VDF) as an RBAC resource: `read`, `publish`,
`membership`, `quarantine:add` and `quarantine:remove`. Its privileges are
evaluated against the pool, so a selector selects pools whatever the action.
2026-11-13 09:32:29 +00:00
Olivier Lambert
0ff845f6d7 fix(web): the firewall overview's donut and its Not protected list agree
The donut's Not protected segment was a subtraction, so it could count
VMs the list left out (e.g. one only on a vdf:unprotected network). Each
running VM now has one class, the list comes from the same one, and VMs
out of the firewall by choice are left out of the donut's total.
2026-11-12 20:54:53 +00:00
Olivier Lambert
2592c5d7c1 fix(xo-server): vm.copyToTemplate destroys a failed copy with its disks
The cleanup used the raw XAPI $destroy(), which left the copy's disks
orphaned; use VM_destroy as vm.clone does.
2026-11-12 20:54:53 +00:00
Olivier Lambert
a5c72279aa fix(xo-server-vdf): keep open quarantine alerts past the last-50 trim
An alert for a quarantine lifted outside XO would drop out of the list
once 50 later entries were recorded, and the VM would look fine again.
Trimming now keeps older open alerts; resolved ones go as before.
2026-11-12 20:54:53 +00:00
Olivier Lambert
5412cac5a4 feat(xo-server-vdf): refuse a group's networks that aren't network tags
Only the XO 6 form checked them; publishing and checking now refuse any
entry not shaped vdf:net=<value>, as for group tags. Published documents
still read.
2026-11-12 20:54:53 +00:00
Olivier Lambert
e499227a74 feat(xo6): Firewall: acknowledge a quarantine lifted outside XO, on the VM too
The plugin now keeps a quarantine lifted outside XO as an alert until the
VM is quarantined again or someone acknowledges it (vates/vdf#141, open
point 3); before, nobody could acknowledge it and it could come back.
- Pool overview: each VM of the alert has Acknowledge, for users who may
  publish the pool's policy (canEdit); the wording is factual: "web-1:
  quarantine lifted outside this XO on <date>".
- The VM's Firewall tab: the same alert for that VM ("The quarantine of
  this VM was lifted outside this XO on <date>"), with Acknowledge on the
  same condition.
- Pool history: the entry says how it ended: "acknowledged by bob on
  <date>", or "put back on <date>".
2026-11-12 20:54:53 +00:00
Olivier Lambert
bbdb4092b7 feat(xo-server-vdf): a quarantine lifted outside XO stays an alert until resolved
quarantineLifted was recomputed on read from the latest change kept, so
nobody could acknowledge it, and it came back: lifted outside XO, put back,
then lifted through XO (not kept) made the old entry look current again
(vates/vdf#141, open point 3).

The alert is now a state on the entry. A removal of vdf:quarantine outside
XO is open until resolved: resolution 'requarantined' when the plugin sees
vdf:quarantine added back on that VM, through XO or outside (it hears
every xo:reservedTagChanged, though it keeps only the outside ones), or
'acknowledged' with resolvedBy, through POST
/pools/{id}/actions/acknowledge_quarantine_lifted { entry }, which takes
administrate on the pool, as a publish. Once resolved, an entry is never
listed again. quarantineLifted lists, per VM XO still has, its last
quarantine change kept if it is an open lift; an alert pushed out of the
last 50 entries goes with them.

Writes to the list, events and acknowledgements, are serialized.
2026-11-12 20:54:53 +00:00
Olivier Lambert
84facca42f feat(xo6): Firewall: the vdf: tags a new VM did not inherit
The plugin now keeps, in a pool's tag changes, the vdf: group tags XO
stripped from a new VM because the user who created it may not add them
(vates/vdf#141, open point 7). Until now nobody saw it: the new VM was
just outside its source's groups.
- Pool history: a row "vdf:role=app not inherited by VM web-1-copy:
  created by alice, who can't add firewall tags", marked "Not inherited".
- The VM's Firewall tab: while the pool's history keeps the entry and the
  VM still lacks some of those tags, a warning says it was created
  without its source's firewall groups, names the tags, and points to
  Groups on the same page to join one.
2026-11-12 20:54:53 +00:00
Olivier Lambert
6d36fe2e72 feat(xo-server-vdf): the vdf: tags a new VM did not inherit in the pool's history
When a user who may not add firewall tags clones, copies, imports or
instantiates a VM, xo-server strips the source's vdf: group tags from the
new VM and now says so (xo:reservedTagsStripped, vates/vdf#141 open point
7). Until then only xo-server's log knew, and the new VM sat outside its
source's groups unnoticed.

While loaded, the plugin keeps each strip in the pool's tag-changes list,
as an entry of kind notInherited: the VM, the tags, and who created it
(their label, as publishedBy; their id if XO no longer knows them). GET
history returns it in tagChanges with the others. Entries without a kind
are the changes made outside XO, as before: those already stored read
unchanged, and only they can raise quarantineLifted.
2026-11-12 20:54:53 +00:00
Olivier Lambert
1ffb9a4a8a feat(xo-server): the vdf: tags stripped from a new VM are emitted
A strip was only a log line: the user who made the VM and the firewall's
owner saw nothing (vates/vdf#141, open point 7). stripInheritedReservedTags
now emits xo:reservedTagsStripped once it has removed tags: the VM's id and
name, its pool, the tags, the user who created it, a timestamp. It is
emitted by the strip itself, not derived from XAPI updates: the new VM may
not be in XO's objects yet. Core does nothing else with it; the VDF plugin
shows it.
2026-11-12 20:54:53 +00:00
Olivier Lambert
db7ecd135a fix(xo-server): a new VM's vdf: tag changes go unreported for 30 seconds, not 2 minutes
Changes nobody noted on a VM whose xo:creation date is recent are taken
as part of its creation and not reported as made outside XO (vates/vdf#141,
open point 2). XO sets a new VM's tags right after setting that date, so
30 seconds is ample; two minutes left a longer window in which an xe
change to a new VM went unreported.
2026-11-12 20:54:53 +00:00
Olivier Lambert
72245f3e06 fix(xo-server): a new VM whose vdf: tags can't be stripped is destroyed
vm.copy, both vm.import paths, REST clone/copy and REST XVA import failed
when stripping a new VM's inherited vdf: group tags failed, but left the
new VM behind with its tags (vates/vdf#141, open point 5); vm.create,
vm.clone, vm.copyToTemplate and REST create_vm already destroyed it.

They now call the reserved-tags mixin's
stripInheritedReservedTagsOrDestroy, which on failure destroys the new VM
with its disks, then fails with the strip's error; a failed destroy is
logged. One shared way instead of a try/catch per path.
2026-11-12 20:54:53 +00:00
Olivier Lambert
8463cf03d3 feat(xo-server): a VM moves to another pool with its vdf: group tags only through the guard
A VM migrated to another pool kept its tags there unchecked: arriving
with vdf:role=app put it into the destination pool's firewall groups
without the right on that pool (vates/vdf#141, open point 1).

Before a migration to another pool, the VM's group tags (every vdf: tag
but vdf:quarantine, which only tightens) go through the guard as if they
were added on the destination pool. Refused, the migration is refused
before anything moves; the tags are never stripped from a migrating VM,
and nothing is noted since they don't change. Same-pool migrations and
calls without a user are not checked.

The rule is the reserved-tags mixin's checkMigration, sharing the
guard's one permission decision; JSON-RPC vm.migrate and REST
POST /vms/{id}/actions/migrate call it, the REST route before creating
its task so that a refusal is a 403.
2026-11-12 20:54:53 +00:00
Olivier Lambert
577aebf4d4 fix(xo-server-vdf): a group on vdf:quarantine is refused as the quarantine tag
The quarantine tag starts with vdf:, so "group tags must start with vdf:"
didn't say why it was refused. It now gets its own sentence, in the
plugin's check and in XO 6's group form.
2026-11-12 20:54:53 +00:00
Olivier Lambert
2d4c552250 fix(xo-server): a vdf: tag change that changes nothing leaves no note
Removing through XO a vdf: tag a VM doesn't have (or adding one it has)
left a note that no XAPI event would ever consume: for two minutes, a
real removal with xe matched it, was logged as made through XO, by the
wrong user, and was not recorded as made outside XO.

checkTagsChange still decides on such a change as before, but notes only
the tags that change the object's current tags. That also drops the notes
left by stripInheritedReservedTags for the group tags a new VM keeps. A
note now waits 30 seconds instead of two minutes: XAPI reports a change
within seconds. The two-minute window of a VM being created is unchanged.
2026-11-12 20:54:53 +00:00
Olivier Lambert
bf2e399222 fix(xo-server): vm.create never drops a template's vdf:quarantine implicitly
Without `tags`, vm.create in a resource set replaced the template's tags
with the resource set's: a user without any right on the firewall got an
instance of a quarantined template that wasn't quarantined, and no check
ran. With `tags`, leaving out a template's group tag was refused as a
removal, although the strip would have removed it anyway.

One rule, in the reserved-tags mixin (checkNewVmTags), compares the
template's tags with the new VM's: a reserved tag added (not by the
resource set) goes through the guard; a group tag left out is fine;
vdf:quarantine stays when `tags` is omitted, and `tags` without it are an
explicit removal, through the guard. The group tags kept are still
stripped after creation.

The REST API's create_vm can't drop it: its body has no tags (extra
properties are refused) and the new VM keeps all the template's tags.
2026-11-12 20:54:53 +00:00
Olivier Lambert
373ba5e5a4 fix(web/vdf): the overview's unprotected VMs list counts uncovered interfaces
The list of running VMs no policy applies to worked per VM from group
membership: a VM with one covered interface and one uncovered was not
listed, and a VM only on a vdf:unprotected network was. It now uses
effectiveTiers, as the VM's Firewall tab does, per interface, and tells
that the traffic is not filtered and that a network policy (a baseline)
would cover it (vdf#141).
2026-11-12 20:54:53 +00:00
Olivier Lambert
6106881b0d feat(xo6): Firewall: vdf: tags changed outside XO, and a quarantine lifted that way
The plugin now keeps the vdf: tags changed outside XO on a pool's VMs and
networks, and the VMs whose quarantine was lifted that way and still is
(vates/vdf#141, section 7).
- Pool history: each such change is a line among the versions, by when XO
  saw it ("vdf:quarantine removed from VM web-1, outside this XO"), marked
  "Outside this XO", with nothing to view or restore.
- Pool overview: a danger alert, beside the one for a document changed
  outside this XO, lists the VMs whose quarantine was lifted outside XO,
  each linked to its Firewall page. Older plugins report none.
2026-11-12 20:54:53 +00:00
Olivier Lambert
f43558ea2c feat(xo-server-vdf): vdf: tags changed outside XO in the pool's history
xo-server now reports each vdf: tag change on a VM or network, through XO
or outside it (xo:reservedTagChanged, vates/vdf#141 section 7). While
loaded, the plugin keeps the changes made outside XO, the last 50 per pool
(tag-changes:<pool> in its store, carried by config export and import):
the object, its name when seen, the tag, added or removed. They are listed
beside the versions (GET history: tagChanges), not among them, so they
neither push versions out nor come between a version and the one it
replaced. Changes made through XO are left to xo-server's logs.

A VM whose vdf:quarantine was last removed outside XO, and which is still
out of quarantine, is an alarm: GET policy lists it in quarantineLifted.
2026-11-12 20:54:53 +00:00
Olivier Lambert
4d8f55f8e5 feat(xo-server): vdf: tag changes made outside XO are detected
XO can't stop xe, another XO or any XAPI client from changing a vdf: tag,
but it receives every change through XAPI events (vates/vdf#141, section 7).

The reserved-tags mixin notes the reserved tag changes checkTagsChange lets
through (the user's id) and those stripInheritedReservedTags makes ('xo'),
for two minutes. It follows XO's VMs and networks: when an object it
already knows changes reserved tags, each tag added or removed is through
XO if a note matches it (consumed), outside XO otherwise. Each is logged
(info, warn outside XO) and emitted as the app event
xo:reservedTagChanged for the firewall plugin.

Objects loaded at startup or on (re)connection, and new ones, are taken as
they are: nothing is reported for them. A VM XO is creating gets its tags
after it appears, so its changes no note matches are not reported while
its xo:creation date is less than two minutes old.
2026-11-12 20:54:53 +00:00
Olivier Lambert
03e4de7661 feat(web/vm): firewall tags in the VM tag editor
`vdf:` tags are special tags: warning accent and a confirmation, with
specific texts for the quarantine tag (the server changes the VM's network
access immediately). When the pool's published policy says the user can't
administrate it, the VM's `vdf:` tags are locked (tooltip) and none is
suggested or created, since the server would refuse.
2026-11-12 20:54:50 +00:00
Olivier Lambert
de71bb04af feat(web-core): VtsTagSelect: tags can be locked
VtsTagSelect takes an optional `tagLock(tag)` function: it returns why a
tag can't be changed, or nothing. A locked tag is disabled in the list and
carries a tooltip with the reason; "Select all" and "Deselect all" leave it
as it is, so a bulk action never asks for a change the caller refuses.
Without the prop, nothing changes.
2026-11-12 20:54:45 +00:00
Olivier Lambert
769850b3bc fix(web/vm): a failed tag change is shown to the user
When adding or removing a VM tag fails, the editor used to log the error
and nothing else: the tag just went back to its previous state. A small
error dialog now says which tag couldn't be added or removed, with the
reason given by the server when there is one (`data.reason` of the error),
else the error's message.
2026-11-12 20:54:41 +00:00
Olivier Lambert
e0ab974aa4 feat(rest-api): a new VM keeps the vdf: tags of its source only through the guard
POST /vms/{id}/actions/clone (clone, copy, cross-pool copy), POST
/pools/{id}/actions/create_vm and POST /pools/{id}/vms (XVA import) call
xo-server's stripInheritedReservedTags on the new VM: the reserved tags it
inherited and the user may not add are removed, vdf:quarantine stays. It
runs before the VM is returned or booted; if it fails, the action fails and
create_vm destroys the VM.
2026-11-12 17:09:22 +00:00
Olivier Lambert
bcf5c54873 feat(xo-server): a new VM keeps the vdf: tags of its source only through the guard
XAPI copies every tag to a clone, a copy or an import, so a user who may
not write vdf:role=app could clone an app VM, instantiate a tagged template
or import an XVA and get a VM inside the group, with its network access
(vates/vdf#141, section 5).

The reserved-tags mixin's stripInheritedReservedTags treats the inherited
tags as added by the user: each reserved tag that checkTagsChange does not
allow on the new VM, in its pool, is removed, except vdf:quarantine, which
always stays. It logs and returns what it stripped. Calls without a user
are not checked.

JSON-RPC: vm.clone, vm.copy, vm.copyToTemplate, vm.import (upload and URL)
and vm.create call it before returning the VM and before any boot; when it
fails the operation fails. vm.create checks only the template's tags the
new VM keeps: tags given in `tags` are still checked as added or removed,
and the resource set's tags are left alone.
2026-11-12 17:09:22 +00:00
Olivier Lambert
d779ac1b86 feat(xo6): Firewall: a group's tags must start with vdf:
The server now refuses to publish a group on another tag. The group form
says so as the tag is typed, and on tags a published group already has,
instead of letting the publish fail.
2026-11-12 17:00:15 +00:00
Olivier Lambert
79496c4f1c feat(xo-server-vdf): group tags must start with vdf:
XO core reserves the vdf: tags to pool administrators, which protects
nothing for a group built on a tag like web. Publishing a pool or global
document is refused when a group has such a tag, or the quarantine tag.
A document already published is still read as it is.
2026-11-12 16:58:42 +00:00
Olivier Lambert
aff914785b feat(rest-api): vdf: tags are reserved to administrators of the pool
Every PUT and DELETE /…/{id}/tags/{tag} route, and PATCH /vms/{id} with
tags, go through xo-server's checkTagsChange: adding or removing a vdf:
tag takes an XO administrator or the administrate permission on the
object's pool, on top of the route's privilege. A refusal is a
forbiddenOperation, answered 403. PATCH checks only the tags it actually
adds or removes, so resending a VM's unchanged vdf: tags is allowed.
2026-11-12 16:56:11 +00:00
Olivier Lambert
b3c3e6e2f7 feat(xo-server): vdf: tags are reserved to administrators of the pool
VDF puts VMs into its groups and quarantines them with tags starting with
vdf:, which host agents follow live. Anyone who could tag a VM could lift
a quarantine or give a VM a group's network access (vates/vdf#141).

Adding or removing a vdf: tag now takes an XO administrator or the
administrate permission on the object's pool; administrate on the VM is
not enough. The rule is in core, not in the plugin, so it holds when the
plugin is unloaded, and in one place: the reserved-tags mixin's
checkTagsChange, which every path writing tags for a user calls. Calls
without a user are not checked.

JSON-RPC: tag.add, tag.remove, and vm.create with tags, which replace the
template's: only the tags that differ from the template's are checked. The
resource set's tags were set by an administrator and are not checked.
2026-11-12 16:56:11 +00:00
Olivier Lambert
33e5eae354 fix(xo-server-vdf): XO 6 can save the plugin's configuration
The REST API takes a plugin's configuration only if it matches one of the
configuration types it knows (@vates/types), and the vdf plugin's wasn't
one: saving it from XO 6's Plugins page failed with "Value does not match
any allowed schema" (422), so the plugin couldn't be enabled there. XO 5's
Plugins page, which goes through JSON-RPC, was not affected.
2026-11-12 14:54:32 +00:00
Olivier Lambert
4f2a73c456 feat(xo6): Firewall: its pages in a side menu, Trace in a drawer, one-line rows
A second row of tabs under the pool's read as the same level. The
Firewall's pages are now a menu beside them, as vSphere lists Configure's:
Status (Overview, Traffic, Rule log) and Configuration (Policies, Groups,
Settings, History). It runs the page's length, its list stays in view, and
it folds to a rail of icons (their label on hover). Open by default, the
toggle at its top; the choice is kept, and below --small it is always
folded. Above each page: its title, its own controls and the tools.

- Trace is a drawer over any page, not a page: the header's "Trace a
  connection" and every "Why?" open it, a trace from inside it replaces
  it. Before a trace, the drawer lists the flows dropped lately. Old
  /firewall/trace links open it on the Overview, run=1 still runs it.
- The rule log is a page of its own; Traffic keeps the flows dropped in
  the last 15 minutes, each a trace away, and its charts side by side.
- Rows on one line: an end's name, first group and address (a long IPv6
  address cut in its middle), what decided as one link (its tier and
  label on hover), the time with its date only when not today. Narrow,
  an end shows its name only and the rule log's outcome is an icon.
- Recently dropped showed its header in each row, not the count.
2026-11-12 14:54:32 +00:00
Olivier Lambert
8c56df2762 fix(xo-server-vdf): a trace takes its family from an end given as an address
A trace from a VM named by uuid to an IPv6 address failed with "<the VM's
IPv4> and <the IPv6> are not of the same family": the family came from the
source end only, so the VM took its first IPv4 address before the
destination was looked at. An end given as an address now sets the family,
whichever end it is, and the VM end takes its address of that family. Two
addresses of different families are still refused.

Same fix in vdf-agent's trace (agent/trace.go, traceFamily).
2026-11-12 14:54:32 +00:00
Olivier Lambert
694139f2b7 feat(web-core): VtsBarChart: square bars, the gradient alone 2026-11-12 14:54:32 +00:00
Olivier Lambert
15030d92af feat(xo6): Firewall Traffic: a shorter flow map description 2026-11-12 14:54:32 +00:00
Olivier Lambert
78ed90fbd2 feat(xo6): Firewall Traffic: the rates and the rule log's minutes side by side
Neither chart needs the page's width: "New connections over time" and the
rule log per minute (the whole log, by the rules and policies that log)
sit side by side, one under the other on a narrow screen. The rule log
table keeps its own chart, of what it shows, elsewhere (the VM page);
on Traffic it leaves it to the page (chart: false). The series and its
caption are a composable both use.
2026-11-12 14:54:32 +00:00
Olivier Lambert
bc887afff5 fix(web-core): VtsBarChart: whole bars at the ends, a stack rounded at its top
A time axis ends at the first and last timestamps: their bars, centered
there, were cut in half. The axis now reaches half an interval past them.
Stacked, only a stack's top segment has rounded corners: a thin segment
rounded under another read as a narrower bar.
2026-11-12 14:54:32 +00:00
Olivier Lambert
8f2e64cda5 feat(web-core): VtsBarChart: bars with a gradient and rounded tops
With colors given, a bar goes from its color at the top to a lighter
shade toward the axis, its top corners rounded (a stack reads as
segments), and glows a little when pointed at.
2026-11-12 14:54:32 +00:00
Olivier Lambert
cdb40818fd feat(xo6): Firewall: a leaner Policies board, drops in red in the charts
- A rule's side that is its policy's own group (the destination inbound,
  the source outbound) is light text, not a chip: the policy's header
  says it once.
- The inbound "When no rule matches" line goes: the header's default
  action says it. The outbound one stays (its own default).
- A run of tiers without a policy is one quiet line, with a way to add a
  policy to each.
- In a pool, a global policy is its header line (with its rule count)
  until opened; all open while searching, or when a link points at it.
- The Log column goes: a logged rule has an icon by its action, its label
  in the tooltip. A rule's buttons show when its row is pointed at or
  focused (always on touch screens).
- The rate charts draw dropped in red, as VDF does everywhere, and
  allowed in the brand color (told from red by red-green colorblind eyes).
2026-11-12 14:54:32 +00:00
Olivier Lambert
a02965037e feat(web-core): VtsBarChart: colors
A color token per series (--color-…), for series whose color means
something, resolved in the current mode; the theme's colors otherwise.
2026-11-12 14:54:32 +00:00
Olivier Lambert
116c072172 feat(xo6): Firewall: rates as stacked bars, the flow map's legend on its title
- The new connections chart (Traffic) and the rule log's per minute are
  stacked bars (VtsBarChart): an interval's bar is its connections,
  allowed under dropped; one without a sample is no bar, where the line
  broke. The y-axis fits the highest sum.
- The flow map's legend is on its title's line, its hint in the
  description: the map gets the height.
2026-11-12 14:54:32 +00:00
Olivier Lambert
48361a7e33 feat(web-core): VtsBarChart
VtsLinearChart's data as bars over time, for counts per interval: an
interval without data is no bar, where a line breaks. stacked puts the
series one on the other.
2026-11-12 14:54:32 +00:00
Olivier Lambert
f185dfa531 feat(xo6): Firewall: a more compact flow map
A flatter ellipse, smaller nodes and labels: the map takes about a third
less height on Traffic.
2026-11-12 14:54:32 +00:00
Olivier Lambert
61607b2faa feat(xo6): Firewall: a flow map on Traffic
The pool's traffic between its groups, live: the groups its policies
apply to (each VM in the most specific, the pool's own before a global
one of the same size), its other VMs and outside (addresses, other
pools' VMs). Open connections in green, moving toward their destination
and thicker with more of them; drops of the last minute (the rule log)
in red, pulsing at their destination. A link says what it carries and
traces one of its flows in the drawer. Plain SVG in XO's colors; still
when paused and when the system asks for less motion.
2026-11-12 14:54:32 +00:00
Olivier Lambert
b7cb037522 feat(xo6): Firewall: Switch to Enforce says what the policy would have dropped
Before a monitored policy enforces (its report's Switch to Enforce), a
confirmation says what it would have dropped in the last 24 hours, as its
report counts it, the busiest three flows with how often the hosts saw
them, or that it would have dropped nothing; saved to the draft. The
board's Switch to Enforce, for a policy published in monitor mode in a
pool, opens its report: the decision is made with it.
2026-11-12 14:54:32 +00:00
Olivier Lambert
d3ecb91b51 feat(xo6): Firewall: a drop just logged slides into the rule log
Drops logged since the table last refreshed come in with a red highlight
that fades (2.5 s), after the first load and whatever the filters; none
when the system asks for less motion.
2026-11-12 14:54:32 +00:00
Olivier Lambert
47eeaf1677 feat(xo6): Firewall: a trace as the path its packet travels
Above a trace's steps, its path: the source, each tier the packet goes
through (the source's outbound ones first), a check that decides outside
the tiers, the destination. The packet travels to the step that decides
and stops there: on to the destination in green, or dropped in red with
the rest greyed; each tier says the rule that decided or that none
matched, the full step in its tooltip. Replay plays it again; no motion
when the system asks for less.
2026-11-12 14:54:32 +00:00
Olivier Lambert
40dcda4c42 fix(xo6): Firewall: an outbound rule log row's trace button names its destination
Its label said "Trace vdf-api to vdf-api": the logged VM, the source of
an outbound entry (vdf#53), as the destination.
2026-11-12 14:54:32 +00:00
Olivier Lambert
dba5f07ad8 feat(xo6): Firewall: a monitored policy's report in a drawer
The Monitor tag of a pool's monitored policy on the Policies board opens
its report in a wide drawer (that policy's, no picker): what it would
drop, Switch to Enforce, and Allow this, which closes it for the rule
drawer. The full report stays a page, from the board's link and the
Overview.
2026-11-12 14:54:32 +00:00
Olivier Lambert
a17846c191 feat(xo6): Firewall: Why? and a VM's traces open in a drawer (vdf#139)
A trace answers a question asked from a row (the rule log's and the
monitor report's Why?) or a VM's page: it opens in a wide drawer, run at
once from a row, and the page stays. The Trace page keeps its form and
recent drops, and its URL (VdfTracePanel, shared). The drawer never writes
the URL: XO closes overlays on any navigation.

A rule log entry from a VM of another pool traced it as one of this
pool's and failed ("no VM ... in the pool"): it's traced as an address.
2026-11-12 14:54:32 +00:00
Olivier Lambert
521edd09a9 feat(web-core): UiDrawer: a wide size
A drawer is half the page: right for a form, too narrow for a table or a
trace's steps. size="wide" makes it up to 90% of the page (120rem at
most) on medium and large screens; the full width on small ones, as now.
2026-11-12 14:54:32 +00:00
Olivier Lambert
70f20065b3 fix(web-core): UiDrawer: dark themes put its body in the cards' color
In the dark themes a drawer's body took the secondary background, the
black of the page behind the cards (#000 in the default dark theme), so
its form sat on black where a page's forms sit on a card. The light
themes are unchanged.
2026-11-12 14:54:32 +00:00
Olivier Lambert
69a7d06ff1 feat(xo6): Firewall: a clearer switch, a Policies board with room to read
- VdfSwitch for the Firewall pages' on/off settings (live hits, the
  monitor report's filters, history's unchanged versions, connection
  grouping, a network left out): the knob inside its track, the label after
  it in sentence case. XO's UiToggle draws its knob larger than the track.
- Policies board: a policy is a block (room above, its name as a heading),
  rules get room to be read, a tier opens a band, an empty tier stays a
  quiet line.
2026-11-12 14:54:32 +00:00
Olivier Lambert
d2a9095621 feat(xo6): Firewall Traffic: one live control, in the tab bar
The page's "Live, every 4 s / Pause" took a row of its own, and the rule
log had a second one. One control now, at the right of the Firewall tab
bar (the dot and the button on narrow screens), and the rule log follows
it; the toolbar row only shows with a draft hint or a VM filter.
2026-11-12 14:54:32 +00:00
Olivier Lambert
e3e22ab231 feat(xo6): Firewall: Live and Rule log merged into a Traffic tab (vdf#139)
One place for what the pool's traffic does: the live counters and chart,
then the rule log (each logged drop and allow, with its rule and Why?),
then open connections and the protected VMs. Live's dropped table goes:
the rule log has its drops with more detail, and a policy that doesn't
log shows its drops in the protected VMs' rates.
2026-11-12 14:54:32 +00:00
Olivier Lambert
ca806ad506 fix(xo6): VM Firewall tab blank on a direct link
A direct link could get the VM before the pool collection: the tab's
setup read the undefined pool's id (useVdfPolicy) and the tab stayed
blank. The page now waits for the pool, then mounts the tab
(VdfVmFirewall, the former page).
2026-11-12 14:54:31 +00:00
Olivier Lambert
9ab5d9090c feat(xo6): Firewall: the agents' alarms apart from their errors
Agents now report alarms (status `alarms`): a bridge tampered with, where
VDF's policy may have been bypassed, and a bond VDF can't read (vdf#136).
- Pool overview: tamper alarms first under Needs attention, bond alarms
  with the warnings; the host problems line counts the other errors only.
- VM page: a tamper alarm on one of the VM's bridges turns its alert red.
- XO-wide overview: the pools' bridges tampered with, in Needs attention.
Older agents report none: their errors show as before.
2026-11-12 14:54:31 +00:00
Olivier Lambert
c9a67ca5f7 feat(xo6): Firewall: Needs attention first, a Settings tab, Monitor under Policies
- Overview: what needs attention first, most serious first (hosts whose
  agent refuses the document, reports errors or keeps restarting, then
  exposed and bypassed VMs, locks, newly dropped VMs), information last.
  The recent drops card leaves it: Live and the rule log show them.
- Onboarding follows the preview guide: group, lock, declare services,
  write a policy (in monitor mode), publish, watch, then enforce.
- A Settings tab: network services (from Groups) and connection limits
  (from the Policies board), what the policies rely on.
- Monitor is no longer a tab: the report is under Policies, from a
  monitored policy's row and the board's "Monitor mode report" link.
2026-11-12 14:54:31 +00:00
Olivier Lambert
ecf8ead378 fix(xo6): Firewall: host errors on the VM page, honest Traffic Rules warnings
- VM page: a host in error still counted its VMs as protected. It now
  lists the errors of the VM's bridges and the host's own, and its
  policies the host doesn't realize, with why.
- Publish and join-group dialogs warn only about the Traffic Rules that
  decide before VDF (an older agent, or found shadowing it): under agents
  that put VDF first, an allow can't bypass it (vdf#40).
- XO-wide overview: the Needs attention column counts hosts reporting
  errors or refusing the document.
- Crash-loop hints point to /var/log/vdf-agent.log: journald keeps
  nothing on XCP-ng.
- Take over shows the plugin's reason when refused.
- Networks are fetched with their bridge.
2026-11-12 14:54:31 +00:00
Olivier Lambert
da17257067 fix(xo-server-vdf): refuse a take-over that would remove the pool's global rules
A fresh XO in place of a lost one has no global configuration: taking a
pool over made it write the pool without the global part it holds, at the
next sync (vdf#81). Same for a pool out of this XO's scope. Refused with
409 (dropsGlobal: true) and a reason, unless dropGlobal.
2026-11-12 14:54:31 +00:00
Olivier Lambert
53a11158ca feat(xo6): Traffic Rules pages say what VDF does with each rule
On an interface VDF protects, a Traffic Rule (SDN Controller) may let
traffic past VDF, drop it before VDF decides, apply after VDF's
decision, or never apply under an older VDF agent. The Traffic Rules
pages of a pool and a VIF give each rule's status, as the host's agent
reports it.
2026-11-12 14:54:31 +00:00
Olivier Lambert
74b157e107 feat(xo6): Firewall card in a network's panel
A network's side panel, in pools VDF manages, says how many protected
interfaces are on the network, which groups are scoped to it, and which
network policy applies to every interface on it; and leaves the network
out of the firewall, or brings it back.
2026-11-12 14:54:31 +00:00
Olivier Lambert
c07d2f8a2d feat(xo6): Firewall overview
The Overview page, now the one /firewall opens:

- A pool: is it protected, what isn't (VMs no policy covers, unlocked or
  untrusted interfaces, VMs newly under a default drop), each host's
  agent, recent drops, and changes made outside this XO.
- The global configuration: its scope, each pool's sync, what a pool
  entering the scope receives, each global policy's reach, and pools
  another XO owns.
2026-11-12 14:54:31 +00:00
Olivier Lambert
c8d7826e7d feat(xo6): Firewall tab of a VM
A VM's Firewall tab, in pools VDF manages: the policies that apply to
it, its groups (join one), each interface with how its addresses are
locked and trusted, Traffic Rules that bypass VDF, its rule log, and
Quarantine from the VM's menu.

Links to a VM from the groups, the rule log and the live view now
open this tab.
2026-11-12 14:54:31 +00:00
Olivier Lambert
6b9b8419c6 feat(xo6): Firewall live view
The Live page of a pool's Firewall: protected VMs with their allowed,
dropped and spoofed traffic rates, open connections with the rule that
let each one in, and blocked attempts.
2026-11-12 14:54:31 +00:00
Olivier Lambert
ef493915f6 feat(xo6): Firewall monitor mode report
The Monitor page: what a policy in monitor mode would drop if it were
enforced, per VM and interface, as suggested rules, with the expected
flows, a trace per flow, and switching the policy to enforce. It also
says where monitor mode can't see, because an enforced policy of the
same tier decides first.

A monitored policy's tag on the Policies page now opens its report.
2026-11-12 14:54:31 +00:00
Olivier Lambert
c2e67b250d feat(xo6): Firewall trace and rule log
- Trace: why is this connection allowed or blocked? From a VM, interface
  or address to another, hop by hop on each host involved, with the
  rule that decides.
- Logs: the hosts' per-rule audit log, with its rate chart (web-core's
  linear chart takes a time formatter and a minimum tick step, for
  windows of minutes), and from each entry a trace
  or quarantine of the VM.
- Quarantine: cut a VM off, from the places an incident starts.
2026-11-12 14:54:31 +00:00
Olivier Lambert
bc80315550 feat(xo6): Firewall history
The History page: the documents XO published to the pool (or the global
configuration), when and by whom, what each one changed object by
object, and the changes made outside this XO. A version can be viewed,
and restored as a draft to review and publish like any edit.
2026-11-12 14:54:31 +00:00
Olivier Lambert
16e227063f feat(xo6): Firewall pages: policies and groups, draft and publish (tech preview)
The Firewall tab of a pool and the XO-wide Firewall (the global
configuration, written into every pool in scope), with their Policies
and Groups pages. They link to each other, so they come together.

- Policies: tiers, policies and rules, with an inbound and an outbound
  side, monitor mode, connection limits, policies on networks alone, and
  the pool's declared network services.
- Groups: groups of VMs (by tag) or networks, their members with their
  addresses and how each one's addresses are trusted, add and remove
  VMs. Interfaces are locked from here: XAPI's lock, or VDF's own lock
  where XAPI refuses one because an SDN controller is registered on the
  pool.
- Every edit goes to a draft kept in the browser, rebased on the
  published document when another user publishes first.
- Publish shows what changes object by object, and what the hosts say
  of it: hosts whose agent is too old ("Publish anyway"), hosts without a
  working agent, and where a global policy's default decides for a
  pool's.
- The route helpers that open a group or a policy (vdf-global.util),
  with their tests.

The tabs show only while the xo-server-vdf plugin is loaded (an opt-in
setting), so nothing changes for an XO without it.

The firewall's other pages come in the next commits, each with its tab.
Until then, /firewall opens Policies, and a group member's name opens
the VM's dashboard rather than its Firewall tab.
2026-11-12 14:54:31 +00:00
Olivier Lambert
658fe567c0 feat(xo6): VDF firewall model, REST resources and jobs
The logic under XO 6's Firewall screens, without any screen yet:

- types.ts: the VDF policy document (groups, policies, tiers, rules,
  network services) and what the xo-server-vdf plugin's routes answer.
- utils: parse and edit a policy document, merge a draft with the
  document published meanwhile, compare two versions object by object,
  the global configuration's part in a pool's document, a host's status
  (realized, stale, too old an agent), VIF locking and which members are
  trusted, monitor mode reports and the policies it can't see past,
  network services, rate series, recent drops.
- remote-resources: the plugin's routes (info, the pool's published
  policy, the global configuration, history, monitor, stats, rule log).
- jobs: publish, trace, VDF lock and unlock, and the VM and network tags
  that make group members.
- use-xo-vif-collection: also fetch a VIF's allowed IPv4 and IPv6
  addresses, which locking and member addresses depend on.

Unit tests cover every util.
2026-11-12 14:54:31 +00:00
Olivier Lambert
2db0c7096a feat(xo-server-vdf): plugin for the Vates Distributed Firewall (tech preview)
The REST routes behind XO 6's Firewall screens, under
/rest/v0/plugins/vdf. VDF is a stateful firewall in front of every VM,
enforced by each host's vdf-agent with Open vSwitch connection tracking
and driven by tags. The policy lives in the pool's other_config
(vdf:policy*); each agent reports what it enforces in its host's
other_config (vdf:status). The plugin never re-implements the agent's
rules: validation, publication and traces go through the hosts' vdf XAPI
plugin, which the agent installs.

- Pools: the enforced policy, check and publish (with a concurrent edit
  check and the hosts whose agent is too old), trace, per-rule stats, the
  rule log, and the history of published documents, including changes
  made outside this XO.
- Global configuration: groups and policies defined once in XO and
  written into every pool in scope, members resolved across pools, one
  XO owning a pool's global part, sync on XAPI events.
- VDF lock: lock a VIF's addresses while an SDN controller is registered
  on the pool, which XAPI's own locking refuses.
- Permissions: administrators edit everything; users with an ACL on a
  pool see its firewall (view) and edit its own part (administrate).
- A config manager, so XO's config export carries the global
  configuration and the histories.

Opt-in: the plugin has a required setting, so xo-server leaves it
unloaded until an administrator enables it. Unloading it is the off
switch for XO: its routes and background sync stop, and XO 6 hides the
firewall. Policies already published stay enforced by the agents.

The README documents every route and the XAPI keys the plugin writes;
index.test.mjs checks that list against the code.
2026-11-12 14:54:31 +00:00
Olivier Lambert
d532e3b99b fix(xo-server-sdn-controller): a pool's first VIF rule gives the bridges the controller first
On the rig (vdf#40), the first rule added on a pool XO didn't manage yet
failed with "connect ECONNREFUSED <host>:6653", and the same call again
worked. _setPoolControllerIfNeeded set the bridges' controller only
through the hosts' existing OVSDB clients: on a pool's first rule there
were none yet, so no bridge had the pssl: controller, and nothing
listened on 6653 when the OpenFlow channel connected. The client was
created right after, so the next rule found the bridges set.

_setBridgeControllerForHost now creates the host's OVSDB client when it
doesn't exist yet.

Test: the bridges get the controller with no OVSDB client created before.
2026-11-12 14:54:31 +00:00
Olivier Lambert
5350ff1a5e fix(openflow): negotiate the OpenFlow version instead of refusing a newer HELLO
A bridge that allows OpenFlow 1.0, 1.1 and 1.4 (XAPI's locking flows use
1.0, VDF uses 1.4) answers XO's HELLO with a HELLO of version 5 and a
version bitmap. unpack() threw "Unsupported OpenFlow version: 5", which
ended the parse loop, so the SDN controller's direct channel never
connected.

Per the OpenFlow specification (1.1 § A.5.1, 1.3.1 § 6.3.1), each side
uses the lower of both versions, and a version bitmap, when sent, says
which versions the peer supports:

- unpack() reads a HELLO of any version: its header, and the versions of
  its version bitmap when it has one (`versions`).
- negotiate(hello): the version to use with that peer among the versions
  this package supports (OpenFlow 1.1), undefined when there's none in
  common: the caller then closes the connection.
- Other messages of an unsupported version are still refused.

Also: packing a flow-mod with a cookie but no cookie_mask threw (the mask
was written at an undefined offset); it's now zeroed as intended.

Tests (node --test in dist): a HELLO captured from OVS 2.17 (version 6, no
bitmap), a version 5 HELLO with bitmap 1.0/1.1/1.4 and one without 1.1, an
OpenFlow 1.0 HELLO, the cookie fields of a flow-mod. Also the version 5
HELLO a bridge allowing OpenFlow 1.0, 1.1 and 1.4 sends, captured from OVS
2.17 (version bitmap 0x26): that OVS answered a version 2 HELLO and
FEATURES_REQUEST with a version 2 FEATURES_REPLY, so negotiating down to
OpenFlow 1.1 works against it.
2026-11-12 14:54:31 +00:00
Olivier Lambert
95d88b5efa chore(xo-server-sdn-controller): link the changelog entry to its PR 2026-11-12 14:48:58 +00:00
Olivier Lambert
d2cb40135c fix(xo-server-sdn-controller): say which host failed a traffic rule with the XAPI plugin
With the XAPI plugin (`useDirectChannel = false`), a traffic rule is
applied by calling `sdncontroller.py` on every host of the network, through
`asyncEach` with `stopOnError: false`. When a host fails, `asyncEach`
rejects with an `AggregateError` that has no message, so the error the user
gets is empty: `{"error": ""}` from the REST API, an empty message in XO 6,
and no useful text in XO 5.

Each host's error is now rethrown with the host's name, keeping the
original as its `cause` (enumerable, so that the XAPI error's code and
params are still serialized, as in the JSON-RPC error's data), and the
`AggregateError` gets these messages, e.g.
`XENAPI_MISSING_PLUGIN(sdncontroller.py) on host-2`; the errors stay in
`errors`. VIF and network rules, added or deleted, share this path.

Test: a rule with the plugin missing on one of two hosts (message, cause,
serialized details), on three hosts with two failing, and a successful one
(the first three fail without the fix).
2026-11-12 14:48:58 +00:00
Olivier Lambert
b4c4ae1322 fix(xo-server-sdn-controller): fail VIF rules the host can't apply
A VIF rule added or deleted through the API or the REST routes reported
success when the host failed to apply it (e.g. the XAPI plugin call
failed), with nothing installed or recorded. `_addRule`/`_deleteRule` take
`{ strict }`, passed by those calls: the call fails and the VIF's
other_config is left as it was. Rules applied again on events keep logging
their errors.
2026-11-12 14:48:58 +00:00
Olivier Lambert
8fa342f799 fix(xo-server-sdn-controller): don't turn OpenFlow 1.0 off on the bridges it controls
Taking a bridge's control inserted OpenFlow11 into Bridge.protocols. On a
bridge with none set, which OVS reads as its defaults (1.0 to 1.5), that
left OpenFlow 1.1 alone, and XAPI's setup-vif-rules (plain ovs-ofctl,
OpenFlow 1.0) could no longer install the VIFs' locking flows.

OpenFlow11 is now inserted only when the bridge's protocols are set and
lack it: an empty set already accepts OpenFlow 1.1.
2026-11-12 14:48:58 +00:00
amouillard-vates
79219f2e1c feat(xo6/vm): add VM tags edition (#10777) 2026-11-12 15:44:42 +01:00
Joris K
67ed8f7eb3 fix(web-core): keep head bar actions next to the title on small screens (#10774) 2026-11-12 14:17:54 +01:00
Killian
2783e60933 feat(fs): lock VM backup folders on S3 and Azure (#10549)
feat(fs): lock VM backup folders on S3 and Azure (#10549)

On S3 and Azure, handler.lock() was a no-op, so two backup jobs, or a job and a
deletion, could write and merge the same VM's backups at the same time. Local, NFS
and SMB repositories have always locked. Both now lock a `<path>.lock` object, with
the contract of LocalHandler#_lock(): ELOCKED at once on contention, and a lost lock
is warned about and taken back.

S3 holds the lock with plain PutObject writes, which every S3-compatible provider
supports, instead of conditional writes that providers handle unevenly. The holder
rewrites the object every 30 seconds, and a lock is stale after 3 minutes without a
rewrite. Its age is read on the provider's clock alone: Last-Modified of the lock
against the Date of the HEAD response. XO's clock takes no part, so a skew of more
than 2.5 minutes no longer takes over a live lock. A provider which omits either
header never lets a lock be taken over. The lock is not atomic: two jobs which find
it free at the same time can both take it.

Azure uses a blob lease, which Azure itself expires: a 60 second lease, renewed every
15 seconds, so there is no clock to compare. A renewal writes nothing, so accounts
with versioning or soft delete do not keep a copy per renewal. Each renewal gives up
after 20 seconds, so a hung request leaves time for the next one before the lease
expires. Releasing deletes the blob only while the lease is still ours
2026-11-12 10:03:18 +01:00
Killian
7679bb0d44 feat(backups): purge the backup journal of a repository (#10703)
feat(backups): purge the backup journal of a repository (#10703)

The journal of a repository (`xo-backup-log/`) is append-only: nothing ever removes its entries,
so it grows with every backup, deletion and merge for as long as the repository lives.

`purgeBackupJournal(handler, before)` removes the day directories before the day of `before`,
whole days at a time, which is what the one-directory-per-day layout was meant for. A day which
cannot be removed (immutable files, a bucket policy refusing deletes) is skipped, so that it does
not keep the later days from being removed, and the next purge tries it again; a purge warns once,
with its first failure and the number of failures. `RemoteAdapter#purgeBackupJournal()` exposes
it, and so does the proxy, as `backup.purgeVmBackupsJournal({ remote, before })`: `before` is
computed by XO, so the horizon is XO's whatever the clock of the proxy.

xo-server purges the journal of every enabled remote when it starts, then once a day, one remote
after another and in the background, of the days older than `backups.journalRetention` (30 days by
default). Whatever that setting says, it keeps what a replay of the listing cache can still read:
the `fullListingInterval` of the remote, plus the clock skew tolerance its cursor is bootstrapped
with. A remote which does not answer within 10 minutes is left to its purge and the next one gets
its turn; a proxy too old to purge fails its remote with a warning.

XO-3039
2026-11-12 09:45:21 +01:00
Joris K
b9f5b0724e feat(xo6/web-core): scroll arrows and keyboard navigation in tab lists (#10759) 2026-11-12 09:37:56 +01:00
Sylvère
7d5c818efb fix(xo6): fix MenuItem display for Pool > Add Network and VM > Add VDI (#10766) 2026-11-12 09:33:19 +01:00
Marie Garde
152365f984 feat(xo6): Add backup log detail page (#10718)
* feat(xo6): Add backup lob detail page

* refacto structure

* refacto for PR

* add tests

* refacto for merge
2026-11-12 09:26:44 +01:00
Florent BEAUCHAMP
83c3e344ba feat(rest-api): expose the endpoint for file level restore (#9941) 2026-11-10 16:59:59 +01:00
Florent BEAUCHAMP
56ffcdba29 feat(rest-api): implement V2V routes (#10597)
implement Rest Api route to do an inventory of a remote esxi/vsphere + the action to import VMs into XCP-ng
2026-11-10 16:59:54 +01:00
Joris K
1d3e1fce39 feat(xo6/web-core): full page menus on mobile and responsive side panel actions (#10745)
* fix(web-core): place menus and tooltips right next to their trigger

- Wrap placement.js in `placeOverlay` which positions the overlay at the top-left corner before measuring, so a temporary scrollbar no longer shifts menus, and compensates when `body` does not cover the whole window
- Submenus open on the same side as their parent menu, unless there is not enough room

* fix(web-core): don't show the side panel for a moment when it is mounted

The panel offset was set through a `v-bind` CSS variable, which Vue only applies once the element is in the page:
the panel was first displayed, then slid out. It is now an inline style, set as soon as the element is created.

* feat(xo6/web-core): open menus in full page on mobile

- New `VtsMobileMenu`: full page menu sliding in like the side panel, with a back button and submenus opening in place
- New `VtsResponsiveMenu`: a `MenuList` on desktop, a `VtsMobileMenu` on mobile
- `VtsActionsMenu` relies on it and gets a `label` prop, it is used by tables, treeview, side panels and the VM and host headers
- Account menu, table bulk actions, "New"/"Add" dropdowns and "Copy all" use `VtsResponsiveMenu`
- On mobile, "Change state" and "Manage in XO-5" move into the "more actions" menu of the VM and host headers and of the VM side panel
- On mobile, an action of a "more actions" menu closes the side panel or the treeview
- New `VmHeaderActions` shared by the VM header and side panel
- Rename "Manage VM lifecycle in XO-5" into "Manage in XO-5"

* feat(xo6/web-core): move side panel actions into "more actions" when space runs out

- `VtsPanel` measures whether its actions fit on one line and moves them one by one, starting from the last one, into its "more actions" menu, created when needed
- New `VtsActionButton`: a button rendered as a menu item when it is inside a menu
- Panel action buttons (connect/disconnect, edit, delete, revert, change state…) use it, and pool, SR and backup repository side panels use the `actions` and `more-actions` slots of `VtsPanel`
- In the VM side panel, "Change state" moves into the "more actions" menu as a submenu

* docs(changelog): add full page menus and side panel actions entries

* refactor(web-core): rename VtsDeleteButton `tooltip` prop into `hint`

Like `VtsActionButton`, it is shown only when the button is disabled.

* fix(web-core): measure again the panel actions when they change

When an action is added, removed or replaced (connect/disconnect), the widths stored to move actions back from the "more actions" menu are no longer valid, so the measure starts over.

* fix(lite): don't show two "more actions" menus in the network side panel

"Copy info JSON" goes into the `more-actions` slot, so the delete button moves into the same menu when the panel is too narrow.

* feat(lite): add stories for VtsMobileMenu, VtsResponsiveMenu and VtsActionButton

The VtsPanel story uses VtsActionButton and MenuItem to show how actions move into the "more actions" menu.

* test(xo6): cover VmHeaderActions and AppHeaderMenu

* feat(web-core): show side panel actions as buttons when there is room

Actions move one by one into the "More actions" menu when they don't fit, and come back out when there is room again.

* feat(web-core): factorize the "Change state" menu and use full page menus on mobile

Add VtsChangeStateMenu and VtsChangeStateMenuItem, rename VmHeaderActions into VmMainActions.

* fix(web-core): don't cut off header actions next to a long name

* feat(xo6,lite): compact PIFs card on narrow side panels

Hosts are links with the master star, and the status is an icon with a tooltip when there is not enough room.

* fix(web-core): keep link icons on the first line and wrap traffic rule cards

* docs(changelog): merge the PR entries
2026-11-10 16:07:58 +01:00
Mathieu
4cd316d23b fix(rest-api): avoid attaching rbac role to admin user (#10694)
introduced by a4b17d5abb
2026-11-10 15:51:37 +01:00
OlivierFL
6e7a6659ab feat(web-core): introduce file uploader components (#10753) 2026-11-10 15:49:03 +01:00
Joris K
2e67209418 feat(xo6): group XO 5 and third party apps links in a mobile header menu (#10744)
* feat(xo6): group XO 5 and third party apps links in a mobile header menu

* docs(changelog): add mobile header menu entry
2026-11-10 10:46:03 +01:00
Enée DI IORIO
03e93498fd feat(web-core): upgrade UiDropdownButton component to v9 (#9850) 2026-11-10 10:05:19 +01:00
Mathieu
50166fe3a4 fix(xo-server): validate authentication token IDs (#10748) 2026-11-10 09:29:37 +01:00
Florent BEAUCHAMP
724402733b feat(rest-api): create backup job routes (#10595)
XO-2189
XO-3282
2026-11-09 17:51:05 +01:00
Thomas Moraine
abbe24631d docs(navbar): make link to vates vms external (#10231)
For legal reasons, the Vates VMS documentation
has to be treated as an external site, relative to
the XCP-ng and Xen Orchestra documentations.

As a result, the link to the Vates VMS doc in the
header of the XO doc has been converted to
an external link.

Clicking the link now opensa new tab instead of
loading the page in the same tab.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-11-09 16:29:58 +01:00
Elise-F
8e81250ca3 fix(xo6,lite): translate VM and host state tooltips in tree view (#10643) 2026-11-09 16:25:00 +01:00
Joris K
c8c11ae63c feat(xo6/web-core): show a loader in VM and host icons during power state changes (#10720) 2026-11-09 10:49:12 +01:00
Elise-F
8df236a512 feat(xo6): displaying traffic rules by priority within network (#10688)
* feat(xo6): displaying traffic rules by priority within network

* update changelog

* clean code
2026-11-09 09:51:51 +01:00
Joris K
8cc3384850 feat(xo6/network): add dedicated network page (#10704) 2026-11-09 09:39:47 +01:00
Olivier Lambert
e4cc8df323 feat(xo6): compact treeview (#10727) 2026-11-06 19:45:39 +01:00
Joris K
43a7e3d3ac feat(web-core): adapt card rows to their content in side panels (#10733)
* feat(web-core): adapt card rows to their content in side panels

- Key and value stay on one line when they fit, the value moves below the key otherwise
- New `values` prop on `VtsCardRowKeyValue`: one copy button per value, a "Copy all"
  action and an optional "see more" limit for fields with multiple values
- New `VtsCardRowList` component holding the 0.8rem spacing between rows
- `UiInfo` text is small inside side panels (icon stays medium), medium elsewhere
- Side panel links use the small size

* docs: add changelog entries
2026-11-06 18:38:09 +01:00
Joris K
2e586f0369 feat(web-core): make the side panel resizable (#10725)
* feat(web-core): make the side panel resizable

- Resize the side panel like the treeview, with a shared `VtsResizeHandle`
  component and `useResizableWidth` composable
- Default width of the treeview and the side panel is 1/5 of the screen
  (280px minimum), max width is 1/3 of the screen (400px minimum)
- Double-click (or Enter) on the handle restores the default width
- The handle is keyboard accessible (arrows, Home, End) and exposed as an
  ARIA separator

* docs: add changelog entries
2026-11-06 18:27:57 +01:00
Bastien Nollet
93dc079f71 docs: remove british spellings (#10724) 2026-11-06 16:58:11 +01:00
Joris K
9c6a59bb89 feat(web-core): show UiButton loader on the right of the label (#10723) 2026-11-06 12:57:52 +01:00
Thomas Moraine
12aa9576b9 docs: use US spelling for licensing (#10666) 2026-11-06 10:21:31 +01:00
Joris K
85541ce5d7 fix(xo6): Fix some issue in the pool tab (#10714) 2026-11-05 15:42:09 +01:00
Joris K
56a33c1639 fix(xo6): change padding tasks (#10713) 2026-11-05 14:49:03 +01:00
Olivier Lambert
7f6ea616ca feat(docs): add optional email to page feedback (#10697) 2026-11-04 16:58:50 +01:00
spacotte-vates
5601a64703 feat(complex-matcher): use complex matcher for backup jobs (#10064)
Use complex matcher for healtcheck and full backup interval in backup jobs

(XO-2435)
2026-11-02 17:49:16 +01:00
MlssFrncJrg
a50776cb7d feat(lite): 0.27.0 (#10690) 2026-10-29 17:34:12 +01:00
MlssFrncJrg
b477167b41 feat: release 9.10.0 (#10689) 2026-10-29 16:52:04 +01:00
Alina D
909fa94161 feat(lite): add possibility to edit a VIF (#10505) 2026-10-29 16:23:38 +01:00
MlssFrncJrg
79e88245da feat: technical release (#10683) 2026-10-29 10:05:20 +01:00
Pierre Donias
4802815bc3 fix(web): move runtime deps to devDependencies (#10678)
Introduced by #10450 and #10631
2026-10-29 09:23:42 +01:00
MlssFrncJrg
7985244daf feat: technical release (#10677)
* feat(@vates/iscsi): 0.4.0

* feat(@vates/types): 2.5.0

* feat(@xen-orchestra/backups): 0.78.0

* feat(@xen-orchestra/backups-cli): 1.1.18

* feat(@xen-orchestra/disk-cli): 2.2.5

* feat(@xen-orchestra/immutable-backups): 3.0.9

* feat(@xen-orchestra/web-core): 0.65.0

* feat(@xen-orchestra/mixins): 0.23.0

* feat(@xen-orchestra/proxy): 0.34.0

* feat(@xen-orchestra/vmware-explorer): 1.2.0

* feat(@xen-orchestra/qa-test): 1.4.0

* feat(@xen-orchestra/rest-api): 0.41.1

* feat(@xen-orchestra/web): 0.65.0

* feat(xo-server): 5.213.0

* feat(xo-server-sdn-controller): 1.6.0

* feat(xo-web): 5.208.0

* chore(CHANGELOG): update next
2026-10-28 16:02:08 +01:00
amouillard-vates
406f4436c6 fix(xo-lite/PIF): update forget PIF modal message (#10676) 2026-10-28 13:25:57 +01:00
MlssFrncJrg
1609d2f825 feat(xo6/xo-lite): add console clipboard to VM and host consoles (#10625) 2026-10-28 11:30:53 +01:00
Florent BEAUCHAMP
a953ca748d fix(web-core/console): stop reconnecting the console on every VM update (#10672)
`VtsRemoteConsole` watched `[consoleContainer, …]` where `consoleContainer`
comes from `useTemplateRef`, i.e. a shallow ref. Vue forces a multi-source
watch to run its callback when one of the sources is a shallow ref, even if
no value has changed. The parent page builds a new (identical) `URL` object
on each VM update, so every SSE update tore down and recreated the VNC
connection.

This was mostly visible while a VM boots (Windows especially), because the VM
gets updated every few hundred milliseconds then.

Watching `() => consoleContainer.value` restores the normal value comparison.
2026-10-28 11:12:49 +01:00
Elise-F
5385f9f229 feat(xo6/VM): migrate action (#10615)
See XO-1758

The migration drawer has two steps:
1. Select the destination host (and optionally the migration network).
2. Map the VM disks to SRs and the VM interfaces to networks on the destination.
2026-10-28 10:59:25 +01:00
Killian
092133f811 feat(sdn-controller): traffic rules ordering (#10525) 2026-10-28 10:19:02 +01:00
spacotte-vates
54c52e8f24 feat(cli): Add disable-remote command (#10587)
* fix(remotes): improve unresponsive remote handling
2026-10-28 10:05:11 +01:00
OlivierFL
9e1fc78f84 feat(web-stack): complete translations of locales below 70% (#10641) 2026-10-28 09:24:55 +01:00
Florent BEAUCHAMP
e2ca66130c feat(live-mount): optional local VDI cache, making the mount read/write (#10434)
* refactor(mixins,proxy,xo-server): pass the XAPI names as one xapiLabels object

The SR name was a lone `nameLabel` threaded through xo-server, xo-proxy and
the mixin; the VDI names are about to join it. No change in behavior: the
proxy API still takes `nameLabel`.

Also export OC_MOUNT from _utils.mjs, the cache module will need it.

* feat(iscsi): FileBlockDevice can serve a block device node

With `size`, the path must be a block device (e.g. a VDI hot-plugged onto the
appliance, /dev/xvdX): its inode reports a size of 0, so the capacity comes
from the caller, and a short read is an I/O error instead of a sparse tail.

Also add unit tests for FileBlockDevice, which had none, and integration
tests against a loop device.

* feat(iscsi): advertise read-only LUNs as write protected

A read-only BlockDevice now sets the Write Protect bit of MODE SENSE, so the
guest mounts the disk read-only instead of failing on its first write. This is
the case of a live mounted disk, served by a read-only DiskBlockDevice.

* feat(live-mount): name the SR and VDI after the backed up VM and restore point

The SR is now `[XO live mount] <VM> (<backup date>)` and the VDI keeps the
name of the backed up disk, so a live mounted disk reads like the disk the
user knows. The VDI is also tagged with `xo:live-mount`, like the SR.

The proxy API takes them as `xapiLabels`; `nameLabel` is still accepted from
older XOs.

* feat(live-mount): optional local VDI cache, making the mount read/write

Without a cache, a live mounted disk is read-only and every read goes to the
backup repository. With `cacheSrId` (xo-server) / `cacheSr` (proxy API), the
disk is materialized block by block into a VDI of that SR, hot-plugged onto
the appliance serving the mount (XOA or the proxy):

- a block is read from the backup repository at most once;
- the LUN is read/write, writes land in the cache VDI and the backup is never
  modified;
- the VDI is created with the mount and destroyed with it: what was written
  is lost on unmount.

The appliance must be a VM of the target pool, and the SR plugged on the host
running it. On a local SR, the appliance must not be migrated while the mount
lasts, since the cache VDI cannot follow it: this is documented, not enforced.

The cache VDI is tagged `[NOBAK] [NOSNAP]` to stay out of the appliance's own
backups, and the SR of a cached mount is named `[XO live restore] …`. The
cache device is opened with retries, since its node may lag behind the
hot-plug.
2026-10-28 09:00:03 +01:00
vates-bot
d618707ac3 feat(web-stack/i18n): update Xen Orchestra 6 (#10464) 2026-10-28 08:53:29 +01:00
OlivierFL
2685f3b95c feat(xo-6/pool-dashboard): refactored duplicated progress bar groups components (#10386) 2026-10-27 18:14:02 +01:00
Marie Garde
4cfe9e81bd feat(xo6): refacto backup folder in backup-job and backup-log (#10636)
* refacto backup job

* refacto ok

* refacto for PR

* rebase ok

* fix bild
2026-10-27 15:58:38 +01:00
Marie Garde
3359eb898f feat(xo6): add backup job side panel (#10631)
* feat(xo6): add backup job side panel

* wip refacto for merge

* refacto for merge

* add tests

* refacto ok

* last refacto

* fix report when field

* fix post rebase
2026-10-27 15:33:30 +01:00
Marie Garde
1ce24b85bd feat(xo6): Add backup job list (#10630)
* add table

* cards info and schedule

* card last 3 runs ok

* add backed up VMs

* add card backup target

* setting card ok

* add cron next schedule

* add warning 2 schedule

* fix style recipient list

* fix style cron

* split into 2 branchs

* refacto for PR

* add tests

* refacto for merge

* last refacto
2026-10-27 14:55:48 +01:00
Joris K
c234800468 feat(lite): add host side panel in pool page (#10580) 2026-10-27 14:21:44 +01:00
Pierre Donias
a846ae68dd feat(xo6/vm): warn that deleting a VM deletes its disks (#10554) 2026-10-27 12:17:18 +01:00
Joris K
666a99706e feat(xo6): add page plugins (#10593)
* feat(xo6): add page plugins

* improve error summary component

* plugin configuration form review

* fix(xo6): adapt plugins to master after rebase

- FrontXoPlugin keeps the plugin types of @vates/types (strict union since #10588)
- tests follow the state checks, task params and loading state of the master backend

* chore(xo6): add plugins page to the changelog

* fix(xo6): pluralize the examples count and unprefix generic plugin i18n keys

* fix(xo6): drop JSDoc from the json-schema form

* test(xo6): align the plugin tests with the existing test patterns

Component tests mock the plugin jobs and assert their run and arguments instead of the REST requests, job tests mock fetch.util directly with run/validate blocks, composable tests use a local mockJob, and createPlugin gets the factory JSDoc.

* refactor(xo6): name the json-schema form conditions and share the object schema guard

* refactor(xo6): expose the summary errors from the json-schema form

* refactor(xo6): name the plugin save and test steps

* fix(xo6): use the delete action icon in the plugin delete buttons

* revert(xo6): drop the capitalize prop of UiToggle

* refactor(xo6): remove the go-to-first-error button of the plugin configuration

* refactor(xo6): use VtsDeleteButton in the plugin configuration deletion

* revert(web-core): keep useValidatedForm and useFormSelect as on master

The plugin forms focus their first missing field through useJsonSchemaForm, so the
error summary and focus-on-validate of useValidatedForm had no consumer and changed
the behaviour of the existing forms.

* feat(web-core): use the puzzle icon for plugins and the window icon for templates

* feat(xo6): plugin load and startup actions in the plugins table, panel and header

- Side panel: Load/Unload button + "Enable/Disable load at startup" in more actions,
  loaded / load at startup as statuses, copiable version and description, no bottom link button
- Table: action column (select + quick actions) replaces the select column
- Detail head bar: same actions, PluginStateToggles removed
- One plugin.change-state job replaces the load and unload jobs; updatePluginAutoload
  and isPluginLoaded move to xo-plugin.util.ts
- useXoPluginUtils no longer exposes description
- Shared test helpers: createJobResult, createActionItem, findTooltipContent

* feat(xo6): read-only plugin detail page, configuration and test in drawers

- Detail page: general info + schema-driven read-only configuration card (VtsColumns)
- Configure: primary action in head bar, side panel and table row, opens a drawer
- Test: drawer from the more-actions menus; delete configuration in the same menus
- Errors follow the app pattern: the drawer stays open on job failure
- Delete configuration is blocked while the plugin is loaded (the API rejects it)
- Secrets: masked by xo-server's name rule or by the value obfuscated by the API
- web-core: VtsActionItems extracted from VtsActionsMenu for menu-slot reuse

* feat(xo6): show the plugin Configure action first, next to Load

* refactor(xo6): remove the unused ApiErrorAlert, useActionError and api-error utils

* refactor(xo6): de-duplicate the plugin drawer composables and test fixtures

- one useXoPluginDrawerJob replaces the near-identical configure and test composables
- useXoPluginUtils exposes the description and loaded state used by the templates
- shared plugin schema fixture and JSON schema item-label test helpers

* refactor(xo6): de-duplicate the JSON schema fields and the plugin tests

- JsonSchemaInputWrapper holds the input wrapper, message and validation of the input, select and XO object fields
- JsonSchemaField wraps the object and array fields in their fieldset, which owns the optional value
- shared test helpers for drawers, plugin actions, titles and JSON schema forms

* refactor(xo6): reuse @vates/obfuscate and cover the JSON schema select fields

- @vates/obfuscate exports isSensitiveParam and ships its types; XO 6 imports it and OBFUSCATED_VALUE instead of copies
- tests for the select, XO object field and useXoObjectOptions; drop the select guard that could never run
- JsonSchemaValue titles its sections in a computed and checks the secret mask

* feat(xo6): show the arrays of a plugin configuration as a list

* refactor(xo6): share the plugin route and the item label type, remove the unused required key

* docs(lite): add a story for VtsActionItems

* feat(xo6): validate the numeric limits, integers and list sizes of the plugin forms

The JSON schema form only checked the required fields, so a value under minimum, over maximum, a decimal number for an integer or a list shorter than minItems was only rejected by the server. oneOf is still left to the server.

* test(xo6): share the drawer, button and tag test helpers, test the preset application

Replaces the local submit, button and tag lookups of the backup repository, plugin, host and VM tests with the src/test helpers, makes PluginMoreActions assert its wiring instead of its mock, and documents the new helpers.

* test(xo6): test the plugin form lists, sections, messages and links, extract the property grouping

* fix(xo6): do not test a plugin while it is being loaded or unloaded

* refactor(xo6): give fetchPut and fetchPatch no default type, like fetchPost

* refactor(xo6): identify a missing plugin in the plugin job argument instead of in defineJob

* refactor(xo6): remove the unused disabled prop from the JSON schema form

* refactor(xo6): detect the plugin secrets in the front only, leave @vates/obfuscate as on master

- the secret property names and the obfuscated value are local to XO 6 again, no @vates/obfuscate export, types or release
- the backup repository and plugin tests import the obfuscated value from the shared constants

* refactor(xo6): use checkboxes instead of toggles in the plugin forms

* test(xo6): get a checkbox by its label with a shared test helper

* test(xo6): reset the state of the array items when the list is replaced from the outside

* refactor(xo6): split the JSON schema form dir into the plugin components, utils, composables and form dirs

---------

Co-authored-by: Olivier Floch <olivier.floch@vates.fr>
2026-10-27 11:42:45 +01:00
Alina D
5dd65197bb feat(xo6): display correct date and time format in all UI (#10547) 2026-10-27 11:01:39 +01:00
Florent BEAUCHAMP
c2bda04112 feat(v2v): enable CBT automatically + use soft shutdown when possible (#10637) 2026-10-27 10:24:16 +01:00
Sebastien Marie
0be905e7cf fix(xo-server-sdn-controller): clean up partial state on network registration failure
When privateNetwork.addNetwork() fails during xapi connection, the network is 
already inserted in privateNetwork.networks and a center may have been set 
without being recorded in _starCenters, leaving a broken PrivateNetwork 
registered.

After the registration loop, remove the network that failed to register, drop 
the orphaned center, and delete the PrivateNetwork once it has no network left, 
like _handleDisconnectedXapi already does.

Also defer pushing to noVniNetworks until the network is registered, so a failed 
network gets no VNI and does not trigger _electNewCenter outside of any 
try/catch.
2026-10-26 17:54:01 +01:00
Sebastien Rodot
fd5b54f972 fix(xo-server-sdn-controller): CHANGELOG.unreleased.md
add CHANGELOG.unreleased.md entry for GPN reliability issues.
2026-10-26 17:54:01 +01:00
Sebastien Rodot
248a6e025c test(xo-server-sdn-controller): Add reliability tests for stale center and unplugged PIF
- handles a stale preferred-center host gracefully
- handles PIF unplugged when center is undefined
2026-10-26 17:54:01 +01:00
Sebastien Rodot
d4dd0791f2 fix(xo-server-sdn-controller): Guard center.$ref access in _pifUpdated and _hostUnreachable
Avoid to dereference privateNetwork.center without checking it first.
2026-10-26 17:54:01 +01:00
Sebastien Rodot
1329b7bc69 fix(xo-server-sdn-controller): Wrap per-network processing in try/catch
Wrap the entire per-network registration body (after the privateNetwork creation 
check) in a try/catch so one network's failure doesn't affect others.

Reorder `xo:sdn-controller:vni` to after `privateNetwork.addNetwork()` call to 
manage more easily the undo.
2026-10-26 17:54:01 +01:00
Sebastien Rodot
b9fa292b9d fix(xo-server-sdn-controller): Wrap preferred-center lookup in try/catch
The value stored for preferred-center could be wrong or out-of-date. If no XAPI 
object exists with such uuid (the host designed as preferred center was 
removed from XAPI), avoid to hard fail, and consider the value is absent.
2026-10-26 17:54:01 +01:00
Mathieu
767fc2cb76 fix(xo-server): fix cannot read properties of undefined (#10668)
Introduced by e58587aa1d
2026-10-26 11:02:03 +01:00
Bastien Nollet
d2ffe1ad56 feat: technical release (#10667) 2026-10-23 12:04:08 +02:00
Bastien Nollet
a508ca7897 fix(xo-web): limit concurency on mass orphan VDI deletion (#10663) 2026-10-23 10:28:26 +02:00
Mathieu
96e789c35e feat(@xen-orchestra/web): backup-log and backup-archive collection use SSE (#10604) 2026-10-23 10:18:41 +02:00
amouillard-vates
f013adf7e4 feat(lite): add ability to forget a PIF from the network table (#10239) 2026-10-23 10:09:45 +02:00
Bastien Nollet
3923756ff5 fix(backups): delete orphan VDIs (#10546)
XO-2839
2026-10-22 12:16:03 +02:00
Killian
988d466cba feat(backups) : sets the backup archives as a collection and streams changes as they happen (#10567)
The `backup-archive` SSE collection (#10472) only announced a change when a client listed the
repository: XO never read one on its own, so a subscriber learnt of a backup created, merged or
deleted at the next listing by any client, and after a restart heard nothing from a repository
until a client listed it.

XO now reads the repositories itself:
- every enabled repository is listed once XO has started, one after another, and a repository
  enabled later as soon as `updateRemote()` has stored it;
- the changes this XO makes are pushed at once, for a repository whose journal can be replayed: at
  the end of each VM export to it (worker, no-worker mode and proxy alike), at the end of the job
  and after a deletion. A read in progress when `refresh()` is called reads again, since it may
  have missed the change, at most 3 times and never when that means listing the repository in full;
- the other changes (the size the merge worker rewrites, another XO sharing the repository) are
  read by two timers, 2 repositories at a time and through the listing backoff: every minute from
  the journal, every 10 minutes by a full listing for a repository whose journal cannot be read,
  e.g. behind a proxy which does not expose it;
- a repository is listed in full again an hour after its entry was built, rather than when it
  crosses a UTC day, which bounds the drift from what the journal does not record. The
  `fullListingInterval` parameter of its URL sets this delay, in whole minutes; the XO 6 and XO 5
  repository forms now keep it when the repository is edited.

The archives are registered through `registerCollection` like any other collection, which drops
the REST API special case for them. Their events now carry an array of archives (`add`, `update`)
or of ids (`remove`); SSE subscribers still receive one event per archive.
2026-10-22 11:41:10 +02:00
Thierry Goettelmann
e0c9112429 feat(web-core): add UiMenuItem component (#10642) 2026-10-22 10:53:45 +02:00
Marie Garde
61f3cbc9ce feat(xo6): add multi edit BRs (#10555)
* rebase ok

* tests ok

* fix build
2026-10-22 10:05:17 +02:00
Marie Garde
eac0b1e2c9 feat(xo6): add actions on BRs (#10532)
* feat(xo6): add actions on BRs

* change monitorTask

* actions jobs ok

* fix monitorTask and add actions buttons on detail and side panel

* action bar wip

* table actions change state ok

* multi test speed ok

* add type to confirm modal

* forget ok and create new component bulk

* fix tests

* refacto for PR

* refacto for PR

* refacto for PR ok

* refacto for merge

* add and update tests

* last refacto
2026-10-22 09:55:46 +02:00
Thierry Goettelmann
5a384d60c4 fix(web-core): show state hero illustrations in color in Safari and Firefox (#10639)
Introduced by a883cef0d6
2026-10-21 15:21:02 +02:00
Florent BEAUCHAMP
bc927fdffb fix(xo6): wrong ETA for task , show transferred size for backup and v2v tasks (#10622)
* feat(xo6): add size on task quick view
* feat(v2v): show transferred size in task
2026-10-21 13:46:00 +02:00
Elise-F
7c3f861a98 feat(rest-api): add bypassCpuCheck parameter to VM migrate action (#10614) 2026-10-21 11:29:46 +02:00
Joris K
0ebc92b50f feat(rest-api): add endpoint to plugins (#10588)
* feat(rest-api): add endpoint to plugins

* add changelog

* review

* mathieu's review

* remove patch endpoint

---------

Co-authored-by: MathieuRA <contact@mathieu-raisin.fr>
2026-10-21 11:25:06 +02:00
Florent BEAUCHAMP
57282c0457 feat(backup/incremental mirror): don't recompress block if the compression is the same on source and destination (#10581)
* feat(backup/incremental mirror): don't recompress block if the compression is the same on source and destination
2026-10-21 11:07:43 +02:00
Mathieu
ee54d78259 docs(rest-api): add some guideline for stream endpoint (#10628) 2026-10-21 10:03:43 +02:00
Florent BEAUCHAMP
10208cbfe0 feat(vhd-lib): add fast custom compression removing only zero chunks (#10557)
typical performance of this filter is 1% cpu at 1GB/s, compression level is 20-40% size reduction depending on data
2026-10-20 15:56:20 +02:00
Florent BEAUCHAMP
6f5caa409c feat(backup): support zstd compression for vhd blocks (#10545)
XO-3133
2026-10-20 15:21:47 +02:00
Florent BEAUCHAMP
2233bad7a7 fix(qa-test): don't try to export disk > 2TB in vhd (#10607)
* fix(qa-test): don't try to export disk > 2TB in vhd
2026-10-20 14:17:45 +02:00
vmpr
b630456835 fix(xo-server-transport-icinga2): send credentials via auth instead of URL (#10594)
Fixes #10498
Introduced by 3293e69cc2
2026-10-20 12:48:35 +02:00
MlssFrncJrg
c77d22f422 feat(xo-server): persist backup repository size on the remote object (#10406) 2026-10-20 09:48:00 +02:00
Mathieu
191317630d feat(vectura): read disks from ESXi 6.x hosts (#10590) 2026-10-19 16:50:18 +02:00
ayoub-el-kajji-v
d32c9879b1 fix(hosts): multipathing should not require disconnecting srs on local storage (#10592)
XO-2336
2026-10-19 16:07:07 +02:00
Thomas Moraine
ab4d372ce2 docs(v2v): replace VDDK screenshots with vectura ones (#10620)
Update the prerequisite check and transfer form screenshots, and remove the unused VDDK/nbdinfo ones.
2026-10-19 14:23:11 +02:00
Pierre Donias
5cc6500def fix(lite/vm-creation): fix MAP_DUPLICATE_KEY on cores-per-socket (#10552)
Introduced by #8491
See #10201 (same workaround)

`VM.set_VCPUs_at_startup` ran concurrently and re-added `cores-per-socket`
between its removal and its re-addition, making the VM creation fail.
2026-10-19 14:14:05 +02:00
Olivier Lambert
8a70c4abc0 docs(xoa): align sizing guidance with the deploy form presets (#10617)
The deploy form now offers Small (2 vCPUs, 4 GiB), Medium (4 vCPUs,
8 GiB) and Large (8 vCPUs, 16 GiB, default) presets. Update the sizing
table to match, and drop the manual --max-old-space-size procedure:
since 6.8, xo-server sets its heap limit to 70% of the VM RAM.

Signed-off-by: Olivier Lambert <olivier.lambert@vates.tech>
2026-10-19 13:28:32 +02:00
Elise-F
1639a5f6a6 fix(xo6): opening console VM in new tab (#10616) 2026-10-19 11:35:34 +02:00
Olivier Lambert
d525a7d5bc feat(@vates/iscsi): bound what a peer can make the target do (#10566)
* fix(@vates/iscsi): only replace the session once the newcomer has logged in

A new TCP connection used to tear down the active session as soon as it
was accepted, before sending anything. Any peer able to reach the port,
a port scan, a failed CHAP login or a Discovery session, interrupted the
disk in use.

New connections are now kept aside until they establish a Normal session
(CHAP included when configured), and only then replace the active one.
That keeps the ability to recover from a stale connection, which is why
connections are replaced rather than refused.

Connections without a session are bounded in time (`loginTimeoutMs`,
15 s by default, also closing lingering Discovery sessions) and in number
(`maxPendingConnections`, 8 by default, oldest dropped first).

* chore: add PR number to changelog

* fix(@vates/iscsi): refuse extra connections and guard the login race

Review follow-up:
- let net.Server refuse connections past maxPendingConnections (+1 for
  the one serving the LUN) instead of evicting the oldest pending one
- a connection destroyed while its final Login Response is being written
  must not go on and replace the current session

* feat(@vates/iscsi): bound what a peer can make the target do

- allowedAddresses: close connections from other initiators as soon as
  they are accepted, so that they cannot take the pending connection
  slots and block a legitimate login
- reject a PDU whose data segment exceeds 8 KiB during login, or the
  declared MaxRecvDataSegmentLength afterwards, before reading it
- maxTransferLength (8 MiB by default): advertised in the block limits VPD
  page, longer READ/WRITE refused, which also bounds the WRITE buffer
- BlockDevice.readOnly: report the LUN write-protected and refuse WRITE
  with DATA PROTECT before soliciting any data; DiskBlockDevice sets it

* chore: add PR number to changelog

* feat(@vates/iscsi): rename allowedInitiatorAddresses, report refused connections

`allowedAddresses` did not tell whose addresses: these are the source
addresses of the initiators. `onConnectionRefused` reports each refused
one, e.g. so that a caller can tell that an initiator connects through a
NAT.

---------

Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2026-10-19 10:03:00 +02:00
Mathieu
776993e119 fix(vates/types): fix vm and server type (#10598)
XO-3258
fix #10514
introduced by 0ef95d9 and b078d3d
2026-10-19 09:59:23 +02:00
Pierre Donias
8ae4ea25b2 feat(netbox): support Netbox v4.7.x (#10574)
Fixes #10565
See https://github.com/netbox-community/netbox/releases
2026-10-19 09:21:11 +02:00
Thomas Moraine
eecba81f22 docs(readme): replace XO 5 screenshots with XO 6 ones (#10600)
The last screenshot (SR statistics) is kept from XO 5, as XO 6 has no equivalent stats view yet.
2026-10-17 11:51:40 +02:00
Florent BEAUCHAMP
05aeff101b fix(immutable-backups): fix test not waiting for immutability (#10606) 2026-10-16 17:09:48 +02:00
spacotte-vates
b7af59b6ed feat(events): improve xo5 event handling (#10568)
XO-3152

Improves XO5 event handling

Event improvements - Front end test.md

Event improvements - Performance impact.md
2026-10-16 16:38:04 +02:00
Thierry Goettelmann
12e1a10394 fix(@xen-orchestra/acl): drop node:assert dependency (#10608)
node:assert is not available in the browser, so importing @xen-orchestra/acl
from a front-end bundle failed. Input checks now throw plain TypeErrors.
2026-10-16 14:24:08 +02:00
Killian
954a2034e6 test(@vates/iscsi): wait until the open-iscsi disk can be opened (#10576)
The CHAP interop test failed on CI with ENXIO on /dev/sdb (master 641f98878,
and #10549), right after waitForDevice() returned the disk.

waitForDevice() took a nonzero /sys/class/block/<name>/size as proof the disk
was ready, but the sd driver sets the size before device_add_disk(), which
creates the sysfs entry, then the /dev node, then registers the block device:
open() fails with ENOENT or ENXIO until then. Polling sysfs in a tight loop
after login hit that window in 15 of 100 logins.

isDeviceReady() now opens the device, and treats ENXIO and ENOENT as not ready
yet.
2026-10-16 11:35:10 +02:00
Thierry Goettelmann
b819ba1527 feat(web-core): make query builder model live and add search event (#10602) 2026-10-16 11:02:49 +02:00
Mathieu
ff90a80e99 feat(rest-api): expose backup-log in the SSE (#10563)
* feat(rest-api): expose backup-log in the SSE

* noop instead undefined

* fix: handle sequance

* changelog

* fix

* final fix

* emit remove event from the gc

* Kilian's feedback

---------

Co-authored-by: Killian <killian.allegrain@vates.tech>
2026-10-16 10:59:12 +02:00
Florent BEAUCHAMP
cfa75ef652 feat(xo6,rest-api): export vdi as vmdk (#10591) 2026-10-16 10:57:42 +02:00
Florent BEAUCHAMP
a350c62cc3 feat(fs, backups): don't concatenate buffers but write them directly (#10538)
* feat(fs, backups): don't concatenate buffers but write them directly
maximum throughput with a perfect source and no write goes from
4.5 to 9.6GB/s

* fix: remove Buffer concat in qcow2 stream generationonly needed for last block
2026-10-16 10:57:00 +02:00
Florent BEAUCHAMP
bb5a79e6c6 feat(xo-server, xo-web): support insecure nbd (#10540)
* feat(nbd-client): support nbd insecure networks
* feat(xo-web): support insecure nbd

also add a confirm on insecure enabling
The dialog is titled "Enable insecure NBD?" and says: "The disk data of the backups will travel unencrypted on this network. Only enable it on a network dedicated to backups and isolated from the VMs and the users (dedicated VLAN or physical network)." Both texts are new entries in messages.js, and the dialog itself is a new confirmInsecureNbd() helper in xo/index.js.

It appears in two places:

Pool Network tab (tab-network.js):
The dialog only appears when switching a network to insecure NBD. Reselecting it on a network that already has it, or switching to "NBD" or "No NBD", doesn't ask.
On cancel, nothing is sent and the drop-down keeps showing the network's current setting.
Network creation form (new/network/index.js):
The dialog appears on "Create" when insecure NBD is selected (bonded and private networks are never affected).
On cancel, the network isn't created and the form stays as is, with no error and no redirect.

* feat: update xo5,xo6 and lite ui for this settings

---------

Co-authored-by: MlssFrncJrg <119158464+MelissaFrncJrg@users.noreply.github.com>
2026-10-16 10:45:54 +02:00
Pierre Brunet
e4e7c05104 fix(fs/s3/azure): better handling of NotFound in _getSize and _unlink (#10548)
* fix: handle error codes

* chore: update changelog

* fix: apply suggestions from code review

Co-authored-by: spacotte-vates <simon.pacotte@vates.tech>

* fix: small typo

---------

Co-authored-by: spacotte-vates <simon.pacotte@vates.tech>
Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@vates.tech>
2026-10-15 18:16:51 +02:00
Marie Garde
52c68a28cc feat(xo6): add edit form BR (#10478)
* feat(xo6): add edit form BR

* type disable

* wip

* fix post rebase

* form initial data ok

* form ok

* add tests dans finalize edit

* refacto for PR

* refacto for RP

* rebase ok

* feedback

* PR ok

* add changelog
2026-10-15 17:26:44 +02:00
Marie Garde
635f02bcae feat(xo6): Add backup job list (#10572) 2026-10-15 16:39:32 +02:00
Olivier Lambert
db5cb01fc9 fix(xo-server): cancel the disconnection grace period when the connection drops (#10577) 2026-10-15 16:15:00 +02:00
Florent BEAUCHAMP
928408fe8e feat(vmdk): first version of the package with streamoptimized export (#10403) 2026-10-15 15:14:34 +02:00
Mathieu
db98f15bc1 docs(rest-api): add a naming section in guidelines (#10571) 2026-10-15 13:58:39 +02:00
Mathieu
decdf61919 fix(vectura): update rustls to 0.23.46 (#10586)
cargo deny fails on master since RUSTSEC-2026-0346, 0347, 0348 and 0349
were published against rustls 0.23.45. 0.23.46 fixes all four.

vectura has no custom session store, no ECH and no brotli feature, so
only 0349 (later connections downgraded to classical key exchange through
the resumption cache) applies, in theory.

vectura 0.1.3, bumped and rebuilt with scripts/release.sh patch.
2026-10-15 11:26:00 +02:00
Thierry Goettelmann
6115c74bbe fix(web-core): keep an open select's Escape from closing its overlay (#10582) 2026-10-15 10:48:37 +02:00
Mathieu
025f6eb533 feat(vectura): bump and build in the pull request, not at release (#10573)
The binary in git is now the build of its sources. build.sh keeps the paths
of the machine out of the binary and refuses another zig or cargo-zigbuild
version, and the Vectura workflow fails when its build differs from the
committed binary. The binary is rebuilt with this build.sh: only its paths
change, the version stays 0.1.2.

vectura is no longer released on its own. release.sh runs once at the end of
the pull request which changes vectura and no longer commits. bump-pkg,
gen-deps-list.js and check-changed-pkgs.js drop the vectura pseudo-package: a
change under vectura/ requires a release of @xen-orchestra/vmware-explorer,
which ships the committed binary as is. release.sh puts Cargo.toml and
Cargo.lock back when the build or the check of the binary fails.
2026-10-15 10:43:00 +02:00
Thierry Goettelmann
3e74189479 fix(web): generate route-map.d.ts only in development mode (#10583) 2026-10-15 10:34:37 +02:00
Thierry Goettelmann
0695c6b44c fix(web-core): hide the focus outline of readonly inputs on Firefox (#10584) 2026-10-15 10:31:29 +02:00
Bastien Nollet
0bb2f12b2b fix(xo-server): avoid sequence abort on error with proxy (#10504) 2026-10-14 12:30:05 +02:00
Thierry Goettelmann
865b6a84c7 fix(web-core): use solid border and body typography in UiTextarea (#10579) 2026-10-14 09:34:13 +02:00
Thierry Goettelmann
6b82340fdc fix(web-core): balance remote resource subscriptions on url and isEnabled changes (#10473) 2026-10-14 08:59:26 +02:00
Marie Garde
d846e00dff feat(xo6): BR add detail page (#10454)
* feat(xo6): BR add detail page

* Prepare PR

* add tests

* fix breadcrum

* fix post rebase

* fix admin path

* fix link side panel

# Conflicts:
#	@xen-orchestra/web/src/modules/backup-repository/components/list/panel/cards/BackupRepositoryInfosCard.unit.test.ts
#	@xen-orchestra/web/src/modules/backup-repository/components/list/panel/cards/BackupRepositoryInfosCard.vue

* feedback

* feedback

* rebase ok
2026-10-14 08:50:18 +02:00
Florent BEAUCHAMP
f93bb67409 fix(xen-api): really throw import error while importing VM (#10551) 2026-10-13 17:13:44 +02:00
Florent BEAUCHAMP
51bb88a1ce feat(vmware-explorer): method to list/attach/detail disk to satellites (#10449)
* feat(vmware-explorer): implement attaching and detaching disks
* feat(vmware-explorer): implement a check to see if a disk can be attached to a VM
* feat(vmware-explorer): implement a method to detect what host are candidates to deploy a satellite
* refactor: computing the datastore of a datapath
2026-10-13 13:58:23 +02:00
Pierre Donias
11335b465b fix(xo-web/self): fix removing a tag from a resource set (#10550)
Fixes Zammad#65918
2026-10-13 11:05:35 +02:00
Killian
295b8838a6 docs(backups): explain when reverse replication updates the VM in place (#10558) 2026-10-13 10:01:20 +02:00
Pierre Briffaux
e0b0c75682 feat(xo6): add copy button for multiple information (#9976) 2026-10-13 09:16:35 +02:00
Thomas Moraine
286dafa267 docs: document warm migration and fix broken link in architecture page (#10570) 2026-10-12 18:20:37 +02:00
Thierry Goettelmann
b08ad1669f feat(web-core): add UiLinkList component (#10287) 2026-10-12 17:36:54 +02:00
Thomas Moraine
4731cbd629 docs(advanced.md): document estimated per-VM power consumption metric (#10569) 2026-10-12 17:13:27 +02:00
Florent BEAUCHAMP
1c4890be10 fix(xo-cli): error when using output on a rest api query (#10541) 2026-10-12 16:59:56 +02:00
Pierre Donias
4abb9d80bc feat(xo-web/patches): warn when pool master is not up to date (#10523) 2026-10-12 16:24:18 +02:00
Mathieu
87fe4e2b53 docs(rest-api): enhance REST API guideline (#10416) 2026-10-12 15:37:17 +02:00
ayoub-el-kajji-v
6058f07813 feat(api-rest): add sr probe endpoints (#10099) 2026-10-12 14:43:07 +02:00
Sebastien Rodot
d451d5c35b feat(xo-server-sdn-controller): add unit tests
- check that objects listeners are properly attached/detached on Xapi connect/disconnect
- check that xapi instance doesn't leak on reconnect
2026-10-12 14:34:55 +02:00
Sebastien Rodot
375b9fe91b fix(xo-server-sdn-controller): avoid leaking listeners on disconnect
Objects listeners are added to Xapi objects by the sdn-controller plugin, but 
they are only removed at plugin `unload()` time.

Prefer to remove them at Xapi disconnect time.
2026-10-12 14:34:55 +02:00
Sebastien Rodot
d53294f15c fix(xo-server-sdn-controller): re-watch XAPI events after reconnect
When a pool's XAPI connection drops and reconnects, xo-server destroys the
old Xapi instance and creates a new one (same pool UUID, same XAPI object
$refs). The plugin tracked managed pools by pool UUID, so _manageXapi()
short-circuited on reconnect and never attached its add/update/remove
listeners to the new Xapi instance.

From then on the plugin was blind to XAPI events: re-applying VIF traffic
rules on VM start, handling VIF/PIF plug and star-center election all
stopped silently until the plugin was reloaded.

Track the managed Xapi instances instead of pool UUIDs, and drop a dead
instance on disconnect so its replacement is managed again.
2026-10-12 14:34:55 +02:00
Sebastien Rodot
40e5ca9a73 fix(xo-server-sdn-controller): recreate stale OpenFlow controllers
When a XAPI connection drops and reconnects, xo-server creates a new
Xapi instance while the XAPI objects (and their $refs) are kept. The
ofChannels cache still held the controllers of the previous
connection, so traffic rule add/delete calls were silently skipped
without reaching the host.

Rework _getOrCreateOfChannel() to key the cache on the Xapi instance:
entries from a previous connection are dropped and the controller is
recreated, concurrent calls share the same instantiation, and a
failed instantiation clears the cache.

Also drop the cached controller when the host is disabled or its XAPI
is not connected, so it is recreated as soon as it comes back.
2026-10-12 14:34:55 +02:00
ayoub-el-kajji-v
cf760d26cd feat(backup): support ndb through http proxy (#10544) 2026-10-12 11:23:29 +02:00
Thierry Goettelmann
d4b85421fe fix(web-core): make query builder usable inside a form (#10475) 2026-10-12 09:59:46 +02:00
Thierry Goettelmann
61a06e817b feat(web-core): add queryFilter rule and backTo to multi-step form (#10477) 2026-10-12 09:31:32 +02:00
Thierry Goettelmann
1b1ae506b6 feat(web-core): add checkbox column to host, SR and VM column sets (#10485) 2026-10-12 09:28:58 +02:00
Thierry Goettelmann
641f988781 feat(web-core): add ignoreRoute option to table pagination and query builder filter (#10476) 2026-10-12 09:27:43 +02:00
Thierry Goettelmann
b6b94a80b1 refactor(xo6): extract VM, host and SR query builder schemas into composables (#10488) 2026-10-12 09:25:17 +02:00
Florent BEAUCHAMP
70b0824dc4 feat(backups): improve full mirror backup performances (#10537)
XO-3135 and XO-3134
2026-10-09 19:04:29 +02:00
Marie Garde
4a9d434559 feat(xo6): add emergency shutdown button host (#10412) 2026-10-09 08:42:23 +02:00
Alina D
341bd0301d feat(xo6/new-vm-form): allow resizing existing template disks when a … (#10359) 2026-10-08 20:13:29 +02:00
Pierre Donias
70be55528f fix(lite): do not reuse window.origin protocol and port when overriding master host (#9955)
When overriding the pool master with `?master=<host>`, XO Lite kept the page's
protocol and port and only swapped the hostname (e.g. `https://proxy:8443` +
`?master=192.168.1.10` → `https://192.168.1.10:8443`). Requests sent directly to
hosts also always used the page's protocol, regardless of the master's.

Everything now follows the XAPI connection (one commit each):

1. **Master override** (`xen-api.store.ts`): the value is parsed as a URL,
   `https://` by default, port and protocol allowed. An invalid value falls back
   to the default host instead of crashing XO Lite.
2. **RRD stats** (`xen-api.ts`, `getResource`): use the XAPI protocol.
3. **VM and host consoles**: `wss:` if XAPI is `https:`, `ws:` otherwise.
4. **Unreachable hosts check**: pings hosts with the XAPI protocol, so the
   certificates modal checks the right URLs.
2026-10-08 16:08:03 +02:00
Joris K
71ff8b0d24 feat(xo6/lite): add shortcut to focus the treeview search (#10492)
* feat(lite): add searchbar in the treeview

* add changelog and improvement

* feedback

* review feedback

* feat(xo6/lite): add shortcut to focus the treeview search

* add changelog

* added mac shortcut

* feedback
2026-10-08 16:03:37 +02:00
Florent BEAUCHAMP
33caeacb49 fix(nbd-client): better handling of reconnection (#10456)
* fix(nbd-client): better handling of reconnection

don't let a failure of an already remplaced trasnport impact the
current transport
2026-10-08 14:19:20 +02:00
Elise-F
854e5dbd89 fix(xo6): add missing condition to the detach job (#10543) 2026-10-08 13:48:27 +02:00
Pierre Donias
c069b7e9c7 fix(lite/vm-creation): keep UEFI firmware when setting boot order (#10521)
Introduced by #8317

`VM.set_HVM_boot_params` replaces the whole map, so setting the boot order for a
network install or a VM without disk dropped the firmware set just before, and
the VM booted in BIOS mode.
2026-10-08 11:18:30 +02:00
Thierry Goettelmann
8338851c4c feat(web-core,web,lite): add UiPanelCardTitle component (#10289)
* feat(web-core,web,lite): add UiPanelCardTitle component

Figma's "Panel title v3" describes three formats — plain text, linked
title, title with an id — which the existing VtsCardObjectTitle only
covered partially, and which several cards had reimplemented inline with
their own flex/gap rules.

Rather than three components, the parts are independent and optional: a
`to` or `href` makes the title a link, an icon or an id is shown when
passed, and `size` replaces Figma's `level`, which described the
appearance in terms of hierarchy.

Adopting it in the side panel cards drops their local title styles, and
brings them back to the 0.8rem title-to-content gap the design system
specifies.

* chore(lite): add the ui-panel-card-title story route

The generated route-map entry was omitted when the story was added.

* fix(web): address review feedback

Task infos and warnings are arrays, so their counters read `length`
directly instead of going through `Object.keys` and a non-null assertion
that only held thanks to the parent's `v-if`.

Add the changelog entries for the side panel card titles.

See XO-2885

* feat(web-core,web,lite): migrate side panel cards to UiPanelCardTitle

Replace the remaining `UiCardTitle` usages in side panel cards with
`UiPanelCardTitle`, so every panel card title shares the same layout.

To cover the cases the old titles handled, `UiPanelCardTitle` gains
`label`, `action` and `more-actions` slots, and tighter spacing between
the title and its id.

See XO-2885
2026-10-08 11:01:20 +02:00
Elise-F
7edeb7a253 feat(lite): add detach action on host (#10520)
* update changelog file

* feedback one

* feat(host): add action reboot on host

* feat(lite): add forget action on host

* feedback

* feat(lite): add detach action on host

* update changelog

* fix modal translations

* add 'host is running' condition to the job and fix translations

* feedback
2026-10-08 10:16:29 +02:00
Mathieu
1a012bdd94 fix(web-core/web): handle SSE for non XAPI objects (#10450) 2026-10-08 09:15:10 +02:00
Sandrine Barrucand
493aa7f15d feat(xo6/pool): add validation to pool connection form (#10484) 2026-10-07 16:40:31 +02:00
Alina D
cb1e45b6b0 feat(xo6): allow removing existing (template) disks in the New VM form (#10292) 2026-10-07 16:02:11 +02:00
Killian
0ff6196e72 feat(xo-server/backups,rest-api): stream changes of the VM backup archives over SSE (#10472)
The cache now announces what a listing finds and what a journal replay
changes, as `add`, `update` and `remove` events carrying the archive and its
previous value, on an emitter the app exposes as `app.vmBackupArchives`. What
was last announced for a repository outlives its entry, so a repository read
again from scratch only announces what really changed. A repository which is
removed or disabled goes through the new `forgetVmBackupRepository()`, which
announces all its archives as removed. A listener which throws is logged and no
longer fails the listing.

The REST API adds `backup-archive` to the SSE collections. It is bound to
`app.vmBackupArchives` the way `task` is, and filtered by `backup-archive:read`
through the same path as every other non-XAPI collection.

Nothing polls the repositories: changes are pushed whenever any client lists a
repository. The first listing after a restart sends every archive it already
holds as `add`.

XO-2992
XO-2993
2026-10-07 11:58:55 +02:00
Bastien Nollet
2ee51a3694 fix(load-balancer): process tags in each pool separately & other fixes 2026-10-07 11:25:38 +02:00
Bastien Nollet
87aca3ff5e feat(load-balancer): smarter migration restrictions 2026-10-07 11:25:38 +02:00
Bastien Nollet
305e66becc feat(load-balancer): vm-to-host affinity 2026-10-07 11:25:38 +02:00
b-Nollet
d3fea1296d feat(load-balancer): various minor improvements 2026-10-07 11:25:38 +02:00
Elise-F
8c4e4d110a feat(lite): add action restart toolstack on host (#10480)
* update changelog file

* feedback one

* feat(host): add action reboot on host

* feat(lite): add forget action on host

* feat(host): Add restart toolstack action for hosts with HA disabled

* feat(icon status): Add a power status icon to the treeview and to the host header

* update changelog files

* fix rebase

* feedback
2026-10-06 15:29:41 +02:00
Florent BEAUCHAMP
cc8f039341 fix(backups): allow a deadlocked merge to recover (#10424)
* fix(backups): properly throw an error if a disk is missing from a chain to merge
but accept it if the merge was interrupted at the cleanup phase
* fix(backups): a merge with 0 or 1 disk is a warning
* fix(backups): don't try to merge chain that are out of retention
this fix will allow broken to recover after a while ( depending on
retention and full backup interval )
* fix: defense in depth against deletion
 every path this class compares it against is normalized (disk paths, alias targets), so
 it must be too: a raw path would e.g. make #cleanOrphanDataFiles fail to exclude the VDI
 directory from the data directories it sweeps, and delete every unclaimed file in it
* fix(backups): better handling of missing disks during a merge
* ensure we never delete ancestries of a differential disk
2026-10-06 15:05:55 +02:00
ayoub-el-kajji-v
59c8935e37 fix(backups): delete first not working in replication (#10452)
* fix(backups): delete old entries before starting a new replication
2026-10-06 14:52:50 +02:00
Thomas Moraine
779e448f21 docs(users): recommend enabling OTP authentication (#10539)
Strongly recommend OTP (TOTP) authentication to mitigate brute force attacks on passwords.
2026-10-06 14:43:39 +02:00
Elise-F
3a64afb226 feat(lite): add enable and disable actions on host (#10337)
* update changelog file

* feedback one

* feat(host): add action reboot on host

* add header state

* feat(lite): add forget action on host

* feat(lite): add enable and disable actions on host

* update changelog file

* fix rebase

* fix rebase

* feedback one

* add redirect after forget action
2026-10-06 13:55:14 +02:00
Marie Garde
cfdf0de9ee feat(xo6): Add create form backup repository (#10271)
* feat(xo6): create and edit backup repository form

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix post rebase

* add tests

* feedback

* feedback

* rebase ok

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 11:39:18 +02:00
Pierre Donias
d6e6e963bf feat(xo-web/ip-pools): show IP format example (#10522) 2026-10-06 11:09:50 +02:00
Pierre Donias
29c7c0e6f1 fix(web-core/remote-resource): clear error after a successful fetch (#10516)
Introduced by #8855

lastError was never reset, so one failed poll kept hasError true and
dashboard cards in error state until reload.
2026-10-06 11:07:05 +02:00
Joris K
72a9959a3c fix(xo6): update modal events to handle confirm actions correctly (#10417) 2026-10-06 11:03:16 +02:00
Sylvère
e5d4c665a4 refactor(web/core): tasks-related refactor for ongoing My Tasks work (#10265) 2026-10-06 10:53:01 +02:00
MlssFrncJrg
0070c7f709 fix(xo-lite/XO6): exclude ISO SR guest tools from selector in VM creation (#10430) 2026-10-06 10:36:47 +02:00
Mathieu
a84584d24e feat(rest-api): POST acl-roles support list of privileges (#10470) 2026-10-06 10:21:35 +02:00
Joris K
786610cba3 feat(lite): add search bar in the treeview (#10414)
* feat(lite): add searchbar in the treeview

* improve open close treeview, and a pool dashboard that displays a scrollbar when the screen is minimized

* add changelog and improvement

* feedback

* feedback

* review feedback

* feedback add story
2026-10-06 10:09:35 +02:00
Mathieu
08e6e1bf49 ci(vectura): only run on the changes of the branch (#10528)
GitHub filters a push on the diff with the previous push of the branch: after
a rebase on `master`, or a merge of it, this diff contains the vectura changes
landed on `master` meanwhile, and the workflow rebuilds vectura on branches
which do not touch it.

A pull request is filtered on the diff with its merge base instead, i.e. the
changes of the branch only. Pushes still run the workflow on `master`.
2026-10-06 09:27:15 +02:00
Elise-F
5999d53e0b feat: release 6.9.1 (#10533) 2026-10-05 17:18:51 +02:00
Elise-F
decc09117d feat: patch technical release (#10529) 2026-10-05 15:47:30 +02:00
Sylvère
d11cf2d2fc fix(xo6): prevent unwanted inherited route params (#10511) 2026-10-05 14:35:02 +02:00
Mathieu
abdddcf105 fix(web/web-core): fix invalid VM-snapshot's VDIs (#10510)
introduced by 57414dafce
2026-10-05 14:01:13 +02:00
spacotte-vates
8c2f38c004 fix(backups): fix synchronized backup snapshop showing as vm in task log (#10518)
* fix(backups): fix synchronized backup snapshop showing as vm in task log
2026-10-05 11:32:42 +02:00
Pierre Donias
5de02b6c19 docs(PULL_REQUEST_TEMPLATE): point to internal review process instead of duplicating it (#10517)
The review process was documented both here and in the internal PR process
doc, and the two had drifted apart. Keep the internal doc as the single
source of truth and only give external contributors what they need.
2026-10-05 09:59:05 +02:00
Florent BEAUCHAMP
440624a09c Fix(backups): incremental distributed replication (#10519)
`AggregatedIncrementalXapiWriter.checkBaseVdis()` didn't forward
`contentKeys` to its sub-writers, so the second run of a multi-SR
replication failed with `Cannot read properties of undefined (reading 'get')`.

Also adds QA tests for distributed replication and copyRetention.

Introduced by 680b81088
2026-10-05 09:54:47 +02:00
ayoub-el-kajji-v
718b348e85 fix(backups): use schedule.id instead of scheduleId in replica retention (#10491)
fix(backups): use schedule from props for replica retention instead of scheduleId
2026-10-02 17:07:55 +02:00
Mathieu
278f7d1706 fix(vectura): build against glibc 2.31 so V2V runs on Debian 11 XOAs (#10513)
XOA appliances still run Debian 11, whose glibc is 2.31. The vectura binary
bundled with @xen-orchestra/vmware-explorer was linked against glibc 2.34,
so it did not start there (`GLIBC_2.32 not found`): the prerequisite check
of the VMware import page failed and V2V imports could not run.

zig now links it against glibc 2.31. Below 2.34, libpthread and libdl are
libraries of their own, so check-binary.sh allows them. The bundled binary
is rebuilt with the same version, and the docs say Debian 11 and Ubuntu
20.04 instead of Debian 12 and Ubuntu 22.04.

Introduced by dc57333f5
2026-10-02 16:34:10 +02:00
ayoub-el-kajji-v
b59c8f3d20 fix(xo-web): disable restat-toolstack for HA_enabled pool hosts in home (#10497) 2026-10-01 17:33:54 +02:00
Joris K
7711b6d6f5 fix(xo6/lite): fix video RAM displayed in bytes instead of MiB (#10486) 2026-10-01 16:40:18 +02:00
Mathieu
b0c0d9361a fix(xen-api): fix HTTP/2 for server behind HTTP proxy (#10507)
Introduced by 11eed70
fix Forum#12502
2026-10-01 14:51:46 +02:00
Mathieu
83fdb2ed1f docs(changelog): vCenter is still supported by V2V (#10502)
The 6.9.0 entry for the VDDK removal said a vCenter address is no longer
supported. vectura reads the disks through a vCenter as well as directly
from an ESXi host; only ESXi 6.x is refused, because its data port does not
announce "SHA256 supported".
2026-10-01 09:47:14 +02:00
Elise-F
c9eaa7a617 feat(lite): 0.26.0 (#10501) 2026-10-01 08:54:52 +02:00
Elise-F
a2bd181d35 feat: release 6.9.0 (#10499) 2026-09-30 18:12:53 +02:00
Elise-F
989cd48559 feat: technical release (#10496) 2026-09-30 15:53:45 +02:00
ayoub-el-kajji-v
b2f0bfc907 fix(xo-web): disable restart toolstack button when HA enabled on pool (#10340) 2026-09-30 14:57:50 +02:00
spacotte-vates
ce26dc2231 fix(backups): prevent failed CR to accumulate replicas (#10404)
* fix(backups): prevent failed CR to accumulate replicas
2026-09-30 14:44:42 +02:00
ayoub-el-kajji-v
2bd4e2a747 fix(xo-server): disable restart toolstack on HA pools (#10344) 2026-09-30 11:31:37 +02:00
Pierre Donias
75098c3cda feat(xo-web/xo-cloud-config): require acknowledgment to enable backup to Vates servers
XO-3030
2026-09-30 11:25:49 +02:00
Pierre Donias
6f091b9380 feat(xo-web/xo-cloud-config): confirm before disabling backup to Vates servers
XO-3031
2026-09-30 11:25:49 +02:00
Elise-F
4b578d439f fix(xo-server): add reactivity on pif metrics (#10438) 2026-09-30 11:13:00 +02:00
amouillard-vates
93e9069614 feat(xo-lite): add network delete action for host internal networks (#10332) 2026-09-30 11:03:53 +02:00
Joris K
c17bcead07 fix(xo6): update transfer task size calculation (#10448) 2026-09-30 10:34:34 +02:00
Pierre Brunet
3dbf8738f1 fix(security): Add safeguard to v2v (#10489)
https://github.com/vatesfr/xen-orchestra/security/advisories/GHSA-mc4m-hc98-rhmv
2026-09-30 10:21:51 +02:00
ludalm13
f187b79319 fix(xo-server): trust forwarded headers from loopback so /v5 proxies … (#10461) 2026-09-30 10:03:34 +02:00
Bastien Nollet
0ae1cf917c fix(xo-web): fix log modal not closing (#10487) 2026-09-30 10:00:22 +02:00
Marie Garde
e287eaeb55 feat(xo6): create page backup repository list (#10247)
* feat(xo6): merge base master

* returns from PR

* fix build

* fix build

* add custom cards side panel

* return form url util

* fix build

* return form PR

* fix rebase

* Remove packages format and parse url

* fix from IA review

* wip rebase

* wip

* opencode review

* add tests utils

* Add changelog

* feedback

* feedback

* feedback

* add tests

* feedback

* feedback

* feedback

* feedback

* fix after rebase
2026-09-29 17:22:24 +02:00
Florent BEAUCHAMP
5a8393aa17 feat(xoa): auto open ufw port for livemount (#10468)
only open the specific route from source to xo / xo-proxy

port openned only live in memory : restarting ufw will drop them, thus making all the live-mount disk unavailable

doc updated
2026-09-29 17:08:38 +02:00
Florent BEAUCHAMP
323f69d773 feat(live-mount): detect disk removing and cleanup sr (#10432)
the live mount feature create a lot of SR , each with 1 disk
this PR introduce an automatic cleanup that will forget a SR as soon as the disk is removed, thus removing any user action needed to cleanup thins in the nominal case

crashing XO can leave SR pluggued
2026-09-29 17:03:57 +02:00
Alina D
a883cef0d6 feat(xo6): replace images with new SVG design (#10380) 2026-09-29 16:26:30 +02:00
Bastien Nollet
d3b3248937 docs: remove "XOA unreachable" troubleshooting section (#10469)
See #9526
2026-09-29 10:30:22 +02:00
Killian
45ca13ba5d feat(xo-server/backups): replay the journal of the repositories on a proxy (#10437)
Phase 4 of the journal-replayed VM backups cache: brings proxy-attached repositories onto the same fast/fresh listing path as directly-attached ones, so a proxy repository is also listed once and then kept up to date by replaying its journal, instead of being re-listed in full on every refresh window.

What changes

RemoteAdapter#readBackupJournalEvents() reduces a journal read to the last event of each backup and resolves the added/changed ones back to their current metadata, so a listing can be brought up to date from a single call. A backup rewritten several times costs one metadata read; one written then deleted costs none.
VmBackupsCache no longer knows how a repository is read: that now lives entirely behind a new VmBackupsSource, which reads a directly-attached repository through the local adapter and a proxy-attached one through backup.listVmBackups (cold build, reused as-is) and the new backup.listVmBackupsJournal RPC (journal replay).
The proxy exposes backup.listVmBackupsJournal, returning its journal events with the backups already formatted, so XO gets a listing update in one round-trip instead of one metadata read per file. A proxy too old to know the method is warned about once and falls back to a full listing, same as before this change; it starts being replayed as soon as it's upgraded, with no restart needed on the XO side.
The cache's staleness check now compares its own clock only against when this process last refreshed an entry (refreshedAt), never against the watermark returned by a journal read (cursor), which is stamped by whichever process wrote it — proxy or not. Comparing the two directly would silently over- or under-refresh a repository depending on clock skew.
An entry also rebuilds from scratch if its repository moves to or from a proxy, or if the source reports it can no longer be replayed (e.g. an old proxy), rather than being purged outright.
The old proxy-only listing path in BackupNg (_listVmBackupsOnProxy, bypassing the cache) is removed: proxy and local repositories now go through the exact same cache.
A single corrupt/unreadable metadata no longer fails a whole journal read; it's kept behind the watermark so only that entry is retried on the next call instead of the caller falling back to listing the whole repository until the file is fixed.
The backup repository cache is now also invalidated on remote update/delete (previously only handled for backup repositories proper).
No behavior change for directly-attached repositories — this phase only extends the existing cache to proxy-attached ones.

XO-2852
2026-09-29 08:58:43 +02:00
MathieuRA
2f846bf07e fix(xo-server): interruptOnDisconnect does't update server error anymore 2026-09-28 19:53:59 +02:00
MathieuRA
ae3e04fb3d fix(xo-server): server status connected only if objects are fetched 2026-09-28 19:53:59 +02:00
MathieuRA
986c6b8687 fix(rest-api): fix SSE event order 2026-09-28 19:53:59 +02:00
Thomas Moraine
386e2afe12 docs(advanced.md): remove outdated 'coming soon' mention (#10378) 2026-09-28 18:30:41 +02:00
Elise-F
d7a75b943b feat: technical release (#10467) 2026-09-28 10:43:56 +02:00
Mathieu
aea3717f81 feat(xo-server,rest-api,xo-web,types/rpu): keep RPU recovery to XCP-ng and XenServer 8.4+ (#10447) 2026-09-28 09:59:27 +02:00
Pierre Brunet
5972cf5c59 fix(vhd-lib): update changelog (#10462) 2026-09-28 09:47:03 +02:00
MathieuRA
2b466d6ac8 fix(perf-alert): fix SR dead link
introduced by 779876ede9
2026-09-28 09:42:52 +02:00
MathieuRA
8ac91f9d7d fix(perf-alert): fix test plugin
introduced by 779876ede9
2026-09-28 09:42:52 +02:00
Killian
f9cafe6a0f fix(backups): actually delete interrupted replication transfers (#10436)
* fix(backups): actually delete interrupted replication transfers on next run

IncrementalXapiWriter#_prepare's "delete previous interrupted copies" cleanup
was dead code: `vm => vm.$destroy` never called the method, and it filtered
with `srUuid: undefined`, which can never match since every replica has had
REPLICATED_TO_SR_UUID set since c610951450. So a transfer interrupted mid-way
(process crash, killed job, ...) left its VM/VDI on the target SR forever,
still carrying decorateVmMetadata's DATETIME epoch sentinel.

Fix the filter to use the real srUuid and match only entries still carrying
that sentinel, then actually call $destroy(). Exclude this._targetVmRef: when
checkBaseVdis (which runs earlier) already validated the same VM as reusable
for this run, it must survive to be chained onto, not be destroyed under it.
2026-09-28 09:33:46 +02:00
Florent BEAUCHAMP
a9caea651c fix(xo-server): ignore tests on startup (#10463)
detected on previous technical release
2026-09-28 09:04:15 +02:00
vates-bot
27882fb1dc feat(web-stack/i18n): update Xen Orchestra 6 (#10396) 2026-09-25 17:03:34 +02:00
Pierre Brunet
99af2031c4 feat(backups): improve doc (#10390)
improve backup documentation for developers
2026-09-25 16:28:16 +02:00
Elise-F
22706164d6 feat: technical release (#10458) 2026-09-25 15:03:27 +02:00
Florent BEAUCHAMP
dc57333f58 feat(v2v): remove vddk dependency (#10421)
Co-authored-by: Mathieu <mathieu.piton@vates.tech>
2026-09-25 10:38:26 +02:00
spacotte-vates
3fea79d899 feat(xo-web): add support for IS_XOA_BUILD var (#10429) 2026-09-25 10:34:55 +02:00
Florent BEAUCHAMP
f4fa020d2e feat(backups): make live mount restore works with xo-proxy (#10431) 2026-09-25 10:15:17 +02:00
Florent BEAUCHAMP
b5f119f1d4 feat(nbd-client): implement a nbd-client for stdin/stdout instead of a tcp connection (#10381) 2026-09-25 09:57:53 +02:00
Florent BEAUCHAMP
d1029ffb24 docs(live-mount): add documentation of live mount (#10445)
* docs(live-mount): add documentation of live mount

Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-09-25 09:48:11 +02:00
Mathieu
a4b17d5abb fix(rest-api): triggers updates for objects that are unmodified but computed (#10441) 2026-09-24 15:43:41 +02:00
Elise-F
b1b5097b6b feat(lite) : add action forget on host (#10315)
* update changelog file

* feedback one

* feat(lite): add forget action on host

* udpate changelog file

* fix rebase

* feedback

* rebase
2026-09-24 15:15:09 +02:00
Marie Garde
5ba393ca2b feat(xo6): fix imports vif form (#10451) 2026-09-24 14:59:32 +02:00
Marie Garde
b3b02b1fef feat(lite): Create Vif from VM Network (#10216) 2026-09-24 14:23:18 +02:00
Killian
6b4a7e5ebd fix(jobs): updates a sequence's runner user on save (#10423)
* fix(jobs): updates a sequence's runner user on save
2026-09-24 13:14:00 +02:00
MlssFrncJrg
9e38c0bac2 fix(xo-web): VDI migration now shows all pool SRs for admins, even on VMs bound to a Resource Set (#10422) 2026-09-24 12:14:16 +02:00
amouillard-vates
4d6a5ffa60 feat(web-core): add accent prop to MenuItem (#10348) 2026-09-24 09:01:20 +02:00
Mathieu
fc571c00d4 feat(xo-server,rest-api,xo-web/rpu): force-close an incomplete rolling pool update record (#10418)
* feat(xo-server,rest-api,xo-web/rpu): force-close an incomplete rolling pool update record

A pool whose previous rolling pool update left a recovery record refuses a
new run until that record is dealt with. Once the operator has reviewed the
pool, `pool.finalizeRollingUpdate` and
`POST /rest/v0/pools/{id}/actions/finalize_rolling_update` close it.

Without `force`, the finalization is refused (`incorrectState`, property
`unrestoredItems`) while the record says the run changed something the pool
still shows changed: HA, auto power-on, WLB, load balancer, backup
schedules, hosts left disabled, running VMs away from their home host,
pinned VMs shut down and not started again. A record that cannot be read
is refused too, with an unknown list.

With `force`, the record is deleted, the abandoned items are written to the
task and the trace, nothing is restored nor changed in the pool.

To tell what the run changed from what the operator did, the run now
persists, before each change, the settings it touches (strict write) and
whether each host was enabled before the run.

XO 5 shows a "Finalize" action on the incomplete update banner, which is
now also shown for a run that succeeded but whose record could not be
removed. Forcing asks for a second confirmation listing the items.
2026-09-23 16:53:43 +02:00
amouillard-vates
974c2cfee9 feat(xo6): sort networks by name on hosts > networks table (#10367) 2026-09-23 15:51:42 +02:00
Thierry Goettelmann
fc6124b1b8 feat(web-core/table): add row selection, checkbox column and controls bar (#10435) 2026-09-23 15:30:29 +02:00
Thomas Moraine
97d338710d doc(xo): remove SECURITY.md to apply global Vates security policy (#10405)
This commit removes the local `SECURITY.md` file from the Xen Orchestra repository.

The new global Vates security policy is now in place, so the local file
is no longer needed and should be replaced by the central policy:
https://github.com/vatesfr/.github/blob/main/SECURITY.md

This ensures consistent security reporting guidelines across Vates projects.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-09-23 13:27:17 +02:00
Grandalf
e957c2d821 test(backups): cover VM backups cache invalidation and local mutations (#10261)
test(xo-server/backups): cover VM backups cache invalidation and local mutations
2026-09-23 11:22:36 +02:00
Elise-F
a1dba1ed4e feat(lite): add start shutdown action host (#10314)
* update changelog file

* feedback one

* update changelog file

* feat(lite): add start shutdown action host

* update changelog file

* feeback 1

* feedback 2

* feedback 3
2026-09-22 15:32:35 +02:00
Marie Garde
26a71c4d4b feat(xo-remote-parser): migration JS to TS and update tests (#10399) 2026-09-22 14:50:44 +02:00
ayoub-el-kajji-v
c40741ccf4 feat(rest-api): add reclaim space method per vm or backup repository (#10262) 2026-09-21 19:02:09 +02:00
ayoub-el-kajji-v
c95dec183e Fix(backup): immediate delta vm backup deletion (#10273) 2026-09-21 16:07:58 +02:00
Thierry Goettelmann
ee0b8f0468 feat(web-core): enhance collection flags configuration and add clearFlag (#10428) 2026-09-21 14:05:57 +02:00
Florent BEAUCHAMP
75d9e4afda feat(xo-web): implement read only live mount as restore target (#10345)
following the backend PR this PR implement the xo5 ui to use live mount for restoring a VM .

    the popup is enlarged to ensure there is enough space
    it fixes a ui feature where removing a VDI make it unrestorable without closing and reopening 
    A disk row is complete when its action has the destination it needs: Live mount → a host, Restore → its own SR or the main SR, Do not restore → nothing.
    An incomplete row shows a red message under its selector, and since the rows sit in a collapsed panel, a red "Some disks have no destination yet" line is also shown outside it.
    The restore SR selector's placeholder now reads "Use main SR" once a main SR is set, so an empty field reads as a deliberate fallback rather than an oversight.
    For live moiunt the picker is prefilled with the host that can use the SR
2026-09-21 11:07:56 +02:00
Sylvère
eb7471a248 chore(web-core,web,lite): update FontAwesome, clean unused icon packages imports (#10365) 2026-09-21 09:52:34 +02:00
Grandalf
cae1b72571 feat(xo-server/backups): list the backups from a journal-replayed cache (#10257)
[XO-2850]
2026-09-21 09:36:25 +02:00
Olivier Lambert
f56150c25b Revert "feat(browser-media): prototype local ISO streaming in XO 5 and XO 6"
This reverts commit 5d1d0eba2f.
2026-09-18 18:40:28 +02:00
Olivier Lambert
5d1d0eba2f feat(browser-media): prototype local ISO streaming in XO 5 and XO 6 2026-09-18 18:37:16 +02:00
Mathieu
fe69a029e1 fix(openmetrics): don't block the scrape on stale collector caches (#10409)
Introduced by 2b7b6ba
See XO-2984
2026-09-18 17:29:46 +02:00
Patrice Ferlet
12a32dce22 fix(@xen-orchestra/mcp): use undici fetch to fix Node 26 dispatcher crash (#10413)
* fix(@xen-orchestra/mcp): use undici fetch to fix Node 26 dispatcher crash

Node's global fetch is backed by Node's bundled undici and rejects a
dispatcher from the npm undici dependency with UND_ERR_INVALID_ARG on
some Node versions (e.g. Node 26), surfacing as "fetch failed". The MCP
passed an EnvHttpProxyAgent dispatcher to the global fetch, so it crashed
at startup.

Route every request through undici's own fetch (proxyFetch) so the
dispatcher and the fetch come from the same undici copy. Add a FetchFn
injection point so unit tests can mock the fetch instead of globalThis.fetch.

Fixes #10411
2026-09-18 17:14:39 +02:00
Elise-F
cafea84fee feat(lite) : add action force reboot on host (#10311)
* update changelog file

* feedback one

* update changelog file

* feat(lite): add force reboot action on host

* update changelog file

* feedback 1

* feedback loader
2026-09-18 17:02:30 +02:00
Bastien Nollet
13480c3457 fix(xo-server): prevent incidental static_memory_min edit (#10285)
XO-3040
2026-09-18 11:26:32 +02:00
Bastien Nollet
4fc992da52 feat(users): add firstname, lastname and username properties (#10163) 2026-09-17 16:53:07 +02:00
Florent BEAUCHAMP
6f73bde276 feat(backups): backend of using live disk mount for restore (#10343)
* feat(backups): implement backend for live mounting disk

when restoring a VM, the vdi to SR map is extended to support a
'live-mount' mode , additionaly to the now explicit 'restore'
and 'ignore' mode

live-mount will need the host. no check is done on backend to
ensure if the VM is bootable
 
add e2e test
2026-09-17 16:30:27 +02:00
Florent BEAUCHAMP
70cbf5cda3 feat(xo-server): implement live mounting a backup archive disk (#10302) 2026-09-17 16:10:06 +02:00
Florent BEAUCHAMP
44b7db2fd9 feat(@vates/iscsi): extract iscsi package code (#10242)
implementation of an iscsi target and initiator in typescript

this will serve as a basis of 2 projects:

    connecting directly to 3rd party san
    exposing a disk from a backup archive to the host, effectively enabling direct mount to a running VM. Final target is to be able to boot completely the VM

@vates/iscsi/VALIDATION.md file is a journal of the additional test I run on my lab
@vates/iscsi/PROTOCOL.md describe the subset of the protocol implemented . For example we only support 1 LUN per target, not multiple connection , only one type of auth and keep alive
2026-09-17 15:07:15 +02:00
Mathieu
f56d241c18 feat(xo-server,rest-api,xo-web/rpu): safe start on a partially updated pool (#10394)
* feat(xo-server,rest-api,xo-web/rpu): safe start on a partially updated pool

A rolling pool update used to overwrite the recovery record left by the
previous run and to look only at the master to decide whether there was
anything to update. Both hid a pool left half updated: an interrupted run,
or updates applied by hand, could be silently completed or forgotten.

- `startRpuRecoveryRun` now refuses a new run while the pool still has a
  record, whatever its status, with an `incorrectState` error (property
  `rollingUpdateRecovery`). The RPU guard held by the caller makes the
  check-then-write safe. A run which fails before any host was handled
  drops its record instead of marking it failed: the orchestrator restores
  what it changed on its way out, and a refused precondition (a guidance to
  accept, a pinned VM to shut down) must not block the retry the operator
  just consented to.
- The update refuses a pool whose master is already up to date while
  another host is not (`incorrectState`, property `partiallyUpdatedPool`,
  `actual` listing the outdated hosts) unless the new
  `acceptCurrentStateAsBaseline` option is set. The option is exposed on
  `pool.rollingUpdate` and on `POST /rest/v0/pools/{id}/actions/rolling_update`.
- XO5 enables the rolling pool update button as soon as any host of the
  pool has missing patches, not only the master, disables it while a
  recovery record exists, and asks for confirmation before accepting the
  current state as the baseline.

Until the force-close operation ships, a leftover record can only be
removed by hand: it must land in the same release.

* feat(xo-server,rest-api,xo-web/rpu): changelog entry for #10394

* fix(xo-server/rpu): getRollingUpdateRecovery returns null on no record

* fix(xo-server/rpu): persist acceptCurrentStateAsBaseline, mark record succeeded on delete failure

* fix(xo-server/rpu): drop an empty record after the pool is restored, update LINSTOR only past the guards
2026-09-17 14:17:06 +02:00
MlssFrncJrg
ce7d94ec77 fix(xo-server): enhance CSP handling for Netdata (#10275)
Introduced by #10101
2026-09-16 17:01:09 +02:00
Mathieu
f91d64235a fix(xo-server): fix vm topology when editing incompatible cpu max number (#10398)
* fix(xo-server): fix vm topology when editing incompatible cpu number
2026-09-16 15:51:31 +02:00
ayoub-el-kajji-v
116f50821e fix(backups): change prefix name for health checks (#10361)
[XO-2797] This PR is for changing the displayed name during backups health check from [Importing...] to [Health Check...]
Introduced by 03865bc
2026-09-16 15:28:24 +02:00
Enée DI IORIO
43f63ee784 feat(web-core): update UiLegend component to v6 (#9509) 2026-09-16 14:58:12 +02:00
Grandalf
11ec9fd5da fix(backups): don't destroy the job's own rolling snapshots (#10395)
listReplicatedVms() stopped excluding snapshots in c1e9b83 (#9524) so that new-style replication, where each transfer is a snapshot of the target VM, would be covered. But a backup snapshot of the source VM matches the very same filter: it carries xo:backup:schedule and xo:backup:vm, and setVmOtherConfig passes srUuid = null, which removes xo:backup:sr and therefore matches the srUuid = undefined argument of the "delete previous interrupted copies" sweep in FullXapiWriter._run.

A full backup job replicating to an SR of the source pool thus destroyed both the rolling snapshot retained from the previous run and the one it had just taken, silently: the sweep is fire-and-forget under ignoreErrors, and XAPI holds the destroy of the snapshot being exported until the transfer ends instead of refusing it.

A replication snapshot is a snapshot of a replica, so a snapshot of the VM designated by xo:backup:vm can never be one.
2026-09-16 14:16:26 +02:00
spacotte-vates
ea2acac77b fix(backups): don't destroy the delta base after failed transfer (#10369)
XO-2994

Customer reported that after a failed replication transfer, the next run was a full and a VM was left orphaned on the target.

This is due to _removeUnusedSnapshots() protecting the last snapshot irrelevant of if it was successfully transferred and thus, in the case of a failed transfer, breaking the delta chain
2026-09-16 10:53:02 +02:00
Pierre Brunet
14b7115cf1 fix(openmetrics): share metrics collection on slow pools (#10401) 2026-09-16 10:19:42 +02:00
MlssFrncJrg
d0645598b9 feat(xo-server/proxy): check proxy licenses from XOA instead of blocking backups on it (#10280)
XO-2768
2026-09-15 20:42:18 +02:00
Thomas Moraine
83b2fd9056 docs(releases.md): document security support (#10379)
Document which Xen Orchestra versions receive security fixes
in the actual XO doc, not just in SECURITY.md.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-09-15 17:13:13 +02:00
Florent BEAUCHAMP
0c04076324 feat(v2v): use CBT to discover allocated blocks instead of vddk (#10384)
Faster (25 GiB test VM: the map goes from 2.5s to 50ms) and fewer moving
parts: the vddk map needs nbdkit, the vddk library and nbdinfo, all
version-coupled, where CBT is a handful of documented calls every backup
product has relied on for years.

Checked on ESXi 8.0.1 that enabling CBT on a disk which already holds data
backfills the tracking file, so the first query reports what was already
written and not only what is written afterwards. Against `nbdinfo --map` the
two maps differ by 960 KiB which reads back as zeros: CBT reports the blocks
that were written where the vddk reports the ones VMFS allocated.

CBT must be enabled before the migration, it cannot usefully be turned on
during the run:
- powered off VM: nothing else to do, the reconfigure writes the tracking
  file and the query answers straight away
- running VM: the query needs a snapshot, and tracking only starts at the
  next stun (snapshot create/delete, power cycle, vMotion)
2026-09-15 16:47:03 +02:00
spacotte-vates
c37c4a286d fix(xo-server): fix cache issue on vmBackupOnRemote (#10277) 2026-09-15 15:56:09 +02:00
Mathieu Lantin
21d99ee856 fix(xo-server-netbox): use numeric comparison to find the most specific prefix (#10297)
Fixes #10240
2026-09-15 14:59:32 +02:00
Bastien Nollet
fec5508ce8 chore(xo-server): remove unused files (#10397) 2026-09-15 14:46:00 +02:00
Florent BEAUCHAMP
f55c77a11a Feat: check changed packages list in CI (#10392) 2026-09-15 11:17:58 +02:00
Pierre Brunet
76225348fa fix: add error when encryption key removed from url (#10339) 2026-09-15 11:10:43 +02:00
Marie Garde
67327eb791 feat(xo6): add disabled icon host (#10188) 2026-09-15 11:09:03 +02:00
Pierre Brunet
e87f4555dd fix(backups): validate data on alias to vhd, not directly on alias (#10393) 2026-09-15 09:53:41 +02:00
vates-bot
3d0ba7ca34 feat(web-stack/i18n): update Xen Orchestra 6 (#10316) 2026-09-14 13:32:06 +02:00
Mathieu
9f337ace71 feat(xo-server,rest-api,xo-web/rpu): persist a recovery record across restarts (#10331)
* feat(xo-server,rest-api,xo-web/rpu): persist a recovery record across restarts

A rolling pool update now writes a `rpuRecovery` record (LevelDB, one per
pool) before its first side effect and keeps it updated during the run:
operator options, update variant, host order, initial VM placement, per-host
step statuses, last filtered error, and the pinned VMs shut down for their
host's reboot (previously only tracked in memory, so an xo-server crash left
them halted with nothing to restart them). A record left in a live status at
xo-server boot is flipped to `interrupted`; a successful run deletes its
record; an unreadable or unknown-version record is reported as `blocked`
without touching the stored value.

The status is exposed through `pool.getRollingUpdateRecovery` and
`GET /rest/v0/pools/{id}/rolling_update_recovery` (filtered view: never the
raw intent, reusing the existing pool permissions), and the pool Patches tab
in XO5 shows a warning with per-host progress, the last error and the pinned
VMs still halted.

First card of the RPU resume-after-failure spike: the record only reports
for now, resuming an interrupted run comes separately.

* docs(changelog): entry for the RPU recovery record (#10331)

* fix(xo-server,rest-api,xo-web/rpu): updates after review

* test(xo-server/rpu): cover the recovery record during the migrate back phase

The migrate back phase now retries the rejected migrations before giving
up, so a host is only recorded as failed once the retry passes are over,
and a pool which opted out of the phase must still close its `restoreVms`
steps, otherwise a finished run reads as still running in the record.

Both are asserted through a recorder spying on the step transitions.

* fix(xo-server/rpu): keep per-key reads in the recovery reconciliation

* test(xo-server/rpu): stub the recovery recorder in the rolling pool update tests

Since #10313, XenServers.rollingPoolUpdate runs the backup guard itself and
its tests drive it with a fake app. With the recovery record, the run also
calls app.startRpuRecoveryRun, which the fake app did not have.

The stub records its call, so the tests also check that a refusal from the
backup guard leaves no record behind.

* fix(xo-server/rpu): do not fail a successful run when its record cannot be deleted
2026-09-14 12:09:15 +02:00
Thomas Moraine
3b08f236a4 docs(xo): fix redirections (#10387)
* docs(docusaurus.config.ts): fix redirection inconsistencies

A few page redirections were missing, some were duplicates, and some pointed to the wrong URLs.

This commit cleans up redirections, updates the target URLs and adds the missing redirections.
2026-09-14 09:42:26 +02:00
Florent BEAUCHAMP
99312eca2b docs(backups): document object lock (#10373)
* docs(backups): document object lock
2026-09-11 17:47:47 +02:00
Florent BEAUCHAMP
3bf50c38ee fix(vmware-explorer): fixes omnibus (#10363)
This branch is a reliability overhaul of the ESXi/vSphere source side of V2V, plus a scope reduction of the package. The SOAP plumbing is rewritten:

    VimClient wraps @vates/node-vsphere-soap in a promise API with one emitter per call (no more cross-talk between concurrent calls, no listener leak) and a pTimeout;
    vim25 faults are now parsed (_parseFault.mjs) so callers get a real fault type instead of task execution failed;
    the library no longer throws from async callbacks (which killed xo-server) and its beforeExit logout hook is per-client instead of removeAllListeners.
    Request specs are built in explicit schema order (soap/specs.mjs) with xsi:type, and property values are normalized (soap/normalize.mjs) so search() and the new #retrieveProperty share one shape.
    search() now paginates, cancels the retrieval and destroys the container view.
    Datastore HTTP downloads get session-cookie reuse, retry with jittered backoff, a headers-only timeout, and range-honouring validation.
    Task waiting polls with backoff and per-operation timeouts (6 h for snapshot consolidation).
    nbdkit servers are memoized by a key that includes the export options, the readiness wait races the process death instead of sleeping 2 s
    the password file is 0600 and unlinked
    the thumbprint is read over TLS in-process instead of two openssl spawns.
    The old VHD-over-datastore readers (VhdEsxi*, Datastore*, openDeltaVmdkAsVhd) are deleted, hence the majo, rleaving the vddk/NBD path with a COWD grain-directory parser as data-map fallback.
    ~1.7k lines of unit tests are added.
2026-09-11 17:36:38 +02:00
Mathieu
ebbd9e42e2 fix(xo-server,rest-api): apply the backup guard to rolling pool operations started over REST (#10313)
* fix(xo-server,rest-api): apply the backup guard to rolling pool operations started over REST

The JSON-RPC methods pool.rollingUpdate and pool.rollingReboot refused to run
while a backup job targets the pool, but the REST actions
POST /pools/:id/actions/rolling_update and rolling_reboot called the
orchestrators directly and skipped that check.

The guard now lives in XenServers.rollingPoolUpdate and Pools.rollingPoolReboot,
so every entry point (JSON-RPC, REST, scheduled jobs) applies the same rule.
The bypass moves into backupGuard(poolId, { bypassBackupCheck, operation }),
which logs a single warning with the operation and the user, and the other
callers (host.restart, host.restartAgent, host.stop, sr.reclaimSpace and their
REST counterparts) use it instead of their own if/else and log.

The REST rolling actions accept bypassBackupCheck in their body and record the
body in the task params, like the host actions do.

* fix(xo-server,rest-api): pass the targeted object ID to the backup guard
2026-09-11 16:49:14 +02:00
Pierre Brunet
f9049754b4 fix(plugin/ldap): fix changelog (#10383) 2026-09-11 15:37:08 +02:00
Mathieu
b61e0dacba fix(xo-server): destroy proxied request when the client is gone (#10388)
Introduced by 9470349
2026-09-11 15:13:46 +02:00
Grandalf
df7b62654c feat(rest-api): migrate functions to registerRestRoutes (#9895) 2026-09-11 15:06:16 +02:00
Pierre Brunet
adecd7306e feat(backups): replace vhd-lib calls with abstract methods (#10124)
* feat: add setChainToParent in order to point header/footer to a new parent

* feat: add getCompressionType and use it in isMergeableParent in RemoteAdapter

* feat: use openDisposableDisk instead of openVhd

* fixup! feat: add setChainToParent in order to point header/footer to a new parent

* feat: add tests for setChainToParent

* fix: from review

* feat: make isMergeableParent contained in remoteDisk

* fix: remove chainToParent now useless

* feat: remove compressionType and isMergeableParent where unneeded

* feat: update test

* feat: handle compressionType in remoteAdapter and fs

* fix: use remoteHandler as config getter

* feat: remove useless isDirectory in remoteDisk

* fix: small change on mode option
2026-09-11 11:42:51 +02:00
Florent BEAUCHAMP
595687dabe docs(backups): precise the prerequisite for NBD (#10377)
* docs(backups): precise the prerequisite for NBD
Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>
Co-authored-by: Florent BEAUCHAMP <florent.beauchamp@gmail.com>

---------

Co-authored-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-09-11 10:02:11 +02:00
Thomas Moraine
79118de1ad docs(xo): add redirections following doc url changes (#10385)
* docs(xo): add page redirections

Since we changed how the sidebar was generated,
many page URLs changed in the process,
but redirections weren't implemented yet.
This broke several links to the documentation.
This commit adds redirections so the links don't result in 404 errors anymore.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(xo): update outdated URLs in the XO app

Following the recent overhaul of the documentation URLs, some links in the XO app were still pointing to obsolete pages.

This commit updates them to use the current URLs and prevents 404 errors.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(intro.md): update link to V2V migration guide

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

---------

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-09-11 09:57:39 +02:00
Thomas Moraine
12d77efe17 docs(xo): fix broken links in the xo app (#10382)
Since the XO documentation structure was changed recently,
so did the URLs to each documentation pages.
This broke multiple links in the XO app,
which pointed to the XO doc.

This commit updates the links to the current URLs,
so that the link work as intended.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-09-11 07:46:42 +02:00
MathieuRA
280c05ac48 feat(rest-api/SSE): SSE support now XO objects changes 2026-09-10 16:41:55 +02:00
MathieuRA
d13fad62bd fix(rest-api): use previousObj instead of object for message SSE events 2026-09-10 16:41:55 +02:00
MathieuRA
e0fb16a421 feat(xo-server): server status become a persistant property 2026-09-10 16:41:55 +02:00
Florent BEAUCHAMP
db8fe8d472 feat(rest-api): export vdi and vdi snapshot content as qcow2 + vdi export fixes (#10350) 2026-09-10 14:38:47 +02:00
Grandalf
2aa3bd602e feat/backups/journal log (#10256)
* fix fbeauchamp comments

* feat(backups): add a journal

* fix cache writing on immutable remotes

* add typing

* feat(backups): shard the journal per UTC day, one entry per time

`readBackupJournal()` listed the whole `xo-backup-log/` and filtered the names
client-side, so the cost of a read grew with the whole history of the repository.

Store the entries as `/xo-backup-log/<YYYYMMDD>/<HHMMSS.mmmZ>-<rand>-<event>-<vmUuid>-<file>`
and list only the days which can hold entries newer than `since`: a read now costs one
listing of the journal root plus one listing per day it is missing — normally a single
day, at most two, since a reader rebuilds from scratch when its watermark crosses a UTC
day. Purging old entries also becomes a directory removal instead of a per-object sweep.

The day is not repeated in the entry name, so `since` bounds the time of its own day
only; every later day is taken whole.

* fix pipeline

---------

Co-authored-by: All-Ki <killian.allegrain@vates.tech>
2026-09-10 14:26:41 +02:00
Florent BEAUCHAMP
b32fd39c19 fix(qcow2): improve qcow2 stream index computation (used for backup/replication and export) (#10333)
* feat(qcow2): improve generation speed

the idea is to build a bit index of the clusters to store them efficiently instead
of going through all the possible blocks 3 times at different phases of the generation.
The memory consumption is 2MB/TB of disk

this index generation can still be quite long , so we use setImmediate from
time to time to le the rest of the nodejs event loop run

also make this index phase abortable
2026-09-10 13:49:32 +02:00
Thomas Moraine
0115680129 doc(restapi.md): document maximum token validity (#10347) 2026-09-09 15:22:57 +02:00
Dan Pollak
e50f033361 fix(xo-server): correct spelling of "across" in migration logic and messages (#10372) 2026-09-09 14:21:10 +02:00
OlivierFL
5cf93b3c3c feat(web-core/tab-item): update UiTabItem component to v7 (#10338) 2026-09-09 12:39:43 +02:00
Mathieu
505ac6caf5 fix(xo-server): keep localhost proxy targets on wildcard binds (#10335)
* fix(xo-server): keep localhost proxy targets on wildcard binds

Since 82f047ef9 (#9572), [http.proxies] targets pointing at localhost are
rewritten to the hostname of the HTTPS listen config. With the default XOA
config (hostname = '0.0.0.0') this produced targets such as
http://0.0.0.0:9004/metrics and, worse, routed these internal requests
through the proxy-aware HTTP agent: with an httpProxy configured,
/openmetrics scrapes left the appliance for the corporate proxy and were
denied there.

Only rewrite when the bind address is a specific one (the #9500 case, where
localhost is not reachable) and pick the agent from the original target
locality, so requests to the machine itself never go through the HTTP proxy.

Seen in support ticket 7763112.

* chore(CHANGELOG.unreleased): entry for #10335

* chore(CHANGELOG.unreleased): clarify localhost targets always bypass the HTTP proxy
2026-09-09 12:00:54 +02:00
OlivierFL
e428304aea feat(xo6/web-core): implement tests on frontend (#10012) 2026-09-09 11:48:41 +02:00
Pierre Brunet
11eed70342 fix(xen-api): update undici to prevent timeout error (#10282) 2026-09-09 11:16:07 +02:00
Sandrine Barrucand
699d78b1ca fix(web-stack): use correct color for no-data stars (#10327) 2026-09-09 10:43:09 +02:00
Pierre Donias
43f81f87e3 feat(xo-web/vm/tab-backups): open edit form in same tab (#10342)
Opening the form in a new tab reloaded the whole app, so every object
was fetched again from scratch. Until that fetch finished, the form
could not be properly auto-filled. Navigating in the same tab reuses
the already loaded objects, so the form is populated right away.
2026-09-08 16:38:00 +02:00
Dan Pollak
77d07cf4ac Docs: Fix typo (#10362) 2026-09-08 15:43:02 +02:00
ayoub-el-kajji-v
dc107da91a fix(xo-server): change current_operations display for xo host and poo… (#10283) 2026-09-08 14:54:24 +02:00
Thomas Moraine
c9391b075e docs(xo): fix wrong page order and restore missing external links (#10370)
docs(xo): fix wrong page order and restore missing links

After ve changed how the sidebar
was generated (from manual declarations to automatic generation)
Some pages went out of order and didn't
have the proper sidebar title.
This commit fixes that.
Also, the external links in the sidebar disappeared.
We can't add them in an auto-generated sidebar, so we circumvent
that by creating dedicated pages for external resources
and re-adding the links there.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-09-08 14:33:29 +02:00
Mathieu
91d0f0fbcb feat(xo-server): make the RPU/RPR migrate back phase reliable and optional (#10295)
Fixes #10260.
Introduced by #7242.
Takes over #10274.
2026-09-08 12:00:33 +02:00
Thomas Moraine
f99d7ab761 doc(xo): sidebar refactor (#10354)
We now generate the sidebar automatically (based on the documentation folder structure and _category.json files), instead of declaring the doc structure manually in sidebars.ts.
Simpler management

This makes the sidebar easier to understand and maintain. It is now managed the exact same way as the sidebars for the Vates VMS and XCP-ng documentations.
Clickable breadcrumbs

The autogenerated sidebar also creates category index pages, which makes breadcrumb categories clickable, as they already are in the Vates VMS and XCP-ng documentations.
Preview
Capture d'écran 2026-09-08 094848
Unlisted pages

Pages that should remain accessible but be hidden from the sidebar are now hidden using the unlisted: true metadata, instead of simply being omitted from sidebars.ts.
Internal links and redirection

Finally, the PR updates internal links and redirects to reflect the new folder structure and URLs.
2026-09-08 11:40:01 +02:00
Florent BEAUCHAMP
e58587aa1d fix(xo-server): auto reconnection infinite loop when pool is already connectd (#10355)
* don't rearm after a fatal error before connection setting changed ( does not affect the normal route on non fatal error like newtork issue)
* serialize error correctly in database to ensure the "avoid a database write per auto-reconnect" corectly trigger
a lot of more cleanup to ensure that the connection is really dropped even if the host changed its id
* fix a mishnalding in the grace period before marking a pool disconnected . this will keep the ui in sync AND not redownload all the xapi object for a transient issue
* fix an error transmitted as raw, so the user was only seeing {}
2026-09-08 11:25:51 +02:00
Grandalf
067aba2bfc feat(backups): add a cache guard and migrate required to backup-archive (#10243)
[XO-2789]
* Extracts the backup migration logic into backup-archive and adds a cache guard for future performance improvements
* don't generate or read cache file on immutable backup repository
* clean cache file on immutable backup repository
2026-09-08 11:11:47 +02:00
Sandrine Barrucand
bbeb67092e fix(web-stack): use correct color for 404 (#10325) 2026-09-08 10:15:14 +02:00
Alina D
a974448339 feat(xo6): add the VM name to actions that open a modal (#10310) 2026-09-08 10:11:09 +02:00
amouillard-vates
069bb79d4a feat(xo6): sr dedicated page hosts tab (#10140)
* Update CHANGELOG.unreleased.md

* Hosts tab

* Hosts side panel

* Update CHANGELOG.unreleased.md

* Route + fix import

* The dev routes

* Simplify SrHeader

* Refactoring

* Fix type

* locale

* Simplify pbd connection status

* Refactor srs get

* New import rules

* Fixes

* Singular locale

* Refactor + sr status

* Push generated routes

* Switch order

* PR Fixes

* Comment fixes

* Changelog packages

* Cleanup changelog

* Keep unreleased
2026-09-08 10:04:07 +02:00
Sandrine Barrucand
ead446fcdb fix(web-stack): use correct color for console offline SVG sparks (#10309) 2026-09-07 17:40:44 +02:00
Sandrine Barrucand
03865bc5fe feat(xo6/host): reconfigure the management interface to another PIF (#10110) 2026-09-04 14:20:23 +02:00
Elise-F
35e8c513d8 feat(lite): add action reboot on host (#10250)
* feat(host): add action reboot on host

* update changelog file

* fix missing ts extension and clean code

* add header state

* feedback one
2026-09-04 14:20:02 +02:00
Florent BEAUCHAMP
00943b92e7 docs(xo-server): rest api guidelines (#10071)
improve the xo guidelines
2026-09-04 13:36:45 +02:00
Bastien Nollet
4416d55d7a feat: release 6.8.2 (#10360) 2026-09-04 12:00:25 +02:00
Bastien Nollet
9f72a1f900 feat: technical release (#10358)
* feat(@xen-orchestra/proxy): 0.31.10

* feat(xo-server): 5.208.3

* chore(CHANGELOG): update next
2026-09-04 11:20:47 +02:00
Florent BEAUCHAMP
003cc6f9cc fix(proxy): fetch failed (#10346)
* fix(backups): fetch failed

disable compresion between xoa on proxy on xoa side

* fix(xo-proxy): disable compression on the proxy

it was disable with hrp, replaced by #10038 since it was never
sending accept-encoding header

compression can be put back later on a route by route basis after
careful checks ( for example : file restore, backup listing )
2026-09-03 12:56:36 +02:00
Bastien Nollet
bca8f02b8a feat: release 6.8.1 (#10341) 2026-09-02 11:18:21 +02:00
Bastien Nollet
7a18762da4 feat: technical release (#10334)
* feat(@xen-orchestra/xapi): 8.10.5

* feat(@xen-orchestra/backups): 0.74.2

* feat(@xen-orchestra/proxy): 0.31.9

* feat(xo-server): 5.208.2

* chore(CHANGELOG): update next
2026-09-01 16:26:11 +02:00
Florent BEAUCHAMP
1004edb51f fix(backups): failing replication of qcow2 disks (#10319)
* fix(backups):  task label NBT->NBD

* fix: still force a list block only call for NBD + stream scenario , that was failing to transfer disk on fallback
2026-09-01 15:28:47 +02:00
Marie Garde
a67dd73304 feat(xo6/traffic rule): edit a traffic rule (#10056) 2026-09-01 11:22:03 +02:00
Mathieu
a2ce3bbc48 fix(@xen-orchestra/qa-test): downgrade engine to 22 to avoid build issue (#10323)
* fix(@xen-orchestra/qa-test): downgrade engine to 22 to avoid build issue

* fix >=
2026-09-01 09:14:56 +02:00
Bastien Nollet
3034955277 feat(lite): 0.25.0 (#10329) 2026-08-31 18:37:24 +02:00
Bastien Nollet
faf6745471 feat: release 6.8 (#10328) 2026-08-31 17:59:40 +02:00
Bastien Nollet
7e022940c3 feat: technical release (#10326)
* feat(@vates/types): 2.0.1

* feat(@xen-orchestra/web): 0.60.1

* feat(xo-server): 5.208.1

* chore(CHANGELOG): update next
2026-08-31 15:55:42 +02:00
Bastien Nollet
b33894c067 feat : technical release (#10322)
* feat(@xen-orchestra/web): 0.60.0

* feat(xo-server): 5.208.0

* feat(xo-server-openmetrics): 1.9.2

* feat(xo-web): 5.203.0

* chore(CHANGELOG): update next
2026-08-31 15:15:41 +02:00
Mathieu
2fba0ff4df fix(openmetrics): keep the Prometheus secret across xo-server restarts (#10290)
* fix(openmetrics): keep the Prometheus secret across xo-server restarts

The `secret` property of `configurationSchema` used a random `default`. That
expression is re-evaluated every time the module is loaded, and xo-server never
persists the values ajv fills in from schema defaults, so each restart handed
the metrics endpoint a brand new bearer token and Prometheus started getting
401s.

Drop the default and generate the secret in `load()` instead, saving it through
`xo.configurePlugin()` so it survives a restart.

Introduced by #9323
See https://xcp-ng.org/forum/topic/12415

* chore(changelog): add openmetrics secret entry

* fix(xo-server-openmetrics): generate a 256-bit Prometheus secret

Buffer.from(getRandomValues(new Uint32Array(8))) coerces each 32-bit
element to a single byte, so the generated secret held 64 bits of
entropy instead of 256. Use randomBytes(32) instead.

Introduced by fe29bd0d4
2026-08-31 13:45:02 +02:00
Pierre Donias
62cc3e3f5d feat(xo-web/new VM): allow admin to create VM in resource set (#10259) 2026-08-31 10:55:00 +02:00
Elise-F
d2d2d57c81 feat(lite): add action create internal network (#10235)
* feat(lite): add create network and network bonded actions

* remove vlan required and info message

* feedback number one

* feat(lite): add create network and network bonded actions

* feat(lite): add action create internal network

* update changelog file

* fix rebase

* feedback number one

* remove as const in cancelRoute

* feedback number two
2026-08-31 10:51:27 +02:00
Mathieu
c7bf0c59ef fix(@xen-orchestra/qa-test): fix node version (#10320) 2026-08-31 10:48:19 +02:00
Marie Garde
62013d4e79 feat(xo6): add detach host button (#10179)
* feat(xo6): rebase detach

* add redirect when detach ok

* fix post rebase

* remove dev route

* returns from MR

* add check master host

* use linked translation

* rebase ok
2026-08-31 09:03:33 +02:00
Bastien Nollet
6a4415ea1d feat: technical release (#10318)
* feat(@vates/nbd-client): 3.5.1

* feat(@vates/types): 2.0.0

* feat(@xen-orchestra/acl): 1.6.1

* feat(@xen-orchestra/backup-archive): 1.1.2

* feat(@xen-orchestra/xapi): 8.10.4

* feat(@xen-orchestra/backups): 0.74.1

* feat(@xen-orchestra/web-core): 0.60.0

* feat(@xen-orchestra/mcp): 1.4.2

* feat(@xen-orchestra/proxy): 0.31.8

* feat(@xen-orchestra/rest-api): 0.39.0

* feat(@xen-orchestra/web): 0.59.0

* feat(xo-server): 5.207.4

* feat(xo-server-ipmi-sensors): 2.1.1

* feat(xo-server-openmetrics): 1.9.1

* feat(xo-server-sdn-controller): 1.5.0

* chore(CHANGELOG): update next
2026-08-28 14:56:02 +02:00
Florent BEAUCHAMP
14e4c7d62c fix(backups): timeout NBD transfer omnibus fixes (#10306)
* fix(backups): timeout was not used correctly for NBD transfers
* fix(nbd-client): evict a dead client in multi instead
actual code was retrying 5 times, each time with a 60s timeout
multi also have its own retry on conneciton
since the client used is determnistic for a block, any stalled client
will be hammerred for a lot of time before giving up

this PR evict a nbd client that is failing to read a block (with its
retry logic)

* fix(nbd-client/multi): better spread read accross client
the actual code derivate the clientId from the block index
we have no information on the block distribution, only hopes that
it's random enough
this PR introduce a mechanism to use a round robin to select
the next client

* fix: correct fall back to full with /without nbd on qcow2
2026-08-28 10:57:17 +02:00
Marie Garde
d541af2c41 feat(xo6): add restart toolstack host button (#10178)
* feat(xo6): rebase from master

* add check restart toolstack on buttons

* add tooltip message

* Fix for PR

* remove loader

* add chengelog

* remove route map

* return form PR
2026-08-28 10:37:38 +02:00
Sandrine Barrucand
cdc4ddb652 feat(xo6/backups): add header title (#10303) 2026-08-27 11:49:53 +02:00
Joris K
2ebf18a636 feat(xo-lite): add vm tree actions and loader (#10304) 2026-08-27 10:56:34 +02:00
Grandalf
dfb4c925ec fix(sdn-controller): delete saved port on traffic rule patch (#10307) 2026-08-27 10:17:20 +02:00
Enée DI IORIO
c0d92f3061 feat(web-core): upgrade UiCheckbox component to v8 (#9777) 2026-08-27 10:12:31 +02:00
vates-bot
642dce30ef feat(web-stack/i18n): update Xen Orchestra 6 (#10181) 2026-08-27 10:11:02 +02:00
Thierry Goettelmann
970eca135e fix(@vates/types,@xen-orchestra/rest-api): remove ACL type parameters from XoRecord (#10308)
Introduced by 63d4334837
2026-08-26 18:12:53 +02:00
Bastien Nollet
9cdaf6bb41 feat: technical release (#10305)
* feat(@xen-orchestra/async-map): 0.1.4

* feat(xo-common): 0.11.0

* feat(xen-api): 4.9.0

* feat(@vates/types): 1.30.2

* feat(@xen-orchestra/acl): 1.6.0

* feat(@xen-orchestra/xapi): 8.10.3

* feat(@xen-orchestra/backups): 0.74.0

* feat(@xen-orchestra/backups-cli): 1.1.14

* feat(@xen-orchestra/disk-cli): 2.2.1

* feat(@xen-orchestra/immutable-backups): 3.0.5

* feat(@xen-orchestra/web-core): 0.59.0

* feat(@xen-orchestra/mixins): 0.20.2

* feat(@xen-orchestra/proxy): 0.31.7

* feat(@xen-orchestra/proxy-cli): 0.3.3

* feat(@xen-orchestra/qa-test): 1.2.0

* feat(@xen-orchestra/rest-api): 0.38.0

* feat(@xen-orchestra/upload-ova): 0.1.9

* feat(@xen-orchestra/web): 0.58.0

* feat(xo-acl-resolver): 0.5.4

* feat(xo-cli): 0.32.4

* feat(xo-server): 5.207.3

* feat(xo-server-audit): 0.15.2

* feat(xo-server-ipmi-sensors): 2.1.0

* feat(xo-server-netbox): 1.13.2

* feat(xo-server-sdn-controller): 1.4.1

* feat(xo-web): 5.202.0

* chore(CHANGELOG): update next
2026-08-25 16:19:35 +02:00
Florent BEAUCHAMP
9b4e1aee3e fix(xo-server): task controller leaking memory (#10237) 2026-08-25 15:42:12 +02:00
Thomas Moraine
ab7b2fc7a9 docs(xo): refine page titles and navigation structure (#10276)
* docs(xo): refine page titles and navigation structure

Make page titles clearer and more consistent with their content and
purpose. Reorder some pages (and sections) and align their sidebar labels with
their titles to make the documentation easier to navigate and
understand.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(sidebars.ts): move discover xo category back to where it was

Applying the change suggested by olivierlambert.
The 'Discover Xen Orchestra' category now follows
the 'Getting started' category.
The 'architecture' page/ 'How XO 6 is organized' stays
at the end of the Getting Started category.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(xo): specify navbar titles

Applying changes suggested by olivierlambert.

Some titles in the navigation bar were made too long,
in an attempt to make them identical to the titles in the corresponding doc pages.
Having different titles in the navbar and in the actual page
is not a defect, and can indeed help readers scan the
document structure quicker.
As a result, we're (re-)adding sidebar_label metadata
to specify a navbar title where the page title was too long.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(xo): revert individual page renames

Applying changes suggested by olivierlambert, following,
the reasoning he explained in the pull request.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(intro_support.md) change title to 'support'

Apply change suggested by olivierlambert.
'Support' instead of 'Support options' is shorter and
carries the same meaning overall.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(xo): update outdated internal link labels

A few commits ago, some page titles were changed.

This did not break any internal links, but it left some link labels outdated. This commit updates those labels to match the current page titles.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs: convert page titles to sentence case

Applying changes suggested by olivierlambert

Some page titles were still using title case, even though we use sentence case throughout the XO documentation.

This commit updates those titles to follow the same convention.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

* docs(xo): switch gerund titles to noun forms

Some page titles used gerunds (the -ing form), while others used nouns or noun phrases, resulting in an inconsistent style.

This commit changes the gerund titles to noun-based titles for consistency.

The "Troubleshooting" title is kept as-is because it is a noun, despite ending in -ing.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>

---------

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-08-25 14:23:34 +02:00
Thierry Goettelmann
9cdd8e9d45 chore(web-core,web,lite): enforce .ts extension in imports (#10299) 2026-08-25 11:01:00 +02:00
Sandrine Barrucand
14e5352a40 feat(xo6/vm): add guest tools status (#10077) 2026-08-25 10:40:50 +02:00
Marie Garde
d955ffc4d1 feat(xo6): add smart reboot host button (#10177)
* feat(xo6): reset smart reboot branch ok

* Last returns

* Remove route map

* rebase ok

* add check onConfirm smart reboot
2026-08-25 09:56:16 +02:00
spacotte-vates
f812eaaeb6 feat(backup): implement synchronized snapshot option back (#10136)
Adds a synchronizedSnapshot backup setting. When enabled, all of a job's VM snapshots are taken up front in a batch before any transfer begins, and each VM's backup then reuses its pre-taken snapshot instead of snapshotting again.

Settings:
synchronizedSnapshot:

    false (default)
    true (synchronize every VM in the job)
    <tag> (synchronize only VMs carrying that tag, others are backed up normally).

snapshotConcurrency: max concurrent snapshots during the batch phase (default 2).

A batch member that fails to open is silently dropped from synchronization and backed up normally.

Unit tested where applicable and e2e tests added in qa-test.

Checkbox added in xo5 backup jobs to synchronize snapshots, tags not supported yet.
2026-08-24 17:58:03 +02:00
amouillard-vates
814bc96a8b feat(xo6): sr dedicated page general tab (#10100)
* feat(xo6): sr dedicated page general tab

* Remove dev routes

* Update CHANGELOG.unreleased.md

* Rename sr usage warning

* Rename vdi allocated space warning

* Remove useless variable

* Remove useless destructuring

* Remove useless destructuring

* Use UiInfo component and normalize margins

* Change variable name

* Add pool/host breadcrumb options

* Normalize fromContext with vdi page

* SR space unwritable alert

* Side panel space card info message

* Usage warning color

* Fix PR comments

* Add sr uuid resolver

* Fix CI

* Update getters name

* Full width UiInfo

* Consistent srUsagePercentage check

* Remove superfluous destructuration

* Reuse existing SR_SCOPE_TYPE

* Remove dev routes

* CSS nesting

* Fix PR comments

* Dev routes + UiPanelCard

* Move getSrPageLocation
2026-08-24 11:13:13 +02:00
Alina D
9ef53890b8 feat(xo-lite): add vdi page (#10269) 2026-08-21 17:31:45 +02:00
spacotte-vates
8b4505a2d5 fix(xo-server): changing vif mac adress destroy traffic rules (#10284) 2026-08-21 17:15:48 +02:00
amouillard-vates
a68c9534e9 fix(lite): add hasChildren prop to xo-lite tree view elements (#10244) 2026-08-21 14:59:23 +02:00
Bastien Nollet
46b0dc67bb chore(backups): use set_memory_limits instead of convoluted code (#10286) 2026-08-21 14:36:22 +02:00
Thierry Goettelmann
249137793f fix(web-core,web,lite): align side panel card spacing with the design system (#10279) 2026-08-20 12:03:44 +02:00
amouillard-vates
96e7e23f40 feat(xo-lite): enable scan PIFs button on host network page (#10147) 2026-08-19 14:01:04 +02:00
Anupam Mediratta
6c3e3e781d fix(backups): upgrade tar to 7.5.19 (CVE-2026-59873) (#10159) 2026-08-19 10:42:23 +02:00
Marie Garde
bcb0a7041f feat(xo6): add force reboot host button (#10175)
* manual rebase

* returns from stack PR

* rebase ok

* returns from PR

* fix import

* remove unescessary params

* remove reoute map

* Last return
2026-08-18 11:48:42 +02:00
Alina D
db0d009d5b feat(xo6/pool): add forget action (#10130) 2026-08-18 11:17:18 +02:00
Elise-F
de1c231873 feat(lite): add create network and network bonded actions (#10145)
* feat(lite): add create network and network bonded actions

* update changelog file

* remove vlan required and info message

* feedback number one

* feedback number two
2026-08-18 10:49:27 +02:00
Marie Garde
4235644950 feat(xo6): add reboot host button (#10141)
* continuefeat(xo6/host): add reboot host button

* fix fr.json

* Add loader on evacuate and reboot

* add check message evacuate VMs

* Fix message

* Returns from PR

* Remove vm migrate from host operations

* Change accent modal

* Rebase master ok

* Fix rebase
2026-08-17 09:40:40 +02:00
Marie Garde
7942d6a84a feat(xo6): add disable host and evacuate VMs (#10090)
* feat(xo6): add disable and enable host button

* Add button enable disable

* remove route map file

* Refacto for PR

* Add modal

* refacto for PR

* fix i18n

* fix route map

* change modale to VtsActionModale

* refacto for MR

* fix type VtsActionModal

* refacto for MR

* add enable disable un treeview

* refacto treeview

* wip

* add shutdown and start button and job

* Update changelog and remove route map

* wip i18n

* tests ok

* fix build

* refacto for mr

* refacto for PR

* add check pending action

* add check host not halted

* feat(xo6): add disable and enable host button

* Add button enable disable

* Refacto for PR

* Add modal

* refacto for PR

* change key i18n

* change modale to VtsActionModale

* refacto for MR

* fix type VtsActionModal

* refacto for MR

* refacto treeview

* wip

* Update changelog and remove route map

* wip

* wip forget

* test and refacto for PR

* fix error rebase

* fx rebase missing

* fix rebase

* fix rebase

* wip

* add shutdown and start button and job

* tests ok

* refacto for mr

* add check host not halted

* wip

* rebase for PR

* fx rebase missing

* wip

* change order button

* wip

* wip forget

* wip

* button and job ok

* add modal disable host and evacuate vm but tests not done

* refacto for PR

* fix fr json and route map

* fix rebase

* fix PR

* remove route map

* last fix

* Returns from PR

* Returns from PR

* Add ? on confirm i18n

* Fix job disable and enable running message

* Fix enable job

* Fix enable job

* Returns from PR

* Fix post rebase

* Fix post rebase

* Fix post rebase

* fix from rebase

* Returns from PR

* Returns from PR

* rebase ok
2026-08-17 08:33:33 +02:00
amouillard-vates
c5faec4cf6 fix(lite): remove multi-selection from pool networks table (#10268) 2026-08-14 15:39:33 +02:00
Thomas Moraine
c9149618f8 docs(xo): rename security tab into traffic rules (#10267)
The Pools -> Security tab in Xen Orchestra was
recently renamed as 'Traffic rules'.
This commit updates the XO documentation (screenshots included) to reflect that change.

Signed-off-by: Thomas Moraine <thomas.moraine@vates.tech>
2026-08-14 13:42:38 +02:00
Elise-F
438415a882 feat(xo6/traffic rules): added the error message related to the method and updated UI (#10202) 2026-08-14 10:41:45 +02:00
OlivierFL
ed78cc10cf feat(xo-6/vif): add VIF name in head bar on VIF detail page (#10252) 2026-08-14 09:05:02 +02:00
Marie Garde
1c18215bd1 feat(xo6): add host action button forget (#10089)
* feat(xo6): add disable and enable host button

* Add button enable disable

* remove route map file

* Refacto for PR

* Add modal

* refacto for PR

* change key i18n

* fix i18n

* fix route map

* change modale to VtsActionModale

* refacto for MR

* fix type VtsActionModal

* refacto for MR

* add enable disable un treeview

* refacto treeview

* wip

* add shutdown and start button and job

* Update changelog and remove route map

* wip i18n

* tests ok

* fix build

* refacto for mr

* refacto for PR

* add check pending action

* add check host not halted

* feat(xo6): add disable and enable host button

* Add button enable disable

* Refacto for PR

* Add modal

* refacto for PR

* change key i18n

* change modale to VtsActionModale

* refacto for MR

* fix type VtsActionModal

* refacto for MR

* refacto treeview

* wip

* Update changelog and remove route map

* refacto for mr

* wip

* add forget button host

* update button for menuitem

* wip forget

* test and refacto for PR

* fix error rebase

* fx rebase missing

* add changelog

* fix rebase

* fix rebase

* Add ? on confirm i18n

* Returns from PR

* rebase overlay ok

* fix changelog

* Add redirect if still on host page after forget

* Add redirect after delete
2026-08-14 09:00:50 +02:00
Pierre Donias
7cdb792bfe fix(xo-web/vm/console): make collapsed header state reliable (#10007)
The console tab kept its own `minimalLayout` state to track whether the page header was collapsed, separate from the VM page's `collapsedHeader`. The two desynced because the tab remounts on navigation while the VM page stays mounted.

Fix:
- Single source of truth `collapsedHeader` in the parent index.js
- Pass it to the child tab-console.js along with `setCollapsedHeader`
- Replace the `_toggleHeader` toggle with an explicit `_setCollapsedHeader(value)` setter
- Remove the duplicate local `minimalLayout` state that could get out of sync and use `collapsedHeader` instead
- Force off on VM shutdown (console no longer displayed) and on unmount (user changed page)
2026-08-13 15:15:25 +02:00
Grandalf
0601716fa4 fix(backups): return backups repository info one by one instead of erroring on first failure (#10205) 2026-08-13 15:09:53 +02:00
spacotte-vates
d53d2fc1c2 fix(xo-server): fix target not found error for warm migration (#10210) 2026-08-13 09:56:10 +02:00
Bruno Verachten
11ff21097f docs(snapshots): note that reverting restores VM metadata too (#10196)
Surfaced via https://xcp-ng.org/forum/topic/12148.
2026-08-12 17:39:33 +02:00
Greg-hub-dev
828c15cbb4 docs(backups.md) : Delete future RAM backup feature details (#10223) 2026-08-12 15:03:33 +02:00
Grandalf
798732e62a fix(async-map): allow return type to be different than input type (#10227) 2026-08-12 10:50:44 +02:00
Mathieu
1bfcc50b23 fix(xen-api): update platform for cores-per-socket update (#10201)
XO-2814
2026-08-10 15:52:07 +02:00
Olivier Lambert
c5d791a7de docs: full documentation revamp (#10212) 2026-08-07 16:36:09 +02:00
Grandalf
3293e69cc2 refactor(xo-server): remove hrp dependency (#10038)
replace http-request-plus external depedency by a a native wrapper around fetch
2026-08-07 11:58:20 +02:00
Grandalf
a0193085f5 feat(ipmi): add get sensors route (#10003) 2026-08-07 11:38:28 +02:00
3072 changed files with 213815 additions and 30613 deletions

View File

@@ -85,6 +85,21 @@ module.exports = {
XO_LITE_GIT_HEAD: true,
},
rules: {
'import/extensions': [
'error',
'ignorePackages',
{
pattern: { ts: 'always', vue: 'always' },
checkTypeImports: true,
// '@core/*' and '@/*' look like scoped packages to the plugin, which 'ignorePackages' would skip
pathGroupOverrides: [
{ pattern: '@core/packages/*', action: 'ignore' }, // barrel files
{ pattern: '@core/icons', action: 'ignore' },
{ pattern: '@core/**', action: 'enforce' },
{ pattern: '@/**', action: 'enforce' },
],
},
],
'import/order': [
'error',
{
@@ -254,6 +269,8 @@ module.exports = {
'zh-Hans',
'sk',
'pt',
'ar',
'tr',
],
},
],

36
.github/workflows/changelog-check.yml vendored Normal file
View File

@@ -0,0 +1,36 @@
name: Changelog
# only pull requests targeting `master`: the check compares the branch to
# `origin/master`, it would report unrelated packages on a PR based on another
# branch
on:
pull_request:
branches:
- master
jobs:
changed-packages:
# the release PR bumps every listed package and empties the list
if: github.head_ref != 'release'
runs-on: ubuntu-latest
steps:
# https://github.com/actions/checkout
- uses: actions/checkout@v4
with:
# the whole history is required to compute the merge base with `master`
fetch-depth: 0
# the check only needs commits and trees: `git diff --name-only
# --no-renames` compares tree entries and never reads file contents,
# skipping the historical blobs halves the fetch (~112 MiB → ~53 MiB)
filter: blob:none
# https://github.com/actions/setup-node
- name: Setup Node environment
uses: actions/setup-node@v4
with:
node-version: '>=22.12'
# the script only uses Node builtins, no `yarn install` necessary
- name: Ensure changed packages are listed in CHANGELOG.unreleased.md
run: ./scripts/check-changed-pkgs.js

View File

@@ -10,7 +10,7 @@ jobs:
- name: Install packages
run: |
sudo apt-get update
sudo apt-get install -y curl qemu-utils python3-vmdkstream git libxml2-utils libfuse2 nbdkit
sudo apt-get install -y curl qemu-utils python3-vmdkstream git libxml2-utils libfuse2 nbdkit open-iscsi
# https://turbo.build/repo/docs/guides/ci-vendors/github-actions#caching-with-github-actionscache
- name: Cache Turbo

73
.github/workflows/vectura.yml vendored Normal file
View File

@@ -0,0 +1,73 @@
name: Vectura
# only when vectura changes: the rest of the repository does not build it
#
# pull requests rather than every push: GitHub filters a push on the diff with
# the previous push, which after a rebase on `master` (or a merge of it)
# contains the vectura changes of `master`, whereas it filters a pull request
# on the diff with the merge base, i.e. the changes of the branch only
on:
pull_request:
paths:
- '@xen-orchestra/vmware-explorer/vectura/**'
- .github/workflows/vectura.yml
push:
branches:
- master
paths:
- '@xen-orchestra/vmware-explorer/vectura/**'
- .github/workflows/vectura.yml
defaults:
run:
working-directory: '@xen-orchestra/vmware-explorer/vectura'
jobs:
checks:
runs-on: ubuntu-latest
steps:
# https://github.com/actions/checkout
- uses: actions/checkout@v4
# rustup, on the runner image, installs the toolchain rust-toolchain.toml
# names on the first cargo command
- run: cargo fmt --all --check
- run: cargo clippy --all-targets --locked -- -D warnings
- run: cargo test --locked
# https://github.com/taiki-e/install-action
- uses: taiki-e/install-action@v2
with:
tool: cargo-deny@0.20.2,cargo-llvm-cov@0.8.5
- run: cargo deny --locked check
- run: cargo llvm-cov --locked --json --summary-only --output-path target/coverage.json
- name: Hold the coverage floor
run: cargo test --locked --test coverage -- --ignored
# builds the binary as release.sh does, checks it, then checks the one in git,
# which xo-server runs, is that build
binary:
runs-on: ubuntu-latest
steps:
# https://github.com/actions/checkout
- uses: actions/checkout@v4
# https://github.com/mlugg/setup-zig
- uses: mlugg/setup-zig@v2
with:
version: 0.16.0
# https://github.com/taiki-e/install-action
- uses: taiki-e/install-action@v2
with:
tool: cargo-zigbuild@0.23.4
- run: scripts/build.sh "$RUNNER_TEMP/build"
- run: scripts/check-binary.sh "$RUNNER_TEMP/build/vectura"
- name: The binary in git is the build of these sources
run: |
cmp "$RUNNER_TEMP/build/vectura" vectura || {
echo '::error::vectura is not the build of these sources: run scripts/build.sh . in vectura/ and commit the binary'
exit 1
}

1
.gitignore vendored
View File

@@ -5,6 +5,7 @@
/@vates/*/dist/
/@vates/*/node_modules/
/@vates/node-vsphere-soap/test/config-test.mjs
/@xen-orchestra/*/dist/
/@xen-orchestra/*/node_modules/
/@xen-orchestra/rest-api/open-api/spec/swagger.json

View File

@@ -5,3 +5,7 @@
# Generated files
/@xen-orchestra/web/src/route-map.d.ts
/@xen-orchestra/lite/src/route-map.d.ts
# Docusaurus MDX content: prettier's markdown parser corrupts JSX
# comments ({/* … */} becomes {/_ … _/}) and other MDX constructs
/docs/docs/**/*.md

View File

@@ -24,7 +24,7 @@
},
"dependencies": {
"@vates/multi-key-map": "^0.2.0",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/async-map": "^0.1.5",
"@xen-orchestra/log": "^0.7.2",
"ensure-array": "^1.0.0"
},

View File

@@ -19,12 +19,12 @@
},
"main": "./index.mjs",
"dependencies": {
"@xen-orchestra/fs": "^4.9.3",
"@xen-orchestra/fs": "^4.10.0",
"@xen-orchestra/log": "^0.7.2",
"fuse-native": "^2.2.6",
"lru-cache": "^7.14.0",
"promise-toolbox": "^0.21.0",
"vhd-lib": "^4.16.1"
"vhd-lib": "^4.17.0"
},
"bin": {
"xo-fuse-vhd": "./cli.mjs"

261
@vates/iscsi/.USAGE.md Normal file
View File

@@ -0,0 +1,261 @@
A small, dependency-light iSCSI **target** (server) written in TypeScript. It
exposes exactly one read/write LUN backed by a pluggable byte-range
`BlockDevice`, and interoperates with the standard Linux **open-iscsi**
initiator (the same one XCP-ng/XAPI uses to attach an `lvmoiscsi` SR).
### Scope
The target drives login negotiation onto a single, fixed code path:
- single initiator, **`MaxConnections=1`**, **`ErrorRecoveryLevel=0`**
- no header/data digests (`HeaderDigest=None`, `DataDigest=None`)
- all writes are R2T-solicited (`InitialR2T=Yes`, `ImmediateData=No`)
- one LUN (LUN 0)
It implements SendTargets discovery, login (`AuthMethod=None` or one-way
**CHAP** — see [Authentication](#authentication-chap)), the ~10 SCSI
commands a Linux initiator issues to attach and use a block device
(`INQUIRY` + VPD `0x00`/`0x80`/`0x83`/`0xb0`, `REPORT LUNS`, `READ CAPACITY 10/16`,
`READ`/`WRITE 10/16`, `TEST UNIT READY`, `REQUEST SENSE`, `MODE SENSE`,
`SYNCHRONIZE CACHE`), the full R2T / Data-Out write path, NOP keepalive replies,
and Logout. Commands may be outstanding concurrently (Data-Out PDUs are routed
by Initiator Task Tag), so it does not stall under real guest I/O.
The package also ships a userspace **initiator** (`IscsiInitiator`) and an
`IscsiDisk` adapter that exposes a remote LUN as an `@xen-orchestra/disk-transform`
`RandomAccessDisk` — see [Initiator](#initiator).
Not implemented (yet): mutual (two-way) CHAP, Task Management (abort/reset),
target-initiated NOP keepalives, multiple connections/LUNs.
What a peer can make the target do is bounded:
- connections that have not logged in yet are limited in time (`loginTimeoutMs`)
and number (`maxPendingConnections`), and `allowedInitiatorAddresses` (the
source addresses of the expected initiators) closes the other ones as soon as
they are accepted, reporting them to `onConnectionRefused`
- a PDU data segment is at most 8 KiB during login, then the declared
`MaxRecvDataSegmentLength` (256 KiB), checked before it is read
- a single `READ`/`WRITE` is at most `maxTransferLength` (8 MiB by default),
advertised in the block limits VPD page so that initiators split their I/O
- a `BlockDevice` with `readOnly: true` is reported write-protected, and `WRITE`
is refused with `DATA PROTECT` before any data is solicited
### CLI
The package ships a `vates-iscsi` binary that serves a file as a LUN:
```sh
> vates-iscsi /srv/lun.img
> vates-iscsi /srv/lun.img --serial MYSERIAL01 --port 3260
> vates-iscsi /srv/new.img --size 10G # create/grow the backing file first
```
```
Usage: vates-iscsi <file> [options]
Options:
--serial <serial> LUN serial number (default: derived from the IQN).
--size <bytes> Create/grow the backing file (suffixes K, M, G, T).
--iqn <iqn> Target IQN (default: iqn.2024-01.tech.vates:<file>).
--host <host> Listen address (default: 0.0.0.0).
--port <port> Listen port (default: 3260).
--block-size <bytes> Logical block size (default: 512).
-h, --help Show this help.
```
### Library
```js
import { IscsiTarget, FileBlockDevice } from '@vates/iscsi'
// The backing file must already exist and be sized to the LUN capacity,
// e.g. `fs.truncate('/srv/lun.img', 10 * 1024 ** 3)`.
const target = new IscsiTarget({
iqn: 'iqn.2024-01.tech.vates:lun0',
host: '0.0.0.0',
port: 3260,
lun: new FileBlockDevice({ path: '/srv/lun.img', blockSize: 512 }),
})
await target.listen()
// ... target is now discoverable and attachable by open-iscsi ...
await target.close()
```
#### Adding a target backend
A "backend" is whatever stores the LUN's bytes. The protocol layer translates
SCSI `READ`/`WRITE` CDBs (LBA + block count) into byte offsets/lengths against a
single small interface, so a backend only has to provide flat random access — it
never sees a PDU. `FileBlockDevice` (a sparse file) is the built-in one; implement
`BlockDevice` to expose anything else (a raw device, an object store, an
`@xen-orchestra/disk-transform` `RandomAccessDisk`, …):
```ts
interface BlockDevice {
open?(): Promise<void> // optional, awaited once before serving I/O
getSize(): number // total capacity in bytes; MUST be a multiple of getBlockSize()
getBlockSize(): number // logical block size, typically 512
read(offset: number, length: number): Promise<Buffer> // exactly `length` bytes at byte `offset`
write(offset: number, data: Buffer): Promise<void> // write `data.length` bytes at byte `offset`
flush(): Promise<void> // SYNCHRONIZE CACHE — persist any buffering
close(): Promise<void> // release resources
}
```
Contract to respect:
- **`getSize()` must be a multiple of `getBlockSize()`** and stable for the life of
the device. `open()` is the place to compute it (e.g. `stat` the file).
- **`read` must return exactly `length` bytes** — never short. Zero-fill a sparse
or past-EOF tail rather than returning fewer bytes; the SCSI layer already
bounds `offset + length` to the capacity.
- `offset` and `length` are always block-aligned (multiples of `getBlockSize()`).
- `read`/`write` may be **called concurrently** (commands are interleaved), so the
backend must be safe under overlapping calls; honour back-pressure and don't
buffer unboundedly.
- Throwing from `read`/`write`/`flush` tears the connection down; fail fast and
attach context to the error.
Minimal example — expose a read-only `RandomAccessDisk` (VHD / qcow2 / raw / a
remote NBD or iSCSI LUN) as a target LUN by mapping byte reads onto block reads:
```js
import { IscsiTarget } from '@vates/iscsi'
class RandomAccessDiskLun {
// `disk` is an initialized @xen-orchestra/disk-transform RandomAccessDisk.
constructor(disk) {
this.disk = disk
this.blockSize = disk.getBlockSize()
}
getSize() {
return this.disk.getVirtualSize()
}
getBlockSize() {
return this.blockSize
}
async read(offset, length) {
const out = Buffer.alloc(length) // pre-zeroed: sparse/holes read as zeros
const first = Math.floor(offset / this.blockSize)
const last = Math.floor((offset + length - 1) / this.blockSize)
for (let index = first; index <= last; index++) {
if (!this.disk.hasBlock(index)) {
continue // unallocated: leave zeros
}
const { data } = await this.disk.readBlock(index)
const blockStart = index * this.blockSize
const from = Math.max(0, offset - blockStart)
const to = Math.min(this.blockSize, offset + length - blockStart)
data.copy(out, blockStart + from - offset, from, to)
}
return out
}
// reported write-protected: WRITE is refused before reaching write()
readOnly = true
write() {
throw new Error('read-only LUN')
}
async flush() {}
async close() {
await this.disk.close()
}
}
const target = new IscsiTarget({ iqn: 'iqn.2024-01.tech.vates:disk0', lun: new RandomAccessDiskLun(disk) })
await target.listen()
```
For a writable backend, implement `write(offset, data)` symmetrically and make
`flush()` durable (e.g. `fsync`), as `FileBlockDevice` does.
### Initiator
`IscsiInitiator` is a minimal userspace iSCSI **client**: it connects to one
target portal, logs in, and reads the LUN. `IscsiDisk` wraps it as an
`@xen-orchestra/disk-transform` `RandomAccessDisk`, so a remote LUN drops straight
into the backup/transform pipeline (the iSCSI counterpart of `@vates/nbd-client`'s
`NbdDisk`). It is **read-only**, single-connection, digests off.
```js
import { IscsiInitiator, IscsiDisk } from '@vates/iscsi'
// Low-level byte reads:
const initiator = new IscsiInitiator({
host: '10.0.0.10',
port: 3260,
targetIqn: 'iqn.2024-01.tech.vates:lun0',
chap: { user: 'alice', secret: 's3cr3t' }, // optional, see Authentication
})
await initiator.connect()
console.log(initiator.getSize(), initiator.getBlockSize())
const bytes = await initiator.read(/* offset */ 0, /* length */ 1024 * 1024)
await initiator.close()
// …or as a RandomAccessDisk (blockSize must be a multiple of the LUN block size):
const disk = new IscsiDisk(
{ host: '10.0.0.10', targetIqn: 'iqn.2024-01.tech.vates:lun0' },
{ blockSize: 2 * 1024 * 1024 }
)
await disk.init()
// disk.getVirtualSize(), disk.getBlockIndexes(), await disk.readBlock(i), …
await disk.close()
```
First-cut `IscsiDisk` treats the LUN as fully allocated (every block reported as
present, like `RawDisk`); a changed-block map (e.g. a Pure "Volume Difference")
can narrow it later.
### Authentication (CHAP)
The target and initiator support **one-way CHAP** (MD5) in opposite roles; mutual
(two-way) CHAP is not yet implemented.
**Target as authenticator.** Pass `chap` to require each _data_ session to prove a
credential; the target challenges the initiator and rejects the login (Login
Response status-class _Initiator Error_ / _authentication failure_) on a mismatch:
```js
const target = new IscsiTarget({
iqn: 'iqn.2024-01.tech.vates:lun0',
lun,
chap: { user: 'alice', secret: 's3cr3t' },
})
```
This interoperates with the Linux **open-iscsi** initiator (the one XCP-ng uses);
configure its node record's incoming credential:
```sh
iscsiadm -m node -T <iqn> -p <portal> -o update -n node.session.auth.authmethod -v CHAP
iscsiadm -m node -T <iqn> -p <portal> -o update -n node.session.auth.username -v alice
iscsiadm -m node -T <iqn> -p <portal> -o update -n node.session.auth.password -v s3cr3t
```
CHAP is enforced only on **Normal (data) sessions**; **Discovery** sessions (which
can only run SendTargets and never reach the LUN) stay unauthenticated, so plain
`iscsiadm -m discovery` keeps working. open-iscsi requires the secret to be 12–16
characters.
**Initiator as responder.** Pass `chap` to `IscsiInitiator`/`IscsiDisk` to answer a
target's challenge (e.g. a Pure array acting as authenticator). If the target
_requires_ mutual CHAP the login is cleanly rejected (we never challenge back).
### Security
iSCSI has no data-path encryption of its own. CHAP authenticates the login only;
with no digests, **CDBs and data remain cleartext**. Anyone who can reach an
unauthenticated port can read/write the LUN, and a CHAP-protected one is still
exposed to eavesdropping and tampering on the wire. Expose targets only on a
trusted, isolated storage network, and use **IPsec** (the RFC-specified mechanism)
or equivalent if confidentiality is required.
### Testing
- `yarn test` — fast unit tests (PDU framing, SCSI encoders/decoders, login
negotiation, sequence-number arithmetic).
- `yarn test-integration` — a self-contained loopback test (an in-process
minimal initiator drives discovery, identity, write/read-back, and command
interleaving), plus an open-iscsi interop test that **requires root and
`iscsiadm`** and skips gracefully otherwise.

1
@vates/iscsi/.npmignore Symbolic link
View File

@@ -0,0 +1 @@
../../scripts/npmignore

124
@vates/iscsi/PROTOCOL.md Normal file
View File

@@ -0,0 +1,124 @@
# Protocol coverage
`@vates/iscsi` implements a deliberately narrow subset of iSCSI (RFC 7143, formerly RFC 3720) and the SCSI
command set it carries (SPC-3 primary commands, SBC-3 block commands) — enough to serve or consume exactly one
read/write LUN over a single connection, not a general-purpose iSCSI stack. This is the detailed companion to
the README's "Scope" section, verified directly against the source (`constants.mts`, `login.mts`,
`connection.mts`, `types.mts`, `initiator.mts`, `chap.mts`) rather than inferred.
The package has two independent roles with very different maturity:
- **Target** (`IscsiTarget`) — the primary, production-used role (backs `@xen-orchestra/mixins/live-mount`).
Handles real-world Linux/XCP-ng initiators robustly: concurrent commands, real write ordering, keepalives,
connection recovery.
- **Initiator** (`IscsiInitiator`) — a minimal, read-only client used to read from a third-party array as a
disk source. Built against `IscsiTarget` and standard-tooling targets (`tgt`); interop testing this session
found it does **not** complete login against a LIO kernel target (see below) — a real, previously-unknown
gap, not yet fixed.
## Session and connection
| Aspect | Target | Initiator |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------- |
| Connections per session | `MaxConnections=1`, enforced | opens exactly one |
| Session reinstatement | **no** — no ISID tracking; once a new connection has established a Normal session (CHAP included), it tears down and replaces whatever connection currently exists, rather than RFC 7143's ISID-matched reinstatement (see `index.mts`'s class doc for the reasoning: safe here because the target is ephemeral, single-consumer, and CHAP-guarded). Until then it does not affect the current one, and is bounded by `loginTimeoutMs` and `maxPendingConnections` | n/a (single connection) |
| Multiple LUNs | **no** — exactly one LUN, always LUN 0 | reads are hardcoded to LUN 0 |
| `ErrorRecoveryLevel` | `0` only | requests `0` |
| Header/data digests | `HeaderDigest=None`, `DataDigest=None` — no CRC32C on the wire | same |
| Markers (`IFMarker`/`OFMarker`) | not offered (RFC-deprecated) | not offered |
## Login / text negotiation
**Target** (`login.mts`) reactively answers whatever the initiator proposes, fixed to:
`HeaderDigest=None`, `DataDigest=None`, `MaxConnections=1`, `InitialR2T=Yes`, `ImmediateData=No`,
`ErrorRecoveryLevel=0`, `DataPDUInOrder=Yes`, `DataSequenceInOrder=Yes`, `DefaultTime2Wait=2`,
`DefaultTime2Retain=0`, `MaxOutstandingR2T=1` — `MaxBurstLength`/`FirstBurstLength` are the only two keys
actually negotiated down from the initiator's offer (capped at an internal maximum). Both `Normal` and
`Discovery` session types are supported (`SessionType`), with CHAP skipped for `Discovery` per RFC — text-mode
negotiation beyond that is limited to `SendTargets` discovery (`#handleText`), not general text negotiation.
**Initiator** (`initiator.mts#operationalLogin`) sends exactly **one** operational-stage login PDU proposing
its own fixed key set (`HeaderDigest=None`, `DataDigest=None`, `InitialR2T=Yes`, `ImmediateData=No`,
`ErrorRecoveryLevel=0`, `MaxRecvDataSegmentLength`) with the transit bit (`T=1`) set immediately, and only
checks whether the reply also has `T=1` — there is no second round to answer a target's own counter-proposed
keys if it declines transit on the first exchange.
- Works against `tgt` and against this package's own `IscsiTarget` — both transit on the first round.
- **Does not complete against a LIO kernel target** (`targetcli-fb`): LIO counter-proposes its own operational
keys (`MaxBurstLength`, `DefaultTime2Wait`, `DefaultTime2Retain`, `MaxOutstandingR2T`) in the same response
and withholds transit until they're answered — confirmed directly via `dmesg`:
```
No response for proposed key "MaxBurstLength".
iSCSI Login negotiation failed.
```
This is a real, previously-unexercised limitation of the initiator's login state machine (single
operational round-trip only), not a bug on LIO's side. Worth weighing given the initiator's real intended
use — reading from unknown-strictness third-party array software, not `tgt`/this package's own target.
## Authentication (CHAP)
- One-way only (RFC 1994, algorithm MD5 — the only one RFC 7143 defines, so this isn't a partial
implementation of a wider set).
- **Target as authenticator**: when `chap` is configured, every login must present the right `CHAP_N`/`CHAP_R`
or the login is rejected (`INITIATOR_ERROR`/`AUTHENTICATION_FAILURE`). Interop-tested against the real
`open-iscsi` kernel initiator (both correct-secret and wrong-secret paths).
- **Initiator as responder**: when `chap` is configured, answers a target's challenge with the same MD5
scheme. Never issues its own challenge.
- **Not supported**: mutual (two-way) CHAP in either role — a target that _requires_ mutual auth will reject
the initiator's login, surfaced as a normal login failure rather than attempted.
## SCSI commands
Decoded explicitly (`types.mts#decodeCdb`); anything else maps to `unsupported` → `CHECK CONDITION` /
`ILLEGAL_REQUEST` / `INVALID COMMAND OPERATION CODE`:
| Command | Variants |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `INQUIRY` | standard + EVPD `0x00`/`0x80`/`0x83` (vendor/product/revision/serial from `identity`) and `0xb0` (block limits: `maxTransferLength`) |
| `REPORT LUNS` | |
| `READ CAPACITY` | `(10)` and `(16)` |
| `READ` | `(10)` and `(16)`; longer than `maxTransferLength` → `INVALID FIELD IN CDB` |
| `WRITE` | `(10)` and `(16)`; longer than `maxTransferLength` → `INVALID FIELD IN CDB`; on a `readOnly` LUN → `DATA PROTECT` / `WRITE PROTECTED`, without soliciting data |
| `TEST UNIT READY` | |
| `REQUEST SENSE` | |
| `MODE SENSE` | `(6)` and `(10)`; `WP` set on a `readOnly` LUN |
| `SYNCHRONIZE CACHE` | `(10)` and `(16)` — flushes the backing `BlockDevice` |
**Not implemented**: persistent reservations, VAAI/thin-provisioning primitives (`UNMAP`, `WRITE SAME`, `COMPARE
AND WRITE`, ...), any vendor-specific commands — anything beyond the ~10 commands a Linux initiator actually
issues to attach and use a plain block device.
## Data transfer
- **Read**: `Data-In`, concurrent commands dispatched up to `readConcurrency` (default 16) regardless of the
advertised command window — bounds how many `lun.read()` calls run at once against the real backend.
Residual under/overflow reported against the CDB's requested transfer length.
- **Write**: always solicited (`InitialR2T=Yes`, `ImmediateData=No`) — every write goes through `R2T` →
`Data-Out`, one outstanding R2T at a time (`MaxOutstandingR2T=1`). `Data-Out` PDUs are routed by Initiator
Task Tag, so writes/reads for different commands can interleave; within one write, out-of-order or
overlapping `Data-Out` segments are rejected outright (`writePath.mts`) rather than silently accepted, to
avoid leaking uninitialized staging-buffer memory to the LUN (a real bug this was hardened against — see
`VALIDATION.md`).
- **Not implemented**: unsolicited/immediate data, `MaxOutstandingR2T > 1`.
## Task management, error recovery, keepalives
- **SCSI Task Management** (`ABORT TASK`, `LUN RESET`, etc.) — not implemented; any `SCSI Task Mgmt Request`
is rejected (`COMMAND_NOT_SUPPORTED`).
- **SNACK** (retransmission requests, part of `ErrorRecoveryLevel > 0`) — not implemented; rejected the same
way. Consistent with `ErrorRecoveryLevel=0` — nothing should ever need to SNACK.
- **NOP / keepalives**:
- Target replies to an initiator-sent `NOP-Out` ping, but does **not** proactively send its own periodic
`NOP-In` probes (no `nop_interval` equivalent) — it relies entirely on the initiator (or TCP-level
connection loss) to notice a dead peer.
- Initiator answers a target-initiated `NOP-In` ping with `NOP-Out` and keeps its read loop draining for the
life of the connection specifically so it never misses one (this was a real, fixed bug — see
`VALIDATION.md`, item 5).
## Summary: not implemented
Mutual (two-way) CHAP · SCSI Task Management (abort/reset) · target-initiated NOP-In keepalives · multiple
connections per session · multiple LUNs · session reinstatement (ISID) · header/data digests · unsolicited
data / `MaxOutstandingR2T>1` · SNACK/error recovery beyond level 0 · persistent reservations and VAAI-style
primitives · general text-mode negotiation beyond `SendTargets` · initiator-side multi-round operational login
negotiation (works against this package's own target and `tgt`; does not against LIO — see above).

294
@vates/iscsi/README.md Normal file
View File

@@ -0,0 +1,294 @@
<!-- DO NOT EDIT MANUALLY, THIS FILE HAS BEEN GENERATED -->
# @vates/iscsi
[![Package Version](https://badgen.net/npm/v/@vates/iscsi)](https://npmjs.org/package/@vates/iscsi) ![License](https://badgen.net/npm/license/@vates/iscsi) [![PackagePhobia](https://badgen.net/bundlephobia/minzip/@vates/iscsi)](https://bundlephobia.com/result?p=@vates/iscsi) [![Node compatibility](https://badgen.net/npm/node/@vates/iscsi)](https://npmjs.org/package/@vates/iscsi)
> Minimal iSCSI target and initiator: expose or read a single LUN, with one-way CHAP
## Install
Installation of the [npm package](https://npmjs.org/package/@vates/iscsi):
```sh
npm install --save @vates/iscsi
```
## Usage
A small, dependency-light iSCSI **target** (server) written in TypeScript. It
exposes exactly one read/write LUN backed by a pluggable byte-range
`BlockDevice`, and interoperates with the standard Linux **open-iscsi**
initiator (the same one XCP-ng/XAPI uses to attach an `lvmoiscsi` SR).
### Scope
The target drives login negotiation onto a single, fixed code path:
- single initiator, **`MaxConnections=1`**, **`ErrorRecoveryLevel=0`**
- no header/data digests (`HeaderDigest=None`, `DataDigest=None`)
- all writes are R2T-solicited (`InitialR2T=Yes`, `ImmediateData=No`)
- one LUN (LUN 0)
It implements SendTargets discovery, login (`AuthMethod=None` or one-way
**CHAP** — see [Authentication](#authentication-chap)), the ~10 SCSI
commands a Linux initiator issues to attach and use a block device
(`INQUIRY` + VPD `0x00`/`0x80`/`0x83`/`0xb0`, `REPORT LUNS`, `READ CAPACITY 10/16`,
`READ`/`WRITE 10/16`, `TEST UNIT READY`, `REQUEST SENSE`, `MODE SENSE`,
`SYNCHRONIZE CACHE`), the full R2T / Data-Out write path, NOP keepalive replies,
and Logout. Commands may be outstanding concurrently (Data-Out PDUs are routed
by Initiator Task Tag), so it does not stall under real guest I/O.
The package also ships a userspace **initiator** (`IscsiInitiator`) and an
`IscsiDisk` adapter that exposes a remote LUN as an `@xen-orchestra/disk-transform`
`RandomAccessDisk` — see [Initiator](#initiator).
Not implemented (yet): mutual (two-way) CHAP, Task Management (abort/reset),
target-initiated NOP keepalives, multiple connections/LUNs.
What a peer can make the target do is bounded:
- connections that have not logged in yet are limited in time (`loginTimeoutMs`)
and number (`maxPendingConnections`), and `allowedInitiatorAddresses` (the
source addresses of the expected initiators) closes the other ones as soon as
they are accepted, reporting them to `onConnectionRefused`
- a PDU data segment is at most 8 KiB during login, then the declared
`MaxRecvDataSegmentLength` (256 KiB), checked before it is read
- a single `READ`/`WRITE` is at most `maxTransferLength` (8 MiB by default),
advertised in the block limits VPD page so that initiators split their I/O
- a `BlockDevice` with `readOnly: true` is reported write-protected, and `WRITE`
is refused with `DATA PROTECT` before any data is solicited
### CLI
The package ships a `vates-iscsi` binary that serves a file as a LUN:
```sh
> vates-iscsi /srv/lun.img
> vates-iscsi /srv/lun.img --serial MYSERIAL01 --port 3260
> vates-iscsi /srv/new.img --size 10G # create/grow the backing file first
```
```
Usage: vates-iscsi <file> [options]
Options:
--serial <serial> LUN serial number (default: derived from the IQN).
--size <bytes> Create/grow the backing file (suffixes K, M, G, T).
--iqn <iqn> Target IQN (default: iqn.2024-01.tech.vates:<file>).
--host <host> Listen address (default: 0.0.0.0).
--port <port> Listen port (default: 3260).
--block-size <bytes> Logical block size (default: 512).
-h, --help Show this help.
```
### Library
```js
import { IscsiTarget, FileBlockDevice } from '@vates/iscsi'
// The backing file must already exist and be sized to the LUN capacity,
// e.g. `fs.truncate('/srv/lun.img', 10 * 1024 ** 3)`.
const target = new IscsiTarget({
iqn: 'iqn.2024-01.tech.vates:lun0',
host: '0.0.0.0',
port: 3260,
lun: new FileBlockDevice({ path: '/srv/lun.img', blockSize: 512 }),
})
await target.listen()
// ... target is now discoverable and attachable by open-iscsi ...
await target.close()
```
#### Adding a target backend
A "backend" is whatever stores the LUN's bytes. The protocol layer translates
SCSI `READ`/`WRITE` CDBs (LBA + block count) into byte offsets/lengths against a
single small interface, so a backend only has to provide flat random access — it
never sees a PDU. `FileBlockDevice` (a sparse file) is the built-in one; implement
`BlockDevice` to expose anything else (a raw device, an object store, an
`@xen-orchestra/disk-transform` `RandomAccessDisk`, …):
```ts
interface BlockDevice {
open?(): Promise<void> // optional, awaited once before serving I/O
getSize(): number // total capacity in bytes; MUST be a multiple of getBlockSize()
getBlockSize(): number // logical block size, typically 512
read(offset: number, length: number): Promise<Buffer> // exactly `length` bytes at byte `offset`
write(offset: number, data: Buffer): Promise<void> // write `data.length` bytes at byte `offset`
flush(): Promise<void> // SYNCHRONIZE CACHE — persist any buffering
close(): Promise<void> // release resources
}
```
Contract to respect:
- **`getSize()` must be a multiple of `getBlockSize()`** and stable for the life of
the device. `open()` is the place to compute it (e.g. `stat` the file).
- **`read` must return exactly `length` bytes** — never short. Zero-fill a sparse
or past-EOF tail rather than returning fewer bytes; the SCSI layer already
bounds `offset + length` to the capacity.
- `offset` and `length` are always block-aligned (multiples of `getBlockSize()`).
- `read`/`write` may be **called concurrently** (commands are interleaved), so the
backend must be safe under overlapping calls; honour back-pressure and don't
buffer unboundedly.
- Throwing from `read`/`write`/`flush` tears the connection down; fail fast and
attach context to the error.
Minimal example — expose a read-only `RandomAccessDisk` (VHD / qcow2 / raw / a
remote NBD or iSCSI LUN) as a target LUN by mapping byte reads onto block reads:
```js
import { IscsiTarget } from '@vates/iscsi'
class RandomAccessDiskLun {
// `disk` is an initialized @xen-orchestra/disk-transform RandomAccessDisk.
constructor(disk) {
this.disk = disk
this.blockSize = disk.getBlockSize()
}
getSize() {
return this.disk.getVirtualSize()
}
getBlockSize() {
return this.blockSize
}
async read(offset, length) {
const out = Buffer.alloc(length) // pre-zeroed: sparse/holes read as zeros
const first = Math.floor(offset / this.blockSize)
const last = Math.floor((offset + length - 1) / this.blockSize)
for (let index = first; index <= last; index++) {
if (!this.disk.hasBlock(index)) {
continue // unallocated: leave zeros
}
const { data } = await this.disk.readBlock(index)
const blockStart = index * this.blockSize
const from = Math.max(0, offset - blockStart)
const to = Math.min(this.blockSize, offset + length - blockStart)
data.copy(out, blockStart + from - offset, from, to)
}
return out
}
// reported write-protected: WRITE is refused before reaching write()
readOnly = true
write() {
throw new Error('read-only LUN')
}
async flush() {}
async close() {
await this.disk.close()
}
}
const target = new IscsiTarget({ iqn: 'iqn.2024-01.tech.vates:disk0', lun: new RandomAccessDiskLun(disk) })
await target.listen()
```
For a writable backend, implement `write(offset, data)` symmetrically and make
`flush()` durable (e.g. `fsync`), as `FileBlockDevice` does.
### Initiator
`IscsiInitiator` is a minimal userspace iSCSI **client**: it connects to one
target portal, logs in, and reads the LUN. `IscsiDisk` wraps it as an
`@xen-orchestra/disk-transform` `RandomAccessDisk`, so a remote LUN drops straight
into the backup/transform pipeline (the iSCSI counterpart of `@vates/nbd-client`'s
`NbdDisk`). It is **read-only**, single-connection, digests off.
```js
import { IscsiInitiator, IscsiDisk } from '@vates/iscsi'
// Low-level byte reads:
const initiator = new IscsiInitiator({
host: '10.0.0.10',
port: 3260,
targetIqn: 'iqn.2024-01.tech.vates:lun0',
chap: { user: 'alice', secret: 's3cr3t' }, // optional, see Authentication
})
await initiator.connect()
console.log(initiator.getSize(), initiator.getBlockSize())
const bytes = await initiator.read(/* offset */ 0, /* length */ 1024 * 1024)
await initiator.close()
// …or as a RandomAccessDisk (blockSize must be a multiple of the LUN block size):
const disk = new IscsiDisk(
{ host: '10.0.0.10', targetIqn: 'iqn.2024-01.tech.vates:lun0' },
{ blockSize: 2 * 1024 * 1024 }
)
await disk.init()
// disk.getVirtualSize(), disk.getBlockIndexes(), await disk.readBlock(i), …
await disk.close()
```
First-cut `IscsiDisk` treats the LUN as fully allocated (every block reported as
present, like `RawDisk`); a changed-block map (e.g. a Pure "Volume Difference")
can narrow it later.
### Authentication (CHAP)
The target and initiator support **one-way CHAP** (MD5) in opposite roles; mutual
(two-way) CHAP is not yet implemented.
**Target as authenticator.** Pass `chap` to require each _data_ session to prove a
credential; the target challenges the initiator and rejects the login (Login
Response status-class _Initiator Error_ / _authentication failure_) on a mismatch:
```js
const target = new IscsiTarget({
iqn: 'iqn.2024-01.tech.vates:lun0',
lun,
chap: { user: 'alice', secret: 's3cr3t' },
})
```
This interoperates with the Linux **open-iscsi** initiator (the one XCP-ng uses);
configure its node record's incoming credential:
```sh
iscsiadm -m node -T <iqn> -p <portal> -o update -n node.session.auth.authmethod -v CHAP
iscsiadm -m node -T <iqn> -p <portal> -o update -n node.session.auth.username -v alice
iscsiadm -m node -T <iqn> -p <portal> -o update -n node.session.auth.password -v s3cr3t
```
CHAP is enforced only on **Normal (data) sessions**; **Discovery** sessions (which
can only run SendTargets and never reach the LUN) stay unauthenticated, so plain
`iscsiadm -m discovery` keeps working. open-iscsi requires the secret to be 12–16
characters.
**Initiator as responder.** Pass `chap` to `IscsiInitiator`/`IscsiDisk` to answer a
target's challenge (e.g. a Pure array acting as authenticator). If the target
_requires_ mutual CHAP the login is cleanly rejected (we never challenge back).
### Security
iSCSI has no data-path encryption of its own. CHAP authenticates the login only;
with no digests, **CDBs and data remain cleartext**. Anyone who can reach an
unauthenticated port can read/write the LUN, and a CHAP-protected one is still
exposed to eavesdropping and tampering on the wire. Expose targets only on a
trusted, isolated storage network, and use **IPsec** (the RFC-specified mechanism)
or equivalent if confidentiality is required.
### Testing
- `yarn test` — fast unit tests (PDU framing, SCSI encoders/decoders, login
negotiation, sequence-number arithmetic).
- `yarn test-integration` — a self-contained loopback test (an in-process
minimal initiator drives discovery, identity, write/read-back, and command
interleaving), plus an open-iscsi interop test that **requires root and
`iscsiadm`** and skips gracefully otherwise.
## Contributions
Contributions are _very_ welcomed, either on the documentation or on
the code.
You may:
- report any [issue](https://github.com/vatesfr/xen-orchestra/issues)
you've encountered;
- fork and create a pull request.
## License
[ISC](https://spdx.org/licenses/ISC) © [Vates SAS](https://vates.fr)

170
@vates/iscsi/VALIDATION.md Normal file
View File

@@ -0,0 +1,170 @@
# Validation notes
Extracted from a longer production-readiness pass covering both this package and the `LiveMount` mixin that
consumes it; this file keeps only what's specific to `@vates/iscsi` itself — protocol-level correctness,
concurrency, and interop with standard tooling. Local testing notes covering the LiveMount/XAPI integration,
VM boot mechanics, and deployment specifics live elsewhere and aren't reproduced here.
## Bugs found and fixed
### In this validation pass
1. **Read-path error isolation** (`scsi.mts`) — `lun.read()` failures were previously uncaught, killing the
whole connection for one bad block. Now wrapped in try/catch, reporting `CHECK_CONDITION`/`MEDIUM_ERROR` for
that command only, mirroring the write path's existing `#handleDataOut` error handling.
- Unit test: `scsi.test.mts` — "reports a failed lun.read() as CHECK CONDITION / MEDIUM ERROR, not an
uncaught throw".
- Regression test: `loopback.integ.mts`, `describe('READ error isolation', ...)` — a LUN whose `read()`
throws for one LBA still returns `CHECK_CONDITION` for that command; the connection stays alive and a
different, healthy LBA read afterward still succeeds.
2. **Connection replacement instead of refusal** (`index.mts`) — found via a real end-to-end guest-boot test
against an S3-backed backup repository: `IscsiTarget#onConnection` used to permanently _refuse_ any new
connection while an existing one was tracked (`MaxConnections=1` enforcement), with no way to tell a
genuinely stale/abandoned connection (initiator gave up and reconnected without ever closing the old TCP
socket) from a healthy one still in use. Under a slow backend this wedges the mount forever: the host's
initiator times out and retries, gets refused, retries again, forever — reproduced live as an 8+ minute
guest boot stall, `WARN refusing concurrent connection (MaxConnections=1)` repeating in the log the whole
time. Now a new connection always tears down whatever connection currently exists and takes over, rather
than being refused. Deliberately not full RFC 7143 session reinstatement (no ISID matching) — chosen as the
simpler, sufficient fix for this target's single-consumer, ephemeral, CHAP-guarded design; see the class doc
comment in `index.mts` for the reasoning and the more rigorous alternative considered.
- Regression test: `loopback.integ.mts`, `describe('connection replacement', ...)` — a second connection is
accepted (not refused) while the first is still open, the first's socket gets torn down, and the new
connection is fully usable (read + logout succeed).
- Re-verified end-to-end after deploying the fix: same S3-backed boot scenario completed successfully
(~5 min total, vs ~20-25s for the equivalent local-storage boot — S3 itself is slow, but the mount no
longer wedges).
- Refined afterwards: the replacement used to happen as soon as the TCP connection was accepted, so any
peer reaching the port (a port scan, a failed CHAP login, a Discovery session) interrupted the session in
use. It now only happens once the new connection has established a Normal session; until then,
connections are bounded by `loginTimeoutMs` and `maxPendingConnections`. Regression tests:
`loopback.integ.mts`, `describe('connection admission', ...)`.
### Independently landed, cross-referenced against this pass's coverage
Five more fixes landed on this branch outside this validation effort. Read in full and checked against what
had (and hadn't) actually been exercised black-box, since two are severe:
1. **`CachedDiskBlockDevice.mts` — acknowledged writes silently and permanently reverted.** A write fully
covering a not-yet-cached block used to skip fetching the source (an "optimization" — the write supplies
all the bytes anyway). But if a _read_ to that same block had already triggered an in-flight fetch, that
fetch could land **after** the write and silently overwrite it with stale source data — guest gets `GOOD`,
data reverts moments later. Fixed by always joining any in-flight fetch before writing. Has a white-box
regression test (pauses a fake disk mid-fetch to force the race). This is exactly the class of bug this
whole effort was looking for, and its vulnerable path — read+write racing on the same never-touched block —
was never actually exercised by this pass's own earlier black-box testing (sequential `dd` and single-job
`fio` both avoid same-offset self-overlap by construction), so the "clean" results up to that point were
consistent with the bug being present, not proof of its absence. Closed that gap afterward (see below).
2. **`connection.mts`/`writePath.mts` — uninitialized heap memory leaked to the LUN on a malformed Data-Out.**
Staging buffer is `Buffer.allocUnsafe`, never zeroed; the old completion check summed received bytes, but
`Buffer.copy` silently clamps on out-of-range/overlapping offsets — a malformed/out-of-order Data-Out
sequence could reach "totalLength received" with gaps of recycled heap memory that then get written to the
LUN and read back by any guest. Real info-disclosure bug, needs a misbehaving/malicious initiator to trigger
(a well-behaved one never sends malformed Data-Out). Fixed with strict contiguity validation; extensively
tested (`connection.test.mts` +586 lines, `writePath.test.mts` +168 lines, both white-box, covering the
malformed-input space directly — not something black-box testing with real tooling can easily reach at all).
3. **`connection.mts` — CmdSN window corrupted by every Data-Out (i.e. every write).** Command-window tracking
read bytes 24-27 as CmdSN off every inbound PDU, but those bytes are reserved on Data-Out per RFC 7143
§11.7 — every write rewound the advertised window. Benign at normal scale (initiators discard the rewind as
stale via serial-number comparison until session CmdSN passes 2³¹) — none of this pass's testing came close
to that scale, so nothing here calls prior results into question. Fixed with an opcode allowlist; has its
own regression test.
4. **`CachedDiskBlockDevice.hydrate()` — no longer throws/orphans on a concurrent unmount.** `hydrate()` now
accepts an `AbortSignal`, stopping at the next block boundary and rejecting cleanly once in-flight blocks
settle, rather than the caller having to serialize hydrate against unmount itself (the orchestration half of
this fix — not running hydrate inside the same long-lived task as the mount — lives in the LiveMount mixin,
outside this package). Tested at the `CachedDiskBlockDevice` level: aborting mid-hydrate keeps everything
already cached, rejects rather than silently returning partial success, and a fresh signal that's already
aborted refuses to start at all.
5. **`initiator.mts` — went silent between commands, dropped by a target's ping timeout.** `IscsiInitiator`'s
read loop used to stop once nothing was outstanding, so during any idle period a target's NOP-In keepalive
sat unread — real targets (LIO's `nopin_response_timeout`, 30s by default) drop sessions with unanswered
pings, with no reconnect logic to recover. Fixed with a loop that keeps draining for the life of the
connection and replies to NOP-In pings (`#pump`/`#replyNop`); added `initiator.integ.mts` (white-box,
real loopback). Never exercised by this pass's own black-box testing at all — the whole effort was
LiveMount-focused, which only ever uses the _target_ role. Attempting to close that gap surfaced two
further, independent findings — see "Initiator interop findings" below.
## Black-box protocol-level testing (standalone target, real tooling)
- **Data integrity under concurrent load**: 30-minute `fio --rw=randrw --bs=4k --iodepth=64 --numjobs=1
--verify=crc32c` soak against the standalone target over real `iscsiadm` loopback — 0 verify errors, ~16.4k
read + ~10.6k write IOPS sustained throughout. (An earlier `numjobs=4` run produced ~116 "bad header"
errors; bisected and reproduced identically against plain local file I/O with zero iSCSI involvement —
a known fio limitation where independent jobs pointed at the same file race each other's own bookkeeping,
not a target bug. `numjobs=1` at higher `iodepth` avoids the false positive while still exercising real
queue depth.)
- **Filesystem round-trip across a full session teardown**: real corpus (2419 files, 53 MB) written, hashed,
then a full cold logout/login cycle plus a host-side page-cache drop before re-reading — all 2419 files
byte-identical.
- **Kill the target process mid-write**: `kill -9`'d mid-transfer with a direct-I/O write in flight. The write
correctly blocked (not a fast, silent "success") until the session recovered, then completed with the exact
intended pattern across the full range — no silent data loss, no torn/partial write at the interruption point.
- **Real cross-machine network partition** (two separate VMs, not loopback): a mid-transfer `iptables DROP`
held for 15s against the target's own port. Compared directly to `tgt` as a baseline over the identical
topology — both stalled through the outage and completed with 0 corrupted bytes once restored; no divergent
behavior from a mature reference implementation.
- **Sequential throughput vs `tgt`** (same two-VM link, plain iSCSI, no XAPI/SM in the path): `@vates/iscsi`
reached ~70% (write) / ~76% (read) of `tgt`'s numbers — noticeably slower but the same order of magnitude,
not a large gap. (A separate, much larger ~10x slowdown measured through the full LiveMount/XAPI/SM stack
was isolated to the host's own `iscsi` SM/tapdisk driver by elimination — outside this package.)
- **Targeted regression testing for bug 1 above**, using a single-job, high-`iodepth`, small-range `fio`
pattern (deliberately avoiding the `numjobs>1` false-positive from the first bullet) to force many concurrent
operations onto the same underlying cache block without needing multiple uncoordinated processes: clean
(0 verify errors) against local storage, against a slow S3-backed remote (a wider fetch-latency window gives
a racing write far more time to land mid-fetch — the exact scenario the bug was about), and against an
actively-running `hydrate()` sweep overlapping guest I/O on the same disk. All three passed.
## Flagged, not fixed
- **`FileBlockDevice.read()` zero-fills past a short/EOF read instead of throwing** — unlike `LocalBlockDevice`
(the actual production cache backend), which throws on the identical condition. Confirmed via grep: only the
standalone CLI tool and this package's own tests use `FileBlockDevice` today, so there's no reachable data-
loss path in production right now — but it's a public export, so a future feature backing a LUN directly
with a file would silently inherit the zero-fill-over-error behavior. This is a design-intent question (is
the CLI tool's "sparse tail" leniency deliberate?), not a clear bug — flagging for a decision rather than
silently changing it.
## Initiator interop findings (this session)
Bug 5's fix (above) had never been exercised against a real target under real conditions — only the white-box
`initiator.integ.mts`. Attempting to close that gap with a live idle-period/NOP-In test against two different
real target implementations surfaced two independent, real findings before the actual keepalive behavior could
be exercised:
- **`tgt` reserves LUN 0 as a built-in, zero-size "controller" device** on every target, and this is not
removable via `tgtadm` (`tgtadm: this logical unit number already exists` / `invalid request` on any attempt
to delete or replace it). `IscsiInitiator.connect()`/`read()` are hardcoded to LUN 0 (matching this package's
own single-LUN-at-0 target design), so `connect()` fails immediately with `CHECK_CONDITION` against `tgt`
regardless of where a real data LUN is added (tried LUN 1). Pure tooling mismatch — `tgt`'s own convention
doesn't fit a client that only ever speaks to LUN 0 — not a bug in either side.
- **LIO (`targetcli-fb`/kernel target) does complete login differently, and `IscsiInitiator` can't get through
it.** With a real data LUN at LUN 0 (LIO doesn't reserve it), `connect()` still fails — `#operationalLogin`
(`initiator.mts`) sends exactly one operational-stage login PDU, requests immediate transit (`T=1`), and only
checks the transit bit on the reply; there's no second round to answer a target's own counter-proposal if it
declines transit on the first exchange. LIO's kernel target does exactly that: it counter-proposes several of
its own operational keys (`MaxBurstLength`, `DefaultTime2Wait`, `DefaultTime2Retain`, `MaxOutstandingR2T`) in
the same response and withholds transit until they're answered, confirmed directly in `dmesg`:
```
No response for proposed key "MaxBurstLength".
No response for proposed key "DefaultTime2Wait".
No response for proposed key "DefaultTime2Retain".
No response for proposed key "MaxOutstandingR2T".
iSCSI Login negotiation failed.
```
This works fine against `tgt` and against this package's own `IscsiTarget` (both transit on the first round),
so it's never been visible before — but it's a **real, previously-unknown limitation of `IscsiInitiator`'s
login state machine**: single operational round-trip only, no handling for a target that insists on its own
multi-round negotiation before agreeing to transit. Worth knowing given the initiator role's real intended
use (reading from a third-party storage array as an initiator) targets unknown-strictness third-party target
software, not `tgt` or this package's own target.
Given both blockers are about _login/LUN setup_, not the keepalive mechanism itself, the actual idle-period/
NOP-In behavior could not be exercised live this session. The white-box `initiator.integ.mts` coverage for
bug 5 stands as the only current test of that specific fix.

View File

@@ -0,0 +1,6 @@
// @ts-check
import eslint from '@eslint/js'
import tseslint from 'typescript-eslint'
export default tseslint.config(eslint.configs.recommended, tseslint.configs.recommended)

61
@vates/iscsi/package.json Normal file
View File

@@ -0,0 +1,61 @@
{
"name": "@vates/iscsi",
"version": "0.4.0",
"main": "dist/index.mjs",
"types": "dist/index.d.mts",
"bin": {
"vates-iscsi": "dist/cli.mjs"
},
"license": "ISC",
"private": false,
"type": "module",
"description": "Minimal iSCSI target and initiator: expose or read a single LUN, with one-way CHAP",
"keywords": [
"iscsi",
"target",
"initiator",
"chap",
"scsi",
"lun",
"block",
"storage"
],
"dependencies": {
"@vates/async-each": "^1.0.3",
"@vates/read-chunk": "^1.2.1",
"@xen-orchestra/disk-transform": "^1.3.4",
"@xen-orchestra/log": "^0.7.2",
"promise-toolbox": "^0.21.0"
},
"devDependencies": {
"@eslint/js": "^9.19.0",
"@types/node": "^22.0",
"rimraf": "^6.0.1",
"typescript": "~5.6",
"typescript-eslint": "^8.61.0"
},
"scripts": {
"build": "tsc",
"clean": "rimraf dist/",
"dev": "tsc --watch",
"postversion": "npm publish --access public",
"prebuild": "rimraf dist/",
"predev": "yarn run prebuild",
"test": "tsc && node --test \"dist/**/*.test.mjs\"",
"test-integration": "tsc && node --test \"dist/**/*.integ.mjs\""
},
"homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@vates/iscsi",
"bugs": "https://github.com/vatesfr/xen-orchestra/issues",
"repository": {
"directory": "@vates/iscsi",
"type": "git",
"url": "https://github.com/vatesfr/xen-orchestra.git"
},
"author": {
"name": "Vates SAS",
"url": "https://vates.fr"
},
"engines": {
"node": ">=22"
}
}

View File

@@ -0,0 +1,321 @@
import { asyncEach } from '@vates/async-each'
import { createLogger, type Logger } from '@xen-orchestra/log'
import type { ProgressHandler, RandomAccessDisk } from '@xen-orchestra/disk-transform'
import type { BlockDevice } from './backend.mjs'
const log: Logger = createLogger('vates:iscsi:cached-disk-block-device')
const DEFAULT_BLOCK_SIZE = 512
const DEFAULT_HYDRATE_CONCURRENCY = 8
export interface CachedDiskBlockDeviceOptions {
/**
* Source disk, already `init()`ed by the caller. Only ever read from, one
* whole block at a time.
*/
readonly disk: RandomAccessDisk
/**
* Writable store holding what has been read from {@link disk} so far, already
* `open()`ed and at least as large as the disk's virtual size.
*/
readonly cache: BlockDevice
/**
* Logical block size advertised to initiators, in bytes. Defaults to 512.
* The disk's own block size must be a multiple of it.
*/
readonly blockSize?: number
/**
* Notified with the materialized fraction (0..1) every time a new block is
* cached — on demand or through {@link hydrate} — and via `done()` once
* every block has been. Reused from the `disk-transform` export/import
* pipelines rather than inventing a new shape.
*/
readonly progressHandler?: ProgressHandler
/**
* A previously-persisted bitmap to resume from, one byte per block (see
* {@link CachedDiskBlockDevice}'s class doc). Its length must match the
* block count computed from `disk`/`blockSize`; a mismatch (e.g. the disk
* was resized since it was saved) is treated as stale and discarded rather
* than rejected, so a corrupt/outdated resume file never blocks a mount.
*/
readonly initialBitmap?: Buffer
/**
* Notified with a block's index the moment it is newly cached — narrower
* than `progressHandler`, which only carries an aggregate fraction and
* cannot say *which* block just landed. Meant for persisting the bitmap
* one byte at a time; best-effort like `progressHandler`, a failure here
* must never break block materialization.
*/
readonly onBlockCached?: (index: number) => void
}
/**
* A read-write {@link BlockDevice} over a {@link RandomAccessDisk}, backed by
* a local writable store: each source block is materialized into the store
* the first time it's needed, tracked by a bitmap so repeat access never goes
* back to the source. Call {@link hydrate} to force the whole disk in upfront.
*
* The bitmap is one byte per block, not one bit: a caller that persists it
* incrementally (see `onBlockCached`) writes one block's byte at a time, and
* every block's offset must be independent of every other's for concurrent
* writes to be safe. A real bit-packed bitmap fails that: several blocks'
* bits sharing one byte would need a read-modify-write of that byte, which
* two blocks completing concurrently could race on and lose an update. One
* byte per block costs a little more memory (negligible in absolute terms —
* a few hundred KB even for a huge disk) and buys that independence for free.
*
* Blocks the source doesn't have are marked present without being written —
* relies on the store reading as zero where nothing was ever written.
*
* Writes land in the store only; the source is never written to, so the
* mount diverges from it as soon as anything writes.
*/
export class CachedDiskBlockDevice implements BlockDevice {
readonly #disk: RandomAccessDisk
readonly #cache: BlockDevice
readonly #blockSize: number
readonly #progressHandler?: ProgressHandler
readonly #onBlockCached?: (index: number) => void
readonly #initialBitmap?: Buffer
// one byte per source block, 0 or 1, set once the block is in the cache
#bitmap: Buffer = Buffer.alloc(0)
#blockCount = 0
#cachedCount = 0
// source blocks being materialized right now, so concurrent reads of
// overlapping ranges fetch each block once
readonly #inFlight: Map<number, Promise<void>> = new Map()
#diskBlockSize?: number
#size?: number
constructor({
disk,
cache,
blockSize = DEFAULT_BLOCK_SIZE,
progressHandler,
initialBitmap,
onBlockCached,
}: CachedDiskBlockDeviceOptions) {
if (!Number.isInteger(blockSize) || blockSize <= 0) {
throw new Error(`blockSize must be a positive integer, got ${blockSize}`)
}
this.#disk = disk
this.#cache = cache
this.#blockSize = blockSize
this.#progressHandler = progressHandler
this.#initialBitmap = initialBitmap
this.#onBlockCached = onBlockCached
}
/** Read the source geometry and allocate (or resume) the bitmap. */
async open(): Promise<void> {
const blockSize = this.#blockSize
const diskBlockSize = this.#disk.getBlockSize()
if (diskBlockSize % blockSize !== 0) {
throw new Error(`disk block size (${diskBlockSize}) is not a multiple of the LUN block size (${blockSize})`)
}
const size = this.#disk.getVirtualSize()
if (size <= 0 || size % blockSize !== 0) {
throw new Error(`disk virtual size (${size}) is not a positive multiple of the LUN block size (${blockSize})`)
}
if (this.#cache.getSize() < size) {
throw new Error(`cache (${this.#cache.getSize()} bytes) is smaller than the disk (${size} bytes)`)
}
this.#diskBlockSize = diskBlockSize
this.#size = size
this.#blockCount = Math.ceil(size / diskBlockSize)
const initialBitmap = this.#initialBitmap
if (initialBitmap !== undefined && initialBitmap.length === this.#blockCount) {
this.#bitmap = initialBitmap
let cachedCount = 0
for (const byte of initialBitmap) {
if (byte !== 0) {
cachedCount++
}
}
this.#cachedCount = cachedCount
} else {
if (initialBitmap !== undefined) {
log.warn('discarding stale initial bitmap (block count mismatch)', {
gotLength: initialBitmap.length,
expectedLength: this.#blockCount,
})
}
this.#bitmap = Buffer.alloc(this.#blockCount)
}
log.debug('opened', {
size,
blockSize,
diskBlockSize,
blockCount: this.#blockCount,
resumedBlocks: this.#cachedCount,
})
}
getSize(): number {
const size = this.#size
if (size === undefined) {
throw new Error('CachedDiskBlockDevice.open() must be called before I/O')
}
return size
}
getBlockSize(): number {
return this.#blockSize
}
/** How much of the source has been materialized into the cache. */
getMaterialized(): { blocks: number; total: number } {
return { blocks: this.#cachedCount, total: this.#blockCount }
}
/**
* Force every block into the cache — including holes, so `getMaterialized()`
* reaches `total` once done, matching what a full sequential read would
* naturally mark. Already-materialized blocks are skipped, so this resumes
* rather than redoes whatever on-demand reads (or a previous, interrupted
* hydration) already did.
*
* Hydrating a large disk takes hours, so it must be interruptible: `signal`
* stops it at the next block boundary, rejecting once the blocks already in
* flight settle. Nothing is rolled back — every block cached so far stays
* cached, and because this call resumes, aborting and calling again later
* picks up where it stopped rather than starting over.
*/
async hydrate({
concurrency = DEFAULT_HYDRATE_CONCURRENCY,
signal,
}: { concurrency?: number; signal?: AbortSignal } = {}): Promise<void> {
signal?.throwIfAborted()
const blockCount = this.#blockCount
await asyncEach(
(function* allBlockIndexes() {
for (let index = 0; index < blockCount; index++) {
yield index
}
})(),
index => this.#ensureBlock(index),
{ concurrency, signal }
)
}
#isCached(index: number): boolean {
return this.#bitmap[index] !== 0
}
#setCached(index: number): void {
if (!this.#isCached(index)) {
this.#bitmap[index] = 1
this.#cachedCount++
this.#reportProgress()
// best-effort, same contract as #reportProgress: a synchronous throw
// here must never break block materialization. Any async failure (e.g.
// the persisted-bitmap write itself) is the callback's own job to catch.
try {
this.#onBlockCached?.(index)
} catch (error) {
log.warn('onBlockCached failed', { error, index })
}
}
}
// best-effort: a failure here (sync or async) must never break block
// materialization. Wrapping the body in an `async` function, rather than a
// plain try/catch, is what makes a *synchronous* throw from the handler
// land in the same `.catch()` as an asynchronous rejection.
#reportProgress(): void {
const handler = this.#progressHandler
if (handler === undefined) {
return
}
const fraction = this.#cachedCount / this.#blockCount
;(async () => {
await handler.setProgress(fraction)
// every block is in the cache: nothing more will ever be reported
if (fraction === 1) {
await handler.done()
}
})().catch(error => log.warn('progress handler failed', { error }))
}
/**
* Copy one source block into the cache, unless it is already there. Concurrent
* calls for the same block share a single fetch.
*/
#ensureBlock(index: number): Promise<void> {
if (this.#isCached(index)) {
return Promise.resolve()
}
let pending = this.#inFlight.get(index)
if (pending === undefined) {
// the entry goes away whether the fetch succeeded or not, so a failed
// block is retried later instead of being remembered as broken
pending = this.#fetchBlock(index).finally(() => this.#inFlight.delete(index))
this.#inFlight.set(index, pending)
}
return pending
}
async #fetchBlock(index: number): Promise<void> {
const diskBlockSize = this.#diskBlockSize as number
if (this.#disk.hasBlock(index)) {
const offset = index * diskBlockSize
const { data } = await this.#disk.readBlock(index)
// a source block is always full-size: the last one may run past the end of
// the disk, and the cache must not be written beyond it
const length = Math.min(diskBlockSize, this.getSize() - offset)
await this.#cache.write(offset, length < data.length ? data.subarray(0, length) : data)
}
// a block the source does not have stays as the cache has it, which is
// zeroes as long as nothing has written there
this.#setCached(index)
}
#checkRange(offset: number, length: number): void {
const size = this.getSize()
if (offset < 0 || length < 0 || offset + length > size) {
throw new Error(`access of ${length} bytes at ${offset} is out of range (size ${size})`)
}
}
async read(offset: number, length: number): Promise<Buffer> {
this.#checkRange(offset, length)
if (length === 0) {
return Buffer.alloc(0)
}
const diskBlockSize = this.#diskBlockSize as number
const last = Math.floor((offset + length - 1) / diskBlockSize)
for (let index = Math.floor(offset / diskBlockSize); index <= last; index++) {
await this.#ensureBlock(index)
}
return this.#cache.read(offset, length)
}
async write(offset: number, data: Buffer): Promise<void> {
this.#checkRange(offset, data.length)
if (data.length === 0) {
return
}
// every covered block must be materialized first, even the ones this write
// covers entirely: skipping the fetch would let an already in-flight one
// land *after* the payload and overwrite it with the source's bytes.
// (optimization left out on purpose — it has to join the in-flight fetch)
await this.read(offset, data.length)
await this.#cache.write(offset, data)
}
async flush(): Promise<void> {
await this.#cache.flush()
}
async close(): Promise<void> {
try {
await this.#cache.close()
} finally {
await this.#disk.close()
}
}
}

View File

@@ -0,0 +1,605 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { RandomAccessDisk, type DiskBlock, type ProgressHandler } from '@xen-orchestra/disk-transform'
import { CachedDiskBlockDevice } from './CachedDiskBlockDevice.mjs'
import type { BlockDevice } from './backend.mjs'
const DISK_BLOCK_SIZE = 4096
const BLOCK_COUNT = 4
const DISK_SIZE = BLOCK_COUNT * DISK_BLOCK_SIZE
const bytePattern = (i: number) => (i * 31 + 11) & 0xff
/** Sparse source: only blocks 0 and 2 exist, like a differencing chain. */
class StubDisk extends RandomAccessDisk {
readBlockCalls: Array<number> = []
closed = false
/** set to hold every fetch open until it resolves */
pause?: Promise<void>
constructor(readonly allocated: Set<number> = new Set([0, 2])) {
super()
}
getBlockSize(): number {
return DISK_BLOCK_SIZE
}
getVirtualSize(): number {
return DISK_SIZE
}
isDifferencing(): boolean {
return false
}
getBlockIndexes(): Array<number> {
return [...this.allocated]
}
hasBlock(index: number): boolean {
return this.allocated.has(index)
}
async init(): Promise<void> {}
async close(): Promise<void> {
this.closed = true
}
async readBlock(index: number): Promise<DiskBlock> {
if (!this.allocated.has(index)) {
throw new Error(`Block ${index} not found in chain`)
}
this.readBlockCalls.push(index)
if (this.pause !== undefined) {
await this.pause
}
const data = Buffer.alloc(DISK_BLOCK_SIZE)
for (let i = 0; i < data.length; i++) {
data[i] = bytePattern(index * DISK_BLOCK_SIZE + i)
}
return { index, data }
}
}
/** In-memory stand-in for the local VDI. */
class StubCache implements BlockDevice {
readonly content: Buffer
writes: Array<{ offset: number; length: number }> = []
closed = false
flushed = 0
failNextWrite = false
constructor(size = DISK_SIZE) {
this.content = Buffer.alloc(size)
}
getSize(): number {
return this.content.length
}
getBlockSize(): number {
return 512
}
async read(offset: number, length: number): Promise<Buffer> {
return Buffer.from(this.content.subarray(offset, offset + length))
}
async write(offset: number, data: Buffer): Promise<void> {
if (this.failNextWrite) {
this.failNextWrite = false
throw new Error('cache write failed')
}
this.writes.push({ offset, length: data.length })
data.copy(this.content, offset)
}
async flush(): Promise<void> {
this.flushed++
}
async close(): Promise<void> {
this.closed = true
}
}
const expectedAt = (offset: number, length: number, allocated: Set<number>): Buffer => {
const expected = Buffer.alloc(length)
for (let i = 0; i < length; i++) {
const absolute = offset + i
if (allocated.has(Math.floor(absolute / DISK_BLOCK_SIZE))) {
expected[i] = bytePattern(absolute)
}
}
return expected
}
const make = async (allocated?: Set<number>, cacheSize?: number) => {
const disk = new StubDisk(allocated)
const cache = new StubCache(cacheSize)
const device = new CachedDiskBlockDevice({ disk, cache })
await device.open()
return { cache, device, disk }
}
describe('open', () => {
it('exposes the disk capacity and a fully cold bitmap', async () => {
const { device } = await make()
assert.equal(device.getSize(), DISK_SIZE)
assert.equal(device.getBlockSize(), 512)
assert.deepEqual(device.getMaterialized(), { blocks: 0, total: BLOCK_COUNT })
})
it('refuses a cache smaller than the disk', async () => {
const device = new CachedDiskBlockDevice({ disk: new StubDisk(), cache: new StubCache(DISK_SIZE - 512) })
await assert.rejects(device.open(), /smaller than the disk/)
})
it('resumes from an initial bitmap of the right length', async () => {
// all blocks allocated, so the not-yet-cached one below is a real fetch,
// not a hole (which would never call readBlock regardless of the bitmap)
const disk = new StubDisk(new Set([0, 1, 2, 3]))
const cache = new StubCache()
const initialBitmap = Buffer.alloc(BLOCK_COUNT)
initialBitmap[0] = 1
initialBitmap[2] = 1
const device = new CachedDiskBlockDevice({ disk, cache, initialBitmap })
await device.open()
assert.deepEqual(device.getMaterialized(), { blocks: 2, total: BLOCK_COUNT })
// already marked cached: served from the (empty) cache store, source untouched
assert.deepEqual(await device.read(0, 512), Buffer.alloc(512))
assert.deepEqual(disk.readBlockCalls, [])
// not marked cached: still fetched from the source as usual
assert.deepEqual(await device.read(DISK_BLOCK_SIZE, 512), expectedAt(DISK_BLOCK_SIZE, 512, disk.allocated))
assert.deepEqual(disk.readBlockCalls, [1])
})
it('discards an initial bitmap of the wrong length and starts cold', async () => {
const disk = new StubDisk()
const cache = new StubCache()
const device = new CachedDiskBlockDevice({ disk, cache, initialBitmap: Buffer.alloc(BLOCK_COUNT - 1, 1) })
await device.open()
assert.deepEqual(device.getMaterialized(), { blocks: 0, total: BLOCK_COUNT })
assert.deepEqual(await device.read(0, 512), expectedAt(0, 512, disk.allocated))
assert.deepEqual(disk.readBlockCalls, [0])
})
})
describe('read', () => {
it('materializes a block on first read and serves later ones from the cache', async () => {
const { device, disk, cache } = await make()
assert.deepEqual(await device.read(0, 512), expectedAt(0, 512, disk.allocated))
assert.deepEqual(disk.readBlockCalls, [0])
// the whole source block was written, not just the part that was read
assert.deepEqual(cache.writes, [{ offset: 0, length: DISK_BLOCK_SIZE }])
assert.deepEqual(device.getMaterialized(), { blocks: 1, total: BLOCK_COUNT })
// a second read of a different part of the same block does not go back
assert.deepEqual(await device.read(1024, 512), expectedAt(1024, 512, disk.allocated))
assert.deepEqual(disk.readBlockCalls, [0])
assert.equal(cache.writes.length, 1)
})
it('marks a hole as present without writing it', async () => {
const { device, disk, cache } = await make()
assert.deepEqual(await device.read(DISK_BLOCK_SIZE, 512), Buffer.alloc(512))
assert.deepEqual(disk.readBlockCalls, [])
assert.deepEqual(cache.writes, [])
assert.deepEqual(device.getMaterialized(), { blocks: 1, total: BLOCK_COUNT })
})
it('stitches a range spanning a present and a missing block', async () => {
const { device, disk } = await make()
const offset = DISK_BLOCK_SIZE - 512
const length = DISK_BLOCK_SIZE + 1024
assert.deepEqual(await device.read(offset, length), expectedAt(offset, length, disk.allocated))
assert.deepEqual(disk.readBlockCalls, [0, 2])
})
it('fetches a block once for concurrent overlapping reads', async () => {
const { device, disk } = await make()
// hold the fetch open until both reads are in flight
let release = () => {}
disk.pause = new Promise<void>(resolve => {
release = resolve
})
const first = device.read(0, 1024)
const second = device.read(512, 1024)
await new Promise(resolve => setImmediate(resolve))
release()
assert.deepEqual(await first, expectedAt(0, 1024, disk.allocated))
assert.deepEqual(await second, expectedAt(512, 1024, disk.allocated))
assert.deepEqual(disk.readBlockCalls, [0])
})
it('rejects a read past the end of the disk', async () => {
const { device } = await make()
await assert.rejects(device.read(DISK_SIZE - 256, 512), /out of range/)
})
it('clamps the tail block to the disk size', async () => {
// 3 blocks + half a block, so the last source block runs past the end
class ShortDisk extends StubDisk {
getVirtualSize(): number {
return 3 * DISK_BLOCK_SIZE + 2048
}
}
const disk = new ShortDisk(new Set([3]))
const cache = new StubCache(3 * DISK_BLOCK_SIZE + 2048)
const device = new CachedDiskBlockDevice({ disk, cache })
await device.open()
await device.read(3 * DISK_BLOCK_SIZE, 2048)
assert.deepEqual(cache.writes, [{ offset: 3 * DISK_BLOCK_SIZE, length: 2048 }])
})
})
describe('write', () => {
it('lands in the cache and is readable back, without touching the source', async () => {
const { device, cache, disk } = await make()
const payload = Buffer.alloc(512, 0xee)
await device.write(0, payload)
assert.deepEqual(await device.read(0, 512), payload)
// the block was materialized first, since the write covers only part of it
assert.deepEqual(disk.readBlockCalls, [0])
// ... and the bytes after the write are still the source's
assert.deepEqual(await device.read(512, 512), expectedAt(512, 512, disk.allocated))
assert.equal(cache.closed, false)
})
it('materializes the block even when the write covers it entirely', async () => {
const { device, disk } = await make()
await device.write(0, Buffer.alloc(DISK_BLOCK_SIZE, 0xee))
// the fetch cannot be skipped: see the race covered by the next test
assert.deepEqual(disk.readBlockCalls, [0])
assert.deepEqual(device.getMaterialized(), { blocks: 1, total: BLOCK_COUNT })
})
it('is not overwritten by a fetch already in flight when it lands', async () => {
const { device, disk } = await make()
// hold block 0's fetch open, so the write lands while it is in flight
let release = () => {}
disk.pause = new Promise<void>(resolve => {
release = resolve
})
const reading = device.read(0, 512)
await new Promise(resolve => setImmediate(resolve))
const payload = Buffer.alloc(DISK_BLOCK_SIZE, 0xee)
const writing = device.write(0, payload)
await new Promise(resolve => setImmediate(resolve))
release()
await reading
await writing
// the source block was fetched once, and its bytes did not come back on top
assert.deepEqual(disk.readBlockCalls, [0])
assert.deepEqual(await device.read(0, DISK_BLOCK_SIZE), payload)
})
})
describe('hydrate', () => {
it('materializes every block the source has, holes included', async () => {
const { device, disk, cache } = await make()
await device.hydrate()
assert.deepEqual(device.getMaterialized(), { blocks: BLOCK_COUNT, total: BLOCK_COUNT })
// only the allocated ones were actually fetched...
assert.deepEqual([...disk.readBlockCalls].sort(), [0, 2])
// ...and the whole disk now reads correctly straight from the cache
assert.deepEqual(await cache.read(0, DISK_SIZE), expectedAt(0, DISK_SIZE, disk.allocated))
})
it('skips blocks already materialized by an earlier read', async () => {
const { device, disk } = await make()
await device.read(0, 512)
assert.deepEqual(disk.readBlockCalls, [0])
await device.hydrate()
// block 0 was not fetched a second time
assert.deepEqual(disk.readBlockCalls, [0, 2])
assert.deepEqual(device.getMaterialized(), { blocks: BLOCK_COUNT, total: BLOCK_COUNT })
})
it('is a no-op once fully materialized', async () => {
const { device, disk } = await make()
await device.hydrate()
disk.readBlockCalls.length = 0
await device.hydrate()
assert.deepEqual(disk.readBlockCalls, [])
})
it('stops on an aborted signal, keeping what it already cached', async () => {
const disk = new StubDisk(new Set([0, 1, 2, 3]))
const cache = new StubCache()
const device = new CachedDiskBlockDevice({ disk, cache })
await device.open()
// every block but the first parks, so the hydration is still mid-flight
// when it is aborted
let release = () => {}
const parked = new Promise<void>(resolve => {
release = resolve
})
disk.readBlock = async index => {
if (index > 0) {
await parked
}
return { index, data: Buffer.alloc(DISK_BLOCK_SIZE, index) }
}
const controller = new AbortController()
const hydrating = device.hydrate({ concurrency: 1, signal: controller.signal })
for (let turn = 0; turn < 20 && device.getMaterialized().blocks === 0; turn++) {
await new Promise(resolve => setImmediate(resolve))
}
assert.deepEqual(device.getMaterialized(), { blocks: 1, total: BLOCK_COUNT })
controller.abort()
// aborting is a failure, not a silent partial success
await assert.rejects(hydrating)
// and the block already cached stays cached, so a later hydration resumes
assert.deepEqual(device.getMaterialized(), { blocks: 1, total: BLOCK_COUNT })
release() // let the parked fetch finish rather than leaving it dangling
})
it('refuses to start on an already-aborted signal', async () => {
const { device, disk } = await make()
const controller = new AbortController()
controller.abort()
await assert.rejects(device.hydrate({ signal: controller.signal }))
assert.deepEqual(disk.readBlockCalls, [])
assert.deepEqual(device.getMaterialized(), { blocks: 0, total: BLOCK_COUNT })
})
it('respects the concurrency limit', async () => {
const disk = new StubDisk(new Set([0, 1, 2, 3]))
const cache = new StubCache()
const device = new CachedDiskBlockDevice({ disk, cache })
await device.open()
let inFlight = 0
let maxInFlight = 0
const releases: Array<() => void> = []
disk.readBlock = async index => {
inFlight++
maxInFlight = Math.max(maxInFlight, inFlight)
await new Promise<void>(resolve => releases.push(resolve))
inFlight--
return { index, data: Buffer.alloc(DISK_BLOCK_SIZE) }
}
const hydrating = device.hydrate({ concurrency: 2 })
// let the first batch start
await new Promise(resolve => setImmediate(resolve))
assert.equal(maxInFlight, 2)
while (releases.length > 0) {
releases.shift()!()
await new Promise(resolve => setImmediate(resolve))
}
await hydrating
assert.equal(maxInFlight, 2)
})
})
describe('progress handler', () => {
class StubProgressHandler implements ProgressHandler {
calls: Array<number> = []
doneCalls = 0
throwSync = false
rejectAsync = false
async setProgress(fraction: number): Promise<void> {
this.calls.push(fraction)
if (this.throwSync) {
throw new Error('sync failure')
}
if (this.rejectAsync) {
throw new Error('async failure')
}
}
done(): void {
this.doneCalls++
}
}
it('reports the materialized fraction as blocks are read, once each', async () => {
const disk = new StubDisk()
const cache = new StubCache()
const progressHandler = new StubProgressHandler()
const device = new CachedDiskBlockDevice({ disk, cache, progressHandler })
await device.open()
await device.read(0, 512)
// let the fire-and-forget notification run
await new Promise(resolve => setImmediate(resolve))
assert.deepEqual(progressHandler.calls, [1 / BLOCK_COUNT])
// a hole is a materialization too, just with nothing to write
await device.read(DISK_BLOCK_SIZE, 512)
await new Promise(resolve => setImmediate(resolve))
assert.deepEqual(progressHandler.calls, [1 / BLOCK_COUNT, 2 / BLOCK_COUNT])
// re-reading an already materialized block reports nothing new
await device.read(0, 512)
await new Promise(resolve => setImmediate(resolve))
assert.deepEqual(progressHandler.calls, [1 / BLOCK_COUNT, 2 / BLOCK_COUNT])
})
it('reports a fully-covered write the same way as a read', async () => {
const disk = new StubDisk()
const cache = new StubCache()
const progressHandler = new StubProgressHandler()
const device = new CachedDiskBlockDevice({ disk, cache, progressHandler })
await device.open()
await device.write(0, Buffer.alloc(DISK_BLOCK_SIZE, 0xee))
await new Promise(resolve => setImmediate(resolve))
assert.deepEqual(progressHandler.calls, [1 / BLOCK_COUNT])
})
it('does not let a synchronously throwing handler break the read', async () => {
const disk = new StubDisk()
const cache = new StubCache()
const progressHandler = new StubProgressHandler()
progressHandler.throwSync = true
const device = new CachedDiskBlockDevice({ disk, cache, progressHandler })
await device.open()
assert.deepEqual(await device.read(0, 512), expectedAt(0, 512, disk.allocated))
assert.deepEqual(device.getMaterialized(), { blocks: 1, total: BLOCK_COUNT })
})
it('does not let a rejecting handler surface as an unhandled rejection', async () => {
const disk = new StubDisk()
const cache = new StubCache()
const progressHandler = new StubProgressHandler()
progressHandler.rejectAsync = true
const device = new CachedDiskBlockDevice({ disk, cache, progressHandler })
await device.open()
assert.deepEqual(await device.read(0, 512), expectedAt(0, 512, disk.allocated))
// give the rejected promise's `.catch` a turn to run before the test ends
await new Promise(resolve => setImmediate(resolve))
})
it('calls done() once every block is cached, via hydrate', async () => {
const disk = new StubDisk(new Set([0, 1, 2, 3]))
const cache = new StubCache()
const progressHandler = new StubProgressHandler()
const device = new CachedDiskBlockDevice({ disk, cache, progressHandler })
await device.open()
await device.hydrate()
await new Promise(resolve => setImmediate(resolve))
assert.deepEqual(progressHandler.calls, [0.25, 0.5, 0.75, 1])
assert.equal(progressHandler.doneCalls, 1)
})
it('calls done() once every block is cached, via on-demand reads alone', async () => {
const disk = new StubDisk(new Set([0, 1, 2, 3]))
const cache = new StubCache()
const progressHandler = new StubProgressHandler()
const device = new CachedDiskBlockDevice({ disk, cache, progressHandler })
await device.open()
await device.read(0, DISK_BLOCK_SIZE * BLOCK_COUNT)
await new Promise(resolve => setImmediate(resolve))
assert.equal(progressHandler.doneCalls, 1)
})
it('does not call done() before every block is cached', async () => {
const disk = new StubDisk(new Set([0, 1, 2, 3]))
const cache = new StubCache()
const progressHandler = new StubProgressHandler()
const device = new CachedDiskBlockDevice({ disk, cache, progressHandler })
await device.open()
await device.read(0, 512)
await new Promise(resolve => setImmediate(resolve))
assert.equal(progressHandler.doneCalls, 0)
})
it('does not let a throwing done() surface as an unhandled rejection', async () => {
const disk = new StubDisk(new Set([0, 1, 2, 3]))
const cache = new StubCache()
const progressHandler = new StubProgressHandler()
progressHandler.done = () => {
throw new Error('done failure')
}
const device = new CachedDiskBlockDevice({ disk, cache, progressHandler })
await device.open()
await device.hydrate()
await new Promise(resolve => setImmediate(resolve))
})
})
describe('onBlockCached', () => {
it('fires with the index of each newly cached block, once each', async () => {
const disk = new StubDisk(new Set([0, 1, 2, 3]))
const cache = new StubCache()
const cached: Array<number> = []
const device = new CachedDiskBlockDevice({ disk, cache, onBlockCached: index => cached.push(index) })
await device.open()
await device.read(0, 512)
assert.deepEqual(cached, [0])
// a hole is a materialization too
await device.read(DISK_BLOCK_SIZE, 512)
assert.deepEqual(cached, [0, 1])
// re-reading an already materialized block does not fire again
await device.read(0, 512)
assert.deepEqual(cached, [0, 1])
})
it('does not fire for a block resumed from the initial bitmap', async () => {
const disk = new StubDisk()
const cache = new StubCache()
const initialBitmap = Buffer.alloc(BLOCK_COUNT)
initialBitmap[0] = 1
const cached: Array<number> = []
const device = new CachedDiskBlockDevice({ disk, cache, initialBitmap, onBlockCached: index => cached.push(index) })
await device.open()
await device.read(0, 512)
assert.deepEqual(cached, [])
})
it('does not let a synchronously throwing callback break the read', async () => {
const disk = new StubDisk()
const cache = new StubCache()
const device = new CachedDiskBlockDevice({
disk,
cache,
onBlockCached: () => {
throw new Error('persistence failure')
},
})
await device.open()
assert.deepEqual(await device.read(0, 512), expectedAt(0, 512, disk.allocated))
assert.deepEqual(device.getMaterialized(), { blocks: 1, total: BLOCK_COUNT })
})
})
describe('flush and close', () => {
it('flushes the cache', async () => {
const { device, cache } = await make()
await device.flush()
assert.equal(cache.flushed, 1)
})
it('closes the cache and the disk', async () => {
const { device, cache, disk } = await make()
await device.close()
assert.equal(cache.closed, true)
assert.equal(disk.closed, true)
})
it('still closes the disk when closing the cache fails', async () => {
const { device, cache, disk } = await make()
cache.close = async () => {
throw new Error('nope')
}
await assert.rejects(device.close(), /nope/)
assert.equal(disk.closed, true)
})
})

View File

@@ -0,0 +1,120 @@
import { createLogger, type Logger } from '@xen-orchestra/log'
import type { RandomAccessDisk } from '@xen-orchestra/disk-transform'
import type { BlockDevice } from './backend.mjs'
const log: Logger = createLogger('vates:iscsi:disk-block-device')
const DEFAULT_BLOCK_SIZE = 512
export interface DiskBlockDeviceOptions {
/**
* Source disk, already `init()`ed by the caller (this adapter never calls
* `init()` — the disk may be part of a chain the caller assembled).
*/
readonly disk: RandomAccessDisk
/**
* Logical block size advertised to initiators, in bytes. Defaults to 512.
* The disk's own block size must be a multiple of it.
*/
readonly blockSize?: number
}
/**
* A read-only {@link BlockDevice} for a {@link RandomAccessDisk} (the inverse
* of {@link IscsiDisk}). Unallocated blocks read as zero; the source must
* never be asked for a block it doesn't have (e.g. `RemoteVhdDiskChain`
* throws).
*
* Nothing is cached — every read re-fetches a whole source block. Put a
* caching decorator in front of the source disk, not here.
*/
export class DiskBlockDevice implements BlockDevice {
readonly #disk: RandomAccessDisk
readonly #blockSize: number
#diskBlockSize?: number
#size?: number
constructor({ disk, blockSize = DEFAULT_BLOCK_SIZE }: DiskBlockDeviceOptions) {
if (!Number.isInteger(blockSize) || blockSize <= 0) {
throw new Error(`blockSize must be a positive integer, got ${blockSize}`)
}
this.#disk = disk
this.#blockSize = blockSize
}
/**
* Read the source disk's geometry and check it can be exposed as a LUN.
* Awaited by the target before it serves any I/O.
*/
async open(): Promise<void> {
const blockSize = this.#blockSize
const diskBlockSize = this.#disk.getBlockSize()
if (diskBlockSize % blockSize !== 0) {
throw new Error(`disk block size (${diskBlockSize}) is not a multiple of the LUN block size (${blockSize})`)
}
const size = this.#disk.getVirtualSize()
if (size <= 0 || size % blockSize !== 0) {
throw new Error(`disk virtual size (${size}) is not a positive multiple of the LUN block size (${blockSize})`)
}
this.#diskBlockSize = diskBlockSize
this.#size = size
log.debug('opened', { size, blockSize, diskBlockSize })
}
getSize(): number {
const size = this.#size
if (size === undefined) {
throw new Error('DiskBlockDevice.open() must be called before I/O')
}
return size
}
getBlockSize(): number {
return this.#blockSize
}
async read(offset: number, length: number): Promise<Buffer> {
const size = this.getSize()
if (offset < 0 || length < 0 || offset + length > size) {
throw new Error(`read of ${length} bytes at ${offset} is out of range (size ${size})`)
}
if (length === 0) {
return Buffer.alloc(0)
}
const diskBlockSize = this.#diskBlockSize as number
// zero-filled: unallocated blocks, and any source block shorter than
// announced, are left as zeroes
const result = Buffer.alloc(length)
const end = offset + length
const lastIndex = Math.floor((end - 1) / diskBlockSize)
for (let index = Math.floor(offset / diskBlockSize); index <= lastIndex; index++) {
if (!this.#disk.hasBlock(index)) {
continue
}
const blockStart = index * diskBlockSize
const from = Math.max(offset, blockStart)
const to = Math.min(end, blockStart + diskBlockSize)
const { data } = await this.#disk.readBlock(index)
data.copy(result, from - offset, from - blockStart, to - blockStart)
}
return result
}
// the target refuses WRITE commands before they reach `write()`
readonly readOnly = true
/** Always throws: the source disk is read-only. */
async write(): Promise<void> {
throw new Error('DiskBlockDevice is read-only')
}
async flush(): Promise<void> {
// nothing to flush: read-only
}
async close(): Promise<void> {
await this.#disk.close()
}
}

View File

@@ -0,0 +1,169 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { RandomAccessDisk, type DiskBlock } from '@xen-orchestra/disk-transform'
import { DiskBlockDevice } from './DiskBlockDevice.mjs'
const BLOCK_SIZE = 4096
/**
* Sparse in-memory {@link RandomAccessDisk}: `blocks` maps a block index to its
* content, every other index is unallocated and must never be read (the real
* `RemoteVhdDiskChain` throws in that case, which is what this asserts).
*/
class StubDisk extends RandomAccessDisk {
readBlockCalls: Array<number> = []
closed = false
constructor(
readonly blocks: Map<number, Buffer>,
readonly virtualSize: number,
readonly blockSize: number = BLOCK_SIZE
) {
super()
}
getBlockSize(): number {
return this.blockSize
}
getVirtualSize(): number {
return this.virtualSize
}
isDifferencing(): boolean {
return false
}
getBlockIndexes(): Array<number> {
return [...this.blocks.keys()]
}
hasBlock(index: number): boolean {
return this.blocks.has(index)
}
async init(): Promise<void> {}
async close(): Promise<void> {
this.closed = true
}
async readBlock(index: number): Promise<DiskBlock> {
const data = this.blocks.get(index)
if (data === undefined) {
throw new Error(`Block ${index} not found`)
}
this.readBlockCalls.push(index)
return { index, data }
}
}
const filled = (byte: number, length = BLOCK_SIZE) => Buffer.alloc(length, byte)
// 4 blocks, only 0 and 2 allocated
const makeDevice = async (blockSize?: number) => {
const disk = new StubDisk(
new Map([
[0, filled(0xaa)],
[2, filled(0xcc)],
]),
4 * BLOCK_SIZE
)
const device = new DiskBlockDevice({ disk, blockSize })
await device.open()
return { disk, device }
}
describe('open', () => {
it('exposes the disk capacity and the requested block size', async () => {
const { device } = await makeDevice()
assert.equal(device.getSize(), 4 * BLOCK_SIZE)
assert.equal(device.getBlockSize(), 512)
})
it('rejects a disk block size that is not a multiple of the LUN block size', async () => {
const disk = new StubDisk(new Map(), 4096, 600)
await assert.rejects(new DiskBlockDevice({ disk }).open(), /not a multiple of the LUN block size/)
})
it('rejects a capacity that is not a multiple of the LUN block size', async () => {
const disk = new StubDisk(new Map(), BLOCK_SIZE + 100)
await assert.rejects(new DiskBlockDevice({ disk }).open(), /not a positive multiple/)
})
it('rejects an invalid block size', () => {
const disk = new StubDisk(new Map(), BLOCK_SIZE)
assert.throws(() => new DiskBlockDevice({ disk, blockSize: 0 }), /positive integer/)
})
it('refuses I/O before open()', () => {
const disk = new StubDisk(new Map(), BLOCK_SIZE)
assert.throws(() => new DiskBlockDevice({ disk }).getSize(), /open\(\) must be called/)
})
})
describe('read', () => {
it('reads a whole allocated block', async () => {
const { device } = await makeDevice()
assert.deepEqual(await device.read(0, BLOCK_SIZE), filled(0xaa))
})
it('reads an unaligned slice inside one block', async () => {
const { device, disk } = await makeDevice()
assert.deepEqual(await device.read(BLOCK_SIZE * 2 + 512, 1024), filled(0xcc, 1024))
// a single source block was fetched, not the whole disk
assert.deepEqual(disk.readBlockCalls, [2])
})
it('zero-fills unallocated blocks instead of reading them', async () => {
const { device, disk } = await makeDevice()
assert.deepEqual(await device.read(BLOCK_SIZE, BLOCK_SIZE), Buffer.alloc(BLOCK_SIZE))
assert.deepEqual(disk.readBlockCalls, [])
})
it('stitches a range spanning allocated and unallocated blocks', async () => {
const { device } = await makeDevice()
const data = await device.read(BLOCK_SIZE - 512, BLOCK_SIZE + 1024)
assert.deepEqual(
data,
Buffer.concat([
filled(0xaa, 512), // tail of block 0
Buffer.alloc(BLOCK_SIZE), // block 1, unallocated
filled(0xcc, 512), // head of block 2
])
)
})
it('reads the last block of the disk', async () => {
const { device } = await makeDevice()
assert.deepEqual(await device.read(3 * BLOCK_SIZE, BLOCK_SIZE), Buffer.alloc(BLOCK_SIZE))
})
it('returns an empty buffer for a zero-length read', async () => {
const { device, disk } = await makeDevice()
assert.equal((await device.read(0, 0)).length, 0)
assert.deepEqual(disk.readBlockCalls, [])
})
it('rejects a read past the end of the disk', async () => {
const { device } = await makeDevice()
await assert.rejects(device.read(4 * BLOCK_SIZE - 512, 1024), /out of range/)
await assert.rejects(device.read(-512, 512), /out of range/)
})
it('zero-pads a source block shorter than announced', async () => {
const disk = new StubDisk(new Map([[0, filled(0xaa, 1024)]]), BLOCK_SIZE)
const device = new DiskBlockDevice({ disk })
await device.open()
assert.deepEqual(await device.read(0, BLOCK_SIZE), Buffer.concat([filled(0xaa, 1024)], BLOCK_SIZE))
})
})
describe('write', () => {
it('always fails', async () => {
const { device } = await makeDevice()
await assert.rejects(device.write(), /read-only/)
})
})
describe('close', () => {
it('closes the underlying disk', async () => {
const { device, disk } = await makeDevice()
await device.close()
assert.equal(disk.closed, true)
})
})

View File

@@ -0,0 +1,98 @@
import { RandomAccessDisk, type DiskBlock } from '@xen-orchestra/disk-transform'
import { IscsiInitiator, type IscsiInitiatorOptions } from './initiator.mjs'
export interface IscsiDiskOptions {
/**
* Block size, in bytes, presented to the disk-transform pipeline. Must be a
* multiple of the LUN's logical block size (checked at {@link IscsiDisk.init}).
* A single SCSI READ(16) fetches one whole block.
*/
readonly blockSize: number
}
/**
* A {@link RandomAccessDisk} backed by a remote iSCSI LUN read through an
* {@link IscsiInitiator} — the iSCSI analogue of `@vates/nbd-client`'s `NbdDisk`.
* It plugs a target LUN into the `@xen-orchestra/disk-transform` pipeline
* (`ReadAhead`, `DiskLargerBlock`, block generator) unchanged.
*
* First implementation: the LUN is treated as fully allocated — every block is
* reported as present (dense, like `RawDisk`). A future version can narrow this
* with a changed-block map (e.g. a Pure "Volume Difference"). Read-only.
*/
export class IscsiDisk extends RandomAccessDisk {
readonly #options: IscsiInitiatorOptions
readonly #blockSize: number
#initiator?: IscsiInitiator
constructor(options: IscsiInitiatorOptions, { blockSize }: IscsiDiskOptions) {
super()
this.#options = options
this.#blockSize = blockSize
}
async init(): Promise<void> {
const initiator = new IscsiInitiator(this.#options)
await initiator.connect()
const lunBlockSize = initiator.getBlockSize()
if (this.#blockSize % lunBlockSize !== 0) {
await initiator.close()
throw new Error(`block size ${this.#blockSize} is not a multiple of the LUN block size ${lunBlockSize}`)
}
this.#initiator = initiator
}
async close(): Promise<void> {
const initiator = this.#initiator
this.#initiator = undefined
await initiator?.close()
}
#requireInitiator(): IscsiInitiator {
const initiator = this.#initiator
if (initiator === undefined) {
throw new Error('IscsiDisk.init() must be called first')
}
return initiator
}
getBlockSize(): number {
return this.#blockSize
}
getVirtualSize(): number {
return this.#requireInitiator().getSize()
}
isDifferencing(): boolean {
return false
}
getBlockIndexes(): Array<number> {
const count = Math.ceil(this.getVirtualSize() / this.#blockSize)
const indexes = new Array<number>(count)
for (let i = 0; i < count; i++) {
indexes[i] = i
}
return indexes
}
hasBlock(index: number): boolean {
return index >= 0 && index * this.#blockSize < this.getVirtualSize()
}
async readBlock(index: number): Promise<DiskBlock> {
const initiator = this.#requireInitiator()
const size = this.getVirtualSize()
const offset = index * this.#blockSize
const length = Math.min(this.#blockSize, size - offset)
let data = await initiator.read(offset, length)
if (data.length < this.#blockSize) {
// Last (short) block on a LUN whose capacity is not a multiple of the
// disk block size: zero-pad the tail to a full block.
data = Buffer.concat([data], this.#blockSize)
}
return { index, data }
}
}

View File

@@ -0,0 +1,129 @@
import assert from 'node:assert/strict'
import { execFile, execFileSync } from 'node:child_process'
import { mkdtemp, rm, truncate, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { promisify } from 'node:util'
import { after, before, describe, it } from 'node:test'
import { FileBlockDevice } from './index.mjs'
const execFileP = promisify(execFile)
const BLOCK_SIZE = 512
const SIZE = 4 * 1024 * 1024 // 4 MiB
/**
* Why this cannot run here, or `false` if it can. A real block device is the
* whole point, and the cheapest one to make is a loop device — which needs root
* and `losetup`.
*/
function skipReason(): string | false {
if (process.getuid === undefined || process.getuid() !== 0) {
return 'requires root to attach a loop device'
}
try {
execFileSync('losetup', ['--version'], { stdio: 'ignore' })
} catch {
return 'requires util-linux (losetup not found)'
}
return false
}
const bytePattern = (i: number) => (i * 31 + 11) & 0xff
describe('FileBlockDevice on a loop device', { skip: skipReason() }, () => {
let dir: string
let devicePath: string
before(async () => {
dir = await mkdtemp(join(tmpdir(), 'vates-iscsi-loop-'))
const backing = join(dir, 'backing.img')
await writeFile(backing, '')
await truncate(backing, SIZE)
const { stdout } = await execFileP('losetup', ['--find', '--show', backing])
devicePath = stdout.trim()
})
after(async () => {
if (devicePath !== undefined) {
await execFileP('losetup', ['--detach', devicePath])
}
await rm(dir, { force: true, recursive: true })
})
const withDevice = async (fn: (device: FileBlockDevice) => Promise<void>): Promise<void> => {
const device = new FileBlockDevice({ path: devicePath, size: SIZE })
await device.open()
try {
await fn(device)
} finally {
await device.close()
}
}
it('accepts the device node and reports the capacity it was given', async () => {
await withDevice(async device => {
assert.equal(device.getSize(), SIZE)
assert.equal(device.getBlockSize(), BLOCK_SIZE)
})
})
it('round-trips block-aligned data', async () => {
await withDevice(async device => {
const payload = Buffer.alloc(8 * BLOCK_SIZE)
for (let i = 0; i < payload.length; i++) {
payload[i] = bytePattern(i)
}
await device.write(16 * BLOCK_SIZE, payload)
assert.deepEqual(await device.read(16 * BLOCK_SIZE, payload.length), payload)
})
})
it('round-trips data straddling block boundaries, at an unaligned offset', async () => {
await withDevice(async device => {
const payload = Buffer.from('a payload that is not a whole number of sectors long')
const offset = 3 * BLOCK_SIZE + 17
await device.write(offset, payload)
assert.deepEqual(await device.read(offset, payload.length), payload)
// the surrounding sector was read-modify-written by the kernel, not clobbered
assert.deepEqual(await device.read(3 * BLOCK_SIZE, 17), Buffer.alloc(17))
})
})
it('survives a reopen, which is what makes it a cache', async () => {
const payload = Buffer.alloc(BLOCK_SIZE, 0x5a)
await withDevice(device => device.write(2 * BLOCK_SIZE, payload))
await withDevice(async device => {
await device.flush()
assert.deepEqual(await device.read(2 * BLOCK_SIZE, BLOCK_SIZE), payload)
})
})
it('reads a large range in one call', async () => {
await withDevice(async device => {
const payload = Buffer.alloc(SIZE / 2)
for (let i = 0; i < payload.length; i++) {
payload[i] = bytePattern(i + 7)
}
await device.write(0, payload)
assert.deepEqual(await device.read(0, payload.length), payload)
})
})
it('throws on a read running past the end, instead of zero-filling it', async () => {
// the behaviour that sets the block device mode apart: a short read on a
// device of a known size is an I/O error, never a sparse tail
await withDevice(async device => {
await assert.rejects(device.read(SIZE, BLOCK_SIZE), /ended after 0 bytes/)
await assert.rejects(device.read(SIZE - BLOCK_SIZE, 2 * BLOCK_SIZE), /ended after 512 bytes/)
})
})
it('flushes without error', async () => {
await withDevice(async device => {
await device.write(0, Buffer.alloc(BLOCK_SIZE, 0xff))
await device.flush()
})
})
})

View File

@@ -0,0 +1,175 @@
import { open, type FileHandle } from 'node:fs/promises'
/**
* Byte-range random-access store backing a single iSCSI LUN.
*
* The protocol layer translates SCSI READ/WRITE CDBs (LBA + block count) into
* byte offsets/lengths against this interface, so an implementation only has to
* provide flat random access — it is intentionally decoupled from the wire
* code. `DiskBlockDevice` is such an implementation, over
* `@xen-orchestra/disk-transform`'s `RandomAccessDisk` (VHD/raw/NBD).
*/
export interface BlockDevice {
/** Optional one-time initialization, awaited by the target before serving I/O. */
open?(): Promise<void>
/** Total capacity in bytes. Must be a multiple of {@link BlockDevice.getBlockSize}. */
getSize(): number
/** Logical block size in bytes (typically 512). */
getBlockSize(): number
/**
* When true, the LUN is reported as write-protected and WRITE commands are
* refused up front (DATA PROTECT), without soliciting their data.
*/
readonly readOnly?: boolean
/** Read exactly `length` bytes starting at byte `offset`. */
read(offset: number, length: number): Promise<Buffer>
/** Write `data` starting at byte `offset`. */
write(offset: number, data: Buffer): Promise<void>
/** Flush any buffered data to stable storage (SYNCHRONIZE CACHE). */
flush(): Promise<void>
/** Release underlying resources. */
close(): Promise<void>
}
const DEFAULT_BLOCK_SIZE = 512
export interface FileBlockDeviceOptions {
/**
* Path to the backing file. It must already exist and be sized to capacity —
* or be a block device node, see {@link FileBlockDeviceOptions.size}.
*/
readonly path: string
/** Logical block size in bytes. Defaults to 512. */
readonly blockSize?: number
/**
* Capacity in bytes, for a block device node (e.g. a VDI hot-plugged onto the
* appliance's VM, `/dev/xvdX`), whose inode reports a size of 0.
*
* Setting it switches to the block device mode:
* - `path` must be a block device: a typo naming the wrong node is the
* difference between a cache and a destroyed disk;
* - a short read is an I/O error, not a sparse tail: zero-filling it would hand
* the initiator fabricated data.
*/
readonly size?: number
}
/**
* A {@link BlockDevice} backed by a regular (sparse) file, or a block device
* node, opened read/write.
*
* The file must already exist and be sized to the desired capacity (e.g. via
* `fs.truncate`); this keeps the device a pure data store with no provisioning
* policy of its own.
*
* I/O is buffered, never `O_DIRECT`, which would need buffers aligned to the
* logical block size that Node cannot allocate.
*/
export class FileBlockDevice implements BlockDevice {
readonly #path: string
readonly #blockSize: number
readonly #isBlockDevice: boolean
#handle?: FileHandle
#size = 0
constructor({ path, blockSize = DEFAULT_BLOCK_SIZE, size }: FileBlockDeviceOptions) {
if (!Number.isInteger(blockSize) || blockSize <= 0) {
throw new Error(`blockSize must be a positive integer, got ${blockSize}`)
}
if (size !== undefined) {
if (!Number.isInteger(size) || size <= 0) {
throw new Error(`size must be a positive integer, got ${size}`)
}
if (size % blockSize !== 0) {
throw new Error(`size (${size}) is not a multiple of block size (${blockSize})`)
}
this.#size = size
}
this.#path = path
this.#blockSize = blockSize
this.#isBlockDevice = size !== undefined
}
/** Open the backing file and read its current size. Must be called before any I/O. */
async open(): Promise<void> {
if (this.#handle !== undefined) {
return
}
const handle = await open(this.#path, 'r+')
try {
const stats = await handle.stat()
if (this.#isBlockDevice) {
if (!stats.isBlockDevice()) {
throw new Error(`${this.#path} is not a block device`)
}
} else {
if (stats.size % this.#blockSize !== 0) {
throw new Error(`backing file size (${stats.size}) is not a multiple of block size (${this.#blockSize})`)
}
this.#size = stats.size
}
this.#handle = handle
} catch (error) {
await handle.close()
throw error
}
}
#requireHandle(): FileHandle {
const handle = this.#handle
if (handle === undefined) {
throw new Error('FileBlockDevice.open() must be called before I/O')
}
return handle
}
getSize(): number {
return this.#size
}
getBlockSize(): number {
return this.#blockSize
}
async read(offset: number, length: number): Promise<Buffer> {
const handle = this.#requireHandle()
const buffer = Buffer.alloc(length, 0)
let read = 0
// A single fh.read may return fewer bytes than requested; loop to fill.
while (read < length) {
const { bytesRead } = await handle.read(buffer, read, length - read, offset + read)
if (bytesRead === 0) {
if (this.#isBlockDevice) {
// the device is exactly as long as the caller said: the I/O failed
throw new Error(`read of ${length} bytes at ${offset} on ${this.#path} ended after ${read} bytes`)
}
// Reading past EOF (sparse tail): zero-fill the remainder.
buffer.fill(0, read)
break
}
read += bytesRead
}
return buffer
}
async write(offset: number, data: Buffer): Promise<void> {
const handle = this.#requireHandle()
let written = 0
while (written < data.length) {
const { bytesWritten } = await handle.write(data, written, data.length - written, offset + written)
written += bytesWritten
}
}
async flush(): Promise<void> {
await this.#requireHandle().sync()
}
async close(): Promise<void> {
const handle = this.#handle
if (handle !== undefined) {
this.#handle = undefined
await handle.close()
}
}
}

View File

@@ -0,0 +1,173 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { mkdtemp, rm, truncate, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { FileBlockDevice } from './backend.mjs'
const BLOCK_SIZE = 512
/** A temporary directory removed whatever the test does. */
const withTmpDir = async (fn: (dir: string) => Promise<void>): Promise<void> => {
const dir = await mkdtemp(join(tmpdir(), 'vates-iscsi-backend-'))
try {
await fn(dir)
} finally {
await rm(dir, { force: true, recursive: true })
}
}
/** A sparse file of `size` bytes, as the class requires: existing and pre-sized. */
const makeFile = async (dir: string, size: number, name = 'lun.img'): Promise<string> => {
const path = join(dir, name)
await writeFile(path, '')
await truncate(path, size)
return path
}
describe('FileBlockDevice', () => {
it('rejects a non-positive-integer block size', () => {
assert.throws(() => new FileBlockDevice({ path: '/nope', blockSize: 0 }), /positive integer/)
assert.throws(() => new FileBlockDevice({ path: '/nope', blockSize: -1 }), /positive integer/)
assert.throws(() => new FileBlockDevice({ path: '/nope', blockSize: 1.5 }), /positive integer/)
})
it('reads the capacity from the backing file', async () => {
await withTmpDir(async dir => {
const device = new FileBlockDevice({ path: await makeFile(dir, 4 * BLOCK_SIZE) })
await device.open()
try {
assert.equal(device.getSize(), 4 * BLOCK_SIZE)
assert.equal(device.getBlockSize(), BLOCK_SIZE)
} finally {
await device.close()
}
})
})
it('refuses a file whose size is not a multiple of the block size', async () => {
await withTmpDir(async dir => {
const device = new FileBlockDevice({ path: await makeFile(dir, BLOCK_SIZE + 1) })
await assert.rejects(device.open(), /not a multiple of block size/)
})
})
it('throws on I/O before open()', async () => {
const device = new FileBlockDevice({ path: '/nope' })
await assert.rejects(device.read(0, BLOCK_SIZE), /must be called before I\/O/)
await assert.rejects(device.write(0, Buffer.alloc(BLOCK_SIZE)), /must be called before I\/O/)
await assert.rejects(device.flush(), /must be called before I\/O/)
})
it('round-trips a write, at an offset and unaligned', async () => {
await withTmpDir(async dir => {
const device = new FileBlockDevice({ path: await makeFile(dir, 4 * BLOCK_SIZE) })
await device.open()
try {
const payload = Buffer.from('hello, lun')
await device.write(BLOCK_SIZE + 3, payload)
assert.deepEqual(await device.read(BLOCK_SIZE + 3, payload.length), payload)
// the bytes around it are untouched
assert.deepEqual(await device.read(BLOCK_SIZE, 3), Buffer.alloc(3))
} finally {
await device.close()
}
})
})
it('zero-fills the sparse tail of a read past EOF', async () => {
await withTmpDir(async dir => {
// a hole: nothing was ever written, so the whole file reads as zeroes
const device = new FileBlockDevice({ path: await makeFile(dir, 2 * BLOCK_SIZE) })
await device.open()
try {
assert.deepEqual(await device.read(0, 2 * BLOCK_SIZE), Buffer.alloc(2 * BLOCK_SIZE))
// past the end of the file, `fh.read` reports 0 bytes read: the remainder is
// zero-filled rather than being reported as an error
assert.deepEqual(await device.read(2 * BLOCK_SIZE, BLOCK_SIZE), Buffer.alloc(BLOCK_SIZE))
} finally {
await device.close()
}
})
})
it('reads back a large buffer in full', async () => {
await withTmpDir(async dir => {
const size = 1024 * BLOCK_SIZE
const device = new FileBlockDevice({ path: await makeFile(dir, size) })
await device.open()
try {
const payload = Buffer.alloc(size)
for (let i = 0; i < payload.length; i++) {
payload[i] = (i * 31 + 11) & 0xff
}
await device.write(0, payload)
assert.deepEqual(await device.read(0, size), payload)
} finally {
await device.close()
}
})
})
it('flushes and closes, and tolerates being opened or closed twice', async () => {
await withTmpDir(async dir => {
const device = new FileBlockDevice({ path: await makeFile(dir, BLOCK_SIZE) })
await device.open()
await device.open() // no-op
await device.write(0, Buffer.alloc(BLOCK_SIZE, 0xaa))
await device.flush()
await device.close()
await device.close() // no-op
await assert.rejects(device.read(0, BLOCK_SIZE), /must be called before I\/O/)
})
})
it('leaves no handle behind when open() fails', async () => {
await withTmpDir(async dir => {
const device = new FileBlockDevice({ path: await makeFile(dir, BLOCK_SIZE + 1) })
await assert.rejects(device.open())
// the handle opened before the size check was closed again
await assert.rejects(device.read(0, BLOCK_SIZE), /must be called before I\/O/)
})
})
})
// A real device node cannot be created without root, so what is covered here is
// everything around the I/O: the guards, the capacity, and the lifecycle. The
// I/O itself is covered by `backend.integ.mts`, against a loop device.
describe('FileBlockDevice with a size, over a block device', () => {
it('rejects a non-positive-integer size', () => {
for (const size of [0, -BLOCK_SIZE, 1.5]) {
assert.throws(() => new FileBlockDevice({ path: '/dev/null', size }), /size must be a positive integer/)
}
})
it('rejects a size that is not a multiple of the block size', () => {
assert.throws(
() => new FileBlockDevice({ path: '/dev/null', size: BLOCK_SIZE + 1 }),
/is not a multiple of block size/
)
})
it('reports the capacity it was given, never the inode', () => {
// nothing may be read from the inode, which for a real device node reports 0
const device = new FileBlockDevice({ path: '/dev/null', size: 4 * BLOCK_SIZE })
assert.equal(device.getSize(), 4 * BLOCK_SIZE)
assert.equal(device.getBlockSize(), BLOCK_SIZE)
})
it('refuses to open a path that is not a block device', async () => {
await withTmpDir(async dir => {
const path = await makeFile(dir, 4 * BLOCK_SIZE)
const device = new FileBlockDevice({ path, size: 4 * BLOCK_SIZE })
// the guard against a misconfigured path naming, say, a real disk's
// filesystem instead of the cache device
await assert.rejects(device.open(), new RegExp(`${path} is not a block device`))
// and the handle it opened to check was closed again
await assert.rejects(device.read(0, BLOCK_SIZE), /must be called before I\/O/)
})
})
})

90
@vates/iscsi/src/chap.mts Normal file
View File

@@ -0,0 +1,90 @@
// Shared CHAP (RFC 1994) primitives for both login directions: the target as
// authenticator, the initiator as responder. Kept in one place so the MD5
// computation and large-binary codec agree byte-for-byte between the two
// (cross-tested against each other in loopback).
//
// RFC 7143 §11.1.4 pins CHAP to MD5 (algorithm id 5) — a wire-format
// requirement, not a security choice.
import { createHash, randomBytes, timingSafeEqual } from 'node:crypto'
/** The only CHAP algorithm iSCSI defines / we support: MD5 (RFC 7143 §11.1.4). */
export const CHAP_ALGORITHM_MD5 = 5
/** Default challenge length in bytes; 16 is what open-iscsi and most targets use. */
const DEFAULT_CHALLENGE_LENGTH = 16
/**
* Compute a CHAP response: `MD5(id ‖ secret ‖ challenge)` (RFC 1994 §4.1).
*
* `id` is the single CHAP_I octet (0-255) — the raw byte, NOT its ASCII decimal.
* `secret` is the shared secret (a string is hashed as its UTF-8 bytes).
*/
export function computeResponse(id: number, secret: string | Buffer, challenge: Buffer): Buffer {
return createHash('md5')
.update(Buffer.from([id & 0xff]))
.update(secret)
.update(challenge)
.digest()
}
/**
* Encode a binary value (CHAP_C / CHAP_R) as an iSCSI hex-constant, e.g.
* `0xa1b2…` (RFC 7143 §5.1). We always emit hex; {@link decodeChapValue} accepts
* both hex and base64 so we interoperate with peers that send either.
*/
export function encodeChapValue(value: Buffer): string {
return '0x' + value.toString('hex')
}
/**
* Decode an iSCSI large-binary value: `0x`/`0X` hex-constant or `0b`/`0B`
* base64-constant (RFC 7143 §5.1). Throws on an unrecognized encoding.
*/
export function decodeChapValue(value: string): Buffer {
const prefix = value.slice(0, 2)
const body = value.slice(2)
if (prefix === '0x' || prefix === '0X') {
return Buffer.from(body, 'hex')
}
if (prefix === '0b' || prefix === '0B') {
return Buffer.from(body, 'base64')
}
throw new Error(`unsupported CHAP value encoding: ${value.slice(0, 2)}…`)
}
/** Generate a random CHAP challenge (CHAP_C). */
export function generateChallenge(length: number = DEFAULT_CHALLENGE_LENGTH): Buffer {
return randomBytes(length)
}
/** Generate a random CHAP identifier (CHAP_I), a single octet. */
export function generateId(): number {
return randomBytes(1)[0]
}
/** Parse a CHAP_A value (`"5"` or a `"5,6,7"` preference list) into algorithm ids. */
export function parseAlgorithmList(value: string): number[] {
return value
.split(',')
.map(part => Number.parseInt(part.trim(), 10))
.filter(id => Number.isInteger(id))
}
/**
* Pick the algorithm we will use from the ones the peer offered: MD5 if present,
* else `undefined` (we only implement MD5, so an offer without it must fail —
* never fall back to the first offered id).
*/
export function selectAlgorithm(offered: readonly number[]): number | undefined {
return offered.includes(CHAP_ALGORITHM_MD5) ? CHAP_ALGORITHM_MD5 : undefined
}
/**
* Constant-time comparison of an expected response against a received one. A
* wrong-length `got` (or the length-mismatch `timingSafeEqual` would throw on)
* counts as a failed authentication, not an exception.
*/
export function verifyResponse(expected: Buffer, got: Buffer): boolean {
return expected.length === got.length && timingSafeEqual(expected, got)
}

View File

@@ -0,0 +1,108 @@
import assert from 'node:assert/strict'
import { createHash } from 'node:crypto'
import { describe, it } from 'node:test'
import {
CHAP_ALGORITHM_MD5,
computeResponse,
decodeChapValue,
encodeChapValue,
generateChallenge,
generateId,
parseAlgorithmList,
selectAlgorithm,
verifyResponse,
} from './chap.mjs'
describe('computeResponse', () => {
// Known-answer vector: MD5(0x01 ‖ "s3cr3t" ‖ <16-byte challenge>), computed
// independently so this pins the exact byte layout (id octet first), not MD5
// against itself.
const id = 1
const secret = 's3cr3t'
const challenge = Buffer.from('0102030405060708090a0b0c0d0e0f10', 'hex')
const expected = 'fc14fb49f184c847514a10278d75a10f'
it('hashes id ‖ secret ‖ challenge (RFC 1994)', () => {
assert.equal(computeResponse(id, secret, challenge).toString('hex'), expected)
})
it('treats a string secret as its UTF-8 bytes', () => {
assert.deepEqual(
computeResponse(id, Buffer.from(secret, 'utf8'), challenge),
computeResponse(id, secret, challenge)
)
})
it('uses the id as a raw octet, not its ASCII decimal', () => {
// computeResponse(1, …) must hash the byte 0x01, whereas hashing the ASCII
// text "1" would hash 0x31 — the two must differ.
const rawOctet = createHash('md5')
.update(Buffer.from([1]))
.update(secret)
.update(challenge)
.digest()
const asciiText = createHash('md5').update(Buffer.from('1', 'ascii')).update(secret).update(challenge).digest()
assert.deepEqual(computeResponse(1, secret, challenge), rawOctet)
assert.notDeepEqual(computeResponse(1, secret, challenge), asciiText)
})
})
describe('CHAP value codec', () => {
it('encodes as a 0x hex-constant', () => {
assert.equal(encodeChapValue(Buffer.from([0xab, 0xcd, 0x01])), '0xabcd01')
})
it('round-trips through hex', () => {
const value = generateChallenge()
assert.deepEqual(decodeChapValue(encodeChapValue(value)), value)
})
it('decodes a 0b base64-constant', () => {
const raw = Buffer.from('0102030405060708090a0b0c0d0e0f10', 'hex')
assert.deepEqual(decodeChapValue('0b' + raw.toString('base64')), raw)
})
it('accepts uppercase 0X / 0B prefixes', () => {
assert.deepEqual(decodeChapValue('0Xabcd'), Buffer.from('abcd', 'hex'))
assert.deepEqual(decodeChapValue('0B' + Buffer.from([1, 2]).toString('base64')), Buffer.from([1, 2]))
})
it('rejects an unknown encoding', () => {
assert.throws(() => decodeChapValue('12345'), /unsupported CHAP value encoding/)
})
})
describe('algorithm selection', () => {
it('parses a preference list', () => {
assert.deepEqual(parseAlgorithmList('5,6,7'), [5, 6, 7])
assert.deepEqual(parseAlgorithmList('5'), [5])
})
it('picks MD5 when offered', () => {
assert.equal(selectAlgorithm([1, CHAP_ALGORITHM_MD5, 6]), CHAP_ALGORITHM_MD5)
})
it('returns undefined when MD5 is not offered (never falls back)', () => {
assert.equal(selectAlgorithm([1, 2]), undefined)
})
})
describe('verifyResponse', () => {
const secret = 's3cr3t'
const challenge = generateChallenge()
const id = generateId()
const good = computeResponse(id, secret, challenge)
it('accepts a matching response', () => {
assert.equal(verifyResponse(good, computeResponse(id, secret, challenge)), true)
})
it('rejects a wrong secret', () => {
assert.equal(verifyResponse(good, computeResponse(id, 'wrong', challenge)), false)
})
it('rejects a wrong-length response without throwing', () => {
assert.equal(verifyResponse(good, Buffer.alloc(4)), false)
})
})

141
@vates/iscsi/src/cli.mts Executable file
View File

@@ -0,0 +1,141 @@
#!/usr/bin/env node
import { stat, truncate, writeFile } from 'node:fs/promises'
import { basename } from 'node:path'
import { parseArgs } from 'node:util'
import { FileBlockDevice, IscsiTarget } from './index.mjs'
const USAGE = `Usage: vates-iscsi <file> [options]
Serve a single read/write iSCSI LUN backed by <file>.
Arguments:
<file> Backing file. Must exist and have a size that is a
multiple of the block size, unless --size is given.
Options:
--serial <serial> LUN serial number (default: derived from the IQN).
--size <bytes> Create/grow the backing file to this many bytes.
Accepts suffixes K, M, G, T (e.g. 10G).
--iqn <iqn> Target IQN (default: iqn.2024-01.tech.vates:<file>).
--host <host> Listen address (default: 0.0.0.0).
--port <port> Listen port (default: 3260).
--block-size <bytes> Logical block size (default: 512).
-h, --help Show this help.
`
function fail(message: string): never {
process.stderr.write(`vates-iscsi: ${message}\n`)
process.exit(1)
}
/** Parse a positive integer with optional K/M/G/T suffix (powers of 1024). */
function parseSize(value: string, label: string): number {
const match = /^(\d+)([KMGT]?)$/i.exec(value.trim())
if (match === null) {
fail(`invalid ${label}: ${value}`)
}
const units: Record<string, number> = { '': 1, K: 1024, M: 1024 ** 2, G: 1024 ** 3, T: 1024 ** 4 }
return Number(match[1]) * units[match[2].toUpperCase()]
}
/** Sanitize a string into the characters allowed in the IQN-specific part. */
function sanitizeIqnPart(value: string): string {
const cleaned = value.toLowerCase().replace(/[^a-z0-9.-]+/g, '-').replace(/^-+|-+$/g, '')
return cleaned.length > 0 ? cleaned : 'lun0'
}
const { values, positionals } = parseArgs({
allowPositionals: true,
options: {
serial: { type: 'string' },
size: { type: 'string' },
iqn: { type: 'string' },
host: { type: 'string' },
port: { type: 'string' },
'block-size': { type: 'string' },
help: { type: 'boolean', short: 'h' },
},
})
if (values.help || positionals.length === 0) {
process.stdout.write(USAGE)
process.exit(values.help ? 0 : 1)
}
if (positionals.length > 1) {
fail(`expected a single <file>, got ${positionals.length}`)
}
const path = positionals[0]
const blockSize = values['block-size'] !== undefined ? parseSize(values['block-size'], 'block size') : 512
const port = values.port !== undefined ? Number.parseInt(values.port, 10) : 3260
if (!Number.isInteger(port) || port < 0 || port > 65535) {
fail(`invalid port: ${values.port}`)
}
// Create/grow the backing file when --size is requested.
if (values.size !== undefined) {
const size = parseSize(values.size, 'size')
if (size % blockSize !== 0) {
fail(`--size (${size}) must be a multiple of the block size (${blockSize})`)
}
const exists = await stat(path).then(
() => true,
() => false
)
if (!exists) {
await writeFile(path, Buffer.alloc(0))
}
await truncate(path, size)
}
// Validate the backing file before binding the socket.
const stats = await stat(path).catch(() => fail(`backing file not found: ${path} (create one, e.g. \`truncate -s 10G ${path}\`, or pass --size)`))
if (stats.size === 0) {
fail(`backing file is empty: ${path} (size it first, e.g. \`truncate -s 10G ${path}\`, or pass --size)`)
}
const iqn = values.iqn ?? `iqn.2024-01.tech.vates:${sanitizeIqnPart(basename(path).replace(/\.[^.]+$/, ''))}`
const target = new IscsiTarget({
iqn,
host: values.host,
port,
lun: new FileBlockDevice({ path, blockSize }),
identity: values.serial !== undefined ? { serial: values.serial } : undefined,
})
await target.listen()
const address = target.address()
const listenHost = values.host ?? '0.0.0.0'
const listenPort = address?.port ?? port
process.stdout.write(
[
`iSCSI target listening on ${listenHost}:${listenPort}`,
` IQN: ${iqn}`,
` LUN: ${path} (${stats.size} bytes, ${blockSize}-byte blocks)`,
` serial: ${values.serial ?? iqn}`,
``,
`Discover with: iscsiadm -m discovery -t sendtargets -p ${listenHost === '0.0.0.0' ? '127.0.0.1' : listenHost}:${listenPort}`,
`Press Ctrl-C to stop.`,
``,
].join('\n')
)
let stopping = false
function stop(): void {
if (stopping) {
return
}
stopping = true
target.close().then(
() => process.exit(0),
error => {
process.stderr.write(`vates-iscsi: error on shutdown: ${String(error)}\n`)
process.exit(1)
}
)
}
process.once('SIGINT', stop)
process.once('SIGTERM', stop)

View File

@@ -0,0 +1,622 @@
import type { Socket } from 'node:net'
import { createLogger, type Logger } from '@xen-orchestra/log'
import type { BlockDevice } from './backend.mjs'
import {
Asc,
DATA_IN_FLAG_FINAL,
DATA_IN_FLAG_OVERFLOW,
DATA_IN_FLAG_STATUS,
DATA_IN_FLAG_UNDERFLOW,
FLAG_FINAL,
InitiatorOpcode,
ISCSI_VERSION,
LoginStatusClass,
LogoutResponse,
RejectReason,
RESERVED_TAG,
SCSI_RESP_FLAG_OVERFLOW,
SCSI_RESP_FLAG_UNDERFLOW,
SCSI_RESPONSE_COMMAND_COMPLETED,
ScsiStatus,
SenseKey,
TargetOpcode,
TEXT_FLAG_FINAL,
} from './constants.mjs'
import { buildFixedSense, handleScsiCommand, type ScsiIdentity } from './scsi.mjs'
import {
buildSendTargetsResponse,
LOGIN_MAX_DATA_SEGMENT_LENGTH,
LoginNegotiator,
OUR_MAX_RECV_DATA_SEGMENT_LENGTH,
parseTextKeys,
} from './login.mjs'
import { allocBhs, assemblePdu, type IncomingPdu, readPdu, writePdu } from './pdu.mjs'
import {
decodeCdb,
type ChapCredentials,
type CommandContext,
type NegotiatedParams,
type ScsiResponseOptions,
type SessionType,
} from './types.mjs'
import { buildR2t, receiveDataOut, type PendingWrite } from './writePath.mjs'
import { Semaphore } from './semaphore.mjs'
const log: Logger = createLogger('vates:iscsi')
/** 2^32: the modulus for all iSCSI sequence-number arithmetic. */
export const SERIAL_MODULO = 0x1_0000_0000
/** Increment an iSCSI serial number with 32-bit wraparound. */
export function incrementSerial(value: number): number {
return (value + 1) % SERIAL_MODULO
}
/** Add `delta` to an iSCSI serial number with 32-bit wraparound. */
export function addSerial(value: number, delta: number): number {
return (value + delta) % SERIAL_MODULO
}
/**
* Initiator opcodes whose BHS bytes 24-27 are CmdSN. The two that are missing —
* SCSI Data-Out and SNACK — have those bytes reserved instead; see the comment
* on `Connection`'s `#updateCommandWindow` for what reading them would cost.
*/
const CARRIES_CMD_SN: ReadonlySet<number> = new Set([
InitiatorOpcode.NOP_OUT,
InitiatorOpcode.SCSI_COMMAND,
InitiatorOpcode.SCSI_TASK_MGMT_REQUEST,
InitiatorOpcode.LOGIN_REQUEST,
InitiatorOpcode.TEXT_REQUEST,
InitiatorOpcode.LOGOUT_REQUEST,
])
/** Everything a {@link Connection} needs from the owning target. */
export interface ConnectionDeps {
readonly iqn: string
readonly identity: ScsiIdentity
readonly lun: BlockDevice
/** Drain timeout for outbound PDUs, in ms (0 disables). */
readonly writeTimeoutMs: number
/** Size of the CmdSN command window advertised to the initiator. */
readonly cmdWindow: number
/** Max number of READs served concurrently (see {@link Connection}). */
readonly readConcurrency: number
/** Allocate a fresh, non-zero Target Session Identifying Handle. */
allocateTsih(): number
/** When set, require one-way CHAP: the target challenges and verifies the initiator. */
readonly chap?: ChapCredentials
/**
* Time allowed to establish a Normal session, in ms (0 or unset disables):
* past it, the connection is closed. Also bounds Discovery sessions, which
* never become Normal ones.
*/
readonly loginTimeoutMs?: number
/** Called once this connection has established a Normal session. */
onSessionEstablished?(connection: Connection): void
/** Max length of a single READ or WRITE, in bytes. */
readonly maxTransferLength?: number
}
type Phase = 'login' | 'fullFeature' | 'closed'
/**
* Drives one iSCSI connection: login, command-window/StatSN sequencing, and
* full-feature-phase PDU dispatch. With MaxConnections=1 a connection is also
* the whole session.
*
* Everything but READ runs one at a time; the read loop advances only once a
* command fully completes. READ is dispatched without waiting on its own
* `lun.read()` (can be a slow network fetch) — up to `readConcurrency` at
* once, via `#readGate` — so one slow read can't block every other command.
*
* StatSN is allocated and written to the socket synchronously, with no
* `await` in between, so responses reach the wire in whatever order the event
* loop runs their completions — no extra sequencing needed. This does not
* order a READ against a concurrent WRITE to the same LBA range: it may see
* old or new bytes, like an untagged SCSI command with no ordering barrier.
*/
export class Connection implements CommandContext {
readonly #socket: Socket
readonly #deps: ConnectionDeps
readonly #login: LoginNegotiator
readonly #readGate: Semaphore
#phase: Phase = 'login'
#sessionType: SessionType = 'Normal'
#params: NegotiatedParams = { initiatorMaxRecvDataSegmentLength: 8192, maxBurstLength: 262144 }
#statSN = 0
#expCmdSN = 0
#maxCmdSN = 0
#cmdSNInitialized = false
#tsih = 0
// In-progress WRITEs keyed by Initiator Task Tag. Data-Out PDUs are routed
// here, so writes can be interleaved with (and outstanding alongside) other
// commands without blocking the read loop.
readonly #pendingWrites = new Map<number, PendingWrite>()
#nextTargetTransferTag = 1
#loginTimer?: ReturnType<typeof setTimeout>
constructor(socket: Socket, deps: ConnectionDeps) {
this.#socket = socket
this.#deps = deps
this.#login = new LoginNegotiator(deps.chap)
this.#readGate = new Semaphore(deps.readConcurrency)
}
get lun(): BlockDevice {
return this.#deps.lun
}
/** Forcibly tear down the connection (used when the target is closing). */
destroy(): void {
this.#phase = 'closed'
this.#socket.destroy()
}
/** Read and service PDUs until the peer logs out or the connection drops. */
async serve(): Promise<void> {
const { loginTimeoutMs = 0 } = this.#deps
if (loginTimeoutMs > 0) {
this.#loginTimer = setTimeout(() => {
log.warn('no session established in time, closing the connection', { remote: this.#socket.remoteAddress })
this.destroy()
}, loginTimeoutMs).unref()
}
try {
for (;;) {
const pdu = await readPdu(
this.#socket,
this.#phase === 'login' ? LOGIN_MAX_DATA_SEGMENT_LENGTH : OUR_MAX_RECV_DATA_SEGMENT_LENGTH
)
if (pdu === null) {
break // peer closed the connection
}
const finished = await this.#dispatch(pdu)
if (finished) {
break
}
}
} catch (error) {
this.#fail(error)
} finally {
clearTimeout(this.#loginTimer)
this.#phase = 'closed'
this.#socket.destroy()
}
}
/**
* Log a connection-ending error and tear the socket down — the same outcome
* an uncaught error in the main `serve()` loop already had, now also reached
* by a concurrently-dispatched READ that fails (e.g. a backend read error):
* with nothing else awaiting it directly, it would otherwise surface only as
* an unhandled rejection while the initiator hangs waiting for a response
* that will never come. Safe to call more than once — `socket.destroy()` is
* itself idempotent.
*/
#fail(error: unknown): void {
if (this.#phase === 'closed') {
return
}
const err = error instanceof Error ? error : new Error(String(error))
const code = (err as NodeJS.ErrnoException).code
// A peer dropping the socket (e.g. on logout) is normal, not a fault.
if (code === 'ECONNRESET' || code === 'EPIPE' || code === 'ECONNABORTED') {
log.debug('connection closed by peer', { code })
} else {
log.warn('connection error', err)
}
this.#phase = 'closed'
this.#socket.destroy()
}
// --- sequencing -----------------------------------------------------------
/**
* Advance the command window from a freshly received PDU — but only from one
* that actually carries a CmdSN.
*
* BHS bytes 24-27 hold CmdSN on command PDUs only; on a SCSI Data-Out (RFC
* 7143 §11.7) and a SNACK they are reserved, as is the immediate bit. An
* initiator zeroes them, so treating such a PDU as a command reads CmdSN=0,
* takes the non-immediate branch, and rewinds the window to ExpCmdSN=1 /
* MaxCmdSN=1+window — then advertises it on the very next R2T, SCSI Response,
* or concurrent READ's Data-In. Initiators discard the rewind as stale only
* while the session's CmdSN stays below 2^31; past that it compares as newer
* under serial arithmetic, is adopted, and the window collapses into a stall.
*
* Since every WRITE is R2T-solicited (InitialR2T=Yes), Data-Out is the common
* case, not an edge one. Allowlisting the opcodes that do carry CmdSN, rather
* than skipping the two that don't, keeps a newly handled opcode from
* silently inheriting the bug.
*/
#updateCommandWindow(pdu: IncomingPdu): void {
if (!CARRIES_CMD_SN.has(pdu.opcode)) {
return
}
const cmdSN = pdu.cmdSN
if (!this.#cmdSNInitialized) {
this.#expCmdSN = cmdSN
this.#cmdSNInitialized = true
}
// Immediate PDUs (Login, NOP pings, aborts) do not consume a CmdSN slot.
if (!pdu.immediate) {
this.#expCmdSN = incrementSerial(cmdSN)
}
this.#maxCmdSN = addSerial(this.#expCmdSN, this.#deps.cmdWindow)
}
/** Current StatSN without advancing (for R2T, NOP-In, Reject, non-final Data-In). */
#currentStatSN(): number {
return this.#statSN
}
/** Return the current StatSN and advance it (for status-bearing responses). */
#nextStatSN(): number {
const value = this.#statSN
this.#statSN = incrementSerial(value)
return value
}
#send(buffer: Buffer): Promise<void> {
return writePdu(this.#socket, buffer, this.#deps.writeTimeoutMs)
}
// --- dispatch -------------------------------------------------------------
/** Returns true when the connection should be closed (after a logout). */
async #dispatch(pdu: IncomingPdu): Promise<boolean> {
this.#updateCommandWindow(pdu)
if (this.#phase === 'login') {
if (pdu.opcode === InitiatorOpcode.LOGIN_REQUEST) {
return await this.#handleLogin(pdu)
}
await this.#sendReject(pdu, RejectReason.PROTOCOL_ERROR)
return false
}
switch (pdu.opcode) {
case InitiatorOpcode.SCSI_COMMAND:
if (this.#sessionType === 'Discovery') {
await this.#sendReject(pdu, RejectReason.PROTOCOL_ERROR)
} else {
await this.#handleScsiCommand(pdu)
}
return false
case InitiatorOpcode.TEXT_REQUEST:
await this.#handleText(pdu)
return false
case InitiatorOpcode.NOP_OUT:
await this.#handleNopOut(pdu)
return false
case InitiatorOpcode.LOGOUT_REQUEST:
await this.#handleLogout(pdu)
return true
case InitiatorOpcode.SCSI_DATA_OUT:
await this.#handleDataOut(pdu)
return false
default:
await this.#sendReject(pdu, RejectReason.COMMAND_NOT_SUPPORTED)
return false
}
}
/** Returns true when the login failed and the connection must be closed. */
async #handleLogin(pdu: IncomingPdu): Promise<boolean> {
const { flagsByte, data, statusClass = LoginStatusClass.SUCCESS, statusDetail = 0 } = this.#login.process(pdu)
if (this.#login.complete && this.#tsih === 0) {
this.#tsih = this.#deps.allocateTsih()
}
const bhs = allocBhs(TargetOpcode.LOGIN_RESPONSE)
bhs[1] = flagsByte
bhs[2] = ISCSI_VERSION // Version-max
bhs[3] = ISCSI_VERSION // Version-active
pdu.bhs.copy(bhs, 8, 8, 14) // echo ISID (6 bytes)
bhs.writeUInt16BE(this.#tsih, 14)
bhs.writeUInt32BE(pdu.itt, 16)
bhs.writeUInt32BE(this.#nextStatSN(), 24)
bhs.writeUInt32BE(this.#expCmdSN, 28)
bhs.writeUInt32BE(this.#maxCmdSN, 32)
bhs[36] = statusClass
bhs[37] = statusDetail
await this.#send(assemblePdu(bhs, data))
// destroyed while the response was being written (login timeout, target
// closing): the write still succeeds, but this connection must not go on
// and replace the current session
if (this.#phase === 'closed') {
return true
}
if (statusClass !== LoginStatusClass.SUCCESS) {
// RFC 7143 §6.3: the failure response goes out first (above), then the
// target closes the connection.
log.warn('login rejected', { statusClass, statusDetail, initiator: this.#login.initiatorName })
this.#phase = 'closed'
return true
}
if (this.#login.complete) {
this.#phase = 'fullFeature'
this.#sessionType = this.#login.sessionType
this.#params = this.#login.params
log.info('session established', {
initiator: this.#login.initiatorName,
sessionType: this.#sessionType,
target: this.#login.targetName,
})
if (this.#sessionType === 'Normal') {
clearTimeout(this.#loginTimer)
this.#deps.onSessionEstablished?.(this)
}
}
return false
}
async #handleText(pdu: IncomingPdu): Promise<void> {
const keys = parseTextKeys(pdu.data)
// SendTargets discovery: advertise this target at the address the initiator
// actually connected to.
const data =
keys.get('SendTargets') !== undefined
? buildSendTargetsResponse(
this.#deps.iqn,
this.#socket.localAddress ?? '0.0.0.0',
this.#socket.localPort ?? 3260
)
: Buffer.alloc(0)
const bhs = allocBhs(TargetOpcode.TEXT_RESPONSE)
bhs[1] = TEXT_FLAG_FINAL
bhs.writeUInt32BE(pdu.itt, 16)
bhs.writeUInt32BE(RESERVED_TAG, 20) // Target Transfer Tag: none / final
bhs.writeUInt32BE(this.#nextStatSN(), 24)
bhs.writeUInt32BE(this.#expCmdSN, 28)
bhs.writeUInt32BE(this.#maxCmdSN, 32)
await this.#send(assemblePdu(bhs, data))
}
async #handleNopOut(pdu: IncomingPdu): Promise<void> {
// A NOP-Out with the reserved ITT is the initiator's reply to a NOP-In we
// sent; nothing to do. Otherwise echo it back as a NOP-In ping reply.
if (pdu.itt === RESERVED_TAG) {
return
}
const bhs = allocBhs(TargetOpcode.NOP_IN)
bhs[1] = FLAG_FINAL
pdu.bhs.copy(bhs, 8, 8, 16) // echo LUN
bhs.writeUInt32BE(pdu.itt, 16)
bhs.writeUInt32BE(RESERVED_TAG, 20) // Target Transfer Tag
bhs.writeUInt32BE(this.#currentStatSN(), 24) // NOP-In does not advance StatSN
bhs.writeUInt32BE(this.#expCmdSN, 28)
bhs.writeUInt32BE(this.#maxCmdSN, 32)
await this.#send(assemblePdu(bhs, pdu.data)) // echo ping data
}
async #handleLogout(pdu: IncomingPdu): Promise<void> {
const bhs = allocBhs(TargetOpcode.LOGOUT_RESPONSE)
bhs[1] = FLAG_FINAL
bhs[2] = LogoutResponse.SUCCESS
bhs.writeUInt32BE(pdu.itt, 16)
bhs.writeUInt32BE(this.#nextStatSN(), 24)
bhs.writeUInt32BE(this.#expCmdSN, 28)
bhs.writeUInt32BE(this.#maxCmdSN, 32)
await this.#send(assemblePdu(bhs))
}
async #handleScsiCommand(pdu: IncomingPdu): Promise<void> {
// Login declares ImmediateData=No unconditionally, so a data segment here is
// write data we never solicited and have no path to apply. Ignoring it would
// acknowledge a write missing its first bytes — exactly the silent loss the
// solicited path is built to avoid — so fail the command loudly instead and
// let the initiator retry or give up.
if (pdu.data.length > 0) {
log.warn('immediate data is not supported', { itt: pdu.itt, length: pdu.data.length })
const sense = buildFixedSense({ key: SenseKey.ILLEGAL_REQUEST, ...Asc.INVALID_FIELD_IN_CDB })
await this.sendScsiResponse(pdu.itt, ScsiStatus.CHECK_CONDITION, { sense })
return
}
const cdb = pdu.bhs.subarray(32, 48)
if (decodeCdb(cdb).kind === 'read') {
// Not awaited: the read loop must move on immediately rather than block on
// this `lun.read()`. See the class doc for why no extra ordering is needed.
void this.#readGate
.run(() => handleScsiCommand(cdb, pdu.itt, this, this.#deps.identity, this.#deps.maxTransferLength))
.catch(error => {
this.#fail(error)
})
return
}
await handleScsiCommand(cdb, pdu.itt, this, this.#deps.identity, this.#deps.maxTransferLength)
}
async #sendReject(pdu: IncomingPdu, reason: number): Promise<void> {
log.debug('rejecting PDU', { opcode: pdu.opcode, itt: pdu.itt, reason, phase: this.#phase })
const bhs = allocBhs(TargetOpcode.REJECT)
bhs[1] = FLAG_FINAL
bhs[2] = reason
bhs.writeUInt32BE(RESERVED_TAG, 16)
bhs.writeUInt32BE(this.#currentStatSN(), 24) // Reject snapshots StatSN
bhs.writeUInt32BE(this.#expCmdSN, 28)
bhs.writeUInt32BE(this.#maxCmdSN, 32)
// The rejected PDU's header is returned as the data segment.
await this.#send(assemblePdu(bhs, pdu.bhs))
}
// --- CommandContext implementation ----------------------------------------
async sendReadData(itt: number, data: Buffer, allocationLength: number): Promise<void> {
let payload = data
let residualOverflow = 0
let residualUnderflow = 0
if (data.length > allocationLength) {
payload = data.subarray(0, allocationLength)
residualOverflow = data.length - allocationLength
} else if (data.length < allocationLength) {
residualUnderflow = allocationLength - data.length
}
if (payload.length === 0) {
return this.sendScsiResponse(itt, ScsiStatus.GOOD, { residualOverflow, residualUnderflow })
}
const maxSegment = this.#params.initiatorMaxRecvDataSegmentLength
let offset = 0
let dataSN = 0
while (offset < payload.length) {
const end = Math.min(offset + maxSegment, payload.length)
const chunk = payload.subarray(offset, end)
const isLast = end >= payload.length
const bhs = allocBhs(TargetOpcode.SCSI_DATA_IN)
let flags = 0
if (isLast) {
// Piggyback GOOD status on the final Data-In PDU.
flags |= DATA_IN_FLAG_FINAL | DATA_IN_FLAG_STATUS
if (residualOverflow > 0) {
flags |= DATA_IN_FLAG_OVERFLOW
}
if (residualUnderflow > 0) {
flags |= DATA_IN_FLAG_UNDERFLOW
}
bhs[3] = ScsiStatus.GOOD // Status valid only when the S bit is set
}
bhs[1] = flags
bhs.writeUInt32BE(itt, 16)
bhs.writeUInt32BE(RESERVED_TAG, 20) // Target Transfer Tag
if (isLast) {
bhs.writeUInt32BE(this.#nextStatSN(), 24) // StatSN valid only with the S bit
}
bhs.writeUInt32BE(this.#expCmdSN, 28)
bhs.writeUInt32BE(this.#maxCmdSN, 32)
bhs.writeUInt32BE(dataSN, 36)
bhs.writeUInt32BE(offset, 40) // Buffer Offset
if (isLast) {
bhs.writeUInt32BE(residualOverflow || residualUnderflow, 44)
}
await this.#send(assemblePdu(bhs, chunk))
offset = end
dataSN++
}
}
async beginWrite(itt: number, lunOffset: number, totalLength: number): Promise<void> {
const pending: PendingWrite = {
itt,
targetTransferTag: this.#nextTargetTransferTag++,
lunOffset,
// zero-filled, not `allocUnsafe`: belt to `receiveDataOut`'s braces, so a
// buffer that somehow reaches the LUN partly filled writes zeroes rather
// than recycled heap (which the initiator could then read straight back)
buffer: Buffer.alloc(totalLength),
totalLength,
contiguousReceived: 0,
solicited: 0,
r2tSN: 0,
}
this.#pendingWrites.set(itt, pending)
log.debug('write begin', { itt, lunOffset, totalLength })
await this.#sendNextR2t(pending)
}
/** Solicit the next burst (≤ MaxBurstLength) of a pending write via an R2T. */
async #sendNextR2t(pending: PendingWrite): Promise<void> {
const desiredLength = Math.min(pending.totalLength - pending.solicited, this.#params.maxBurstLength)
const bufferOffset = pending.solicited
const r2tSN = pending.r2tSN
pending.solicited += desiredLength
pending.r2tSN++
await this.#send(
buildR2t(
{ itt: pending.itt, r2tSN, targetTransferTag: pending.targetTransferTag, bufferOffset, desiredLength },
{ statSN: this.#currentStatSN(), expCmdSN: this.#expCmdSN, maxCmdSN: this.#maxCmdSN }
)
)
}
async #handleDataOut(pdu: IncomingPdu): Promise<void> {
const pending = this.#pendingWrites.get(pdu.itt)
if (pending === undefined) {
// Data-Out for an unknown command: protocol desync.
await this.#sendReject(pdu, RejectReason.PROTOCOL_ERROR)
return
}
const bufferOffset = pdu.readU32(40) // BufferOffset is relative to the command's data
const outcome = receiveDataOut(pending, {
targetTransferTag: pdu.readU32(20),
bufferOffset,
data: pdu.data,
})
log.debug('write data-out', {
itt: pending.itt,
offset: bufferOffset,
length: pdu.data.length,
received: pending.contiguousReceived,
total: pending.totalLength,
})
if (!outcome.ok) {
// The command is dropped, not just the PDU: leaving it pending would hang
// the initiator on an R2T that never comes, until its command timeout.
this.#pendingWrites.delete(pdu.itt)
log.warn('unplaceable write data-out', { itt: pending.itt, reason: outcome.reason })
await this.#sendReject(pdu, RejectReason.PROTOCOL_ERROR)
return
}
if (outcome.complete) {
this.#pendingWrites.delete(pdu.itt)
try {
await this.lun.write(pending.lunOffset, pending.buffer)
} catch (error) {
log.warn('LUN write failed', error instanceof Error ? error : new Error(String(error)))
const sense = buildFixedSense({ key: SenseKey.MEDIUM_ERROR, ...Asc.NO_ADDITIONAL_SENSE })
await this.sendScsiResponse(pending.itt, ScsiStatus.CHECK_CONDITION, { sense })
return
}
await this.sendScsiResponse(pending.itt, ScsiStatus.GOOD)
} else if (outcome.burstComplete) {
// Current burst complete and data remains: solicit the next one.
await this.#sendNextR2t(pending)
}
}
async sendScsiResponse(itt: number, status: number, options: ScsiResponseOptions = {}): Promise<void> {
const { sense, residualOverflow = 0, residualUnderflow = 0 } = options
const bhs = allocBhs(TargetOpcode.SCSI_RESPONSE)
let flags = FLAG_FINAL // bit 7 is always set on a SCSI Response
if (residualOverflow > 0) {
flags |= SCSI_RESP_FLAG_OVERFLOW
}
if (residualUnderflow > 0) {
flags |= SCSI_RESP_FLAG_UNDERFLOW
}
bhs[1] = flags
bhs[2] = SCSI_RESPONSE_COMMAND_COMPLETED
bhs[3] = status
bhs.writeUInt32BE(itt, 16)
bhs.writeUInt32BE(this.#nextStatSN(), 24)
bhs.writeUInt32BE(this.#expCmdSN, 28)
bhs.writeUInt32BE(this.#maxCmdSN, 32)
bhs.writeUInt32BE(residualOverflow || residualUnderflow, 44)
let data: Buffer | undefined
if (sense !== undefined) {
// Sense data in a SCSI Response is prefixed with a 2-byte length.
data = Buffer.alloc(2 + sense.length)
data.writeUInt16BE(sense.length, 0)
sense.copy(data, 2)
}
await this.#send(assemblePdu(bhs, data))
}
}

View File

@@ -0,0 +1,712 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import type { Socket } from 'node:net'
import { Duplex } from 'node:stream'
import type { BlockDevice } from './backend.mjs'
import {
DATA_IN_FLAG_FINAL,
DATA_IN_FLAG_OVERFLOW,
DATA_IN_FLAG_STATUS,
DATA_IN_FLAG_UNDERFLOW,
FLAG_FINAL,
InitiatorOpcode,
LOGIN_CSG_SHIFT,
LOGIN_FLAG_TRANSIT,
LoginStage,
LoginStatusClass,
OPCODE_IMMEDIATE,
RejectReason,
RESERVED_TAG,
ScsiStatus,
SenseKey,
TargetOpcode,
} from './constants.mjs'
import { Connection, type ConnectionDeps } from './connection.mjs'
import { serializeTextKeys } from './login.mjs'
import { assemblePdu, IncomingPdu } from './pdu.mjs'
import type { ScsiIdentity } from './scsi.mjs'
const IDENTITY: ScsiIdentity = { vendor: 'VATES', product: 'ISCSI LUN', revision: '0001', serial: 'unit-serial-1' }
const BLOCK_SIZE = 512
const LUN_SIZE = 16 * BLOCK_SIZE
// Small enough that a handful of blocks splits into several PDUs / bursts, so
// the chunking and multi-burst paths are exercised without huge buffers.
const MAX_RECV_DATA_SEGMENT_LENGTH = 512
const MAX_BURST_LENGTH = 512
// --- test doubles -----------------------------------------------------------
/**
* Stands in for the TCP socket. `deliver()` queues a PDU for the connection's
* read loop (the readable side); everything the connection writes is captured
* in `sent` rather than transmitted. `Connection` only ever reads, writes, and
* destroys its socket, plus `localAddress`/`localPort` for SendTargets — hence
* the narrow cast in `asSocket()`.
*/
class FakeSocket extends Duplex {
readonly sent: Array<Buffer> = []
readonly localAddress = '10.0.0.1'
readonly localPort = 3260
override _read(): void {}
override _write(chunk: Buffer, _encoding: BufferEncoding, callback: (error?: Error | null) => void): void {
this.sent.push(Buffer.from(chunk))
callback()
}
/** Hand a PDU to the connection. */
deliver(pdu: Buffer): void {
this.push(pdu)
}
/** Clean end-of-stream, which ends `serve()`'s read loop. */
finish(): void {
this.push(null)
}
asSocket(): Socket {
return this as unknown as Socket
}
}
class MemoryLun implements BlockDevice {
readonly content = Buffer.alloc(LUN_SIZE)
/** Set to make the next `write()` fail, standing in for a backend error. */
failNextWrite = false
getSize(): number {
return this.content.length
}
getBlockSize(): number {
return BLOCK_SIZE
}
async read(offset: number, length: number): Promise<Buffer> {
return Buffer.from(this.content.subarray(offset, offset + length))
}
async write(offset: number, data: Buffer): Promise<void> {
if (this.failNextWrite) {
this.failNextWrite = false
throw new Error('backend write failed')
}
data.copy(this.content, offset)
}
async flush(): Promise<void> {}
async close(): Promise<void> {}
}
// --- initiator-side PDU builders --------------------------------------------
/** A single-PDU operational login that transits straight to Full Feature Phase. */
const loginPdu = (): Buffer => {
const bhs = Buffer.alloc(48)
bhs[0] = InitiatorOpcode.LOGIN_REQUEST | OPCODE_IMMEDIATE
bhs[1] = LOGIN_FLAG_TRANSIT | (LoginStage.OPERATIONAL_NEGOTIATION << LOGIN_CSG_SHIFT) | LoginStage.FULL_FEATURE_PHASE
bhs.writeUInt32BE(1, 16) // ITT
return assemblePdu(
bhs,
serializeTextKeys([
['InitiatorName', 'iqn.2026-01.tech.vates:test-initiator'],
['SessionType', 'Normal'],
['HeaderDigest', 'None'],
['DataDigest', 'None'],
['MaxRecvDataSegmentLength', String(MAX_RECV_DATA_SEGMENT_LENGTH)],
['MaxBurstLength', String(MAX_BURST_LENGTH)],
])
)
}
const scsiCommandPdu = ({
itt,
cmdSN,
cdb,
data,
}: {
itt: number
cmdSN: number
cdb: Buffer
data?: Buffer
}): Buffer => {
const bhs = Buffer.alloc(48)
bhs[0] = InitiatorOpcode.SCSI_COMMAND
bhs[1] = FLAG_FINAL
bhs.writeUInt32BE(itt, 16)
bhs.writeUInt32BE(cmdSN, 24)
cdb.copy(bhs, 32)
return assemblePdu(bhs, data)
}
const dataOutPdu = ({
itt,
targetTransferTag,
bufferOffset,
data,
dataSN = 0,
}: {
itt: number
targetTransferTag: number
bufferOffset: number
data: Buffer
dataSN?: number
}): Buffer => {
const bhs = Buffer.alloc(48)
bhs[0] = InitiatorOpcode.SCSI_DATA_OUT
bhs[1] = FLAG_FINAL
bhs.writeUInt32BE(itt, 16)
bhs.writeUInt32BE(targetTransferTag, 20)
bhs.writeUInt32BE(dataSN, 36)
bhs.writeUInt32BE(bufferOffset, 40)
return assemblePdu(bhs, data)
}
const cdb10 = (opcode: number, lba: number, blocks: number): Buffer => {
const cdb = Buffer.alloc(16)
cdb[0] = opcode
cdb.writeUInt32BE(lba, 2)
cdb.writeUInt16BE(blocks, 7)
return cdb
}
const read10 = (lba: number, blocks: number) => cdb10(0x28, lba, blocks)
const write10 = (lba: number, blocks: number) => cdb10(0x2a, lba, blocks)
const cdb16 = (opcode: number, lba: bigint, blocks: number): Buffer => {
const cdb = Buffer.alloc(16)
cdb[0] = opcode
cdb.writeBigUInt64BE(lba, 2)
cdb.writeUInt32BE(blocks, 10)
return cdb
}
const read16 = (lba: bigint, blocks: number) => cdb16(0x88, lba, blocks)
const write16 = (lba: bigint, blocks: number) => cdb16(0x8a, lba, blocks)
const inquiry = (allocationLength: number): Buffer => {
const cdb = Buffer.alloc(16)
cdb[0] = 0x12
cdb.writeUInt16BE(allocationLength, 3)
return cdb
}
// --- harness ----------------------------------------------------------------
/** Re-read a PDU the target sent, so assertions can talk in field names. */
const parseSent = (buffer: Buffer): IncomingPdu => {
const dataSegmentLength = buffer.readUIntBE(5, 3)
return new IncomingPdu(buffer.subarray(0, 48), Buffer.alloc(0), buffer.subarray(48, 48 + dataSegmentLength))
}
/**
* Sense key of a SCSI Response: its data segment is a 2-byte sense length then
* the fixed-format sense data, whose byte 2 holds the key in its low nibble.
*/
const senseKey = (response: IncomingPdu): number => response.data[4] & 0x0f
class Harness {
readonly socket = new FakeSocket()
readonly lun = new MemoryLun()
readonly serving: Promise<void>
constructor(overrides: Partial<ConnectionDeps> = {}) {
const deps: ConnectionDeps = {
iqn: 'iqn.2026-01.tech.vates:target',
identity: IDENTITY,
lun: this.lun,
writeTimeoutMs: 0,
cmdWindow: 64,
readConcurrency: 4,
allocateTsih: () => 1,
...overrides,
}
this.serving = new Connection(this.socket.asSocket(), deps).serve()
}
/** Every PDU the target has sent, parsed. */
get sent(): Array<IncomingPdu> {
return this.socket.sent.map(parseSent)
}
/**
* Let the read loop run until the target has sent `count` PDUs in total,
* optionally delivering something first. Bounded so a response that never
* comes fails the test instead of hanging it.
*/
async untilSent(count: number, deliver?: () => void): Promise<Array<IncomingPdu>> {
deliver?.()
for (let turn = 0; turn < 500 && this.socket.sent.length < count; turn++) {
await new Promise(resolve => setImmediate(resolve))
}
const sent = this.sent
assert.ok(
sent.length >= count,
`expected ${count} PDUs, got ${sent.length}: ${sent.map(pdu => pdu.opcode.toString(16)).join(',')}`
)
return sent
}
/** Log in and confirm the target accepted it. */
async login(): Promise<void> {
// relative to what has already been sent, so this works mid-connection too
const baseline = this.socket.sent.length
const response = (await this.untilSent(baseline + 1, () => this.socket.deliver(loginPdu())))[baseline]
assert.equal(response.opcode, TargetOpcode.LOGIN_RESPONSE)
assert.equal(response.readU8(36), LoginStatusClass.SUCCESS)
}
async close(): Promise<void> {
this.socket.finish()
await this.serving
}
}
/** Log in, then run `body`, always tearing the connection down afterwards. */
const withSession = async (body: (harness: Harness) => Promise<void>, overrides?: Partial<ConnectionDeps>) => {
const harness = new Harness(overrides)
try {
await harness.login()
await body(harness)
} finally {
await harness.close()
}
}
const CMD_WINDOW = 64
/** ExpCmdSN (bytes 28-31) and MaxCmdSN (32-35), the command window we advertise. */
const commandWindow = (pdu: IncomingPdu) => ({ expCmdSN: pdu.readU32(28), maxCmdSN: pdu.readU32(32) })
describe('command window', () => {
// Bytes 24-27 are CmdSN only on command PDUs. On a Data-Out (RFC 7143 §11.7)
// and a SNACK they are reserved, so reading a window out of them yields the
// zeroes an initiator puts there — advertising ExpCmdSN=1 mid-session. Real
// initiators discard that as stale only while session CmdSN < 2^31; past the
// wrap it compares as newer, is adopted, and the window collapses to a stall.
it('is not recomputed from the reserved bytes of a Data-Out', async () => {
await withSession(async harness => {
// two bursts, so an R2T is emitted *after* a Data-Out has been processed
harness.socket.deliver(scsiCommandPdu({ itt: 31, cmdSN: 50, cdb: write10(0, 2) }))
const [, firstR2t] = await harness.untilSent(2)
const expected = { expCmdSN: 51, maxCmdSN: 51 + CMD_WINDOW }
assert.deepEqual(commandWindow(firstR2t), expected)
const ttt = firstR2t.readU32(20)
harness.socket.deliver(
dataOutPdu({ itt: 31, targetTransferTag: ttt, bufferOffset: 0, data: Buffer.alloc(BLOCK_SIZE, 0x11) })
)
const [, , secondR2t] = await harness.untilSent(3)
assert.equal(secondR2t.opcode, TargetOpcode.R2T)
assert.deepEqual(commandWindow(secondR2t), expected)
harness.socket.deliver(
dataOutPdu({
itt: 31,
targetTransferTag: ttt,
bufferOffset: BLOCK_SIZE,
data: Buffer.alloc(BLOCK_SIZE, 0x22),
dataSN: 1,
})
)
const [, , , response] = await harness.untilSent(4)
assert.equal(response.opcode, TargetOpcode.SCSI_RESPONSE)
assert.deepEqual(commandWindow(response), expected)
})
})
it('is not recomputed from an opcode the target does not implement', async () => {
await withSession(async harness => {
harness.socket.deliver(scsiCommandPdu({ itt: 33, cmdSN: 70, cdb: read10(0, 1) }))
const [, dataIn] = await harness.untilSent(2)
const expected = { expCmdSN: 71, maxCmdSN: 71 + CMD_WINDOW }
assert.deepEqual(commandWindow(dataIn), expected)
// SNACK also has bytes 24-27 reserved; it is rejected, but the Reject it
// gets must still carry the window the session actually reached
const bhs = Buffer.alloc(48)
bhs[0] = InitiatorOpcode.SNACK_REQUEST
harness.socket.deliver(assemblePdu(bhs))
const [, , rejected] = await harness.untilSent(3)
assert.equal(rejected.opcode, TargetOpcode.REJECT)
assert.equal(rejected.readU8(2), RejectReason.COMMAND_NOT_SUPPORTED)
assert.deepEqual(commandWindow(rejected), expected)
})
})
it('advances the window for each command PDU, but not for an immediate one', async () => {
await withSession(async harness => {
harness.socket.deliver(scsiCommandPdu({ itt: 35, cmdSN: 5, cdb: read10(0, 1) }))
const [, dataIn] = await harness.untilSent(2)
assert.deepEqual(commandWindow(dataIn), { expCmdSN: 6, maxCmdSN: 6 + CMD_WINDOW })
// an immediate NOP-Out is answered but consumes no CmdSN slot
const bhs = Buffer.alloc(48)
bhs[0] = InitiatorOpcode.NOP_OUT | OPCODE_IMMEDIATE
bhs[1] = FLAG_FINAL
bhs.writeUInt32BE(37, 16)
bhs.writeUInt32BE(RESERVED_TAG, 20)
bhs.writeUInt32BE(6, 24)
harness.socket.deliver(assemblePdu(bhs))
const [, , nopIn] = await harness.untilSent(3)
assert.equal(nopIn.opcode, TargetOpcode.NOP_IN)
assert.deepEqual(commandWindow(nopIn), { expCmdSN: 6, maxCmdSN: 6 + CMD_WINDOW })
})
})
})
describe('login', () => {
it('reaches full feature phase and only then accepts commands', async () => {
const harness = new Harness()
try {
// a command before login is a protocol error, not a served command
harness.socket.deliver(scsiCommandPdu({ itt: 5, cmdSN: 0, cdb: read10(0, 1) }))
const [rejected] = await harness.untilSent(1)
assert.equal(rejected.opcode, TargetOpcode.REJECT)
assert.equal(rejected.readU8(2), RejectReason.PROTOCOL_ERROR)
await harness.login()
// the same command now gets served
harness.socket.deliver(scsiCommandPdu({ itt: 5, cmdSN: 1, cdb: read10(0, 1) }))
const sent = await harness.untilSent(3)
assert.equal(sent[2].opcode, TargetOpcode.SCSI_DATA_IN)
} finally {
await harness.close()
}
})
})
describe('write path wiring', () => {
it('solicits the data, commits it to the LUN, and answers GOOD', async () => {
await withSession(async harness => {
harness.socket.deliver(scsiCommandPdu({ itt: 9, cmdSN: 1, cdb: write10(2, 1) }))
const [, r2t] = await harness.untilSent(2)
assert.equal(r2t.opcode, TargetOpcode.R2T)
assert.equal(r2t.itt, 9)
assert.equal(r2t.readU32(36), 0) // R2TSN
assert.equal(r2t.readU32(40), 0) // BufferOffset
assert.equal(r2t.readU32(44), BLOCK_SIZE) // DesiredDataTransferLength
const payload = Buffer.alloc(BLOCK_SIZE, 0xab)
harness.socket.deliver(dataOutPdu({ itt: 9, targetTransferTag: r2t.readU32(20), bufferOffset: 0, data: payload }))
const [, , response] = await harness.untilSent(3)
assert.equal(response.opcode, TargetOpcode.SCSI_RESPONSE)
assert.equal(response.readU8(3), ScsiStatus.GOOD)
// ... and the bytes landed at the LBA the CDB asked for, nowhere else
assert.deepEqual(harness.lun.content.subarray(2 * BLOCK_SIZE, 3 * BLOCK_SIZE), payload)
assert.deepEqual(harness.lun.content.subarray(0, 2 * BLOCK_SIZE), Buffer.alloc(2 * BLOCK_SIZE))
assert.deepEqual(harness.lun.content.subarray(3 * BLOCK_SIZE), Buffer.alloc(LUN_SIZE - 3 * BLOCK_SIZE))
})
})
it('solicits one burst at a time for a write larger than MaxBurstLength', async () => {
await withSession(async harness => {
// 2 blocks with a 1-block burst limit: two R2Ts, second only after the first lands
harness.socket.deliver(scsiCommandPdu({ itt: 11, cmdSN: 1, cdb: write10(0, 2) }))
const [, first] = await harness.untilSent(2)
assert.equal(first.readU32(36), 0) // R2TSN
assert.equal(first.readU32(40), 0) // BufferOffset
assert.equal(first.readU32(44), MAX_BURST_LENGTH)
const ttt = first.readU32(20)
harness.socket.deliver(
dataOutPdu({ itt: 11, targetTransferTag: ttt, bufferOffset: 0, data: Buffer.alloc(BLOCK_SIZE, 0x11) })
)
const [, , second] = await harness.untilSent(3)
assert.equal(second.opcode, TargetOpcode.R2T)
assert.equal(second.readU32(36), 1) // R2TSN advanced
assert.equal(second.readU32(40), BLOCK_SIZE) // next burst picks up where the first ended
assert.equal(second.readU32(44), MAX_BURST_LENGTH)
// nothing is committed until the whole command has landed
assert.deepEqual(harness.lun.content.subarray(0, BLOCK_SIZE), Buffer.alloc(BLOCK_SIZE))
harness.socket.deliver(
dataOutPdu({
itt: 11,
targetTransferTag: ttt,
bufferOffset: BLOCK_SIZE,
data: Buffer.alloc(BLOCK_SIZE, 0x22),
dataSN: 1,
})
)
const [, , , response] = await harness.untilSent(4)
assert.equal(response.opcode, TargetOpcode.SCSI_RESPONSE)
assert.equal(response.readU8(3), ScsiStatus.GOOD)
assert.deepEqual(
harness.lun.content.subarray(0, 2 * BLOCK_SIZE),
Buffer.concat([Buffer.alloc(BLOCK_SIZE, 0x11), Buffer.alloc(BLOCK_SIZE, 0x22)])
)
})
})
it('rejects an unplaceable Data-Out and drops the command with it', async () => {
await withSession(async harness => {
harness.socket.deliver(scsiCommandPdu({ itt: 13, cmdSN: 1, cdb: write10(0, 1) }))
const [, r2t] = await harness.untilSent(2)
const ttt = r2t.readU32(20)
// BufferOffset past the end of the command's data: a copy there would land
// nothing, so the write must be refused rather than acknowledged
harness.socket.deliver(
dataOutPdu({
itt: 13,
targetTransferTag: ttt,
bufferOffset: BLOCK_SIZE,
data: Buffer.alloc(BLOCK_SIZE, 0xcc),
})
)
const [, , rejected] = await harness.untilSent(3)
assert.equal(rejected.opcode, TargetOpcode.REJECT)
assert.equal(rejected.readU8(2), RejectReason.PROTOCOL_ERROR)
assert.deepEqual(harness.lun.content, Buffer.alloc(LUN_SIZE))
// the command is gone, not merely stalled: a now-valid Data-Out for it is
// an unknown ITT, and no SCSI Response ever completes it
harness.socket.deliver(
dataOutPdu({ itt: 13, targetTransferTag: ttt, bufferOffset: 0, data: Buffer.alloc(BLOCK_SIZE, 0xdd) })
)
const sent = await harness.untilSent(4)
assert.equal(sent[3].opcode, TargetOpcode.REJECT)
assert.deepEqual(harness.lun.content, Buffer.alloc(LUN_SIZE))
assert.equal(
sent.some(pdu => pdu.opcode === TargetOpcode.SCSI_RESPONSE),
false
)
})
})
it("rejects a Data-Out carrying another command's Target Transfer Tag", async () => {
await withSession(async harness => {
harness.socket.deliver(scsiCommandPdu({ itt: 15, cmdSN: 1, cdb: write10(0, 1) }))
const [, r2t] = await harness.untilSent(2)
harness.socket.deliver(
dataOutPdu({
itt: 15,
targetTransferTag: r2t.readU32(20) + 1,
bufferOffset: 0,
data: Buffer.alloc(BLOCK_SIZE, 0xee),
})
)
const [, , rejected] = await harness.untilSent(3)
assert.equal(rejected.opcode, TargetOpcode.REJECT)
assert.deepEqual(harness.lun.content, Buffer.alloc(LUN_SIZE))
})
})
it('rejects a Data-Out for a command it knows nothing about', async () => {
await withSession(async harness => {
harness.socket.deliver(
dataOutPdu({ itt: 404, targetTransferTag: 1, bufferOffset: 0, data: Buffer.alloc(BLOCK_SIZE, 0xff) })
)
const [, rejected] = await harness.untilSent(2)
assert.equal(rejected.opcode, TargetOpcode.REJECT)
assert.equal(rejected.readU8(2), RejectReason.PROTOCOL_ERROR)
})
})
it('reports a failed backend write as CHECK CONDITION, not a dropped connection', async () => {
await withSession(async harness => {
harness.lun.failNextWrite = true
harness.socket.deliver(scsiCommandPdu({ itt: 17, cmdSN: 1, cdb: write10(0, 1) }))
const [, r2t] = await harness.untilSent(2)
harness.socket.deliver(
dataOutPdu({
itt: 17,
targetTransferTag: r2t.readU32(20),
bufferOffset: 0,
data: Buffer.alloc(BLOCK_SIZE, 0xab),
})
)
const [, , response] = await harness.untilSent(3)
assert.equal(response.opcode, TargetOpcode.SCSI_RESPONSE)
assert.equal(response.readU8(3), ScsiStatus.CHECK_CONDITION)
assert.equal(senseKey(response), SenseKey.MEDIUM_ERROR)
})
})
it('fails a command carrying immediate data instead of silently dropping it', async () => {
await withSession(async harness => {
// ImmediateData=No is declared at login, so this cannot be honoured — and
// must not be ignored either, since the write would be short by these bytes
harness.socket.deliver(
scsiCommandPdu({ itt: 19, cmdSN: 1, cdb: write10(0, 1), data: Buffer.alloc(BLOCK_SIZE, 0x99) })
)
const [, response] = await harness.untilSent(2)
assert.equal(response.opcode, TargetOpcode.SCSI_RESPONSE)
assert.equal(response.readU8(3), ScsiStatus.CHECK_CONDITION)
assert.equal(senseKey(response), SenseKey.ILLEGAL_REQUEST)
assert.deepEqual(harness.lun.content, Buffer.alloc(LUN_SIZE))
// no R2T either: the command never started
assert.equal(
harness.sent.some(pdu => pdu.opcode === TargetOpcode.R2T),
false
)
})
})
})
describe('Data-In chunking', () => {
/** Read `blocks` blocks of a LUN pre-filled with a recognizable pattern. */
const readBack = async (harness: Harness, blocks: number): Promise<Array<IncomingPdu>> => {
for (let i = 0; i < harness.lun.content.length; i++) {
harness.lun.content[i] = (i * 7 + 3) & 0xff
}
harness.socket.deliver(scsiCommandPdu({ itt: 21, cmdSN: 1, cdb: read10(0, blocks) }))
const expected = Math.ceil((blocks * BLOCK_SIZE) / MAX_RECV_DATA_SEGMENT_LENGTH)
const sent = await harness.untilSent(1 + expected)
return sent.slice(1)
}
it('splits the payload at the initiator MaxRecvDataSegmentLength', async () => {
await withSession(async harness => {
const dataIn = await readBack(harness, 4)
assert.equal(dataIn.length, 4)
for (const pdu of dataIn) {
assert.equal(pdu.opcode, TargetOpcode.SCSI_DATA_IN)
assert.equal(pdu.data.length, MAX_RECV_DATA_SEGMENT_LENGTH)
}
// reassembled, it is exactly what the LUN holds
assert.deepEqual(Buffer.concat(dataIn.map(pdu => pdu.data)), harness.lun.content.subarray(0, 4 * BLOCK_SIZE))
})
})
it('numbers the PDUs with a gapless DataSN and stamps each BufferOffset', async () => {
await withSession(async harness => {
const dataIn = await readBack(harness, 3)
assert.deepEqual(
dataIn.map(pdu => pdu.readU32(36)),
[0, 1, 2]
)
assert.deepEqual(
dataIn.map(pdu => pdu.readU32(40)),
[0, MAX_RECV_DATA_SEGMENT_LENGTH, 2 * MAX_RECV_DATA_SEGMENT_LENGTH]
)
// Target Transfer Tag is unused on Data-In
for (const pdu of dataIn) {
assert.equal(pdu.readU32(20), RESERVED_TAG)
}
})
})
it('carries the status only on the final PDU, and advances StatSN once', async () => {
await withSession(async harness => {
const statSNAfterLogin = parseSent(harness.socket.sent[0]).readU32(24) + 1
const dataIn = await readBack(harness, 3)
for (const pdu of dataIn.slice(0, -1)) {
assert.equal(pdu.flags & DATA_IN_FLAG_FINAL, 0)
assert.equal(pdu.flags & DATA_IN_FLAG_STATUS, 0)
// both fields are defined as meaningless without the S bit, and are left
// unset rather than filled with a value an initiator might act on
assert.equal(pdu.readU8(3), 0)
assert.equal(pdu.readU32(24), 0)
}
const last = dataIn[dataIn.length - 1]
assert.equal(last.flags & DATA_IN_FLAG_FINAL, DATA_IN_FLAG_FINAL)
assert.equal(last.flags & DATA_IN_FLAG_STATUS, DATA_IN_FLAG_STATUS)
assert.equal(last.readU8(3), ScsiStatus.GOOD)
// exactly one StatSN consumed for the whole command, on its final PDU
assert.equal(last.readU32(24), statSNAfterLogin)
})
})
it('sends a single Data-In when the payload fits one segment', async () => {
await withSession(async harness => {
const dataIn = await readBack(harness, 1)
assert.equal(dataIn.length, 1)
assert.equal(
dataIn[0].flags & (DATA_IN_FLAG_FINAL | DATA_IN_FLAG_STATUS),
DATA_IN_FLAG_FINAL | DATA_IN_FLAG_STATUS
)
assert.equal(dataIn[0].readU32(36), 0)
})
})
it('flags residual overflow when the allocation length truncates the payload', async () => {
await withSession(async harness => {
// standard INQUIRY data is 36 bytes; ask for 8
harness.socket.deliver(scsiCommandPdu({ itt: 23, cmdSN: 1, cdb: inquiry(8) }))
const [, dataIn] = await harness.untilSent(2)
assert.equal(dataIn.opcode, TargetOpcode.SCSI_DATA_IN)
assert.equal(dataIn.data.length, 8)
assert.equal(dataIn.flags & DATA_IN_FLAG_OVERFLOW, DATA_IN_FLAG_OVERFLOW)
assert.equal(dataIn.flags & DATA_IN_FLAG_UNDERFLOW, 0)
assert.equal(dataIn.readU32(44), 36 - 8) // Residual Count
})
})
it('flags residual underflow when the payload is shorter than requested', async () => {
await withSession(async harness => {
harness.socket.deliver(scsiCommandPdu({ itt: 25, cmdSN: 1, cdb: inquiry(100) }))
const [, dataIn] = await harness.untilSent(2)
assert.equal(dataIn.data.length, 36)
assert.equal(dataIn.flags & DATA_IN_FLAG_UNDERFLOW, DATA_IN_FLAG_UNDERFLOW)
assert.equal(dataIn.flags & DATA_IN_FLAG_OVERFLOW, 0)
assert.equal(dataIn.readU32(44), 100 - 36)
})
})
it('answers with status alone, no Data-In, when nothing is allocated for the data', async () => {
await withSession(async harness => {
harness.socket.deliver(scsiCommandPdu({ itt: 27, cmdSN: 1, cdb: inquiry(0) }))
const [, response] = await harness.untilSent(2)
assert.equal(response.opcode, TargetOpcode.SCSI_RESPONSE)
assert.equal(response.readU8(3), ScsiStatus.GOOD)
assert.equal(
harness.sent.some(pdu => pdu.opcode === TargetOpcode.SCSI_DATA_IN),
false
)
})
})
})
describe('unservable CDBs', () => {
// An LBA above 2^53 cannot be held exactly by a JS number. Decoding it must
// not throw: the error would unwind out of the dispatch loop and destroy the
// socket, turning one bogus CDB — a buggy or hostile initiator's — into the
// loss of the whole session and every command in flight on it.
for (const [name, cdb] of [
['READ(16)', read16(0xffffffffffffffffn, 1)],
['WRITE(16)', write16(0xffffffffffffffffn, 1)],
] as const) {
it(`answers a ${name} past the addressable range with CHECK CONDITION, keeping the session`, async () => {
await withSession(async harness => {
harness.socket.deliver(scsiCommandPdu({ itt: 31, cmdSN: 1, cdb }))
const [, response] = await harness.untilSent(2)
assert.equal(response.opcode, TargetOpcode.SCSI_RESPONSE)
assert.equal(response.readU8(3), ScsiStatus.CHECK_CONDITION)
assert.equal(senseKey(response), SenseKey.ILLEGAL_REQUEST)
assert.equal(
harness.sent.some(pdu => pdu.opcode === TargetOpcode.R2T),
false
)
// the connection is still serving: a following command is answered
harness.socket.deliver(scsiCommandPdu({ itt: 32, cmdSN: 2, cdb: inquiry(36) }))
const [, , dataIn] = await harness.untilSent(3)
assert.equal(dataIn.opcode, TargetOpcode.SCSI_DATA_IN)
})
})
}
})

View File

@@ -0,0 +1,200 @@
// iSCSI (RFC 7143 / RFC 3720) and SCSI (SPC-3 / SBC-3) protocol constants.
//
// Only the subset required by a minimal single-LUN target is defined here.
// --- iSCSI PDU layout -------------------------------------------------------
/** Length of the Basic Header Segment, in bytes. Fixed by the protocol. */
export const BHS_LENGTH = 48
/** Bit set in byte 0 for initiator PDUs that must be processed immediately. */
export const OPCODE_IMMEDIATE = 0x40
/** Mask extracting the 6-bit opcode from byte 0. */
export const OPCODE_MASK = 0x3f
/** Final bit, byte 1, set on the last PDU of a sequence. */
export const FLAG_FINAL = 0x80
/** Sentinel for "no tag" in Initiator/Target Task Tag fields. */
export const RESERVED_TAG = 0xffffffff
// --- iSCSI opcodes ----------------------------------------------------------
/** Opcodes sent by the initiator (request PDUs). */
export const InitiatorOpcode = {
NOP_OUT: 0x00,
SCSI_COMMAND: 0x01,
SCSI_TASK_MGMT_REQUEST: 0x02,
LOGIN_REQUEST: 0x03,
TEXT_REQUEST: 0x04,
SCSI_DATA_OUT: 0x05,
LOGOUT_REQUEST: 0x06,
SNACK_REQUEST: 0x10,
} as const
/** Opcodes sent by the target (response PDUs). */
export const TargetOpcode = {
NOP_IN: 0x20,
SCSI_RESPONSE: 0x21,
SCSI_TASK_MGMT_RESPONSE: 0x22,
LOGIN_RESPONSE: 0x23,
TEXT_RESPONSE: 0x24,
SCSI_DATA_IN: 0x25,
LOGOUT_RESPONSE: 0x26,
R2T: 0x31,
ASYNC_MESSAGE: 0x32,
REJECT: 0x3f,
} as const
// --- Login negotiation ------------------------------------------------------
/** Login/text stage codes used in the CSG/NSG fields. */
export const LoginStage = {
SECURITY_NEGOTIATION: 0,
OPERATIONAL_NEGOTIATION: 1,
FULL_FEATURE_PHASE: 3,
} as const
/** Login Request/Response byte 1 flags. */
export const LOGIN_FLAG_TRANSIT = 0x80
export const LOGIN_FLAG_CONTINUE = 0x40
/** Mask + shift for the Current Stage (CSG) field in byte 1. */
export const LOGIN_CSG_MASK = 0x0c
export const LOGIN_CSG_SHIFT = 2
/** Mask for the Next Stage (NSG) field in byte 1. */
export const LOGIN_NSG_MASK = 0x03
/** iSCSI version (only version 0 exists). */
export const ISCSI_VERSION = 0x00
/** Login Response Status-Class values. */
export const LoginStatusClass = {
SUCCESS: 0x00,
REDIRECTION: 0x01,
INITIATOR_ERROR: 0x02,
TARGET_ERROR: 0x03,
} as const
/**
* Login Response Status-Detail values (byte 37), qualifying the Status-Class.
* `AUTHENTICATION_FAILURE` pairs with `INITIATOR_ERROR` to reject a bad CHAP
* login (RFC 7143 §11.1.5).
*/
export const LoginStatusDetail = {
SUCCESS: 0x00,
AUTHENTICATION_FAILURE: 0x01,
} as const
/** `AuthMethod` login-key values we understand (RFC 7143 §12.1). */
export const AuthMethod = {
NONE: 'None',
CHAP: 'CHAP',
} as const
// --- Text/Data flags --------------------------------------------------------
export const TEXT_FLAG_FINAL = 0x80
export const TEXT_FLAG_CONTINUE = 0x40
/** SCSI Data-In byte 1 flags. */
export const DATA_IN_FLAG_FINAL = 0x80
export const DATA_IN_FLAG_ACK = 0x40
export const DATA_IN_FLAG_OVERFLOW = 0x04
export const DATA_IN_FLAG_UNDERFLOW = 0x02
export const DATA_IN_FLAG_STATUS = 0x01
/** SCSI Command (initiator) byte 1 flags. */
export const SCSI_CMD_FLAG_READ = 0x40
export const SCSI_CMD_FLAG_WRITE = 0x20
/** SCSI Response byte 1 residual flags. */
export const SCSI_RESP_FLAG_OVERFLOW = 0x04
export const SCSI_RESP_FLAG_UNDERFLOW = 0x02
/** SCSI Response "Response" field (byte 2). */
export const SCSI_RESPONSE_COMMAND_COMPLETED = 0x00
/** Logout reason codes (byte 1, low 7 bits). */
export const LogoutReason = {
CLOSE_SESSION: 0,
CLOSE_CONNECTION: 1,
REMOVE_CONNECTION_FOR_RECOVERY: 2,
} as const
/** Logout Response codes (byte 2). */
export const LogoutResponse = {
SUCCESS: 0,
} as const
// --- SCSI status codes ------------------------------------------------------
export const ScsiStatus = {
GOOD: 0x00,
CHECK_CONDITION: 0x02,
} as const
// --- SCSI CDB opcodes -------------------------------------------------------
export const ScsiOpcode = {
TEST_UNIT_READY: 0x00,
REQUEST_SENSE: 0x03,
INQUIRY: 0x12,
MODE_SENSE_6: 0x1a,
READ_CAPACITY_10: 0x25,
READ_10: 0x28,
WRITE_10: 0x2a,
SYNCHRONIZE_CACHE_10: 0x35,
MODE_SENSE_10: 0x5a,
READ_16: 0x88,
WRITE_16: 0x8a,
SYNCHRONIZE_CACHE_16: 0x91,
SERVICE_ACTION_IN_16: 0x9e, // READ CAPACITY(16) lives under this opcode
REPORT_LUNS: 0xa0,
} as const
/** Service action (CDB byte 1, low 5 bits) for SERVICE_ACTION_IN_16. */
export const SERVICE_ACTION_READ_CAPACITY_16 = 0x10
// --- SCSI sense keys / additional sense codes -------------------------------
export const SenseKey = {
NO_SENSE: 0x00,
RECOVERED_ERROR: 0x01,
NOT_READY: 0x02,
MEDIUM_ERROR: 0x03,
HARDWARE_ERROR: 0x04,
ILLEGAL_REQUEST: 0x05,
UNIT_ATTENTION: 0x06,
DATA_PROTECT: 0x07,
} as const
/** Additional Sense Code / Qualifier pairs we may emit. */
export const Asc = {
NO_ADDITIONAL_SENSE: { asc: 0x00, ascq: 0x00 },
INVALID_COMMAND_OPERATION_CODE: { asc: 0x20, ascq: 0x00 },
INVALID_FIELD_IN_CDB: { asc: 0x24, ascq: 0x00 },
LOGICAL_BLOCK_ADDRESS_OUT_OF_RANGE: { asc: 0x21, ascq: 0x00 },
WRITE_PROTECTED: { asc: 0x27, ascq: 0x00 },
} as const
/** Reject reason codes (Reject PDU byte 2). */
export const RejectReason = {
PROTOCOL_ERROR: 0x04,
COMMAND_NOT_SUPPORTED: 0x05,
} as const
// --- debugging --------------------------------------------------------------
// Initiator opcodes (0x00-0x10) and target opcodes (0x20-0x3f) never collide,
// so one reverse map names both directions for logs.
const OPCODE_NAMES: Record<number, string> = {
...Object.fromEntries(Object.entries(InitiatorOpcode).map(([name, code]) => [code, name])),
...Object.fromEntries(Object.entries(TargetOpcode).map(([name, code]) => [code, name])),
}
/** Human-readable name of a PDU opcode (for debug logs). */
export function opcodeName(opcode: number): string {
const code = opcode & OPCODE_MASK
return OPCODE_NAMES[code] ?? `0x${code.toString(16).padStart(2, '0')}`
}

View File

@@ -0,0 +1,30 @@
// Ambient declarations for untyped external packages used by this module.
// (CLAUDE.md: "Add declaration.d.ts for untyped external packages.")
declare module '@vates/read-chunk' {
import type { Readable } from 'node:stream'
/**
* Resolves to exactly `size` bytes read from `stream`, or `null` if the
* stream ends before any byte of the requested chunk is available.
*/
export function readChunk(stream: Readable, size?: number): Promise<Buffer | null>
/**
* Resolves to exactly `size` bytes read from `stream`; rejects if the stream
* ends before `size` bytes have been read.
*/
export function readChunkStrict(stream: Readable, size: number): Promise<Buffer>
}
declare module 'promise-toolbox' {
/**
* Reject the promise it is bound to if it does not settle within `delay`
* milliseconds. Used as `pTimeout.call(promise, delay)`.
*/
interface PTimeout {
<T>(this: Promise<T>, delay: number, onReject?: () => void): Promise<T>
call<T>(promise: Promise<T>, delay: number, onReject?: () => void): Promise<T>
}
export const pTimeout: PTimeout
}

281
@vates/iscsi/src/index.mts Normal file
View File

@@ -0,0 +1,281 @@
import { createServer, type AddressInfo, type Server, type Socket } from 'node:net'
import { createLogger, type Logger } from '@xen-orchestra/log'
import type { BlockDevice } from './backend.mjs'
import { Connection, type ConnectionDeps } from './connection.mjs'
import { DEFAULT_MAX_TRANSFER_LENGTH, type ScsiIdentity } from './scsi.mjs'
import type { ChapCredentials } from './types.mjs'
export { FileBlockDevice } from './backend.mjs'
export type { BlockDevice, FileBlockDeviceOptions } from './backend.mjs'
export type { ScsiIdentity } from './scsi.mjs'
export type { ChapCredentials } from './types.mjs'
export { IscsiInitiator } from './initiator.mjs'
export type { IscsiInitiatorOptions } from './initiator.mjs'
export { IscsiDisk } from './IscsiDisk.mjs'
export type { IscsiDiskOptions } from './IscsiDisk.mjs'
export { DiskBlockDevice } from './DiskBlockDevice.mjs'
export type { DiskBlockDeviceOptions } from './DiskBlockDevice.mjs'
export { CachedDiskBlockDevice } from './CachedDiskBlockDevice.mjs'
export type { CachedDiskBlockDeviceOptions } from './CachedDiskBlockDevice.mjs'
const log: Logger = createLogger('vates:iscsi')
const DEFAULT_PORT = 3260
const DEFAULT_WRITE_TIMEOUT_MS = 30_000
const DEFAULT_CMD_WINDOW = 64
const DEFAULT_READ_CONCURRENCY = 16
const DEFAULT_LOGIN_TIMEOUT_MS = 15_000
const DEFAULT_MAX_PENDING_CONNECTIONS = 8
const DEFAULT_IDENTITY: ScsiIdentity = {
vendor: 'VATES',
product: 'ISCSI LUN',
revision: '0001',
serial: '',
}
export interface IscsiTargetOptions {
/** The target's iSCSI Qualified Name, e.g. `iqn.2024-01.tech.vates:lun0`. */
readonly iqn: string
/** The single LUN exposed by this target. */
readonly lun: BlockDevice
/** Listen address. Defaults to all interfaces. */
readonly host?: string
/** Listen port. Defaults to 3260. */
readonly port?: number
/** Overrides for the SCSI INQUIRY identity strings. */
readonly identity?: Partial<ScsiIdentity>
/** Drain timeout for outbound PDUs in ms (0 disables). Defaults to 30000. */
readonly writeTimeoutMs?: number
/** CmdSN command-window depth advertised to the initiator. Defaults to 64. */
readonly cmdWindow?: number
/**
* Max number of READ commands served concurrently — the command window lets
* the initiator keep many outstanding, but this caps how many `lun.read()`
* calls actually run at once regardless (e.g. so a deep queue depth doesn't
* fire dozens of simultaneous fetches against a real backend). Defaults to 16.
*/
readonly readConcurrency?: number
/**
* When set, require one-way CHAP: the target challenges each initiator and
* rejects the login unless it proves this credential (interop with the
* open-iscsi `node.session.auth` username/password). Omit for no authentication.
*/
readonly chap?: ChapCredentials
/**
* Time a new connection has to establish a Normal session, in ms (0
* disables). Also bounds Discovery sessions. Defaults to 15000.
*/
readonly loginTimeoutMs?: number
/**
* Max number of connections that have not established a Normal session yet;
* past it, new ones are refused. Defaults to 8.
*/
readonly maxPendingConnections?: number
/**
* Source addresses the initiators may connect from. When set, connections
* from any other address are closed as soon as they are accepted, before any
* login: only these initiators can take a pending connection slot.
* IPv4-mapped IPv6 addresses (`::ffff:a.b.c.d`) match their IPv4 form.
*/
readonly allowedInitiatorAddresses?: Iterable<string>
/**
* Called with the source address of each connection closed because it is
* not in `allowedInitiatorAddresses`, e.g. to tell why an initiator behind a
* NAT cannot connect.
*/
readonly onConnectionRefused?: (address: string) => void
/**
* Max length of a single READ or WRITE, in bytes, advertised in the block
* limits VPD page; longer commands are refused. At least one logical block.
* Defaults to 8 MiB.
*/
readonly maxTransferLength?: number
}
const IPV4_MAPPED_PREFIX = '::ffff:'
function normalizeAddress(address: string): string {
return address.startsWith(IPV4_MAPPED_PREFIX) && address.includes('.')
? address.slice(IPV4_MAPPED_PREFIX.length)
: address
}
/**
* A minimal iSCSI target exposing exactly one read/write LUN.
*
* Single initiator, single connection (`MaxConnections=1`),
* `ErrorRecoveryLevel=0`, no digests.
*
* A new connection replaces the current one rather than being refused (see
* `#onSessionEstablished`): there is no way to tell a stale, abandoned
* connection (initiator gave up without closing the socket) from a healthy
* one, so refusing would let one stuck connection wedge the target forever.
* The replacement only happens once the new connection has established a
* Normal session, CHAP included when configured, so that a port scan, a failed
* login or a Discovery session cannot interrupt the current one. Until then,
* connections are bounded in time (`loginTimeoutMs`) and in number
* (`maxPendingConnections`); with `allowedInitiatorAddresses`, only the expected
* initiators can take those slots. Not RFC 7143 session reinstatement (no ISID
* matching) — safe here since the target is ephemeral, CHAP-guarded, and
* single-consumer.
*/
export class IscsiTarget {
readonly #iqn: string
readonly #lun: BlockDevice
readonly #host?: string
readonly #port: number
readonly #identity: ScsiIdentity
readonly #writeTimeoutMs: number
readonly #cmdWindow: number
readonly #readConcurrency: number
readonly #chap?: ChapCredentials
readonly #loginTimeoutMs: number
readonly #maxPendingConnections: number
readonly #allowedInitiatorAddresses?: ReadonlySet<string>
readonly #onConnectionRefused?: (address: string) => void
readonly #maxTransferLength: number
#server?: Server
// the connection with an established Normal session, serving the LUN
#connection?: Connection
// every open connection, so that close() can drop them all
readonly #connections = new Set<Connection>()
#tsih = 0
constructor(options: IscsiTargetOptions) {
this.#iqn = options.iqn
this.#lun = options.lun
this.#host = options.host
this.#port = options.port ?? DEFAULT_PORT
this.#writeTimeoutMs = options.writeTimeoutMs ?? DEFAULT_WRITE_TIMEOUT_MS
this.#cmdWindow = options.cmdWindow ?? DEFAULT_CMD_WINDOW
this.#readConcurrency = options.readConcurrency ?? DEFAULT_READ_CONCURRENCY
this.#chap = options.chap
this.#loginTimeoutMs = options.loginTimeoutMs ?? DEFAULT_LOGIN_TIMEOUT_MS
this.#maxPendingConnections = options.maxPendingConnections ?? DEFAULT_MAX_PENDING_CONNECTIONS
if (options.allowedInitiatorAddresses !== undefined) {
this.#allowedInitiatorAddresses = new Set(Array.from(options.allowedInitiatorAddresses, normalizeAddress))
}
this.#onConnectionRefused = options.onConnectionRefused
this.#maxTransferLength = options.maxTransferLength ?? DEFAULT_MAX_TRANSFER_LENGTH
if (this.#maxTransferLength < options.lun.getBlockSize()) {
throw new RangeError('maxTransferLength must be at least one logical block')
}
this.#identity = {
...DEFAULT_IDENTITY,
// Default the serial to the IQN so the LUN has a stable, unique identity.
serial: options.iqn,
...options.identity,
}
}
#allocateTsih(): number {
this.#tsih = (this.#tsih % 0xffff) + 1 // non-zero, wraps within 16 bits
return this.#tsih
}
#deps(): ConnectionDeps {
return {
iqn: this.#iqn,
identity: this.#identity,
lun: this.#lun,
writeTimeoutMs: this.#writeTimeoutMs,
cmdWindow: this.#cmdWindow,
readConcurrency: this.#readConcurrency,
allocateTsih: () => this.#allocateTsih(),
chap: this.#chap,
loginTimeoutMs: this.#loginTimeoutMs,
onSessionEstablished: connection => this.#onSessionEstablished(connection),
maxTransferLength: this.#maxTransferLength,
}
}
#onConnection(socket: Socket): void {
const allowed = this.#allowedInitiatorAddresses
const remote = normalizeAddress(socket.remoteAddress ?? '')
if (allowed !== undefined && !allowed.has(remote)) {
log.warn('connection from an address that is not allowed, closing it', { remote })
socket.destroy()
this.#onConnectionRefused?.(remote)
return
}
socket.setNoDelay(true)
const connection = new Connection(socket, this.#deps())
this.#connections.add(connection)
// Long-lived per-connection task; serve() handles its own errors and never
// rejects, so detaching it here is safe.
void connection.serve().finally(() => {
this.#connections.delete(connection)
if (this.#connection === connection) {
this.#connection = undefined
}
})
}
#onSessionEstablished(connection: Connection): void {
const existing = this.#connection
this.#connection = connection
if (existing !== undefined) {
// Replaces a possibly-stale connection rather than refusing; see the class doc.
log.warn('replacing existing connection with a newly logged in one')
existing.destroy()
}
}
/** Open the LUN (if needed) and start accepting connections. */
async listen(): Promise<void> {
if (this.#server !== undefined) {
throw new Error('iSCSI target is already listening')
}
await this.#lun.open?.()
const server = createServer(socket => this.#onConnection(socket))
// + 1: the connection serving the LUN does not take a pending slot
server.maxConnections = this.#maxPendingConnections + 1
server.on('drop', data => {
log.warn('too many connections without a session, refusing a new one', { remote: data?.remoteAddress })
})
this.#server = server
try {
await new Promise<void>((resolve, reject) => {
const onError = (error: Error) => reject(error)
server.once('error', onError)
server.listen(this.#port, this.#host, () => {
server.removeListener('error', onError)
resolve()
})
})
} catch (error) {
this.#server = undefined
throw error
}
log.info('target listening', { iqn: this.#iqn, address: this.address() })
}
/** The bound address, or undefined if not listening on an IP socket. */
address(): AddressInfo | undefined {
const address = this.#server?.address()
return address !== null && typeof address === 'object' ? address : undefined
}
/** Stop accepting connections, drop every connection, and close the LUN. */
async close(): Promise<void> {
const server = this.#server
const connections = [...this.#connections]
this.#server = undefined
this.#connection = undefined
this.#connections.clear()
if (server !== undefined) {
await new Promise<void>((resolve, reject) => {
server.close(error => (error ? reject(error) : resolve()))
// Drop the live connections so server.close() can complete.
for (const connection of connections) {
connection.destroy()
}
})
}
await this.#lun.close()
}
}

View File

@@ -0,0 +1,261 @@
import assert from 'node:assert/strict'
import { createServer, type Server, type Socket } from 'node:net'
import { once } from 'node:events'
import { describe, it } from 'node:test'
import {
DATA_IN_FLAG_FINAL,
DATA_IN_FLAG_STATUS,
FLAG_FINAL,
InitiatorOpcode,
ISCSI_VERSION,
LOGIN_FLAG_TRANSIT,
LoginStage,
RESERVED_TAG,
ScsiStatus,
TargetOpcode,
} from './constants.mjs'
import { IscsiInitiator } from './initiator.mjs'
import { allocBhs, assemblePdu, type IncomingPdu, readPdu } from './pdu.mjs'
const IQN = 'iqn.2024-01.tech.vates:nop-test'
const BLOCK_SIZE = 512
const BLOCK_COUNT = 2048
/**
* A target that speaks just enough to get a session up — login, READ
* CAPACITY(16), READ(16) — and can then ping the initiator on demand, which
* the real {@link IscsiTarget} never does. Everything here is about what the
* *initiator* does with an unsolicited NOP-In.
*/
class PingingTarget {
readonly #server: Server
#socket?: Socket
/** PDUs received from the initiator after the session was established. */
readonly received: Array<IncomingPdu> = []
#statSN = 0
constructor() {
this.#server = createServer(socket => {
this.#socket = socket
socket.setNoDelay(true)
this.#serve(socket).catch(() => {}) // the initiator hanging up is normal
})
}
async listen(): Promise<number> {
this.#server.listen(0, '127.0.0.1')
await once(this.#server, 'listening')
return (this.#server.address() as { port: number }).port
}
async close(): Promise<void> {
this.#socket?.destroy()
this.#server.close()
}
#send(socket: Socket, buffer: Buffer): Promise<void> {
return new Promise((resolve, reject) => {
socket.write(buffer, error => (error ? reject(error) : resolve()))
})
}
/** Ping the initiator the way a target does when the session goes quiet. */
async ping(targetTransferTag: number, data = Buffer.alloc(0)): Promise<void> {
const socket = this.#socket
assert.ok(socket !== undefined, 'no initiator connected')
const bhs = allocBhs(TargetOpcode.NOP_IN)
bhs[1] = FLAG_FINAL
bhs.writeUInt32BE(RESERVED_TAG, 16) // ITT: unsolicited, so none
bhs.writeUInt32BE(targetTransferTag, 20)
bhs.writeUInt32BE(this.#statSN, 24) // NOP-In does not advance StatSN
await this.#send(socket, assemblePdu(bhs, data))
}
/** Resolves once the initiator has sent a PDU with `opcode`. */
async waitFor(opcode: number, timeoutMs = 2000): Promise<IncomingPdu> {
const deadline = Date.now() + timeoutMs
for (;;) {
const found = this.received.find(pdu => pdu.opcode === opcode)
if (found !== undefined) {
return found
}
assert.ok(Date.now() < deadline, `initiator never sent opcode 0x${opcode.toString(16)}`)
await new Promise(resolve => setTimeout(resolve, 10))
}
}
async #serve(socket: Socket): Promise<void> {
let established = false
for (;;) {
const pdu = await readPdu(socket)
if (pdu === null) {
return
}
if (established) {
this.received.push(pdu)
}
switch (pdu.opcode) {
case InitiatorOpcode.LOGIN_REQUEST:
await this.#sendLoginResponse(socket, pdu)
established = true
break
case InitiatorOpcode.SCSI_COMMAND:
await this.#sendReadData(socket, pdu)
break
case InitiatorOpcode.LOGOUT_REQUEST: {
const bhs = allocBhs(TargetOpcode.LOGOUT_RESPONSE)
bhs[1] = FLAG_FINAL
bhs.writeUInt32BE(pdu.itt, 16)
bhs.writeUInt32BE(this.#statSN++, 24)
await this.#send(socket, assemblePdu(bhs))
return
}
default: // NOP-Out replies land in `received`; nothing to answer
break
}
}
}
async #sendLoginResponse(socket: Socket, request: IncomingPdu): Promise<void> {
const bhs = allocBhs(TargetOpcode.LOGIN_RESPONSE)
bhs[1] = LOGIN_FLAG_TRANSIT | (LoginStage.OPERATIONAL_NEGOTIATION << 2) | LoginStage.FULL_FEATURE_PHASE
bhs[2] = ISCSI_VERSION
bhs[3] = ISCSI_VERSION
request.bhs.copy(bhs, 8, 8, 14) // echo ISID
bhs.writeUInt16BE(1, 14) // TSIH
bhs.writeUInt32BE(request.itt, 16)
bhs.writeUInt32BE(this.#statSN++, 24)
await this.#send(socket, assemblePdu(bhs))
}
/** Answer any read-type command with a single Data-In carrying GOOD status. */
async #sendReadData(socket: Socket, command: IncomingPdu): Promise<void> {
const cdb = command.bhs.subarray(32, 48)
const expectedLength = command.readU32(20)
let payload
if (cdb[0] === 0x9e) {
// SERVICE ACTION IN(16) / READ CAPACITY(16)
payload = Buffer.alloc(expectedLength)
payload.writeBigUInt64BE(BigInt(BLOCK_COUNT - 1), 0)
payload.writeUInt32BE(BLOCK_SIZE, 8)
} else {
payload = Buffer.alloc(expectedLength, 0x5a)
}
const bhs = allocBhs(TargetOpcode.SCSI_DATA_IN)
bhs[1] = DATA_IN_FLAG_FINAL | DATA_IN_FLAG_STATUS
bhs[3] = ScsiStatus.GOOD
bhs.writeUInt32BE(command.itt, 16)
bhs.writeUInt32BE(RESERVED_TAG, 20)
bhs.writeUInt32BE(this.#statSN++, 24)
await this.#send(socket, assemblePdu(bhs, payload))
}
}
/** Run `body` against a connected initiator, always tearing both ends down. */
const withSession = async (body: (initiator: IscsiInitiator, target: PingingTarget) => Promise<void>) => {
const target = new PingingTarget()
const port = await target.listen()
const initiator = new IscsiInitiator({
host: '127.0.0.1',
port,
targetIqn: IQN,
messageTimeoutMs: 5000,
})
await initiator.connect()
try {
await body(initiator, target)
} finally {
await initiator.close()
await target.close()
}
}
describe('NOP-In keepalives', () => {
// The read loop used to stop as soon as no read was outstanding, and it is
// the only reader of the socket — so a ping arriving while idle sat unread in
// a paused stream. Targets drop a session whose pings go unanswered (30s on
// LIO by default), and nothing here reconnects, so an idle mount died.
it('are answered while the session is idle', async () => {
await withSession(async (initiator, target) => {
// no read in flight, and none will be
await target.ping(7)
const reply = await target.waitFor(InitiatorOpcode.NOP_OUT)
assert.equal(reply.readU32(20), 7) // echoes the Target Transfer Tag
assert.equal(reply.itt, RESERVED_TAG) // unsolicited reply, so no ITT
assert.notEqual(reply.bhs[0] & 0x40, 0) // immediate
})
})
it('echo the ping payload back', async () => {
await withSession(async (initiator, target) => {
const ping = Buffer.from('are you there')
await target.ping(9, ping)
const reply = await target.waitFor(InitiatorOpcode.NOP_OUT)
assert.deepEqual(reply.data, ping)
})
})
it('are answered between reads, without disturbing them', async () => {
await withSession(async (initiator, target) => {
assert.equal((await initiator.read(0, BLOCK_SIZE)).length, BLOCK_SIZE)
await target.ping(11)
await target.waitFor(InitiatorOpcode.NOP_OUT)
// the session is still usable afterwards
const data = await initiator.read(BLOCK_SIZE, BLOCK_SIZE)
assert.deepEqual(data, Buffer.alloc(BLOCK_SIZE, 0x5a))
})
})
it('are ignored when they are a reply to our own ping', async () => {
await withSession(async (initiator, target) => {
// a NOP-In with no Target Transfer Tag is a reply, not a ping: answering
// it would start an endless exchange
await target.ping(RESERVED_TAG)
await new Promise(resolve => setTimeout(resolve, 200))
assert.equal(
target.received.some(pdu => pdu.opcode === InitiatorOpcode.NOP_OUT),
false
)
// and the session still works
assert.equal((await initiator.read(0, BLOCK_SIZE)).length, BLOCK_SIZE)
})
})
})
describe('session teardown', () => {
it('logs out cleanly even though the read loop owns the socket', async () => {
const target = new PingingTarget()
const port = await target.listen()
const initiator = new IscsiInitiator({ host: '127.0.0.1', port, targetIqn: IQN, messageTimeoutMs: 5000 })
await initiator.connect()
await initiator.close()
// the Logout Request reached the target, i.e. `close()` did not give up on
// its own response for lack of a reader
await target.waitFor(InitiatorOpcode.LOGOUT_REQUEST)
await target.close()
})
it('fails a read with the reason the session died, rather than timing out', async () => {
const target = new PingingTarget()
const port = await target.listen()
const initiator = new IscsiInitiator({ host: '127.0.0.1', port, targetIqn: IQN, messageTimeoutMs: 5000 })
await initiator.connect()
await target.close() // target vanishes while the initiator is idle
await new Promise(resolve => setTimeout(resolve, 100))
const started = Date.now()
await assert.rejects(initiator.read(0, BLOCK_SIZE), /connection closed/)
// far below messageTimeoutMs: the dead session is known, not discovered
assert.ok(Date.now() - started < 1000)
})
})

View File

@@ -0,0 +1,501 @@
import { connect, type Socket } from 'node:net'
import { once } from 'node:events'
import { createLogger, type Logger } from '@xen-orchestra/log'
import { pTimeout } from 'promise-toolbox'
import {
AuthMethod,
DATA_IN_FLAG_STATUS,
FLAG_FINAL,
InitiatorOpcode,
LOGIN_CSG_SHIFT,
LOGIN_FLAG_TRANSIT,
LOGIN_NSG_MASK,
LoginStage,
LoginStatusClass,
OPCODE_IMMEDIATE,
RESERVED_TAG,
SCSI_CMD_FLAG_READ,
ScsiOpcode,
ScsiStatus,
SERVICE_ACTION_READ_CAPACITY_16,
TargetOpcode,
} from './constants.mjs'
import { CHAP_ALGORITHM_MD5, computeResponse, decodeChapValue, encodeChapValue } from './chap.mjs'
import { parseTextKeys, serializeTextKeys } from './login.mjs'
import { allocBhs, assemblePdu, type IncomingPdu, readPdu, writePdu } from './pdu.mjs'
import type { ChapCredentials } from './types.mjs'
const log: Logger = createLogger('vates:iscsi:initiator')
const DEFAULT_PORT = 3260
const DEFAULT_MESSAGE_TIMEOUT_MS = 60_000
const DEFAULT_MAX_RECV_DATA_SEGMENT_LENGTH = 262144
const READ_CAPACITY_16_LENGTH = 32
export interface IscsiInitiatorOptions {
/** Portal address to connect to. */
readonly host: string
/** Portal port. Defaults to 3260. */
readonly port?: number
/** Target IQN (`TargetName`) to log into. */
readonly targetIqn: string
/** Our initiator IQN (`InitiatorName`). Defaults to a generated one. */
readonly initiatorIqn?: string
/**
* One-way CHAP responder credential. When the target requests CHAP we answer
* its challenge with this user/secret. Omit for an unauthenticated login.
*/
readonly chap?: ChapCredentials
/** Per-message timeout for login/read responses, in ms. Defaults to 60000. */
readonly messageTimeoutMs?: number
}
/** An outstanding SCSI read, keyed by its Initiator Task Tag. */
interface Pending {
/** Preallocated buffer sized to the expected transfer length. */
readonly buffer: Buffer
resolve(data: Buffer): void
reject(error: Error): void
}
/**
* A minimal userspace iSCSI initiator (client), mirroring {@link IscsiTarget}'s
* framing and negotiation (read-only, single connection, one LUN, digests
* off, `ErrorRecoveryLevel=0`).
*
* A backlog `Map` keyed by ITT, drained by a single read loop, demultiplexes
* responses so several reads can be outstanding at once (e.g. under a
* `ReadAhead` disk wrapper).
*/
export class IscsiInitiator {
readonly #host: string
readonly #port: number
readonly #targetIqn: string
readonly #initiatorIqn: string
readonly #chap?: ChapCredentials
readonly #messageTimeoutMs: number
#socket?: Socket
#itt = 0
#cmdSN = 0
#expStatSN = 0
#blockSize = 0
#capacityBytes = 0
// Outstanding reads, keyed by ITT. A single #readLoop() (guarded by #reading)
// is the only consumer of readPdu, so it is safe to have many entries here.
readonly #backlog = new Map<number, Pending>()
#reading = false
// set once `close()` starts, so the read loop's own end is not reported as a
// fault and it stops instead of racing the socket teardown
#closing = false
// why the read loop stopped, when it was not us closing
#failure?: Error
// resolved by the read loop when the target's Logout Response arrives
#logoutAnswered?: () => void
constructor(options: IscsiInitiatorOptions) {
this.#host = options.host
this.#port = options.port ?? DEFAULT_PORT
this.#targetIqn = options.targetIqn
this.#initiatorIqn = options.initiatorIqn ?? 'iqn.2024-01.tech.vates:initiator'
this.#chap = options.chap
this.#messageTimeoutMs = options.messageTimeoutMs ?? DEFAULT_MESSAGE_TIMEOUT_MS
}
/** LUN logical block size in bytes (valid after {@link connect}). */
getBlockSize(): number {
return this.#blockSize
}
/** LUN capacity in bytes (valid after {@link connect}). */
getSize(): number {
return this.#capacityBytes
}
// --- lifecycle ------------------------------------------------------------
/** Open the TCP connection, log in, and read the LUN capacity. */
async connect(): Promise<void> {
const socket = connect(this.#port, this.#host)
this.#socket = socket
try {
await once(socket, 'connect')
socket.setNoDelay(true)
// A socket error with no read in flight would otherwise go unnoticed.
socket.on('error', error => this.#rejectAll(error instanceof Error ? error : new Error(String(error))))
// `#login` reads its own responses; from here on the read loop owns the
// socket, and it must be running before anything can answer a NOP-In
await this.#login()
void this.#pump()
await this.#readCapacity()
log.info('session established', { target: this.#targetIqn, capacity: this.#capacityBytes })
} catch (error) {
socket.destroy()
this.#socket = undefined
throw error
}
}
#requireSocket(): Socket {
const socket = this.#socket
if (socket === undefined) {
throw new Error('IscsiInitiator.connect() must be called first')
}
return socket
}
#send(buffer: Buffer): Promise<void> {
return writePdu(this.#requireSocket(), buffer, this.#messageTimeoutMs)
}
#nextItt(): number {
this.#itt = (this.#itt + 1) >>> 0
return this.#itt
}
/** Best-effort Logout, then close the socket. */
async close(): Promise<void> {
const socket = this.#socket
if (socket === undefined) {
return
}
// stops the read loop and keeps its own end from being logged as a fault
this.#closing = true
try {
const bhs = allocBhs(InitiatorOpcode.LOGOUT_REQUEST | OPCODE_IMMEDIATE)
bhs[1] = FLAG_FINAL // reason 0: close the session
bhs.writeUInt32BE(this.#nextItt(), 16)
bhs.writeUInt32BE(this.#cmdSN, 24)
bhs.writeUInt32BE(this.#expStatSN, 28)
// the read loop owns the socket, so the response comes through it: reading
// here too would put two consumers on one stream and split PDUs between them
const answered = new Promise<void>(resolve => {
this.#logoutAnswered = resolve
})
await this.#send(assemblePdu(bhs))
await pTimeout.call(answered, this.#messageTimeoutMs)
} catch (error) {
log.debug('logout failed, closing anyway', { error })
} finally {
this.#logoutAnswered = undefined
this.#socket = undefined
socket.destroy()
}
}
// --- login ----------------------------------------------------------------
/** Build the flags byte (T/CSG/NSG) of a Login Request. */
#loginFlags(transit: boolean, csg: number, nsg: number): number {
return (transit ? LOGIN_FLAG_TRANSIT : 0) | (csg << LOGIN_CSG_SHIFT) | (nsg & LOGIN_NSG_MASK)
}
async #sendLogin(itt: number, flags: number, keys: Array<[string, string]>): Promise<IncomingPdu> {
const bhs = allocBhs(InitiatorOpcode.LOGIN_REQUEST | OPCODE_IMMEDIATE)
bhs[1] = flags
Buffer.from([0x80, 0, 0, 0, 0, 1]).copy(bhs, 8) // ISID (fixed; single session)
bhs.writeUInt32BE(itt, 16)
bhs.writeUInt32BE(this.#cmdSN, 24) // login is immediate: does not advance CmdSN
bhs.writeUInt32BE(this.#expStatSN, 28)
log.debug('login request', { flags: `0x${flags.toString(16)}`, keys: Object.fromEntries(keys) })
await this.#send(assemblePdu(bhs, serializeTextKeys(keys)))
const pdu = await pTimeout.call(readPdu(this.#requireSocket()), this.#messageTimeoutMs)
if (pdu === null) {
throw new Error('connection closed during login')
}
if (pdu.opcode !== TargetOpcode.LOGIN_RESPONSE) {
throw new Error(`expected Login Response, got opcode 0x${pdu.opcode.toString(16)}`)
}
const statusClass = pdu.readU8(36)
log.debug('login response', {
statusClass,
statusDetail: pdu.readU8(37),
transit: (pdu.readU8(1) & LOGIN_FLAG_TRANSIT) !== 0,
keys: Object.fromEntries(parseTextKeys(pdu.data)),
})
if (statusClass !== LoginStatusClass.SUCCESS) {
throw new Error(
`login rejected (status-class 0x${statusClass.toString(16)}, detail 0x${pdu.readU8(37).toString(16)})`
)
}
this.#expStatSN = (pdu.readU32(24) + 1) >>> 0
return pdu
}
/**
* Drive the login: an optional CHAP security stage (we are the responder),
* then the operational stage into the full-feature phase. All PDUs of one
* login share a single ITT.
*/
async #login(): Promise<void> {
const itt = this.#nextItt()
const names: Array<[string, string]> = [
['InitiatorName', this.#initiatorIqn],
['TargetName', this.#targetIqn],
['SessionType', 'Normal'],
]
if (this.#chap !== undefined) {
// Round 1: offer CHAP (falling back to None) in the security stage.
const r1 = await this.#sendLogin(
itt,
this.#loginFlags(false, LoginStage.SECURITY_NEGOTIATION, LoginStage.OPERATIONAL_NEGOTIATION),
[...names, ['AuthMethod', `${AuthMethod.CHAP},${AuthMethod.NONE}`]]
)
const method = parseTextKeys(r1.data).get('AuthMethod')
log.debug('chap: target selected AuthMethod', { method })
if (method === AuthMethod.CHAP) {
// Round 2: propose MD5.
const r2 = await this.#sendLogin(
itt,
this.#loginFlags(false, LoginStage.SECURITY_NEGOTIATION, LoginStage.OPERATIONAL_NEGOTIATION),
[['CHAP_A', String(CHAP_ALGORITHM_MD5)]]
)
// Round 3: answer the target's challenge, transiting to operational.
// (Mutual CHAP is not supported: we never challenge the target; a target
// that *requires* mutual auth will reject this login, surfaced above.)
const challenge = parseTextKeys(r2.data)
const id = Number.parseInt(challenge.get('CHAP_I') ?? '', 10)
const challengeValue = challenge.get('CHAP_C')
if (!Number.isInteger(id) || challengeValue === undefined) {
throw new Error('malformed CHAP challenge from target')
}
const response = computeResponse(id, this.#chap.secret, decodeChapValue(challengeValue))
log.debug('chap: answering challenge', { id, user: this.#chap.user })
await this.#sendLogin(
itt,
this.#loginFlags(true, LoginStage.SECURITY_NEGOTIATION, LoginStage.OPERATIONAL_NEGOTIATION),
[
['CHAP_N', this.#chap.user],
['CHAP_R', encodeChapValue(response)],
]
)
await this.#operationalLogin(itt, [])
return
}
if (method !== AuthMethod.NONE && method !== undefined) {
throw new Error(`target selected unsupported AuthMethod=${method}`)
}
// Target did not require CHAP: continue straight to operational.
await this.#operationalLogin(itt, [])
return
}
// No CHAP: a single operational-stage PDU straight to the full-feature phase.
await this.#operationalLogin(itt, names)
}
/** Operational-stage login PDU, transiting to the full-feature phase. */
async #operationalLogin(itt: number, extraKeys: Array<[string, string]>): Promise<void> {
const pdu = await this.#sendLogin(
itt,
this.#loginFlags(true, LoginStage.OPERATIONAL_NEGOTIATION, LoginStage.FULL_FEATURE_PHASE),
[
...extraKeys,
['HeaderDigest', 'None'],
['DataDigest', 'None'],
['InitialR2T', 'Yes'],
['ImmediateData', 'No'],
['ErrorRecoveryLevel', '0'],
['MaxRecvDataSegmentLength', String(DEFAULT_MAX_RECV_DATA_SEGMENT_LENGTH)],
]
)
if ((pdu.readU8(1) & LOGIN_FLAG_TRANSIT) === 0) {
throw new Error('target did not transit to the full-feature phase')
}
}
// --- read path ------------------------------------------------------------
/**
* Read exactly `length` bytes from byte `offset` of the LUN. `offset` and
* `length` must be block-aligned (multiples of {@link getBlockSize}).
*/
read(offset: number, length: number): Promise<Buffer> {
const blockSize = this.#blockSize
if (offset % blockSize !== 0 || length % blockSize !== 0) {
return Promise.reject(new Error(`unaligned read (offset ${offset}, length ${length}, block ${blockSize})`))
}
const cdb = Buffer.alloc(16)
cdb[0] = ScsiOpcode.READ_16
cdb.writeBigUInt64BE(BigInt(offset / blockSize), 2) // LBA
cdb.writeUInt32BE(length / blockSize, 10) // transfer length, in blocks
return this.#scsiRead(cdb, length)
}
/** Issue READ CAPACITY(16) and record the LUN block size and capacity. */
async #readCapacity(): Promise<void> {
const cdb = Buffer.alloc(16)
cdb[0] = ScsiOpcode.SERVICE_ACTION_IN_16
cdb[1] = SERVICE_ACTION_READ_CAPACITY_16
cdb.writeUInt32BE(READ_CAPACITY_16_LENGTH, 10) // allocation length
// Block size is unknown here, so bypass the alignment guard in read().
const data = await this.#scsiRead(cdb, READ_CAPACITY_16_LENGTH)
const maxLba = data.readBigUInt64BE(0)
this.#blockSize = data.readUInt32BE(8)
this.#capacityBytes = Number((maxLba + 1n) * BigInt(this.#blockSize))
log.debug('read capacity', { blockSize: this.#blockSize, capacityBytes: this.#capacityBytes })
}
/** Issue a read-type SCSI command and resolve with its assembled Data-In. */
#scsiRead(cdb: Buffer, expectedLength: number): Promise<Buffer> {
const failure = this.#failure
if (failure !== undefined) {
// the session is already gone: fail with why, rather than writing into a
// half-closed socket and waiting out the message timeout
return Promise.reject(failure)
}
const socket = this.#requireSocket()
const itt = this.#nextItt()
const bhs = allocBhs(InitiatorOpcode.SCSI_COMMAND)
bhs[1] = FLAG_FINAL | SCSI_CMD_FLAG_READ
bhs.writeUInt32BE(itt, 16)
bhs.writeUInt32BE(expectedLength, 20) // Expected Data Transfer Length
bhs.writeUInt32BE(this.#cmdSN, 24)
this.#cmdSN = (this.#cmdSN + 1) >>> 0
bhs.writeUInt32BE(this.#expStatSN, 28)
cdb.copy(bhs, 32)
log.debug('scsi read', { itt, cdb: `0x${cdb[0].toString(16)}`, expectedLength, cmdSN: (this.#cmdSN - 1) >>> 0 })
// Register the pending read BEFORE writing, so a fast response cannot arrive
// before the backlog entry exists. No await between allocate → register →
// send → pump, which keeps concurrent reads correct.
const promise = new Promise<Buffer>((resolve, reject) => {
this.#backlog.set(itt, { buffer: Buffer.alloc(expectedLength), resolve, reject })
})
writePdu(socket, assemblePdu(bhs), this.#messageTimeoutMs).catch(error => {
const pending = this.#backlog.get(itt)
if (pending !== undefined) {
this.#backlog.delete(itt)
pending.reject(error instanceof Error ? error : new Error(String(error)))
}
})
void this.#pump()
return pTimeout.call(promise, this.#messageTimeoutMs)
}
/**
* Drain inbound PDUs for as long as the session lives. Only ever one runs at
* a time, and it is the only reader of the socket after login.
*
* It deliberately does not stop when the backlog empties. A target sends its
* NOP-In keepalives precisely when nothing else is going on, and it is the
* only reader — so a loop that parked between reads would leave those pings
* sitting unread in a paused stream. Targets drop a session whose pings go
* unanswered (LIO's `nopin_response_timeout` is 30s by default, arrays are
* similar), and nothing here reconnects, so the session would die during any
* idle period: right after `connect()`, between reads, or while the consumer
* of a mounted disk stalls.
*/
async #pump(): Promise<void> {
if (this.#reading) {
return
}
this.#reading = true
try {
while (!this.#closing) {
const socket = this.#socket
if (socket === undefined) {
throw new Error('connection closed')
}
const pdu = await readPdu(socket)
if (pdu === null) {
throw new Error('connection closed by target')
}
this.#route(pdu)
}
} catch (error) {
const failure = error instanceof Error ? error : new Error(String(error))
if (!this.#closing) {
// The loop now runs while idle, so it is usually the first to notice the
// session die — with nothing in the backlog to reject and nobody to
// return an error to. Log it where it happens, and keep the reason so
// later reads fail with the original cause instead of whatever the dead
// socket produces.
this.#failure = failure
log.warn('read loop stopped, session is dead', { error: failure })
this.#rejectAll(failure)
}
} finally {
this.#reading = false
}
}
/** Route one inbound PDU to its pending read (by ITT), completing it if done. */
#route(pdu: IncomingPdu): void {
switch (pdu.opcode) {
case TargetOpcode.SCSI_DATA_IN: {
const pending = this.#backlog.get(pdu.itt)
if (pending === undefined) {
return // stale/duplicate; ignore
}
pdu.data.copy(pending.buffer, pdu.readU32(40)) // copy at Buffer Offset
const hasStatus = (pdu.flags & DATA_IN_FLAG_STATUS) !== 0
log.debug('data-in', { itt: pdu.itt, offset: pdu.readU32(40), length: pdu.data.length, final: hasStatus })
if (hasStatus) {
this.#expStatSN = (pdu.readU32(24) + 1) >>> 0
this.#complete(pdu.itt, pending, pdu.readU8(3))
}
return
}
case TargetOpcode.SCSI_RESPONSE: {
const pending = this.#backlog.get(pdu.itt)
if (pending !== undefined) {
this.#expStatSN = (pdu.readU32(24) + 1) >>> 0
this.#complete(pdu.itt, pending, pdu.readU8(3))
}
return
}
case TargetOpcode.NOP_IN: {
// A target-initiated NOP ping (TTT set) must be answered to keep the
// session alive; a NOP reply to our own ping (none here) is ignored.
const ttt = pdu.readU32(20)
if (ttt !== RESERVED_TAG) {
void this.#replyNop(ttt, pdu.data)
}
return
}
case TargetOpcode.LOGOUT_RESPONSE:
// awaited by `close()`, which cannot read the socket itself
this.#logoutAnswered?.()
return
default:
// R2T/Async/Reject are not expected on the read-only path; ignore.
log.debug('ignoring unexpected PDU', { opcode: pdu.opcode })
}
}
#complete(itt: number, pending: Pending, status: number): void {
this.#backlog.delete(itt)
log.debug('read complete', { itt, status, pending: this.#backlog.size })
if (status === ScsiStatus.GOOD) {
pending.resolve(pending.buffer)
} else {
pending.reject(new Error(`SCSI command failed with status 0x${status.toString(16)}`))
}
}
async #replyNop(targetTransferTag: number, ping: Buffer): Promise<void> {
const bhs = allocBhs(InitiatorOpcode.NOP_OUT | OPCODE_IMMEDIATE)
bhs[1] = FLAG_FINAL
bhs.writeUInt32BE(RESERVED_TAG, 16) // ITT: none (this is an unsolicited reply)
bhs.writeUInt32BE(targetTransferTag, 20)
bhs.writeUInt32BE(this.#cmdSN, 24)
bhs.writeUInt32BE(this.#expStatSN, 28)
await this.#send(assemblePdu(bhs, ping)).catch(error => log.debug('NOP reply failed', { error }))
}
#rejectAll(error: Error): void {
if (this.#backlog.size > 0) {
log.debug('rejecting all pending reads', { count: this.#backlog.size, error: error.message })
}
for (const [itt, pending] of this.#backlog) {
this.#backlog.delete(itt)
pending.reject(error)
}
}
}

372
@vates/iscsi/src/login.mts Normal file
View File

@@ -0,0 +1,372 @@
import { isIP } from 'node:net'
import { createLogger, type Logger } from '@xen-orchestra/log'
import {
AuthMethod,
LOGIN_CSG_MASK,
LOGIN_CSG_SHIFT,
LOGIN_FLAG_CONTINUE,
LOGIN_FLAG_TRANSIT,
LOGIN_NSG_MASK,
LoginStage,
LoginStatusClass,
LoginStatusDetail,
} from './constants.mjs'
import {
computeResponse,
encodeChapValue,
decodeChapValue,
generateChallenge,
generateId,
parseAlgorithmList,
selectAlgorithm,
verifyResponse,
} from './chap.mjs'
import type { IncomingPdu } from './pdu.mjs'
import type { ChapCredentials, NegotiatedParams, SessionType } from './types.mjs'
const EMPTY = Buffer.alloc(0)
// Target-side login negotiation trace. Enable the `vates:iscsi:login` debug
// namespace to see every negotiated key and CHAP state change — the usual root
// cause when a particular initiator fails to attach.
const log: Logger = createLogger('vates:iscsi:login')
// Operational values the target imposes. The target drives negotiation, so these
// are pinned to a single code path (see plan): all writes are R2T-solicited
// (InitialR2T=Yes, ImmediateData=No), no digests, one connection.
export const OUR_MAX_RECV_DATA_SEGMENT_LENGTH = 262144
const OUR_MAX_BURST_LENGTH = 262144
const DEFAULT_INITIATOR_MAX_RECV_DATA_SEGMENT_LENGTH = 8192 // RFC 7143 default
// a declared MaxRecvDataSegmentLength only applies once logged in: until then,
// both sides keep to the 8192-byte default (RFC 7143 §13.12)
export const LOGIN_MAX_DATA_SEGMENT_LENGTH = 8192
/** Parse a NUL-delimited iSCSI text key=value data segment. */
export function parseTextKeys(data: Buffer): Map<string, string> {
const keys = new Map<string, string>()
for (const pair of data.toString('utf8').split('\0')) {
if (pair.length === 0) {
continue
}
const eq = pair.indexOf('=')
if (eq !== -1) {
keys.set(pair.slice(0, eq), pair.slice(eq + 1))
}
}
return keys
}
/** Serialize key=value pairs into a NUL-delimited (and NUL-terminated) buffer. */
export function serializeTextKeys(entries: ReadonlyArray<readonly [string, string]>): Buffer {
if (entries.length === 0) {
return Buffer.alloc(0)
}
return Buffer.from(entries.map(([key, value]) => `${key}=${value}\0`).join(''), 'utf8')
}
function parsePositiveInt(value: string | undefined, fallback: number): number {
if (value === undefined) {
return fallback
}
const parsed = Number.parseInt(value, 10)
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback
}
/**
* Format an `address:port,portal-group-tag` value for a SendTargets
* `TargetAddress`, per the iSCSI grammar:
* - unwrap IPv4-mapped IPv6 (`::ffff:1.2.3.4`, produced by a dual-stack socket
* accepting an IPv4 connection) to plain IPv4, so initiators get a clean,
* matchable portal;
* - bracket real IPv6 literals (`[2001:db8::1]:3260,1`).
*/
export function formatPortal(address: string, port: number): string {
const mapped = /^::ffff:(\d{1,3}(?:\.\d{1,3}){3})$/i.exec(address)
const normalized = mapped !== null ? mapped[1] : address
const bracketed = isIP(normalized) === 6 ? `[${normalized}]` : normalized
return `${bracketed}:${port},1`
}
/** Build the SendTargets discovery Text Response data segment. */
export function buildSendTargetsResponse(iqn: string, address: string, port: number): Buffer {
return serializeTextKeys([
['TargetName', iqn],
['TargetAddress', formatPortal(address, port)],
])
}
/** Outcome of processing a single Login Request PDU. */
export interface LoginStepResult {
/** Byte 1 (T/C/CSG/NSG) of the Login Response. */
flagsByte: number
/** Response key=value text segment. */
data: Buffer
/** Login Response Status-Class (byte 36). Defaults to SUCCESS when omitted. */
statusClass?: number
/** Login Response Status-Detail (byte 37). Defaults to 0 when omitted. */
statusDetail?: number
}
/** A rejected login: the Status-Class/Detail bytes the response must carry. */
interface AuthFailure {
statusClass: number
statusDetail: number
}
/** CHAP authentication failure (RFC 7143 §11.1.5): Initiator Error / auth failure. */
const AUTH_FAILURE: AuthFailure = {
statusClass: LoginStatusClass.INITIATOR_ERROR,
statusDetail: LoginStatusDetail.AUTHENTICATION_FAILURE,
}
/** Progress of the target-side one-way CHAP exchange (RFC 7143 §11.1.4). */
type ChapState = 'none' | 'awaitAlgorithm' | 'awaitResponse' | 'authenticated' | 'failed'
/**
* Negotiates one connection's login: the target is authoritative, so every
* key is answered with the pinned value, and RFC 7143's boolean rules
* guarantee the session collapses to the one supported path regardless of
* what the initiator offers.
*
* With {@link ChapCredentials}, the target challenges the initiator during
* security negotiation and refuses to reach the operational stage until it
* proves the secret; without one, it uses the legacy `AuthMethod=None` path.
*/
export class LoginNegotiator {
#firstResponseSent = false
#complete = false
readonly #chap?: ChapCredentials
#chapState: ChapState = 'none'
#chapId = 0
#chapChallenge: Buffer = EMPTY
sessionType: SessionType = 'Normal'
initiatorName?: string
targetName?: string
readonly params: NegotiatedParams = {
initiatorMaxRecvDataSegmentLength: DEFAULT_INITIATOR_MAX_RECV_DATA_SEGMENT_LENGTH,
maxBurstLength: OUR_MAX_BURST_LENGTH,
}
constructor(chap?: ChapCredentials) {
this.#chap = chap
}
/** True once the session has transited to the full-feature phase. */
get complete(): boolean {
return this.#complete
}
/** True once CHAP authentication has definitively failed for this connection. */
get failed(): boolean {
return this.#chapState === 'failed'
}
/** CHAP is enforced only when configured and only for Normal (data) sessions. */
get #chapRequired(): boolean {
return this.#chap !== undefined && this.sessionType === 'Normal'
}
process(pdu: IncomingPdu): LoginStepResult {
const flags = pdu.flags
const csg = (flags & LOGIN_CSG_MASK) >> LOGIN_CSG_SHIFT
const nsg = flags & LOGIN_NSG_MASK
const transit = (flags & LOGIN_FLAG_TRANSIT) !== 0
const cont = (flags & LOGIN_FLAG_CONTINUE) !== 0
const keys = parseTextKeys(pdu.data)
log.debug('login request', { csg, nsg, transit, cont, keys: Object.fromEntries(keys) })
const sessionType = keys.get('SessionType')
if (sessionType !== undefined) {
this.sessionType = sessionType === 'Discovery' ? 'Discovery' : 'Normal'
}
this.initiatorName ??= keys.get('InitiatorName')
this.targetName ??= keys.get('TargetName')
const responseKeys: Array<[string, string]> = []
// A normal session's first Login Response must carry the portal group tag.
if (!this.#firstResponseSent && this.sessionType === 'Normal') {
responseKeys.push(['TargetPortalGroupTag', '1'])
}
let failure: AuthFailure | undefined
if (csg === LoginStage.SECURITY_NEGOTIATION) {
// Act only on a complete key set (C bit clear); while the initiator is
// still sending keys we accumulate without answering the exchange.
if (!cont) {
failure = this.#handleSecurity(keys, responseKeys)
}
} else if (csg === LoginStage.OPERATIONAL_NEGOTIATION) {
if (this.#chapRequired && this.#chapState !== 'authenticated') {
// Normal session jumped to operational negotiation without completing the
// CHAP security stage we require.
this.#chapState = 'failed'
failure = AUTH_FAILURE
} else {
this.#negotiateOperational(keys, responseKeys)
}
}
let flagsByte = csg << LOGIN_CSG_SHIFT
// Transit only when the initiator asked to, is not still sending keys, and
// (if CHAP is required) authentication has succeeded. This single gate blocks
// a premature Transit bit and any transit before/without CHAP.
const authReady = !this.#chapRequired || this.#chapState === 'authenticated'
if (transit && !cont && authReady) {
flagsByte |= LOGIN_FLAG_TRANSIT | nsg
if (nsg === LoginStage.FULL_FEATURE_PHASE) {
this.#complete = true
}
}
this.#firstResponseSent = true
log.debug('login response', {
transit: (flagsByte & LOGIN_FLAG_TRANSIT) !== 0,
nsg: flagsByte & LOGIN_NSG_MASK,
keys: Object.fromEntries(responseKeys),
statusClass: failure?.statusClass ?? 0,
statusDetail: failure?.statusDetail ?? 0,
chapState: this.#chapState,
})
return {
flagsByte,
data: serializeTextKeys(responseKeys),
statusClass: failure?.statusClass,
statusDetail: failure?.statusDetail,
}
}
/**
* Drive one round of the target-side one-way CHAP exchange, mutating
* {@link ChapState} and appending any response keys. Returns an
* {@link AuthFailure} when authentication is rejected, otherwise `undefined`
* (success or "still in progress, awaiting the next PDU").
*/
#handleSecurity(keys: Map<string, string>, responseKeys: Array<[string, string]>): AuthFailure | undefined {
// CHAP applies only to Normal (data) sessions. Discovery sessions can only
// run SendTargets — they never reach the LUN — so they negotiate
// AuthMethod=None, keeping the standard unauthenticated `iscsiadm -m
// discovery` flow working. Without CHAP configured, this is the only path.
const chap = this.#chap
if (chap === undefined || this.sessionType !== 'Normal') {
if (keys.has('AuthMethod')) {
responseKeys.push(['AuthMethod', AuthMethod.NONE])
}
return undefined
}
switch (this.#chapState) {
case 'none': {
const offer = keys.get('AuthMethod')
if (offer === undefined) {
return undefined // still waiting for the AuthMethod offer
}
if (!offer.split(',').includes(AuthMethod.CHAP)) {
// Initiator offered only None (or an unknown method) while we require CHAP.
log.debug('chap: initiator did not offer CHAP', { offered: offer })
this.#chapState = 'failed'
return AUTH_FAILURE
}
responseKeys.push(['AuthMethod', AuthMethod.CHAP])
this.#chapState = 'awaitAlgorithm'
return undefined
}
case 'awaitAlgorithm': {
const offered = keys.get('CHAP_A')
if (offered === undefined) {
return undefined // still waiting for CHAP_A
}
const algorithm = selectAlgorithm(parseAlgorithmList(offered))
if (algorithm === undefined) {
log.debug('chap: no supported algorithm offered', { offered })
this.#chapState = 'failed'
return AUTH_FAILURE
}
this.#chapId = generateId()
this.#chapChallenge = generateChallenge()
responseKeys.push(['CHAP_A', String(algorithm)])
responseKeys.push(['CHAP_I', String(this.#chapId)])
responseKeys.push(['CHAP_C', encodeChapValue(this.#chapChallenge)])
log.debug('chap: issued challenge', { algorithm, id: this.#chapId })
this.#chapState = 'awaitResponse'
return undefined
}
case 'awaitResponse': {
const name = keys.get('CHAP_N')
const response = keys.get('CHAP_R')
if (name === undefined || response === undefined) {
return undefined // still waiting for CHAP_N / CHAP_R
}
let got: Buffer
try {
got = decodeChapValue(response)
} catch {
this.#chapState = 'failed'
return AUTH_FAILURE
}
const expected = computeResponse(this.#chapId, chap.secret, this.#chapChallenge)
if (name === chap.user && verifyResponse(expected, got)) {
log.debug('chap: authenticated', { user: name })
this.#chapState = 'authenticated'
return undefined
}
log.debug('chap: verification failed', { user: name, userMatch: name === chap.user })
this.#chapState = 'failed'
return AUTH_FAILURE
}
default:
return undefined
}
}
#negotiateOperational(keys: Map<string, string>, responseKeys: Array<[string, string]>): void {
// Only answer what the initiator actually offered: for every key below, the
// RFC 7143 default of an unoffered key is already the value we want, so
// staying silent and letting the default stand is equivalent.
const answer = (key: string, value: string) => {
if (keys.has(key)) {
responseKeys.push([key, value])
}
}
answer('HeaderDigest', 'None')
answer('DataDigest', 'None')
answer('MaxConnections', '1')
answer('InitialR2T', 'Yes')
answer('ErrorRecoveryLevel', '0')
// ImmediateData is the one exception: its RFC 7143 default is *Yes*, so an
// initiator that never offers the key may legally put write data in the SCSI
// Command PDU itself — which this target does not implement and would
// otherwise silently drop, acknowledging a write it never applied. Declare
// No whether or not it was offered. Costs nothing: there is no immediate
// data path to lose, and every mainstream initiator offers the key anyway,
// so for them this is byte-identical to before.
responseKeys.push(['ImmediateData', 'No'])
answer('DataPDUInOrder', 'Yes')
answer('DataSequenceInOrder', 'Yes')
answer('DefaultTime2Wait', '2')
answer('DefaultTime2Retain', '0')
answer('MaxOutstandingR2T', '1')
// MaxRecvDataSegmentLength is declarative per direction: capture the
// initiator's (caps our Data-In) and always declare ours (caps its Data-Out).
this.params.initiatorMaxRecvDataSegmentLength = parsePositiveInt(
keys.get('MaxRecvDataSegmentLength'),
this.params.initiatorMaxRecvDataSegmentLength
)
responseKeys.push(['MaxRecvDataSegmentLength', String(OUR_MAX_RECV_DATA_SEGMENT_LENGTH)])
// MaxBurstLength / FirstBurstLength negotiate to the minimum of both offers.
const maxBurst = Math.min(parsePositiveInt(keys.get('MaxBurstLength'), OUR_MAX_BURST_LENGTH), OUR_MAX_BURST_LENGTH)
this.params.maxBurstLength = maxBurst
if (keys.has('MaxBurstLength')) {
responseKeys.push(['MaxBurstLength', String(maxBurst)])
}
if (keys.has('FirstBurstLength')) {
const firstBurst = Math.min(parsePositiveInt(keys.get('FirstBurstLength'), maxBurst), maxBurst)
responseKeys.push(['FirstBurstLength', String(firstBurst)])
}
}
}

View File

@@ -0,0 +1,313 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import {
InitiatorOpcode,
LOGIN_CSG_SHIFT,
LOGIN_FLAG_CONTINUE,
LOGIN_FLAG_TRANSIT,
LOGIN_NSG_MASK,
LoginStage,
LoginStatusClass,
LoginStatusDetail,
OPCODE_IMMEDIATE,
} from './constants.mjs'
import { buildSendTargetsResponse, formatPortal, LoginNegotiator, parseTextKeys, serializeTextKeys } from './login.mjs'
import { computeResponse, decodeChapValue, encodeChapValue } from './chap.mjs'
import { IncomingPdu } from './pdu.mjs'
function loginFlags(options: { transit?: boolean; cont?: boolean; csg: number; nsg: number }): number {
return (
(options.transit ? LOGIN_FLAG_TRANSIT : 0) |
(options.cont ? LOGIN_FLAG_CONTINUE : 0) |
(options.csg << LOGIN_CSG_SHIFT) |
(options.nsg & LOGIN_NSG_MASK)
)
}
function makeLoginPdu(flags: number, keys: Record<string, string>): IncomingPdu {
const bhs = Buffer.alloc(48)
bhs[0] = InitiatorOpcode.LOGIN_REQUEST | OPCODE_IMMEDIATE
bhs[1] = flags
const data = serializeTextKeys(Object.entries(keys))
return new IncomingPdu(bhs, Buffer.alloc(0), data)
}
describe('text key serialization', () => {
it('round-trips NUL-delimited key=value pairs', () => {
const buffer = serializeTextKeys([
['A', '1'],
['HeaderDigest', 'None'],
])
assert.equal(buffer.toString('binary'), 'A=1\0HeaderDigest=None\0')
const keys = parseTextKeys(buffer)
assert.equal(keys.get('A'), '1')
assert.equal(keys.get('HeaderDigest'), 'None')
})
it('ignores empty fragments and malformed pairs', () => {
const keys = parseTextKeys(Buffer.from('\0Key=Value\0bogus\0', 'binary'))
assert.equal(keys.size, 1)
assert.equal(keys.get('Key'), 'Value')
})
})
describe('LoginNegotiator', () => {
it('completes a single-PDU operational login and pins the negotiated values', () => {
const negotiator = new LoginNegotiator()
const pdu = makeLoginPdu(
loginFlags({
transit: true,
csg: LoginStage.OPERATIONAL_NEGOTIATION,
nsg: LoginStage.FULL_FEATURE_PHASE,
}),
{
InitiatorName: 'iqn.1994-05.com.example:initiator',
SessionType: 'Normal',
HeaderDigest: 'None,CRC32C',
DataDigest: 'None,CRC32C',
InitialR2T: 'No',
ImmediateData: 'Yes',
MaxRecvDataSegmentLength: '262144',
MaxBurstLength: '16776192',
}
)
const { flagsByte, data } = negotiator.process(pdu)
assert.equal(negotiator.complete, true)
assert.ok((flagsByte & LOGIN_FLAG_TRANSIT) !== 0)
assert.equal(flagsByte & LOGIN_NSG_MASK, LoginStage.FULL_FEATURE_PHASE)
const keys = parseTextKeys(data)
assert.equal(keys.get('TargetPortalGroupTag'), '1')
assert.equal(keys.get('HeaderDigest'), 'None')
assert.equal(keys.get('DataDigest'), 'None')
// Pinned to the single write code path regardless of the initiator's offer.
assert.equal(keys.get('InitialR2T'), 'Yes')
assert.equal(keys.get('ImmediateData'), 'No')
// MaxBurstLength negotiates down to our cap.
assert.equal(keys.get('MaxBurstLength'), '262144')
assert.equal(negotiator.sessionType, 'Normal')
assert.equal(negotiator.initiatorName, 'iqn.1994-05.com.example:initiator')
assert.equal(negotiator.params.initiatorMaxRecvDataSegmentLength, 262144)
assert.equal(negotiator.params.maxBurstLength, 262144)
})
it('handles a two-stage security then operational login', () => {
const negotiator = new LoginNegotiator()
const security = negotiator.process(
makeLoginPdu(
loginFlags({
transit: true,
csg: LoginStage.SECURITY_NEGOTIATION,
nsg: LoginStage.OPERATIONAL_NEGOTIATION,
}),
{ InitiatorName: 'iqn.x:i', SessionType: 'Normal', AuthMethod: 'None' }
)
)
assert.equal(negotiator.complete, false)
assert.equal(parseTextKeys(security.data).get('AuthMethod'), 'None')
const operational = negotiator.process(
makeLoginPdu(
loginFlags({
transit: true,
csg: LoginStage.OPERATIONAL_NEGOTIATION,
nsg: LoginStage.FULL_FEATURE_PHASE,
}),
{ HeaderDigest: 'None', DataDigest: 'None' }
)
)
assert.equal(negotiator.complete, true)
// The portal group tag is only emitted in the first response.
assert.equal(parseTextKeys(operational.data).get('TargetPortalGroupTag'), undefined)
})
it('omits the portal group tag for a discovery session', () => {
const negotiator = new LoginNegotiator()
const { data } = negotiator.process(
makeLoginPdu(
loginFlags({
transit: true,
csg: LoginStage.OPERATIONAL_NEGOTIATION,
nsg: LoginStage.FULL_FEATURE_PHASE,
}),
{ SessionType: 'Discovery' }
)
)
assert.equal(negotiator.sessionType, 'Discovery')
assert.equal(parseTextKeys(data).get('TargetPortalGroupTag'), undefined)
})
it('does not transit while the initiator is still sending keys (C bit)', () => {
const negotiator = new LoginNegotiator()
const { flagsByte } = negotiator.process(
makeLoginPdu(
loginFlags({
transit: true,
cont: true,
csg: LoginStage.OPERATIONAL_NEGOTIATION,
nsg: LoginStage.FULL_FEATURE_PHASE,
}),
{ HeaderDigest: 'None' }
)
)
assert.equal(negotiator.complete, false)
assert.equal(flagsByte & LOGIN_FLAG_TRANSIT, 0)
})
})
describe('LoginNegotiator CHAP authenticator', () => {
const CHAP = { user: 'alice', secret: 's3cr3t' }
const securityFlags = (transit = false) =>
loginFlags({ transit, csg: LoginStage.SECURITY_NEGOTIATION, nsg: LoginStage.OPERATIONAL_NEGOTIATION })
/** Round 1: offer AuthMethod. */
function offerAuth(negotiator: LoginNegotiator, methods = 'CHAP,None', transit = false) {
return negotiator.process(
makeLoginPdu(securityFlags(transit), { InitiatorName: 'iqn.x:i', SessionType: 'Normal', AuthMethod: methods })
)
}
/** Round 2: offer CHAP_A, return the target's parsed challenge keys. */
function offerAlgorithm(negotiator: LoginNegotiator, list = '5') {
const result = negotiator.process(makeLoginPdu(securityFlags(false), { CHAP_A: list }))
return { result, keys: parseTextKeys(result.data) }
}
/** Round 3: answer with CHAP_N/CHAP_R for the given challenge. */
function answerChallenge(
negotiator: LoginNegotiator,
challengeKeys: Map<string, string>,
{ user = CHAP.user, secret = CHAP.secret } = {}
) {
const id = Number.parseInt(challengeKeys.get('CHAP_I') ?? '', 10)
const challenge = decodeChapValue(challengeKeys.get('CHAP_C') ?? '')
const response = encodeChapValue(computeResponse(id, secret, challenge))
return negotiator.process(makeLoginPdu(securityFlags(true), { CHAP_N: user, CHAP_R: response }))
}
it('challenges then transits only after a correct response', () => {
const negotiator = new LoginNegotiator(CHAP)
// Round 1: we answer AuthMethod=CHAP and do NOT transit.
const r1 = offerAuth(negotiator)
assert.equal(parseTextKeys(r1.data).get('AuthMethod'), 'CHAP')
assert.equal(r1.flagsByte & LOGIN_FLAG_TRANSIT, 0)
assert.equal(r1.statusClass ?? LoginStatusClass.SUCCESS, LoginStatusClass.SUCCESS)
// Round 2: we issue CHAP_A=5, an identifier and a challenge, still no transit.
const { result: r2, keys: challenge } = offerAlgorithm(negotiator)
assert.equal(challenge.get('CHAP_A'), '5')
assert.ok(challenge.has('CHAP_I'))
assert.match(challenge.get('CHAP_C') ?? '', /^0x[0-9a-f]+$/)
assert.equal(r2.flagsByte & LOGIN_FLAG_TRANSIT, 0)
// Round 3: a correct response transits to the operational stage.
const r3 = answerChallenge(negotiator, challenge)
assert.equal(r3.statusClass ?? LoginStatusClass.SUCCESS, LoginStatusClass.SUCCESS)
assert.ok((r3.flagsByte & LOGIN_FLAG_TRANSIT) !== 0, 'transit granted after auth')
assert.equal(r3.flagsByte & LOGIN_NSG_MASK, LoginStage.OPERATIONAL_NEGOTIATION)
assert.equal(negotiator.failed, false)
// The operational stage then completes as usual.
const op = negotiator.process(
makeLoginPdu(
loginFlags({ transit: true, csg: LoginStage.OPERATIONAL_NEGOTIATION, nsg: LoginStage.FULL_FEATURE_PHASE }),
{ HeaderDigest: 'None', DataDigest: 'None' }
)
)
assert.equal(negotiator.complete, true)
assert.equal(parseTextKeys(op.data).get('HeaderDigest'), 'None')
})
it('rejects a wrong secret with Initiator Error / authentication failure', () => {
const negotiator = new LoginNegotiator(CHAP)
offerAuth(negotiator)
const { keys: challenge } = offerAlgorithm(negotiator)
const r3 = answerChallenge(negotiator, challenge, { secret: 'wrong' })
assert.equal(r3.statusClass, LoginStatusClass.INITIATOR_ERROR)
assert.equal(r3.statusDetail, LoginStatusDetail.AUTHENTICATION_FAILURE)
assert.equal(r3.flagsByte & LOGIN_FLAG_TRANSIT, 0)
assert.equal(negotiator.complete, false)
assert.equal(negotiator.failed, true)
})
it('rejects a wrong username even with the right secret', () => {
const negotiator = new LoginNegotiator(CHAP)
offerAuth(negotiator)
const { keys: challenge } = offerAlgorithm(negotiator)
const r3 = answerChallenge(negotiator, challenge, { user: 'mallory' })
assert.equal(r3.statusClass, LoginStatusClass.INITIATOR_ERROR)
assert.equal(negotiator.failed, true)
})
it('fails when the initiator offers only AuthMethod=None', () => {
const negotiator = new LoginNegotiator(CHAP)
const r1 = offerAuth(negotiator, 'None')
assert.equal(r1.statusClass, LoginStatusClass.INITIATOR_ERROR)
assert.equal(r1.statusDetail, LoginStatusDetail.AUTHENTICATION_FAILURE)
assert.equal(negotiator.failed, true)
})
it('fails when CHAP_A does not offer MD5', () => {
const negotiator = new LoginNegotiator(CHAP)
offerAuth(negotiator)
const { result } = offerAlgorithm(negotiator, '6,7')
assert.equal(result.statusClass, LoginStatusClass.INITIATOR_ERROR)
assert.equal(negotiator.failed, true)
})
it('does not transit if the initiator sets the T bit before authenticating', () => {
const negotiator = new LoginNegotiator(CHAP)
// Round 1 with the Transit bit set: must be ignored until CHAP completes.
const r1 = offerAuth(negotiator, 'CHAP,None', true)
assert.equal(r1.flagsByte & LOGIN_FLAG_TRANSIT, 0)
assert.equal(negotiator.complete, false)
})
it('fails if the initiator skips security and jumps to operational negotiation', () => {
const negotiator = new LoginNegotiator(CHAP)
const result = negotiator.process(
makeLoginPdu(
loginFlags({ transit: true, csg: LoginStage.OPERATIONAL_NEGOTIATION, nsg: LoginStage.FULL_FEATURE_PHASE }),
{ HeaderDigest: 'None' }
)
)
assert.equal(result.statusClass, LoginStatusClass.INITIATOR_ERROR)
assert.equal(result.flagsByte & LOGIN_FLAG_TRANSIT, 0)
assert.equal(negotiator.complete, false)
assert.equal(negotiator.failed, true)
})
})
describe('formatPortal', () => {
it('keeps plain IPv4 addresses', () => {
assert.equal(formatPortal('10.0.0.1', 3260), '10.0.0.1:3260,1')
})
it('unwraps IPv4-mapped IPv6 (dual-stack socket artifact)', () => {
assert.equal(formatPortal('::ffff:192.168.1.8', 3260), '192.168.1.8:3260,1')
})
it('brackets real IPv6 literals', () => {
assert.equal(formatPortal('2a01:e0a:e6a:9500::1', 3260), '[2a01:e0a:e6a:9500::1]:3260,1')
})
})
describe('buildSendTargetsResponse', () => {
it('advertises the target name and address', () => {
const keys = parseTextKeys(buildSendTargetsResponse('iqn.2024-01.tech.vates:lun0', '10.0.0.1', 3260))
assert.equal(keys.get('TargetName'), 'iqn.2024-01.tech.vates:lun0')
assert.equal(keys.get('TargetAddress'), '10.0.0.1:3260,1')
})
it('advertises a clean IPv4 portal for an IPv4-mapped local address', () => {
const keys = parseTextKeys(buildSendTargetsResponse('iqn.x:y', '::ffff:192.168.1.8', 3260))
assert.equal(keys.get('TargetAddress'), '192.168.1.8:3260,1')
})
})

File diff suppressed because it is too large Load Diff

174
@vates/iscsi/src/pdu.mts Normal file
View File

@@ -0,0 +1,174 @@
import type { Socket } from 'node:net'
import type { Readable } from 'node:stream'
import { readChunk, readChunkStrict } from '@vates/read-chunk'
import { pTimeout } from 'promise-toolbox'
import { createLogger, type Logger } from '@xen-orchestra/log'
import { BHS_LENGTH, opcodeName, OPCODE_IMMEDIATE, OPCODE_MASK, FLAG_FINAL } from './constants.mjs'
const EMPTY = Buffer.alloc(0)
// Framing-level wire trace, shared by target and initiator. Enable with the
// `vates:iscsi:pdu` debug namespace to see every PDU that crosses the socket —
// the first thing to look at when a new initiator/target combination misbehaves.
const log: Logger = createLogger('vates:iscsi:pdu')
/** Round `n` up to the next multiple of 4 (iSCSI segments are 4-byte padded). */
export function roundUp4(n: number): number {
return (n + 3) & ~3
}
/**
* A fully-received inbound PDU: the 48-byte Basic Header Segment, any Additional
* Header Segment, and the (unpadded) data segment. Field accessors read the BHS
* at the offsets defined by RFC 7143; callers pick the ones relevant to the
* opcode they are handling.
*/
export class IncomingPdu {
constructor(
readonly bhs: Buffer,
readonly ahs: Buffer,
readonly data: Buffer
) {}
/** 6-bit opcode (immediate bit masked off). */
get opcode(): number {
return this.bhs[0] & OPCODE_MASK
}
/** Whether the immediate bit is set (initiator requests immediate processing). */
get immediate(): boolean {
return (this.bhs[0] & OPCODE_IMMEDIATE) !== 0
}
/** Raw opcode-specific flags byte (byte 1). */
get flags(): number {
return this.bhs[1]
}
get finalBit(): boolean {
return (this.bhs[1] & FLAG_FINAL) !== 0
}
readU8(offset: number): number {
return this.bhs.readUInt8(offset)
}
readU16(offset: number): number {
return this.bhs.readUInt16BE(offset)
}
readU32(offset: number): number {
return this.bhs.readUInt32BE(offset)
}
/** Initiator Task Tag (bytes 16-19), present in every initiator PDU we handle. */
get itt(): number {
return this.bhs.readUInt32BE(16)
}
/** CmdSN (bytes 24-27) — meaningful on command-carrying PDUs. */
get cmdSN(): number {
return this.bhs.readUInt32BE(24)
}
/** ExpStatSN (bytes 28-31). */
get expStatSN(): number {
return this.bhs.readUInt32BE(28)
}
}
/**
* Read exactly one PDU from `stream`, reassembling across TCP boundaries.
* Resolves to `null` on a clean EOF between PDUs; throws on a truncated PDU
* or read error. No header/data digest bytes: digests are pinned off.
*
* No per-read timeout, intentionally — a target may sit idle between commands
* indefinitely. Liveness is a transport concern (socket timeout / NOP
* keepalives), handled elsewhere.
*
* A PDU announcing a data segment longer than `maxDataSegmentLength` is
* rejected before its payload is read, so a peer cannot make us buffer up to
* the 16 MiB the 24-bit length field allows.
*/
export async function readPdu(stream: Readable, maxDataSegmentLength = Infinity): Promise<IncomingPdu | null> {
const bhs = await readChunk(stream, BHS_LENGTH)
if (bhs === null) {
return null // clean EOF between PDUs
}
if (bhs.length !== BHS_LENGTH) {
throw new Error(`truncated iSCSI BHS (got ${bhs.length} of ${BHS_LENGTH} bytes)`)
}
const totalAhsLength = bhs.readUInt8(4)
const dataSegmentLength = bhs.readUIntBE(5, 3)
if (dataSegmentLength > maxDataSegmentLength) {
throw new Error(`iSCSI data segment too long (${dataSegmentLength} > ${maxDataSegmentLength} bytes)`)
}
const ahsBytes = totalAhsLength * 4
const dataBytesPadded = roundUp4(dataSegmentLength)
let ahs = EMPTY
let data = EMPTY
if (ahsBytes + dataBytesPadded > 0) {
const rest = await readChunkStrict(stream, ahsBytes + dataBytesPadded)
if (ahsBytes > 0) {
ahs = rest.subarray(0, ahsBytes)
}
data = rest.subarray(ahsBytes, ahsBytes + dataSegmentLength) // strip 4-byte padding
}
const pdu = new IncomingPdu(bhs, ahs, data)
log.debug('rx', {
opcode: opcodeName(bhs[0]),
itt: pdu.itt,
flags: `0x${bhs[1].toString(16).padStart(2, '0')}`,
ahs: ahsBytes,
data: dataSegmentLength,
})
return pdu
}
/** Allocate a zeroed 48-byte BHS with byte 0 set to `opcode`. */
export function allocBhs(opcode: number): Buffer {
const bhs = Buffer.alloc(BHS_LENGTH)
bhs[0] = opcode
return bhs
}
/** Write the 24-bit DataSegmentLength field (bytes 5-7) of a BHS. */
export function setDataSegmentLength(bhs: Buffer, length: number): void {
bhs.writeUIntBE(length, 5, 3)
}
/**
* Assemble a complete PDU buffer from a 48-byte BHS and an optional data
* segment, setting DataSegmentLength and applying 4-byte padding.
*/
export function assemblePdu(bhs: Buffer, data?: Buffer): Buffer {
const length = data?.length ?? 0
setDataSegmentLength(bhs, length)
if (length === 0) {
return bhs
}
const padding = roundUp4(length) - length
return padding === 0 ? Buffer.concat([bhs, data!]) : Buffer.concat([bhs, data!, Buffer.alloc(padding)])
}
/**
* Write a PDU to the socket, resolving once the kernel has accepted it (honoring
* back-pressure via the write callback). Rejects if the write errors or does not
* drain within `timeoutMs` (0 disables the timeout).
*/
export function writePdu(socket: Socket, buffer: Buffer, timeoutMs: number): Promise<void> {
log.debug('tx', {
opcode: opcodeName(buffer[0]),
itt: buffer.readUInt32BE(16),
flags: `0x${buffer[1].toString(16).padStart(2, '0')}`,
length: buffer.length,
})
const promise = new Promise<void>((resolve, reject) => {
socket.write(buffer, error => (error ? reject(error) : resolve()))
})
return timeoutMs > 0 ? pTimeout.call(promise, timeoutMs) : promise
}

View File

@@ -0,0 +1,118 @@
import assert from 'node:assert/strict'
import { PassThrough } from 'node:stream'
import { describe, it } from 'node:test'
import { InitiatorOpcode } from './constants.mjs'
import { allocBhs, assemblePdu, IncomingPdu, readPdu, roundUp4 } from './pdu.mjs'
function buildScsiCommandPdu(itt: number, cmdSN: number, cdb: Buffer, data?: Buffer): Buffer {
const bhs = allocBhs(InitiatorOpcode.SCSI_COMMAND)
bhs[1] = 0x80 // Final
bhs.writeUInt32BE(itt, 16)
bhs.writeUInt32BE(0, 20) // EDTL
bhs.writeUInt32BE(cmdSN, 24)
cdb.copy(bhs, 32)
return assemblePdu(bhs, data)
}
async function readOne(buffer: Buffer): Promise<IncomingPdu> {
const stream = new PassThrough()
stream.end(buffer)
const pdu = await readPdu(stream)
assert.ok(pdu !== null)
return pdu
}
describe('roundUp4', () => {
it('rounds up to the next multiple of 4', () => {
assert.equal(roundUp4(0), 0)
assert.equal(roundUp4(1), 4)
assert.equal(roundUp4(4), 4)
assert.equal(roundUp4(5), 8)
assert.equal(roundUp4(15), 16)
})
})
describe('assemblePdu / readPdu round-trip', () => {
it('preserves opcode and header fields', async () => {
const cdb = Buffer.alloc(16)
cdb[0] = 0x28 // READ(10)
const pdu = await readOne(buildScsiCommandPdu(0xdeadbeef, 42, cdb))
assert.equal(pdu.opcode, InitiatorOpcode.SCSI_COMMAND)
assert.equal(pdu.immediate, false)
assert.equal(pdu.finalBit, true)
assert.equal(pdu.itt, 0xdeadbeef)
assert.equal(pdu.cmdSN, 42)
assert.equal(pdu.bhs.subarray(32, 48)[0], 0x28)
})
it('carries a data segment and strips its 4-byte padding', async () => {
const data = Buffer.from([1, 2, 3, 4, 5]) // 5 bytes -> padded to 8 on the wire
const wire = buildScsiCommandPdu(1, 1, Buffer.alloc(16), data)
assert.equal(wire.length, 48 + 8, 'data segment is padded to a 4-byte boundary')
const pdu = await readOne(wire)
assert.deepEqual(pdu.data, data, 'padding is stripped from the parsed data')
})
})
describe('readPdu framing', () => {
it('rejects a data segment longer than the limit before reading it', async () => {
const bhs = allocBhs(InitiatorOpcode.SCSI_COMMAND)
bhs.writeUIntBE(16 * 1024 * 1024 - 1, 5, 3) // announced data segment length
const stream = new PassThrough()
stream.write(bhs) // the announced payload is never sent
await assert.rejects(readPdu(stream, 8192), /data segment too long/)
})
it('accepts a data segment of exactly the limit', async () => {
const data = Buffer.alloc(8192, 0x5a)
const pdu = await (async () => {
const stream = new PassThrough()
stream.end(buildScsiCommandPdu(1, 1, Buffer.alloc(16), data))
return readPdu(stream, 8192)
})()
assert.deepEqual(pdu?.data, data)
})
it('reassembles a PDU split across multiple TCP chunks', async () => {
const data = Buffer.from('hello world!!') // 13 bytes -> padded to 16
const wire = buildScsiCommandPdu(7, 7, Buffer.alloc(16), data)
const stream = new PassThrough()
const pduPromise = readPdu(stream)
// Feed the bytes in awkward fragments.
stream.write(wire.subarray(0, 10))
stream.write(wire.subarray(10, 48))
stream.write(wire.subarray(48, 55))
stream.end(wire.subarray(55))
const pdu = await pduPromise
assert.ok(pdu !== null)
assert.equal(pdu.itt, 7)
assert.deepEqual(pdu.data, data)
})
it('reads two PDUs coalesced into one chunk', async () => {
const first = buildScsiCommandPdu(1, 1, Buffer.alloc(16), Buffer.from([0xaa]))
const second = buildScsiCommandPdu(2, 2, Buffer.alloc(16), Buffer.from([0xbb, 0xcc]))
const stream = new PassThrough()
stream.end(Buffer.concat([first, second]))
const a = await readPdu(stream)
const b = await readPdu(stream)
assert.equal(a?.itt, 1)
assert.deepEqual(a?.data, Buffer.from([0xaa]))
assert.equal(b?.itt, 2)
assert.deepEqual(b?.data, Buffer.from([0xbb, 0xcc]))
})
it('returns null on a clean end-of-stream', async () => {
const stream = new PassThrough()
stream.end()
assert.equal(await readPdu(stream), null)
})
it('throws on a truncated BHS', async () => {
const stream = new PassThrough()
stream.end(Buffer.alloc(20)) // fewer than 48 bytes, then EOF
await assert.rejects(readPdu(stream), /truncated/)
})
})

293
@vates/iscsi/src/scsi.mts Normal file
View File

@@ -0,0 +1,293 @@
import { createLogger, type Logger } from '@xen-orchestra/log'
import { Asc, ScsiStatus, SenseKey } from './constants.mjs'
import { decodeCdb, type CommandContext, type SenseInfo } from './types.mjs'
// SCSI command trace. Enable `vates:iscsi:scsi` to see every decoded CDB the
// target services — useful when an initiator issues a command we mishandle.
const log: Logger = createLogger('vates:iscsi:scsi')
/** SCSI INQUIRY identity strings advertised by the LUN. */
export interface ScsiIdentity {
/** T10 vendor identification (≤ 8 ASCII chars). */
vendor: string
/** Product identification (≤ 16 ASCII chars). */
product: string
/** Product revision level (≤ 4 ASCII chars). */
revision: string
/** Unit serial number; also seeds the VPD 0x83 device identifier. */
serial: string
}
// --- low-level encoders -----------------------------------------------------
function writeAscii(buffer: Buffer, offset: number, length: number, value: string): void {
buffer.fill(0x20, offset, offset + length) // space pad
buffer.write(value.slice(0, length), offset, 'ascii')
}
/** Standard INQUIRY data (36 bytes) for a direct-access block device. */
export function buildStandardInquiry(identity: ScsiIdentity): Buffer {
const buffer = Buffer.alloc(36)
buffer[0] = 0x00 // peripheral qualifier 0, device type 0x00 (direct-access block device)
buffer[1] = 0x00 // not removable
buffer[2] = 0x05 // version: SPC-3
buffer[3] = 0x02 // response data format 2
buffer[4] = 31 // additional length (total 36 - 5)
writeAscii(buffer, 8, 8, identity.vendor)
writeAscii(buffer, 16, 16, identity.product)
writeAscii(buffer, 32, 4, identity.revision)
return buffer
}
const SUPPORTED_VPD_PAGES = [0x00, 0x80, 0x83, 0xb0]
/** Default max length of a single READ or WRITE, in bytes. */
export const DEFAULT_MAX_TRANSFER_LENGTH = 8 * 1024 * 1024
/**
* Build an INQUIRY EVPD page, or `undefined` for an unsupported page code (the
* caller then returns CHECK CONDITION / INVALID FIELD IN CDB).
*/
export function buildVpd(
pageCode: number,
identity: ScsiIdentity,
maxTransferBlocks = DEFAULT_MAX_TRANSFER_LENGTH / 512
): Buffer | undefined {
switch (pageCode) {
case 0x00: {
// Supported VPD pages.
const buffer = Buffer.alloc(4 + SUPPORTED_VPD_PAGES.length)
buffer[1] = 0x00
buffer.writeUInt16BE(SUPPORTED_VPD_PAGES.length, 2)
SUPPORTED_VPD_PAGES.forEach((page, i) => {
buffer[4 + i] = page
})
return buffer
}
case 0x80: {
// Unit serial number.
const serial = Buffer.from(identity.serial, 'ascii')
const buffer = Buffer.alloc(4 + serial.length)
buffer[1] = 0x80
buffer.writeUInt16BE(serial.length, 2)
serial.copy(buffer, 4)
return buffer
}
case 0x83: {
// Device identification: one T10-vendor-ID-based designator (ASCII).
const designator = Buffer.alloc(8 + identity.serial.length)
writeAscii(designator, 0, 8, identity.vendor)
designator.write(identity.serial, 8, 'ascii')
const descriptor = Buffer.alloc(4 + designator.length)
descriptor[0] = 0x02 // code set: ASCII
descriptor[1] = 0x01 // association: logical unit, designator type: T10 vendor ID based
descriptor[3] = designator.length
designator.copy(descriptor, 4)
const buffer = Buffer.alloc(4 + descriptor.length)
buffer[1] = 0x83
buffer.writeUInt16BE(descriptor.length, 2)
descriptor.copy(buffer, 4)
return buffer
}
case 0xb0: {
// Block limits: initiators split their I/O to the max transfer length
// instead of seeing it refused.
const buffer = Buffer.alloc(64)
buffer[1] = 0xb0
buffer.writeUInt16BE(0x3c, 2)
buffer.writeUInt32BE(maxTransferBlocks, 8) // maximum transfer length
return buffer
}
default:
return undefined
}
}
/** READ CAPACITY(10) parameter data (8 bytes). */
export function buildReadCapacity10(blockCount: number, blockSize: number): Buffer {
const buffer = Buffer.alloc(8)
const maxLba = blockCount - 1
// If the capacity overflows 32 bits, return 0xffffffff to force the initiator
// to issue READ CAPACITY(16).
buffer.writeUInt32BE(maxLba > 0xffffffff ? 0xffffffff : maxLba, 0)
buffer.writeUInt32BE(blockSize, 4)
return buffer
}
/** READ CAPACITY(16) parameter data (32 bytes). */
export function buildReadCapacity16(blockCount: number, blockSize: number): Buffer {
const buffer = Buffer.alloc(32)
buffer.writeBigUInt64BE(BigInt(blockCount - 1), 0)
buffer.writeUInt32BE(blockSize, 8)
return buffer
}
/** REPORT LUNS parameter data advertising a single LUN 0 (16 bytes). */
export function buildReportLuns(): Buffer {
const buffer = Buffer.alloc(16)
buffer.writeUInt32BE(8, 0) // LUN list length: one 8-byte LUN entry
return buffer // bytes 8-15 are LUN 0 (all zero)
}
/** Fixed-format sense data (18 bytes). */
export function buildFixedSense({ key, asc, ascq }: SenseInfo): Buffer {
const buffer = Buffer.alloc(18)
buffer[0] = 0x70 // response code: current error, fixed format
buffer[2] = key
buffer[7] = 10 // additional sense length (18 - 8)
buffer[12] = asc
buffer[13] = ascq
return buffer
}
// WP bit of the device-specific parameter in the mode parameter header
const MODE_SENSE_WRITE_PROTECT = 0x80
/** Minimal MODE SENSE parameter data: a header with no block descriptor or pages. */
export function buildModeSense(long: boolean, writeProtected = false): Buffer {
const deviceSpecific = writeProtected ? MODE_SENSE_WRITE_PROTECT : 0
if (long) {
const buffer = Buffer.alloc(8)
buffer.writeUInt16BE(6, 0) // mode data length (8 - 2)
buffer[3] = deviceSpecific
return buffer
}
const buffer = Buffer.alloc(4)
buffer[0] = 3 // mode data length (4 - 1)
buffer[2] = deviceSpecific
return buffer
}
const ILLEGAL_REQUEST_INVALID_OPCODE: SenseInfo = {
key: SenseKey.ILLEGAL_REQUEST,
...Asc.INVALID_COMMAND_OPERATION_CODE,
}
const ILLEGAL_REQUEST_INVALID_FIELD: SenseInfo = {
key: SenseKey.ILLEGAL_REQUEST,
...Asc.INVALID_FIELD_IN_CDB,
}
const ILLEGAL_REQUEST_LBA_OUT_OF_RANGE: SenseInfo = {
key: SenseKey.ILLEGAL_REQUEST,
...Asc.LOGICAL_BLOCK_ADDRESS_OUT_OF_RANGE,
}
const MEDIUM_ERROR_NO_ADDITIONAL_SENSE: SenseInfo = { key: SenseKey.MEDIUM_ERROR, ...Asc.NO_ADDITIONAL_SENSE }
const DATA_PROTECT_WRITE_PROTECTED: SenseInfo = { key: SenseKey.DATA_PROTECT, ...Asc.WRITE_PROTECTED }
function checkCondition(ctx: CommandContext, itt: number, sense: SenseInfo): Promise<void> {
return ctx.sendScsiResponse(itt, ScsiStatus.CHECK_CONDITION, { sense: buildFixedSense(sense) })
}
/**
* Decode and execute one SCSI Command, driving completion through `ctx`.
*
* `cdb` is the 16-byte CDB from the SCSI Command PDU and `itt` its Initiator
* Task Tag. Read commands stream Data-In; WRITE solicits Data-Out via R2T;
* everything else returns status (with sense on error).
*
* A READ or WRITE longer than `maxTransferLength` bytes, advertised in the
* block limits VPD page, is refused with INVALID FIELD IN CDB: it also bounds
* the buffer a WRITE allocates.
*/
export async function handleScsiCommand(
cdb: Buffer,
itt: number,
ctx: CommandContext,
identity: ScsiIdentity,
maxTransferLength = DEFAULT_MAX_TRANSFER_LENGTH
): Promise<void> {
const { lun } = ctx
const blockSize = lun.getBlockSize()
const blockCount = lun.getSize() / blockSize
const maxTransferBlocks = Math.floor(maxTransferLength / blockSize)
const request = decodeCdb(cdb)
log.debug('scsi command', { itt, ...request })
switch (request.kind) {
case 'inquiry': {
if (!request.evpd) {
if (request.pageCode !== 0) {
// EVPD clear but a page code was given: invalid per SPC.
return checkCondition(ctx, itt, ILLEGAL_REQUEST_INVALID_FIELD)
}
return ctx.sendReadData(itt, buildStandardInquiry(identity), request.allocationLength)
}
const page = buildVpd(request.pageCode, identity, maxTransferBlocks)
if (page === undefined) {
return checkCondition(ctx, itt, ILLEGAL_REQUEST_INVALID_FIELD)
}
return ctx.sendReadData(itt, page, request.allocationLength)
}
case 'reportLuns':
return ctx.sendReadData(itt, buildReportLuns(), request.allocationLength)
case 'readCapacity10':
return ctx.sendReadData(itt, buildReadCapacity10(blockCount, blockSize), 8)
case 'readCapacity16':
return ctx.sendReadData(itt, buildReadCapacity16(blockCount, blockSize), request.allocationLength)
case 'read': {
if (request.lba + request.blocks > blockCount) {
return checkCondition(ctx, itt, ILLEGAL_REQUEST_LBA_OUT_OF_RANGE)
}
if (request.blocks > maxTransferBlocks) {
return checkCondition(ctx, itt, ILLEGAL_REQUEST_INVALID_FIELD)
}
const byteLength = request.blocks * blockSize
if (byteLength === 0) {
return ctx.sendReadData(itt, Buffer.alloc(0), 0)
}
let data: Buffer
try {
data = await lun.read(request.lba * blockSize, byteLength)
} catch (error) {
// A single bad block (backend hiccup, corrupt source) must fail only
// this command, not the whole connection — mirrors how a failed
// lun.write() is handled in connection.mts's #handleDataOut.
log.warn('LUN read failed', error instanceof Error ? error : new Error(String(error)))
return checkCondition(ctx, itt, MEDIUM_ERROR_NO_ADDITIONAL_SENSE)
}
return ctx.sendReadData(itt, data, byteLength)
}
case 'write': {
if (lun.readOnly === true) {
return checkCondition(ctx, itt, DATA_PROTECT_WRITE_PROTECTED)
}
if (request.lba + request.blocks > blockCount) {
return checkCondition(ctx, itt, ILLEGAL_REQUEST_LBA_OUT_OF_RANGE)
}
if (request.blocks > maxTransferBlocks) {
return checkCondition(ctx, itt, ILLEGAL_REQUEST_INVALID_FIELD)
}
const byteLength = request.blocks * blockSize
if (byteLength === 0) {
return ctx.sendScsiResponse(itt, ScsiStatus.GOOD)
}
// Hand off to the connection: it solicits the data via R2T and sends the
// SCSI Response once the (possibly interleaved) Data-Out PDUs complete.
return ctx.beginWrite(itt, request.lba * blockSize, byteLength)
}
case 'testUnitReady':
return ctx.sendScsiResponse(itt, ScsiStatus.GOOD)
case 'requestSense': {
// No deferred error model in v1: always report NO SENSE.
const sense = buildFixedSense({ key: SenseKey.NO_SENSE, ...Asc.NO_ADDITIONAL_SENSE })
return ctx.sendReadData(itt, sense, request.allocationLength)
}
case 'modeSense':
return ctx.sendReadData(itt, buildModeSense(request.long, lun.readOnly === true), request.allocationLength)
case 'syncCache':
await lun.flush()
return ctx.sendScsiResponse(itt, ScsiStatus.GOOD)
case 'unsupported':
return checkCondition(ctx, itt, ILLEGAL_REQUEST_INVALID_OPCODE)
}
}

View File

@@ -0,0 +1,302 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import type { BlockDevice } from './backend.mjs'
import { ScsiStatus, SenseKey } from './constants.mjs'
import {
buildReadCapacity10,
buildReadCapacity16,
buildReportLuns,
buildStandardInquiry,
buildVpd,
handleScsiCommand,
type ScsiIdentity,
} from './scsi.mjs'
import { decodeCdb, type CommandContext, type ScsiResponseOptions } from './types.mjs'
const IDENTITY: ScsiIdentity = { vendor: 'VATES', product: 'ISCSI LUN', revision: '0001', serial: 'unit-serial-1' }
// --- a trivial in-memory LUN ------------------------------------------------
class MemoryBlockDevice implements BlockDevice {
readonly #buffer: Buffer
readonly #blockSize: number
constructor(sizeBytes: number, blockSize = 512) {
this.#buffer = Buffer.alloc(sizeBytes)
this.#blockSize = blockSize
}
get buffer(): Buffer {
return this.#buffer
}
getSize(): number {
return this.#buffer.length
}
getBlockSize(): number {
return this.#blockSize
}
async read(offset: number, length: number): Promise<Buffer> {
return Buffer.from(this.#buffer.subarray(offset, offset + length))
}
async write(offset: number, data: Buffer): Promise<void> {
data.copy(this.#buffer, offset)
}
async flush(): Promise<void> {}
async close(): Promise<void> {}
}
class FakeContext implements CommandContext {
readonly reads: Array<{ data: Buffer; allocationLength: number }> = []
readonly responses: Array<{ status: number; options?: ScsiResponseOptions }> = []
readonly writes: Array<{ itt: number; lunOffset: number; totalLength: number }> = []
constructor(readonly lun: BlockDevice) {}
async sendReadData(_itt: number, data: Buffer, allocationLength: number): Promise<void> {
this.reads.push({ data, allocationLength })
}
async beginWrite(itt: number, lunOffset: number, totalLength: number): Promise<void> {
this.writes.push({ itt, lunOffset, totalLength })
}
async sendScsiResponse(_itt: number, status: number, options?: ScsiResponseOptions): Promise<void> {
this.responses.push({ status, options })
}
}
function cdb(...bytes: number[]): Buffer {
const buffer = Buffer.alloc(16)
Buffer.from(bytes).copy(buffer)
return buffer
}
describe('decodeCdb', () => {
it('decodes READ(10) LBA and transfer length', () => {
const buffer = Buffer.alloc(16)
buffer[0] = 0x28
buffer.writeUInt32BE(0x01020304, 2)
buffer.writeUInt16BE(8, 7)
assert.deepEqual(decodeCdb(buffer), { kind: 'read', lba: 0x01020304, blocks: 8 })
})
it('decodes WRITE(16) LBA and transfer length', () => {
const buffer = Buffer.alloc(16)
buffer[0] = 0x8a
buffer.writeBigUInt64BE(123456789n, 2)
buffer.writeUInt32BE(16, 10)
assert.deepEqual(decodeCdb(buffer), { kind: 'write', lba: 123456789, blocks: 16 })
})
it('decodes READ CAPACITY(16) only for the right service action', () => {
assert.equal(decodeCdb(cdb(0x9e, 0x10)).kind, 'readCapacity16')
assert.equal(decodeCdb(cdb(0x9e, 0x12)).kind, 'unsupported')
})
it('clamps an out-of-range 64-bit LBA instead of throwing', () => {
// Past 2^53 an LBA no longer fits a JS number — and is past any capacity we
// can serve, so it must come back as an unservable LBA the range check
// rejects, not as an error that would tear the connection down.
const buffer = Buffer.alloc(16)
buffer[0] = 0x88
buffer.writeBigUInt64BE(0xffffffffffffffffn, 2)
buffer.writeUInt32BE(1, 10)
assert.deepEqual(decodeCdb(buffer), { kind: 'read', lba: Number.MAX_SAFE_INTEGER, blocks: 1 })
})
it('maps unknown opcodes to unsupported', () => {
assert.deepEqual(decodeCdb(cdb(0xff)), { kind: 'unsupported', opcode: 0xff })
})
})
describe('response encoders', () => {
it('builds 36-byte standard INQUIRY data for a block device', () => {
const data = buildStandardInquiry(IDENTITY)
assert.equal(data.length, 36)
assert.equal(data[0], 0x00) // direct-access block device
assert.equal(data[4], 31) // additional length
assert.equal(data.subarray(8, 16).toString('ascii').trim(), 'VATES')
assert.equal(data.subarray(16, 32).toString('ascii').trim(), 'ISCSI LUN')
})
it('builds READ CAPACITY(10) as (maxLba, blockSize)', () => {
const data = buildReadCapacity10(2048, 512)
assert.equal(data.readUInt32BE(0), 2047)
assert.equal(data.readUInt32BE(4), 512)
})
it('clamps READ CAPACITY(10) max LBA to force RC16 on large disks', () => {
const data = buildReadCapacity10(0x1_0000_0001, 512)
assert.equal(data.readUInt32BE(0), 0xffffffff)
})
it('builds READ CAPACITY(16) with a 64-bit max LBA', () => {
const data = buildReadCapacity16(0x1_0000_0001, 512)
assert.equal(data.readBigUInt64BE(0), 0x1_0000_0000n)
assert.equal(data.readUInt32BE(8), 512)
})
it('reports a single LUN 0', () => {
const data = buildReportLuns()
assert.equal(data.readUInt32BE(0), 8) // one 8-byte LUN entry
assert.deepEqual(data.subarray(8, 16), Buffer.alloc(8))
})
it('advertises supported VPD pages and rejects unknown ones', () => {
const page00 = buildVpd(0x00, IDENTITY)
assert.ok(page00 !== undefined)
assert.deepEqual([...page00.subarray(4)], [0x00, 0x80, 0x83, 0xb0])
assert.equal(buildVpd(0x55, IDENTITY), undefined)
})
it('advertises the max transfer length in the block limits page', () => {
const page = buildVpd(0xb0, IDENTITY, 2048)
assert.ok(page !== undefined)
assert.equal(page[1], 0xb0)
assert.equal(page.readUInt16BE(2), 0x3c)
assert.equal(page.readUInt32BE(8), 2048)
})
})
describe('handleScsiCommand', () => {
it('INQUIRY returns standard data trimmed to the allocation length', async () => {
const ctx = new FakeContext(new MemoryBlockDevice(1024 * 1024))
await handleScsiCommand(cdb(0x12, 0x00, 0x00, 0x00, 36), 1, ctx, IDENTITY)
assert.equal(ctx.reads.length, 1)
assert.equal(ctx.reads[0].data.length, 36)
assert.equal(ctx.reads[0].allocationLength, 36)
})
it('READ(10) returns the backing bytes for the requested blocks', async () => {
const lun = new MemoryBlockDevice(4096)
lun.buffer.fill(0xab, 512, 1024) // block 1
const ctx = new FakeContext(lun)
const buffer = Buffer.alloc(16)
buffer[0] = 0x28
buffer.writeUInt32BE(1, 2) // LBA 1
buffer.writeUInt16BE(1, 7) // 1 block
await handleScsiCommand(buffer, 1, ctx, IDENTITY)
assert.equal(ctx.reads.length, 1)
assert.deepEqual(ctx.reads[0].data, Buffer.alloc(512, 0xab))
})
it('WRITE(10) begins a solicited write at the right LUN offset and length', async () => {
const ctx = new FakeContext(new MemoryBlockDevice(4096))
const buffer = Buffer.alloc(16)
buffer[0] = 0x2a
buffer.writeUInt32BE(2, 2) // LBA 2
buffer.writeUInt16BE(3, 7) // 3 blocks
await handleScsiCommand(buffer, 7, ctx, IDENTITY)
// The data transfer + GOOD response are completed later by the connection.
assert.deepEqual(ctx.writes, [{ itt: 7, lunOffset: 2 * 512, totalLength: 3 * 512 }])
assert.equal(ctx.responses.length, 0)
})
it('WRITE(10) of zero blocks returns GOOD immediately with no transfer', async () => {
const ctx = new FakeContext(new MemoryBlockDevice(4096))
const buffer = Buffer.alloc(16)
buffer[0] = 0x2a // LBA 0, 0 blocks
await handleScsiCommand(buffer, 1, ctx, IDENTITY)
assert.equal(ctx.writes.length, 0)
assert.deepEqual(ctx.responses, [{ status: ScsiStatus.GOOD, options: undefined }])
})
it('rejects an out-of-range READ with CHECK CONDITION', async () => {
const ctx = new FakeContext(new MemoryBlockDevice(4096)) // 8 blocks
const buffer = Buffer.alloc(16)
buffer[0] = 0x28
buffer.writeUInt32BE(7, 2) // LBA 7
buffer.writeUInt16BE(4, 7) // 4 blocks -> past end
await handleScsiCommand(buffer, 1, ctx, IDENTITY)
assert.equal(ctx.responses.length, 1)
assert.equal(ctx.responses[0].status, ScsiStatus.CHECK_CONDITION)
assert.equal(ctx.responses[0].options?.sense?.[2], SenseKey.ILLEGAL_REQUEST)
})
it('reports a failed lun.read() as CHECK CONDITION / MEDIUM ERROR, not an uncaught throw', async () => {
const lun: BlockDevice = {
getSize: () => 4096,
getBlockSize: () => 512,
read: async () => {
throw new Error('backend read failed')
},
write: async () => {},
flush: async () => {},
close: async () => {},
}
const ctx = new FakeContext(lun)
const buffer = Buffer.alloc(16)
buffer[0] = 0x28
buffer.writeUInt32BE(1, 2) // LBA 1
buffer.writeUInt16BE(1, 7) // 1 block
await handleScsiCommand(buffer, 1, ctx, IDENTITY)
assert.equal(ctx.responses.length, 1)
assert.equal(ctx.responses[0].status, ScsiStatus.CHECK_CONDITION)
assert.equal(ctx.responses[0].options?.sense?.[2], SenseKey.MEDIUM_ERROR)
assert.equal(ctx.reads.length, 0) // sendReadData was never reached
})
it('returns CHECK CONDITION for an unsupported opcode', async () => {
const ctx = new FakeContext(new MemoryBlockDevice(4096))
await handleScsiCommand(cdb(0xff), 1, ctx, IDENTITY)
assert.equal(ctx.responses[0].status, ScsiStatus.CHECK_CONDITION)
assert.equal(ctx.responses[0].options?.sense?.[12], 0x20) // INVALID COMMAND OPERATION CODE
})
for (const [name, opcode] of [
['READ(10)', 0x28],
['WRITE(10)', 0x2a],
] as const) {
it(`refuses a ${name} longer than the max transfer length, without reading or soliciting data`, async () => {
let reads = 0
const lun = new MemoryBlockDevice(8192)
const read = lun.read.bind(lun)
lun.read = (offset, length) => {
reads++
return read(offset, length)
}
const ctx = new FakeContext(lun)
const buffer = Buffer.alloc(16)
buffer[0] = opcode
buffer.writeUInt16BE(3, 7) // 3 blocks, the limit is 2
await handleScsiCommand(buffer, 1, ctx, IDENTITY, 1024)
assert.equal(ctx.responses[0].status, ScsiStatus.CHECK_CONDITION)
assert.equal(ctx.responses[0].options?.sense?.[2], SenseKey.ILLEGAL_REQUEST)
assert.equal(ctx.responses[0].options?.sense?.[12], 0x24) // INVALID FIELD IN CDB
assert.equal(reads, 0)
assert.equal(ctx.writes.length, 0)
})
}
describe('read-only LUN', () => {
class ReadOnlyBlockDevice extends MemoryBlockDevice {
readonly readOnly = true
}
it('refuses WRITE(10) with DATA PROTECT, without soliciting its data', async () => {
const ctx = new FakeContext(new ReadOnlyBlockDevice(4096))
const buffer = Buffer.alloc(16)
buffer[0] = 0x2a
buffer.writeUInt16BE(1, 7)
await handleScsiCommand(buffer, 1, ctx, IDENTITY)
assert.equal(ctx.writes.length, 0)
assert.equal(ctx.responses[0].status, ScsiStatus.CHECK_CONDITION)
assert.equal(ctx.responses[0].options?.sense?.[2], SenseKey.DATA_PROTECT)
assert.equal(ctx.responses[0].options?.sense?.[12], 0x27) // WRITE PROTECTED
})
it('reports write protection in MODE SENSE(6) and MODE SENSE(10)', async () => {
const ctx = new FakeContext(new ReadOnlyBlockDevice(4096))
await handleScsiCommand(cdb(0x1a, 0x00, 0x3f, 0x00, 255), 1, ctx, IDENTITY)
await handleScsiCommand(cdb(0x5a, 0x00, 0x3f, 0, 0, 0, 0, 0x01, 0x00), 2, ctx, IDENTITY)
assert.equal(ctx.reads[0].data[2] & 0x80, 0x80)
assert.equal(ctx.reads[1].data[3] & 0x80, 0x80)
})
it('does not report a writable LUN as write-protected', async () => {
const ctx = new FakeContext(new MemoryBlockDevice(4096))
await handleScsiCommand(cdb(0x1a, 0x00, 0x3f, 0x00, 255), 1, ctx, IDENTITY)
assert.equal(ctx.reads[0].data[2] & 0x80, 0)
})
})
})

View File

@@ -0,0 +1,33 @@
/**
* A minimal counting semaphore: bounds how many `run()`-wrapped tasks execute
* at once, queueing the rest in FIFO order until a slot frees up.
*/
export class Semaphore {
#available: number
readonly #waiters: Array<() => void> = []
constructor(concurrency: number) {
if (!Number.isInteger(concurrency) || concurrency <= 0) {
throw new Error(`concurrency must be a positive integer, got ${concurrency}`)
}
this.#available = concurrency
}
async run<T>(fn: () => Promise<T>): Promise<T> {
if (this.#available > 0) {
this.#available--
} else {
await new Promise<void>(resolve => this.#waiters.push(resolve))
}
try {
return await fn()
} finally {
const next = this.#waiters.shift()
if (next !== undefined) {
next()
} else {
this.#available++
}
}
}
}

View File

@@ -0,0 +1,70 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { Semaphore } from './semaphore.mjs'
describe('Semaphore', () => {
it('rejects a non-positive-integer concurrency', () => {
assert.throws(() => new Semaphore(0), /positive integer/)
assert.throws(() => new Semaphore(-1), /positive integer/)
assert.throws(() => new Semaphore(1.5), /positive integer/)
})
it('runs tasks immediately while under the limit', async () => {
const semaphore = new Semaphore(2)
const started: Array<number> = []
await Promise.all(
[0, 1].map(i =>
semaphore.run(async () => {
started.push(i)
})
)
)
assert.deepEqual(started.sort(), [0, 1])
})
it('caps concurrency and admits the next task as soon as a slot frees', async () => {
const semaphore = new Semaphore(2)
let inFlight = 0
let maxInFlight = 0
const releases: Array<() => void> = []
const run = () =>
semaphore.run(async () => {
inFlight++
maxInFlight = Math.max(maxInFlight, inFlight)
await new Promise<void>(resolve => releases.push(resolve))
inFlight--
})
const tasks = [run(), run(), run(), run()]
// let the first batch (2, the concurrency limit) start
await new Promise(resolve => setImmediate(resolve))
assert.equal(inFlight, 2)
assert.equal(releases.length, 2)
while (releases.length > 0) {
releases.shift()!()
await new Promise(resolve => setImmediate(resolve))
}
await Promise.all(tasks)
assert.equal(maxInFlight, 2)
})
it('propagates a task failure without leaking its slot', async () => {
const semaphore = new Semaphore(1)
await assert.rejects(
semaphore.run(async () => {
throw new Error('boom')
}),
/boom/
)
// the slot must have been released despite the failure
let ran = false
await semaphore.run(async () => {
ran = true
})
assert.equal(ran, true)
})
})

View File

@@ -0,0 +1,24 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { addSerial, incrementSerial, SERIAL_MODULO } from './connection.mjs'
describe('iSCSI serial-number arithmetic', () => {
it('increments within range', () => {
assert.equal(incrementSerial(0), 1)
assert.equal(incrementSerial(41), 42)
})
it('wraps at 2^32', () => {
assert.equal(SERIAL_MODULO, 0x1_0000_0000)
assert.equal(incrementSerial(0xffffffff), 0)
assert.equal(addSerial(0xfffffffe, 5), 3)
assert.equal(addSerial(0xffffffff, 1), 0)
})
it('advertises a command window above the next expected CmdSN, wrapping cleanly', () => {
const expCmdSN = 0xfffffff0
const window = 64
assert.equal(addSerial(expCmdSN, window), (0xfffffff0 + 64) % SERIAL_MODULO)
})
})

View File

@@ -0,0 +1,244 @@
import assert from 'node:assert/strict'
import { execFile, execFileSync } from 'node:child_process'
import { mkdtemp, open as openFile, readFile, rm, truncate, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { setTimeout as delay } from 'node:timers/promises'
import { promisify } from 'node:util'
import { after, before, describe, it } from 'node:test'
import { FileBlockDevice, IscsiTarget } from './index.mjs'
const execFileP = promisify(execFile)
const IQN = 'iqn.2024-01.tech.vates:integ'
const LUN_SIZE = 8 * 1024 * 1024 // 8 MiB
const BLOCK_SIZE = 512
/**
* Determine why the open-iscsi interop test cannot run here (or `false` if it
* can). It needs root (to manage iSCSI sessions and flush block buffers) and the
* open-iscsi initiator — the exact software XCP-ng uses. As a lighter, root-free
* local alternative, the `libiscsi` userspace tools (`iscsi-inq`, `iscsi-ls`)
* can smoke-test discovery/INQUIRY without touching the kernel initiator.
*/
function skipReason(): string | false {
if (process.getuid === undefined || process.getuid() !== 0) {
return 'requires root to manage iSCSI sessions'
}
try {
execFileSync('iscsiadm', ['--version'], { stdio: 'ignore' })
} catch {
return 'requires open-iscsi (iscsiadm not found)'
}
return false
}
async function iscsiadm(args: string[]): Promise<string> {
const { stdout } = await execFileP('iscsiadm', args)
return stdout
}
/** Parse `iscsiadm -m session -P 3` for the attached SCSI disk's name (e.g. `sdb`) of our target. */
function parseAttachedDisk(sessionOutput: string): string | undefined {
const lines = sessionOutput.split('\n')
const targetLine = lines.findIndex(line => line.includes(IQN))
if (targetLine === -1) {
return undefined
}
for (let i = targetLine; i < lines.length; i++) {
const match = lines[i].match(/Attached scsi disk (sd\w+)/)
if (match !== null) {
return match[1]
}
}
return undefined
}
/**
* Neither the device node nor a nonzero `/sys/class/block/<name>/size` means
* the disk can be opened: the sd driver sets the size before
* `device_add_disk()`, which creates the sysfs entry, then the `/dev` node, then
* registers the block device: until then `open()` fails with ENOENT or ENXIO.
* Only an `open()` that succeeds means ready.
*/
async function isDeviceReady(name: string): Promise<boolean> {
try {
await (await openFile(`/dev/${name}`, 'r')).close()
return true
} catch (error) {
if (error instanceof Error && 'code' in error && (error.code === 'ENXIO' || error.code === 'ENOENT')) {
return false
}
throw error
}
}
async function waitForDevice(): Promise<string> {
for (let attempt = 0; attempt < 40; attempt++) {
const session = await iscsiadm(['-m', 'session', '-P', '3']).catch(() => '')
const name = parseAttachedDisk(session)
if (name !== undefined && (await isDeviceReady(name))) {
return `/dev/${name}`
}
await delay(250)
}
throw new Error('iSCSI block device did not appear')
}
describe('open-iscsi interop', { skip: skipReason() }, () => {
let dir: string
let backingPath: string
let target: IscsiTarget
let portal: string
before(async () => {
dir = await mkdtemp(join(tmpdir(), 'vates-iscsi-integ-'))
backingPath = join(dir, 'lun.img')
await writeFile(backingPath, Buffer.alloc(0))
await truncate(backingPath, LUN_SIZE)
target = new IscsiTarget({
iqn: IQN,
host: '127.0.0.1',
port: 0,
lun: new FileBlockDevice({ path: backingPath, blockSize: BLOCK_SIZE }),
})
await target.listen()
const address = target.address()
assert.ok(address !== undefined)
portal = `127.0.0.1:${address.port}`
})
after(async () => {
// Best-effort teardown: logout + drop the node record so the host is clean.
await iscsiadm(['-m', 'node', '-T', IQN, '-p', portal, '--logout']).catch(() => {})
await iscsiadm(['-m', 'node', '-o', 'delete', '-T', IQN, '-p', portal]).catch(() => {})
await target?.close()
await rm(dir, { recursive: true, force: true })
})
it('discovers, logs in, writes and reads back a LUN', async () => {
const discovery = await iscsiadm(['-m', 'discovery', '-t', 'sendtargets', '-p', portal])
assert.match(discovery, new RegExp(IQN), 'SendTargets discovery should list our IQN')
await iscsiadm(['-m', 'node', '-T', IQN, '-p', portal, '--login'])
const device = await waitForDevice()
// Write a recognizable pattern at block 10 through the kernel initiator.
const lba = 10
const offset = lba * BLOCK_SIZE
const pattern = Buffer.alloc(BLOCK_SIZE)
for (let i = 0; i < pattern.length; i++) {
pattern[i] = (i * 3 + 5) & 0xff
}
const writeHandle = await openFile(device, 'r+')
try {
await writeHandle.write(pattern, 0, pattern.length, offset)
await writeHandle.sync()
} finally {
await writeHandle.close()
}
// The WRITE path must have landed the bytes in the backing file.
const onDisk = await readFile(backingPath)
assert.deepEqual(onDisk.subarray(offset, offset + BLOCK_SIZE), pattern, 'backing file updated by WRITE')
// Drop the page cache so the read traverses the target (READ path).
await execFileP('blockdev', ['--flushbufs', device]).catch(() => {})
const readHandle = await openFile(device, 'r')
try {
const buffer = Buffer.alloc(BLOCK_SIZE)
await readHandle.read(buffer, 0, BLOCK_SIZE, offset)
assert.deepEqual(buffer, pattern, 'data read back through the target matches')
} finally {
await readHandle.close()
}
})
})
// The target as CHAP authenticator against the real open-iscsi initiator (the
// XCP-ng interop role). Credentials go into the node record's node.session.auth
// keys, exactly as XCP-ng's `device-config:chapuser`/`chappassword` do.
describe('open-iscsi CHAP interop', { skip: skipReason() }, () => {
// open-iscsi requires the initiator (outgoing) secret to be 12-16 characters.
const CHAP = { user: 'alice', secret: 'supersecret1234' }
let dir: string
let backingPath: string
let target: IscsiTarget
let portal: string
async function configureAuth(user: string, secret: string): Promise<void> {
const set = (name: string, value: string) =>
iscsiadm(['-m', 'node', '-T', IQN, '-p', portal, '-o', 'update', '-n', name, '-v', value])
await set('node.session.auth.authmethod', 'CHAP')
await set('node.session.auth.username', user)
await set('node.session.auth.password', secret)
}
before(async () => {
dir = await mkdtemp(join(tmpdir(), 'vates-iscsi-chap-integ-'))
backingPath = join(dir, 'lun.img')
await writeFile(backingPath, Buffer.alloc(0))
await truncate(backingPath, LUN_SIZE)
target = new IscsiTarget({
iqn: IQN,
host: '127.0.0.1',
port: 0,
lun: new FileBlockDevice({ path: backingPath, blockSize: BLOCK_SIZE }),
chap: CHAP,
})
await target.listen()
const address = target.address()
assert.ok(address !== undefined)
portal = `127.0.0.1:${address.port}`
// Discovery creates the node record we then attach auth to.
await iscsiadm(['-m', 'discovery', '-t', 'sendtargets', '-p', portal])
})
after(async () => {
await iscsiadm(['-m', 'node', '-T', IQN, '-p', portal, '--logout']).catch(() => {})
await iscsiadm(['-m', 'node', '-o', 'delete', '-T', IQN, '-p', portal]).catch(() => {})
await target?.close()
await rm(dir, { recursive: true, force: true })
})
it('rejects login with a wrong secret', async () => {
await configureAuth(CHAP.user, 'wrongsecret12345')
await assert.rejects(
iscsiadm(['-m', 'node', '-T', IQN, '-p', portal, '--login']),
'open-iscsi login must fail when the CHAP secret is wrong'
)
})
it('authenticates with the right secret and does IO', async () => {
await configureAuth(CHAP.user, CHAP.secret)
await iscsiadm(['-m', 'node', '-T', IQN, '-p', portal, '--login'])
const device = await waitForDevice()
const lba = 7
const offset = lba * BLOCK_SIZE
const pattern = Buffer.alloc(BLOCK_SIZE)
for (let i = 0; i < pattern.length; i++) {
pattern[i] = (i * 5 + 9) & 0xff
}
const writeHandle = await openFile(device, 'r+')
try {
await writeHandle.write(pattern, 0, pattern.length, offset)
await writeHandle.sync()
} finally {
await writeHandle.close()
}
await execFileP('blockdev', ['--flushbufs', device]).catch(() => {})
const readHandle = await openFile(device, 'r')
try {
const buffer = Buffer.alloc(BLOCK_SIZE)
await readHandle.read(buffer, 0, BLOCK_SIZE, offset)
assert.deepEqual(buffer, pattern, 'data read back over an authenticated session matches')
} finally {
await readHandle.close()
}
})
})

149
@vates/iscsi/src/types.mts Normal file
View File

@@ -0,0 +1,149 @@
import type { BlockDevice } from './backend.mjs'
import { ScsiOpcode, SERVICE_ACTION_READ_CAPACITY_16 } from './constants.mjs'
// --- Decoded SCSI requests --------------------------------------------------
/** A SCSI command we recognize, decoded from its CDB. */
export type ScsiRequest =
| { kind: 'inquiry'; evpd: boolean; pageCode: number; allocationLength: number }
| { kind: 'reportLuns'; allocationLength: number }
| { kind: 'readCapacity10' }
| { kind: 'readCapacity16'; allocationLength: number }
| { kind: 'read'; lba: number; blocks: number }
| { kind: 'write'; lba: number; blocks: number }
| { kind: 'testUnitReady' }
| { kind: 'requestSense'; allocationLength: number }
| { kind: 'modeSense'; long: boolean; pageCode: number; allocationLength: number }
| { kind: 'syncCache' }
| { kind: 'unsupported'; opcode: number }
/** Largest LBA representable as a safe JS integer; guards 64-bit CDB decoding. */
const MAX_SAFE_LBA = Number.MAX_SAFE_INTEGER
/**
* Read a 64-bit LBA, clamped to what a JS number holds exactly.
*
* An LBA above that is past the capacity of any device we can serve, so the
* clamp cannot collide with an addressable block: the caller's
* `lba + blocks > blockCount` check rejects it with LBA OUT OF RANGE, which is
* the answer SCSI wants anyway. Throwing here instead would take the whole
* connection down (the error unwinds to `serve()`, which destroys the socket)
* over a single bogus CDB.
*/
function readLba64(cdb: Buffer, offset: number): number {
const lba = cdb.readBigUInt64BE(offset)
return lba > BigInt(MAX_SAFE_LBA) ? MAX_SAFE_LBA : Number(lba)
}
/**
* Decode a SCSI CDB into a {@link ScsiRequest}. Unknown opcodes (and READ
* CAPACITY(16)'s sibling service actions we do not implement) map to
* `{ kind: 'unsupported' }` so the caller can return CHECK CONDITION. Never
* throws: every 16-byte CDB decodes to something the caller can answer.
*/
export function decodeCdb(cdb: Buffer): ScsiRequest {
const opcode = cdb[0]
switch (opcode) {
case ScsiOpcode.INQUIRY:
return {
kind: 'inquiry',
evpd: (cdb[1] & 0x01) !== 0,
pageCode: cdb[2],
allocationLength: cdb.readUInt16BE(3),
}
case ScsiOpcode.REPORT_LUNS:
return { kind: 'reportLuns', allocationLength: cdb.readUInt32BE(6) }
case ScsiOpcode.READ_CAPACITY_10:
return { kind: 'readCapacity10' }
case ScsiOpcode.SERVICE_ACTION_IN_16:
if ((cdb[1] & 0x1f) === SERVICE_ACTION_READ_CAPACITY_16) {
return { kind: 'readCapacity16', allocationLength: cdb.readUInt32BE(10) }
}
return { kind: 'unsupported', opcode }
case ScsiOpcode.READ_10:
return { kind: 'read', lba: cdb.readUInt32BE(2), blocks: cdb.readUInt16BE(7) }
case ScsiOpcode.WRITE_10:
return { kind: 'write', lba: cdb.readUInt32BE(2), blocks: cdb.readUInt16BE(7) }
case ScsiOpcode.READ_16:
return { kind: 'read', lba: readLba64(cdb, 2), blocks: cdb.readUInt32BE(10) }
case ScsiOpcode.WRITE_16:
return { kind: 'write', lba: readLba64(cdb, 2), blocks: cdb.readUInt32BE(10) }
case ScsiOpcode.TEST_UNIT_READY:
return { kind: 'testUnitReady' }
case ScsiOpcode.REQUEST_SENSE:
return { kind: 'requestSense', allocationLength: cdb[4] }
case ScsiOpcode.MODE_SENSE_6:
return { kind: 'modeSense', long: false, pageCode: cdb[2] & 0x3f, allocationLength: cdb[4] }
case ScsiOpcode.MODE_SENSE_10:
return { kind: 'modeSense', long: true, pageCode: cdb[2] & 0x3f, allocationLength: cdb.readUInt16BE(7) }
case ScsiOpcode.SYNCHRONIZE_CACHE_10:
case ScsiOpcode.SYNCHRONIZE_CACHE_16:
return { kind: 'syncCache' }
default:
return { kind: 'unsupported', opcode }
}
}
// --- Session / connection state shared across modules -----------------------
/** Operational parameters fixed during login that the SCSI layer depends on. */
export interface NegotiatedParams {
/** Initiator's MaxRecvDataSegmentLength — the cap on each Data-In PDU we send. */
initiatorMaxRecvDataSegmentLength: number
/** Max bytes solicited per R2T. */
maxBurstLength: number
}
export type SessionType = 'Discovery' | 'Normal'
/**
* A single one-way CHAP credential. On the target it is the incoming credential
* an initiator must present; on the initiator it is the credential presented to
* the target when challenged. Mutual (two-way) CHAP is not yet supported.
*/
export interface ChapCredentials {
/** CHAP username (`CHAP_N`). */
readonly user: string
/** CHAP shared secret. */
readonly secret: string
}
/** Sense data to attach to a CHECK CONDITION response. */
export interface SenseInfo {
key: number
asc: number
ascq: number
}
export interface ScsiResponseOptions {
/** Fixed-format sense data buffer (already serialized). */
sense?: Buffer
/** Residual byte counts for under/overflow reporting. */
residualOverflow?: number
residualUnderflow?: number
}
/**
* The capabilities the SCSI command handler needs from its connection: the LUN
* backend plus the three ways a command can complete (stream read data, begin a
* solicited write, or just return status). The connection implements this;
* keeping it an interface avoids a dependency cycle between `scsi` and
* `connection`.
*/
export interface CommandContext {
readonly lun: BlockDevice
/**
* Send `data` as Data-In PDU(s) with GOOD status piggybacked on the last,
* trimmed/residual-reported against `allocationLength`.
*/
sendReadData(itt: number, data: Buffer, allocationLength: number): Promise<void>
/**
* Begin a solicited write of `totalLength` bytes landing at `lunOffset`: the
* connection registers the transfer and sends the first R2T, then returns. The
* write executes and its SCSI Response is sent once the Data-Out PDUs arrive
* (which may be interleaved with other commands).
*/
beginWrite(itt: number, lunOffset: number, totalLength: number): Promise<void>
/** Send a standalone SCSI Response carrying `status` (+ optional sense/residual). */
sendScsiResponse(itt: number, status: number, options?: ScsiResponseOptions): Promise<void>
}

View File

@@ -0,0 +1,151 @@
import { FLAG_FINAL, TargetOpcode } from './constants.mjs'
import { allocBhs, assemblePdu } from './pdu.mjs'
/** Sequence numbers stamped into an outbound R2T (which does not advance StatSN). */
export interface SequenceSnapshot {
statSN: number
expCmdSN: number
maxCmdSN: number
}
/**
* Build a Ready To Transfer (R2T) PDU soliciting `desiredLength` bytes of write
* data starting at `bufferOffset`.
*/
export function buildR2t(
params: {
itt: number
r2tSN: number
targetTransferTag: number
bufferOffset: number
desiredLength: number
},
sequence: SequenceSnapshot
): Buffer {
const bhs = allocBhs(TargetOpcode.R2T)
bhs[1] = FLAG_FINAL // R2T always has the Final bit set
bhs.writeUInt32BE(params.itt, 16)
bhs.writeUInt32BE(params.targetTransferTag, 20)
bhs.writeUInt32BE(sequence.statSN, 24)
bhs.writeUInt32BE(sequence.expCmdSN, 28)
bhs.writeUInt32BE(sequence.maxCmdSN, 32)
bhs.writeUInt32BE(params.r2tSN, 36)
bhs.writeUInt32BE(params.bufferOffset, 40)
bhs.writeUInt32BE(params.desiredLength, 44)
return assemblePdu(bhs)
}
/**
* State of one in-progress WRITE: data is solicited one burst at a time (we pin
* MaxOutstandingR2T=1) and accumulated as Data-Out PDUs arrive, interleaved with
* any other commands on the connection.
*/
export interface PendingWrite {
/** Initiator Task Tag of the WRITE command; Data-Out PDUs are routed by it. */
readonly itt: number
/** Target Transfer Tag echoed in this command's R2Ts / Data-Out PDUs. */
readonly targetTransferTag: number
/** Byte offset within the LUN where the data must land. */
readonly lunOffset: number
/** Assembled write payload. */
readonly buffer: Buffer
/** Total bytes expected. */
readonly totalLength: number
/**
* Bytes of {@link buffer} filled, counted from offset 0 and contiguous by
* construction — see {@link receiveDataOut}. Never a plain count of bytes
* received off the wire, which is not the same thing.
*/
contiguousReceived: number
/** Bytes solicited so far (one burst ahead of `contiguousReceived`). */
solicited: number
/** Next R2TSN to emit. */
r2tSN: number
}
/** One inbound Data-Out PDU, reduced to what the write path needs from it. */
export interface DataOutSegment {
/** Target Transfer Tag echoed by the initiator (BHS bytes 20-23). */
readonly targetTransferTag: number
/** Where these bytes belong, relative to the start of the command's data (BHS bytes 40-43). */
readonly bufferOffset: number
readonly data: Buffer
}
/** The PDU was placed in the staging buffer. */
export interface PlacedDataOut {
readonly ok: true
/** Every byte of the command has landed: the write can be committed to the LUN. */
readonly complete: boolean
/** The solicited burst is fully received: the next one can be solicited. */
readonly burstComplete: boolean
}
/** The PDU cannot be placed; the command must be aborted rather than committed. */
export interface RefusedDataOut {
readonly ok: false
/** Why, for the log line accompanying the Reject. */
readonly reason: string
}
export type DataOutOutcome = PlacedDataOut | RefusedDataOut
/**
* Validate one Data-Out PDU against its pending write and, only if it belongs
* exactly where the buffer is still unfilled, copy it in. Validation and copy
* live together on purpose: the checks are what make the copy in-bounds, so a
* caller must not be able to do one without the other.
*
* A Data-Out must start exactly where the previous one ended. That strictness
* is what makes `contiguousReceived` mean "bytes of the buffer filled" — and
* only that meaning makes the completion test sound. Counting bytes as they
* arrive off the wire does not work, because `Buffer.copy` silently clamps:
*
* - 512 bytes at BufferOffset 512 of a 512-byte write copies *nothing*,
* - 512 bytes at BufferOffset 256 of a 512-byte write copies only 256,
* - the same 256 bytes sent twice copies 256 bytes over themselves.
*
* Each of those reaches `totalLength` bytes received with part of the buffer
* never written, so the target would commit a partly-applied write and answer
* GOOD — an acknowledged write silently losing data, which is worse than
* failing it. Before this was a contiguity check the staging buffer was also
* `allocUnsafe`, so the unwritten part leaked recycled heap to the initiator.
*
* Requiring contiguity is legitimate only because of what login declares in
* `#negotiateOperational`: `InitialR2T=Yes` (no unsolicited data, so every
* Data-Out is covered by an R2T and its TTT must match), `DataPDUInOrder=Yes`
* (RFC 7143: within a sequence, Data-Out PDUs MUST be in increasing
* BufferOffset order) and `MaxOutstandingR2T=1` (only one burst is ever
* solicited, so no other range can legitimately be in flight). Each of those
* is the winning side of its negotiation rule, so an initiator cannot talk us
* out of them. Should any of the three ever change, this must become per-burst
* range accounting instead of a single high-water mark.
*/
export function receiveDataOut(pending: PendingWrite, segment: DataOutSegment): DataOutOutcome {
const { targetTransferTag, bufferOffset, data } = segment
if (targetTransferTag !== pending.targetTransferTag) {
return {
ok: false,
reason: `Target Transfer Tag ${targetTransferTag} is not this command's ${pending.targetTransferTag}`,
}
}
if (bufferOffset !== pending.contiguousReceived) {
return { ok: false, reason: `BufferOffset ${bufferOffset} is not the expected ${pending.contiguousReceived}` }
}
// `solicited` only ever grows to at most `totalLength`, so bounding by it also
// keeps the copy inside the buffer — no clamping, no silent short copy.
if (bufferOffset + data.length > pending.solicited) {
return {
ok: false,
reason: `${data.length} bytes at ${bufferOffset} exceed the ${pending.solicited} bytes solicited`,
}
}
data.copy(pending.buffer, bufferOffset)
pending.contiguousReceived += data.length
return {
ok: true,
complete: pending.contiguousReceived === pending.totalLength,
burstComplete: pending.contiguousReceived === pending.solicited,
}
}

View File

@@ -0,0 +1,168 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { FLAG_FINAL, TargetOpcode } from './constants.mjs'
import { buildR2t, receiveDataOut, type PendingWrite } from './writePath.mjs'
const TTT = 7
const BLOCK = 512
/**
* A pending WRITE with its first burst already solicited, as `beginWrite`
* leaves it. `solicited` defaults to the whole command (one burst), which is
* what a write below MaxBurstLength gets.
*/
const pending = ({
totalLength = BLOCK,
solicited = totalLength,
}: { totalLength?: number; solicited?: number } = {}): PendingWrite => ({
itt: 1,
targetTransferTag: TTT,
lunOffset: 0,
buffer: Buffer.alloc(totalLength),
totalLength,
contiguousReceived: 0,
solicited,
r2tSN: 1,
})
const dataOut = (bufferOffset: number, length: number, fill = 0xab) => ({
targetTransferTag: TTT,
bufferOffset,
data: Buffer.alloc(length, fill),
})
describe('buildR2t', () => {
it('solicits the requested range, always with the final bit set', () => {
const r2t = buildR2t(
{ itt: 42, r2tSN: 3, targetTransferTag: TTT, bufferOffset: 1024, desiredLength: 512 },
{ statSN: 9, expCmdSN: 10, maxCmdSN: 74 }
)
assert.equal(r2t[0], TargetOpcode.R2T)
assert.equal(r2t[1], FLAG_FINAL)
assert.equal(r2t.readUInt32BE(16), 42)
assert.equal(r2t.readUInt32BE(20), TTT)
assert.equal(r2t.readUInt32BE(36), 3)
assert.equal(r2t.readUInt32BE(40), 1024)
assert.equal(r2t.readUInt32BE(44), 512)
})
})
describe('receiveDataOut', () => {
it('places a burst delivered in one PDU and reports it complete', () => {
const write = pending()
const outcome = receiveDataOut(write, dataOut(0, BLOCK))
assert.deepEqual(outcome, { ok: true, complete: true, burstComplete: true })
assert.equal(write.contiguousReceived, BLOCK)
assert.deepEqual(write.buffer, Buffer.alloc(BLOCK, 0xab))
})
it('places consecutive PDUs of one burst, completing only on the last', () => {
const write = pending({ totalLength: 3 * BLOCK })
const first = receiveDataOut(write, dataOut(0, BLOCK, 0x11))
const second = receiveDataOut(write, dataOut(BLOCK, BLOCK, 0x22))
const third = receiveDataOut(write, dataOut(2 * BLOCK, BLOCK, 0x33))
assert.deepEqual(first, { ok: true, complete: false, burstComplete: false })
assert.deepEqual(second, { ok: true, complete: false, burstComplete: false })
assert.deepEqual(third, { ok: true, complete: true, burstComplete: true })
assert.deepEqual(
write.buffer,
Buffer.concat([Buffer.alloc(BLOCK, 0x11), Buffer.alloc(BLOCK, 0x22), Buffer.alloc(BLOCK, 0x33)])
)
})
it('reports the burst complete, but not the command, when more remains to solicit', () => {
const write = pending({ totalLength: 2 * BLOCK, solicited: BLOCK })
const outcome = receiveDataOut(write, dataOut(0, BLOCK))
assert.deepEqual(outcome, { ok: true, complete: false, burstComplete: true })
})
// The three ways a plain count of received bytes reached `totalLength` with
// part of the staging buffer never written — each one an acknowledged write
// that silently lost data, since `Buffer.copy` clamps instead of throwing.
describe('refuses a PDU that would leave a hole in the buffer', () => {
it('past the end of the buffer, where the copy would land nothing', () => {
const write = pending()
const outcome = receiveDataOut(write, dataOut(BLOCK, BLOCK))
assert.equal(outcome.ok, false)
assert.equal(write.contiguousReceived, 0)
assert.deepEqual(write.buffer, Buffer.alloc(BLOCK))
})
it('straddling the end, where the copy would be silently short', () => {
const write = pending()
const outcome = receiveDataOut(write, dataOut(BLOCK / 2, BLOCK))
assert.equal(outcome.ok, false)
assert.equal(write.contiguousReceived, 0)
})
it('repeating a range already received, which would never fill the rest', () => {
const write = pending()
assert.equal(receiveDataOut(write, dataOut(0, BLOCK / 2, 0x11)).ok, true)
const outcome = receiveDataOut(write, dataOut(0, BLOCK / 2, 0x22))
assert.equal(outcome.ok, false)
// the first PDU's bytes are still there, and the rest is still unfilled
assert.equal(write.contiguousReceived, BLOCK / 2)
assert.deepEqual(write.buffer, Buffer.concat([Buffer.alloc(BLOCK / 2, 0x11), Buffer.alloc(BLOCK / 2)]))
})
it('skipping ahead, leaving a gap before it', () => {
const write = pending({ totalLength: 2 * BLOCK })
const outcome = receiveDataOut(write, dataOut(BLOCK, BLOCK))
assert.equal(outcome.ok, false)
assert.equal(write.contiguousReceived, 0)
})
})
it('refuses bytes beyond the current burst, even when the buffer could hold them', () => {
// 2 blocks solicited of a 4-block write: the last 2 are only legitimate
// after the next R2T, so accepting them now would break burst accounting
const write = pending({ totalLength: 4 * BLOCK, solicited: 2 * BLOCK })
const outcome = receiveDataOut(write, dataOut(0, 3 * BLOCK))
assert.equal(outcome.ok, false)
assert.equal(write.contiguousReceived, 0)
})
it("refuses a PDU carrying another command's Target Transfer Tag", () => {
const write = pending()
const outcome = receiveDataOut(write, { ...dataOut(0, BLOCK), targetTransferTag: TTT + 1 })
assert.equal(outcome.ok, false)
assert.equal(write.contiguousReceived, 0)
})
it('explains why it refused, for the log line next to the Reject', () => {
const outcome = receiveDataOut(pending(), dataOut(BLOCK, BLOCK))
assert.equal(outcome.ok, false)
assert.match(outcome.ok === false ? outcome.reason : '', /BufferOffset/)
})
it('accepts a zero-length PDU without disturbing the coverage it has', () => {
const write = pending()
receiveDataOut(write, dataOut(0, BLOCK / 2, 0x11))
const outcome = receiveDataOut(write, dataOut(BLOCK / 2, 0))
assert.deepEqual(outcome, { ok: true, complete: false, burstComplete: false })
assert.equal(write.contiguousReceived, BLOCK / 2)
})
})

View File

@@ -0,0 +1,14 @@
{
"compilerOptions": {
"target": "es2022",
"module": "ESNext",
"types": ["node"], // Only use types from Node.js and packages that are explicitly imported
"moduleResolution": "bundler",
"strict": true,
"skipLibCheck": true, // Skip type-checking dependency .d.ts (e.g. @xen-orchestra/disk-transform), as it does itself
"declaration": true, // Generates `.d.ts` files for type safety
"outDir": "dist" // Outputs compiled code to the `dist` directory
},
"include": ["src/**/*"],
"exclude": ["node_modules", "dist"]
}

View File

@@ -1,6 +1,12 @@
### `new NdbClient({address, exportname, secure = true, port = 10809})`
This package provides a NBD client, in two flavors sharing the same protocol
implementation (`AbstractNbdClient`): over the network, or over the standard
streams of a NBD server run as a child process.
create a new nbd client
### `new NbdClient({address, exportname, cert, port = 10809})`
Create a new nbd client connecting to a server over TCP.
`NbdClient` is also exported as `NbdTcpClient`.
```js
import NbdClient from '@vates/nbd-client'
@@ -14,3 +20,55 @@ await client.connect()
const block = await client.readBlock(blockIndex, BlockSize)
await client.disconnect()
```
### `new NbdStdioClient({command, args, exportname})`
Create a new nbd client talking to a server through its standard streams, like
`nbdkit --single` or any server reading its queries on stdin and writing its
answers on stdout.
The process is spawned by `connect()` and stopped by `disconnect()`.
```js
import { NbdStdioClient } from '@vates/nbd-client'
const client = new NbdStdioClient({
command: 'nbdkit',
args: ['--single', '--exit-with-parent', '--read-only', 'file', '/path/to/disk.raw'],
exportname: '', // optional, the default export of the server
})
await client.connect()
const block = await client.readBlock(blockIndex, BlockSize)
await client.disconnect()
```
There is no TLS here: the transport is a pair of pipes, there is nothing to
encrypt.
`readBlock()` retries on error, and a retry reconnects: with this client that
means killing the server and spawning a new one, so the command must be
restartable and must serve the same content on each run.
`getMap()` is not implemented, since `nbdinfo` needs either an URI or a server
supporting systemd socket activation: compute the map on the caller side and
pass it explicitly, for example `new NbdDisk(infos, blockSize, { dataMap })`.
The tail of the server stderr is available as `client.stderr` and attached to
the errors raised when the server dies unexpectedly.
### Writing another client
Subclass `AbstractNbdClient` and implement the transport:
- `_openTransport()`: **required**, returns `{ readable, writable }`, called on
every (re)connection.
- `_secureTransport(transport)`: optional, called during the handshake, the only
place where the transport can be upgraded (`NBD_OPT_STARTTLS`).
- `_closeTransport(transport, lastMessage)`: optional, must hand over
`lastMessage` (`NBD_CMD_DISC`) before closing.
- `_destroyTransport(transport)`: optional, forceful cleanup, must not throw.
- `getMap(signal)`: optional, the base implementation rejects with the code
`NBD_MAP_UNSUPPORTED`.
Any extra property set on the transport object is passed back to
`_closeTransport()` and `_destroyTransport()`.

View File

@@ -0,0 +1,586 @@
import assert from 'node:assert'
import { pRetry, pDelay, pTimeout, pFromCallback } from 'promise-toolbox'
import { readChunkStrict } from '@vates/read-chunk'
import { createLogger } from '@xen-orchestra/log'
import {
INIT_PASSWD,
NBD_CMD_DISC,
NBD_CMD_READ,
NBD_DEFAULT_BLOCK_SIZE,
NBD_FLAG_FIXED_NEWSTYLE,
NBD_FLAG_HAS_FLAGS,
NBD_OPT_EXPORT_NAME,
NBD_OPT_REPLY_MAGIC,
NBD_REPLY_ACK,
NBD_REPLY_MAGIC,
NBD_REQUEST_MAGIC,
OPTS_MAGIC,
} from './constants.mjs'
const { debug, warn } = createLogger('vates:nbd-client')
// documentation is here : https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md
/**
* A pair of streams carrying the NBD protocol.
*
* For a TCP client both are the same socket, for a client talking to a server
* through its standard streams they are the two ends of two different pipes.
*
* @typedef {object} NbdTransport
* @property {import('node:stream').Readable} readable - where the server answers are read from
* @property {import('node:stream').Writable} writable - where the client queries are written to
*/
/**
* Transport agnostic NBD client: it implements the handshake and the
* transmission phases, and delegates the opening/closing of the underlying
* streams to its subclasses.
*
* Subclasses MUST implement `_openTransport()` and MAY override
* `_secureTransport()`, `_closeTransport()`, `_destroyTransport()` and `getMap()`.
*
* @abstract
*/
export default class AbstractNbdClient {
#exportName
#exportSize
/** @type {NbdTransport|undefined} */
#transport
// incremented on each connection attempt: an attempt which is not the last
// one anymore has been superseded (its caller timed out on it) and must not
// publish itself
#connectGeneration = 0
// generation of the attempt which published the current transport, undefined
// when not connected: an error on a transport of another generation is stale
// and must be ignored
#connectedGeneration
#waitBeforeReconnect
#readBlockRetries
#reconnectRetry
#connectTimeout
#messageTimeout
// AFAIK, there is no guaranty the server answers in the same order as the queries
// so we handle a backlog of command waiting for response and handle concurrency manually
#waitingForResponse // there is already a listener waiting for a response
#nextCommandQueryId = BigInt(0)
// map of command waiting for a response queryId => { size/*in byte*/, resolve, reject}
#commandQueryBacklog = new Map()
#reconnectingPromise
/**
* @param {object} settings
* @param {string} [settings.exportname] - the NBD export to open, empty for the default one
* @param {object} [options]
* @param {number} [options.connectTimeout]
* @param {number} [options.messageTimeout]
* @param {number} [options.waitBeforeReconnect]
* @param {number} [options.readBlockRetries]
* @param {number} [options.reconnectRetry]
*/
constructor(
{ exportname = '' },
{
connectTimeout = 6e4,
messageTimeout = 6e4,
waitBeforeReconnect = 1e3,
readBlockRetries = 5,
reconnectRetry = 5,
} = {}
) {
this.#exportName = exportname
this.#waitBeforeReconnect = waitBeforeReconnect
this.#readBlockRetries = readBlockRetries
this.#reconnectRetry = reconnectRetry
this.#connectTimeout = connectTimeout
this.#messageTimeout = messageTimeout
}
get exportSize() {
return this.#exportSize
}
get exportName() {
return this.#exportName
}
get connected() {
return this.#connectedGeneration !== undefined
}
/* ---------------------------------------------------------------------------
* transport, to be implemented by the subclasses
* ------------------------------------------------------------------------- */
/**
* Open the streams used to talk to the NBD server.
*
* Called on every (re)connection: it must be able to open a brand new
* transport each time it is called.
*
* @abstract
* @returns {Promise<NbdTransport>}
*/
async _openTransport() {
throw new Error(`${this.constructor.name} must implement _openTransport()`)
}
/**
* Hook called during the handshake, after the client flags have been sent and
* before the export is selected: this is the only moment where the transport
* can be upgraded (NBD_OPT_STARTTLS).
*
* Returning another transport than the one received replaces it, returning
* the same one is a no-op.
*
* @param {NbdTransport} transport
* @returns {Promise<NbdTransport>}
*/
async _secureTransport(transport) {
return transport
}
/**
* Gracefully close the transport.
*
* `lastMessage` is a ready to send NBD_CMD_DISC: implementations must hand it
* over to the server (it is the only clean way to tell it we're done) before
* closing their side.
*
* Failing or timing out here is not fatal: `_destroyTransport()` will be
* called as a fallback.
*
* @param {NbdTransport} transport
* @param {Buffer} lastMessage
* @returns {Promise<void>}
*/
async _closeTransport(transport, lastMessage) {
await pFromCallback(cb => transport.writable.end(lastMessage, cb))
}
/**
* Forcefully release the transport and everything backing it.
*
* Must be idempotent, and must not throw.
*
* @param {NbdTransport} transport
* @returns {void}
*/
_destroyTransport(transport) {
transport.readable.destroy()
if (transport.writable !== transport.readable) {
transport.writable.destroy()
}
}
/**
* Returns the map of the file with holes, zeros and data, useful to handle
* efficiently sparse sources.
*
* The base implementation is unable to compute it: subclasses able to do so
* must override this method.
*
* To implement this here: use structured replies if the server supports them,
* and then ask for BLOCK_STATUS.
*
* @returns {Promise<{ offset: number, length: number, type: number }[]>}
* A promise that resolves to an array where each object represents a segment:
* - `offset` — The byte offset from the start.
* - `length` — The size of the segment in bytes.
* - `type` — A numeric code indicating the segment type ( 0 means data).
*/
async getMap() {
const error = new Error(`${this.constructor.name} does not support getMap()`)
error.code = 'NBD_MAP_UNSUPPORTED'
throw error
}
/* ---------------------------------------------------------------------------
* connection
* ------------------------------------------------------------------------- */
#onTransportError(generation, error) {
// without this listener an error on the transport would be thrown as an
// uncaught exception, and pending reads would only fail on message timeout
//
// an error on a transport which is not the current one is expected (a
// closed transport can be reset late by the other end, after a reconnection)
// and must not reject the reads of the current transport
//
// an error during the handshake is reported by the handshake reads
// themselves, through the connect() rejection
if (generation !== this.#connectedGeneration) {
debug('error on a stale nbd transport', { error })
return
}
warn('error on the nbd transport', { error })
this.#rejectAll(error)
}
// the generation is used instead of the transport itself since an attempt can
// own multiple transports (the raw one and the one upgraded to TLS)
#watchTransport(transport, generation) {
const onError = error => this.#onTransportError(generation, error)
transport.readable.on('error', onError)
if (transport.writable !== transport.readable) {
transport.writable.on('error', onError)
}
}
async #connect() {
const generation = ++this.#connectGeneration
// the transport is kept local until the client is connected: connect() does
// not cancel on timeout, so a superseded attempt must not be able to use,
// replace or destroy the transport of the attempt which replaced it
let transport = await this._openTransport()
this.#watchTransport(transport, generation)
try {
// the transport can be replaced during the handshake (TLS upgrade)
transport = await this.#handshake(transport, generation)
if (generation !== this.#connectGeneration) {
const error = new Error('this connection has been superseded by a newer one')
error.code = 'NBD_CONNECT_SUPERSEDED'
throw error
}
} catch (error) {
// don't leak the transport (a socket, a child process, ...): disconnect()
// is a no-op as long as we're not connected, so nobody else will close it
this.#destroyTransport(transport)
throw error
}
this.#transport = transport
this.#connectedGeneration = generation
// reset internal state if we reconnected a nbd client
this.#commandQueryBacklog = new Map()
this.#waitingForResponse = false
}
async connect() {
const promise = this.#connect()
try {
return await pTimeout.call(promise, this.#connectTimeout)
} catch (error) {
// pTimeout does not cancel: #connect() may still be running and end up
// with an opened transport nobody owns anymore
promise.then(
() =>
this.disconnect().catch(cleanupError => warn('error while cleaning up a late connection', { cleanupError })),
() => {} // the failure is already reported through `error`
)
throw error
}
}
async disconnect() {
if (this.#connectedGeneration === undefined) {
return
}
this.#connectedGeneration = undefined
const transport = this.#transport
this.#transport = undefined
if (transport === undefined) {
return
}
const queryId = this.#nextCommandQueryId
this.#nextCommandQueryId++
const buffer = Buffer.alloc(28)
buffer.writeInt32BE(NBD_REQUEST_MAGIC, 0) // it is a nbd request
buffer.writeInt16BE(0, 4) // no command flags for a disconnect
buffer.writeInt16BE(NBD_CMD_DISC, 6) // we want to disconnect from nbd server
buffer.writeBigUInt64BE(queryId, 8)
buffer.writeBigUInt64BE(0n, 16)
buffer.writeInt32BE(0, 24)
try {
await pTimeout.call(this._closeTransport(transport, buffer), this.#messageTimeout)
} catch (error) {
// expected when the other end is already gone, nothing actionable: the
// transport is destroyed instead of being closed gracefully
debug('error while closing the nbd transport, destroying it', { error })
this.#destroyTransport(transport)
}
}
#destroyTransport(transport) {
try {
this._destroyTransport(transport)
} catch (error) {
warn('error while destroying the nbd transport', { error })
}
}
#clearReconnectPromise = () => {
this.#reconnectingPromise = undefined
}
async #reconnect() {
await this.disconnect().catch(() => {})
await pDelay(this.#waitBeforeReconnect) // need to let the xapi clean things on its side
await this.connect()
}
async reconnect() {
// we need to ensure reconnections do not occur in parallel
if (this.#reconnectingPromise === undefined) {
this.#reconnectingPromise = pRetry(() => this.#reconnect(), {
tries: this.#reconnectRetry,
})
this.#reconnectingPromise.then(this.#clearReconnectPromise, this.#clearReconnectPromise)
}
return this.#reconnectingPromise
}
/* ---------------------------------------------------------------------------
* handshake
* ------------------------------------------------------------------------- */
/**
* Send an option to the server and check it acknowledged it.
*
* We can use individual read/write from the transport here since there is no
* concurrency during the handshake.
*
* The transport is passed explicitly since the client is not connected yet:
* it is only published once the handshake succeeded.
*
* protected: subclasses need it to negotiate their own options (NBD_OPT_STARTTLS)
*
* @param {NbdTransport} transport
* @param {number} option
* @param {Buffer} [buffer] - the payload of the option
*/
async _sendOption(transport, option, buffer = Buffer.alloc(0)) {
await this.#write(transport, OPTS_MAGIC)
await this.#writeInt32(transport, option)
await this.#writeInt32(transport, buffer.length)
await this.#write(transport, buffer)
assert.strictEqual(await this.#readInt64(transport), NBD_OPT_REPLY_MAGIC) // magic number everywhere
assert.strictEqual(await this.#readInt32(transport), option) // the option passed
assert.strictEqual(await this.#readInt32(transport), NBD_REPLY_ACK) // ACK
const length = await this.#readInt32(transport)
assert.strictEqual(length, 0) // length
}
// we can use individual read/write from the transport here since there is only one handshake at once, no concurrency
//
// it works on the transport it is given and returns the one to use for the
// transmission phase, which is not necessarily the same (TLS upgrade)
//
/**
* @param {NbdTransport} transport
* @param {number} generation - the connection attempt owning the transport
* @returns {Promise<NbdTransport>}
*/
async #handshake(transport, generation) {
assert((await this.#read(transport, 8)).equals(INIT_PASSWD))
assert((await this.#read(transport, 8)).equals(OPTS_MAGIC))
const flagsBuffer = await this.#read(transport, 2)
const flags = flagsBuffer.readInt16BE(0)
assert.strictEqual(flags & NBD_FLAG_FIXED_NEWSTYLE, NBD_FLAG_FIXED_NEWSTYLE) // only FIXED_NEWSTYLE one is supported from the server options
await this.#writeInt32(transport, NBD_FLAG_FIXED_NEWSTYLE) // client also support NBD_FLAG_C_FIXED_NEWSTYLE
// let the subclass upgrade the transport if it needs to (TLS)
const secured = await this._secureTransport(transport)
if (secured !== transport) {
this.#watchTransport(secured, generation)
transport = secured
}
// send export name we want to access.
// it's implicitly closing the negotiation phase.
await this.#write(transport, OPTS_MAGIC)
await this.#writeInt32(transport, NBD_OPT_EXPORT_NAME)
const exportNameBuffer = Buffer.from(this.#exportName)
await this.#writeInt32(transport, exportNameBuffer.length)
await this.#write(transport, exportNameBuffer)
// 8 (export size ) + 2 (flags) + 124 zero = 134
// must read all to ensure nothing stays in the buffer
const answer = await this.#read(transport, 134)
this.#exportSize = answer.readBigUInt64BE(0)
const transmissionFlags = answer.readInt16BE(8)
assert.strictEqual(transmissionFlags & NBD_FLAG_HAS_FLAGS, NBD_FLAG_HAS_FLAGS, 'NBD_FLAG_HAS_FLAGS') // must always be 1 by the norm
// note : xapi server always send NBD_FLAG_READ_ONLY (3) as a flag
return transport
}
/* ---------------------------------------------------------------------------
* raw read/write
* ------------------------------------------------------------------------- */
#getTransport() {
const transport = this.#transport
if (transport === undefined) {
const error = new Error('the nbd client is not connected')
error.code = 'NBD_NOT_CONNECTED'
throw error
}
return transport
}
#read(transport, length) {
const promise = readChunkStrict(transport.readable, length)
return pTimeout.call(promise, this.#messageTimeout)
}
#write(transport, buffer) {
let timeout
const messageTimeout = this.#messageTimeout
const { writable } = transport
return Promise.race([
new Promise((resolve, reject) => {
timeout = setTimeout(() => {
reject(new Error('timeout'))
}, messageTimeout)
}),
new Promise(resolve =>
writable.write(buffer, () => {
clearTimeout(timeout)
resolve()
})
),
])
}
async #readInt32(transport) {
const buffer = await this.#read(transport, 4)
return buffer.readInt32BE(0)
}
async #readInt64(transport) {
const buffer = await this.#read(transport, 8)
return buffer.readBigUInt64BE(0)
}
#writeInt32(transport, int) {
const buffer = Buffer.alloc(4)
buffer.writeInt32BE(int)
return this.#write(transport, buffer)
}
/* ---------------------------------------------------------------------------
* transmission
* ------------------------------------------------------------------------- */
// when one read fail ,stop everything
// (sync: it must be usable from an event handler without leaving a floating promise)
#rejectAll(error) {
this.#commandQueryBacklog.forEach(({ reject }) => {
reject(error)
})
}
async #readBlockResponse() {
// ensure at most one read occur in parallel
if (this.#waitingForResponse) {
return
}
try {
this.#waitingForResponse = true
const transport = this.#getTransport()
const buffer = await this.#read(transport, 16)
const magic = buffer.readInt32BE(0)
if (magic !== NBD_REPLY_MAGIC) {
throw new Error(`magic number for block answer is wrong : ${magic} ${NBD_REPLY_MAGIC}`)
}
const error = buffer.readInt32BE(4)
if (error !== 0) {
// @todo use error code from constants.mjs
throw new Error(`GOT ERROR CODE : ${error}`)
}
const blockQueryId = buffer.readBigUInt64BE(8)
const query = this.#commandQueryBacklog.get(blockQueryId)
if (!query) {
throw new Error(` no query associated with id ${blockQueryId}`)
}
this.#commandQueryBacklog.delete(blockQueryId)
const data = await this.#read(transport, query.size)
query.resolve(data)
this.#waitingForResponse = false
if (this.#commandQueryBacklog.size > 0) {
// it doesn't throw directly but will throw all relevant promise on failure
this.#readBlockResponse()
}
} catch (error) {
// reject all the promises
// we don't need to call readBlockResponse on failure
// since we will empty the backlog
this.#rejectAll(error)
}
}
async #readBlock(index, size) {
// we don't want to add anything in backlog while reconnecting
if (this.#reconnectingPromise) {
await this.#reconnectingPromise
}
const queryId = this.#nextCommandQueryId
this.#nextCommandQueryId++
// create and send command at once to ensure there is no concurrency issue
const buffer = Buffer.alloc(28)
buffer.writeInt32BE(NBD_REQUEST_MAGIC, 0) // it is a nbd request
buffer.writeInt16BE(0, 4) // no command flags for a simple block read
buffer.writeInt16BE(NBD_CMD_READ, 6) // we want to read a data block
buffer.writeBigUInt64BE(queryId, 8)
// byte offset in the raw disk
const offset = BigInt(index) * BigInt(size)
const remaining = this.#exportSize - offset
if (remaining < BigInt(size)) {
size = Number(remaining)
}
buffer.writeBigUInt64BE(offset, 16)
buffer.writeInt32BE(size, 24)
return new Promise((resolve, reject) => {
function decoratedReject(error) {
error.index = index
error.size = size
reject(error)
}
// this will handle one block response, but it can be another block
// since server does not guaranty to handle query in order
this.#commandQueryBacklog.set(queryId, {
size,
resolve,
reject: decoratedReject,
})
// really send the command to the server
this.#write(this.#getTransport(), buffer).catch(decoratedReject)
// #readBlockResponse never throws directly
// but if it fails it will reject all the promises in the backlog
this.#readBlockResponse()
})
}
async readBlock(index, size = NBD_DEFAULT_BLOCK_SIZE) {
return pRetry(() => this.#readBlock(index, size), {
tries: this.#readBlockRetries,
when: error => error.code !== 'ERR_ABORTED',
onRetry: async err => {
warn('will retry reading block ', index, err)
await this.reconnect()
},
})
}
}

View File

@@ -0,0 +1,199 @@
import assert from 'node:assert/strict'
import { PassThrough } from 'node:stream'
import { setTimeout as pSleep } from 'node:timers/promises'
import { describe, it } from 'node:test'
import AbstractNbdClient from './AbstractNbdClient.mjs'
import { serveNbd } from './tests/fake-nbd-server.mjs'
const BLOCK_SIZE = 512
// non aligned size, to check the truncated last block
const DATA = Buffer.alloc(BLOCK_SIZE * 3 + 100)
for (let i = 0; i < DATA.length; i += 4) {
DATA.writeUInt32BE(i, i)
}
// exercises AbstractNbdClient without any I/O: the fake server runs in the same
// process, on the other end of two in-memory streams
class InMemoryNbdClient extends AbstractNbdClient {
#serverOptions
/** @type {Array<object>} */
transports = []
/** @type {Array<object>} */
destroyedTransports = []
secureTransportCalls = 0
constructor({ exportname, ...serverOptions } = {}, options) {
super({ exportname }, options)
this.#serverOptions = serverOptions
}
async _openTransport() {
const toServer = new PassThrough()
const toClient = new PassThrough()
const transport = { readable: toClient, writable: toServer }
this.transports.push(transport)
// the server runs alongside the client, a failure on its side is reported
// to the client the same way a network error would be
serveNbd({ readable: toServer, writable: toClient, data: DATA, ...this.#serverOptions }).catch(error =>
toClient.destroy(error)
)
return transport
}
async _secureTransport(transport) {
this.secureTransportCalls++
return super._secureTransport(transport)
}
_destroyTransport(transport) {
this.destroyedTransports.push(transport)
super._destroyTransport(transport)
}
}
// the first connection is opened on a transport nobody ever answers on: its
// handshake stalls until the message timeout, well after connect() gave up
class StallingFirstConnectClient extends InMemoryNbdClient {
#openCalls = 0
async _openTransport() {
if (this.#openCalls++ === 0) {
const transport = { readable: new PassThrough(), writable: new PassThrough() }
this.transports.push(transport)
return transport
}
return super._openTransport()
}
}
describe('AbstractNbdClient', () => {
it('handshakes and reads blocks', async () => {
const client = new InMemoryNbdClient()
await client.connect()
try {
assert.equal(client.exportSize, BigInt(DATA.length))
assert.equal(client.connected, true)
const block = await client.readBlock(1, BLOCK_SIZE)
assert.ok(block.equals(DATA.subarray(BLOCK_SIZE, 2 * BLOCK_SIZE)))
} finally {
await client.disconnect()
}
assert.equal(client.connected, false)
})
it('truncates the last block of a non aligned export', async () => {
const client = new InMemoryNbdClient()
await client.connect()
try {
const block = await client.readBlock(3, BLOCK_SIZE)
assert.equal(block.length, DATA.length - 3 * BLOCK_SIZE)
assert.ok(block.equals(DATA.subarray(3 * BLOCK_SIZE)))
} finally {
await client.disconnect()
}
})
it('handles answers coming out of order', async () => {
const client = new InMemoryNbdClient({ answerInReverse: true })
await client.connect()
try {
const [first, second] = await Promise.all([client.readBlock(0, BLOCK_SIZE), client.readBlock(1, BLOCK_SIZE)])
assert.ok(first.equals(DATA.subarray(0, BLOCK_SIZE)))
assert.ok(second.equals(DATA.subarray(BLOCK_SIZE, 2 * BLOCK_SIZE)))
} finally {
await client.disconnect()
}
})
it('calls _secureTransport during the handshake', async () => {
const client = new InMemoryNbdClient()
await client.connect()
assert.equal(client.secureTransportCalls, 1)
await client.disconnect()
})
it('does not leak the transport when the handshake fails', async () => {
// the server expects another export name, and fails the handshake
const client = new InMemoryNbdClient({ exportname: 'wrong', exportName: 'expected' })
await assert.rejects(client.connect())
assert.equal(client.connected, false)
assert.equal(client.transports.length, 1)
assert.deepEqual(client.destroyedTransports, client.transports)
assert.equal(client.transports[0].writable.destroyed, true)
})
it('reports the errors answered by the server, after having retried', async () => {
const client = new InMemoryNbdClient(
{ errorCode: () => 5 /* EIO */ },
{ readBlockRetries: 2, reconnectRetry: 1, waitBeforeReconnect: 0 }
)
await client.connect()
try {
await assert.rejects(client.readBlock(0, BLOCK_SIZE), /ERROR CODE/)
// one transport for the initial connection, one for the retry
assert.equal(client.transports.length, 2)
} finally {
await client.disconnect()
}
})
it('does not let a connection which timed out disturb the next one', async () => {
const client = new StallingFirstConnectClient({}, { connectTimeout: 50, messageTimeout: 150 })
await assert.rejects(client.connect())
// the previous attempt is still running: it must not publish itself, nor
// touch the transport of this one
await client.connect()
try {
// let the stalled handshake reach its message timeout
await pSleep(250)
assert.equal(client.connected, true)
assert.equal(client.transports.length, 2)
// only the stalled transport has been destroyed
assert.deepEqual(client.destroyedTransports, [client.transports[0]])
const block = await client.readBlock(0, BLOCK_SIZE)
assert.ok(block.equals(DATA.subarray(0, BLOCK_SIZE)))
} finally {
await client.disconnect()
}
})
it('does not let an error on a superseded transport disturb the current one', async () => {
const client = new InMemoryNbdClient({}, { readBlockRetries: 1, waitBeforeReconnect: 0 })
await client.connect()
await client.reconnect()
try {
assert.equal(client.transports.length, 2)
const [supersededTransport] = client.transports
// the read is queued synchronously, the in memory server answers later
const promise = client.readBlock(0, BLOCK_SIZE)
// the other end of the closed transport resets it late
supersededTransport.readable.emit('error', Object.assign(new Error('read ECONNRESET'), { code: 'ECONNRESET' }))
const block = await promise
assert.ok(block.equals(DATA.subarray(0, BLOCK_SIZE)))
// no reconnection has been triggered
assert.equal(client.transports.length, 2)
} finally {
await client.disconnect()
}
})
it('does not support getMap()', async () => {
const client = new InMemoryNbdClient()
await assert.rejects(client.getMap(), ({ code }) => code === 'NBD_MAP_UNSUPPORTED')
})
it('can be disconnected without being connected', async () => {
await new InMemoryNbdClient().disconnect()
})
})

View File

@@ -5,12 +5,18 @@ import NbdClient from '@vates/nbd-client'
* @typedef {{ offset: number, length: number }} DataRange
*/
/**
* any NBD client, whatever transport it uses
*
* @typedef {import('@vates/nbd-client').AbstractNbdClient} AbstractNbdClient
*/
/**
* @extends {RandomAccessDisk}
*/
export class NbdDisk extends RandomAccessDisk {
#nbdInfos
/** @type {NbdClient|undefined} */
/** @type {AbstractNbdClient|undefined} */
#nbdClient
/** @type {Array<DataRange> | undefined} */
@@ -25,10 +31,21 @@ export class NbdDisk extends RandomAccessDisk {
/** @type {number | undefined} */
#hasBlockPreviousIndex
constructor(nbdInfos, blockSize, { dataMap } = {}) {
/** @type {typeof AbstractNbdClient} */
#ClientClass
/**
* @param {object} nbdInfos - the settings of the client, depends on `ClientClass`
* @param {number} blockSize
* @param {object} [options]
* @param {Array<{offset: number, length: number, type: number}>} [options.dataMap] - computed through `getMap()` when not provided
* @param {typeof AbstractNbdClient} [options.ClientClass] - to talk to a NBD server through another transport
*/
constructor(nbdInfos, blockSize, { dataMap, ClientClass = NbdClient } = {}) {
super()
this.#blockSize = blockSize
this.#nbdInfos = nbdInfos
this.#ClientClass = ClientClass
this.#dataMap = dataMap && this.#processDatamap(dataMap)
}
@@ -111,7 +128,7 @@ export class NbdDisk extends RandomAccessDisk {
* @returns {Promise<void>}
*/
async init() {
this.#nbdClient = new NbdClient(this.#nbdInfos)
this.#nbdClient = new this.#ClientClass(this.#nbdInfos)
await this.#nbdClient.connect()
if (this.#dataMap === undefined) {
this.#dataMap = this.#processDatamap(await this.#nbdClient.getMap())

View File

@@ -0,0 +1,157 @@
import assert from 'node:assert'
import { spawn } from 'node:child_process'
import { pFromCallback } from 'promise-toolbox'
import { createLogger } from '@xen-orchestra/log'
import AbstractNbdClient from './AbstractNbdClient.mjs'
const { warn } = createLogger('vates:nbd-client:stdio')
// only the tail is kept: it's the part describing why the server died
const STDERR_MAX_LENGTH = 4096
/**
* NBD client talking to a server through its standard streams, like
* `nbdkit -s` or `qemu-nbd` in `--fd` mode: the queries are written to the
* server stdin and its answers are read from its stdout.
*
* The process is spawned on `connect()` and killed on `disconnect()`, and
* `reconnect()` (used by `readBlock()` retries) spawns a brand new one: the
* server must be restartable, and must serve the same content on each run.
*
* There is no TLS here: the transport is a pair of pipes, nothing to encrypt.
*
* @extends {AbstractNbdClient}
*/
export default class NbdStdioClient extends AbstractNbdClient {
#args
#command
#cwd
#env
/** @type {{ child: import('node:child_process').ChildProcess, closing: boolean, stderr: string }|undefined} */
#state
/**
* @param {object} settings
* @param {string} settings.command - the NBD server to run
* @param {ReadonlyArray<string>} [settings.args] - its arguments
* @param {string} [settings.exportname] - empty (the default export) for most single connection servers
* @param {NodeJS.ProcessEnv} [settings.env]
* @param {string} [settings.cwd]
* @param {object} [options] - see {@link AbstractNbdClient}
*/
constructor({ command, args = [], exportname, env, cwd }, options) {
super({ exportname }, options)
assert.strictEqual(typeof command, 'string', 'command must be a string')
this.#command = command
this.#args = args
this.#env = env
this.#cwd = cwd
}
/**
* the tail of what the server wrote on stderr since it was spawned, useful to
* report why it failed
*
* @returns {string}
*/
get stderr() {
return this.#state?.stderr ?? ''
}
/**
* the pid of the current server process, undefined if it's not running
*
* @returns {number|undefined}
*/
get pid() {
return this.#state?.child.pid
}
async _openTransport() {
const child = spawn(this.#command, this.#args, {
cwd: this.#cwd,
env: this.#env,
// the protocol goes through stdin/stdout, stderr is only used for diagnostics
stdio: ['pipe', 'pipe', 'pipe'],
})
const state = { child, closing: false, stderr: '' }
this.#state = state
child.stderr.setEncoding('utf8')
child.stderr.on('data', chunk => {
state.stderr = (state.stderr + chunk).slice(-STDERR_MAX_LENGTH)
})
// spawn failures (ENOENT, EACCES, ...) are reported asynchronously
await new Promise((resolve, reject) => {
const onError = error => {
error.command = this.#command
reject(error)
}
child.once('error', onError)
child.once('spawn', () => {
child.removeListener('error', onError)
// from now on, an error on the process itself (a failed kill for
// example) must not be thrown as an uncaught exception
child.on('error', error => warn('error on the nbd server process', { error, command: this.#command }))
resolve()
})
})
child.once('exit', (code, signal) => {
if (state.closing) {
// expected: disconnect() asked it to quit and is waiting for this
return
}
const error = new Error(`the nbd server process exited (code: ${code}, signal: ${signal})`)
error.code = 'NBD_SERVER_EXITED'
error.exitCode = code
error.signal = signal
error.command = this.#command
error.stderr = state.stderr
warn('the nbd server process exited unexpectedly', { error })
// make the pending and future reads fail with this error instead of
// waiting for the message timeout on a stream that will never answer
child.stdout.destroy(error)
child.stdin.destroy()
})
return { readable: child.stdout, writable: child.stdin, state }
}
async _closeTransport(transport, lastMessage) {
const { state } = transport
state.closing = true
// hand over NBD_CMD_DISC, then close stdin: that's how the server knows
// it can quit
await pFromCallback(cb => transport.writable.end(lastMessage, cb))
// don't let the server block on a write while we wait for its exit
transport.readable.resume()
const { child } = state
if (child.exitCode === null && child.signalCode === null) {
await new Promise(resolve => child.once('exit', () => resolve()))
}
}
_destroyTransport(transport) {
const { state } = transport
const { child } = state
// we're the one tearing it down: the exit that follows is expected, not a
// crash, and destroying the streams can be enough to make the server quit
state.closing = true
transport.readable.destroy()
transport.writable.destroy()
if (child.exitCode === null && child.signalCode === null) {
child.kill('SIGKILL')
}
}
// getMap() is not implemented: `nbdinfo` needs an URI or a server supporting
// systemd socket activation, and this one only speaks through its standard
// streams. Compute the map on the caller side and pass it explicitly (
// `new NbdDisk(infos, blockSize, { dataMap })`).
}

View File

@@ -0,0 +1,197 @@
import { once } from 'node:events'
import { request as httpRequest } from 'node:http'
import { request as httpsRequest } from 'node:https'
import { isIPv6, Socket } from 'node:net'
import { connect } from 'node:tls'
import { spawn } from 'node:child_process'
import AbstractNbdClient from './AbstractNbdClient.mjs'
import { NBD_DEFAULT_PORT, NBD_OPT_STARTTLS } from './constants.mjs'
/**
* NBD client talking to a server over TCP, optionally upgrading the connection
* to TLS during the handshake.
*
* @extends {AbstractNbdClient}
*/
export default class NbdTcpClient extends AbstractNbdClient {
#httpProxy
#proxyRejectUnauthorized
#proxyTimeout
#serverAddress
#serverCert
#serverPort
/**
* @param {object} settings
* @param {string} settings.address
* @param {number} [settings.port]
* @param {string} [settings.exportname]
* @param {string} [settings.cert] - PEM certificate, enables TLS when set
* @param {string} [settings.httpProxy] - URL of an HTTP(S) proxy, the connection is tunneled through it with `CONNECT`
* @param {boolean} [settings.proxyRejectUnauthorized=true] - whether to check the certificate of an HTTPS proxy
* @param {number} [settings.proxyTimeout=6e4] - delay in ms after which an unanswered `CONNECT` is aborted
* @param {object} [options] - see {@link AbstractNbdClient}
*/
constructor(
{
address,
port = NBD_DEFAULT_PORT,
exportname,
cert,
httpProxy,
proxyRejectUnauthorized = true,
proxyTimeout = 6e4,
},
options
) {
super({ exportname }, options)
// other proxy protocols (e.g. SOCKS) are not supported for now
if (httpProxy !== undefined) {
const { protocol } = new URL(httpProxy)
if (protocol !== 'http:' && protocol !== 'https:') {
const error = new Error(`unsupported proxy protocol ${protocol}, only http: and https: are supported`)
error.code = 'NBD_PROXY_UNSUPPORTED_PROTOCOL'
throw error
}
}
this.#serverAddress = address
this.#serverPort = port
this.#serverCert = cert
this.#httpProxy = httpProxy
this.#proxyRejectUnauthorized = proxyRejectUnauthorized
this.#proxyTimeout = proxyTimeout
}
/**
* Open a raw TCP tunnel to the NBD server through an HTTP proxy
*
* https://datatracker.ietf.org/doc/html/rfc9110#section-9.3.6
*
* @returns {Promise<import('node:net').Socket>}
*/
async #connectThroughHttpProxy() {
const proxy = new URL(this.#httpProxy)
const isHttps = proxy.protocol === 'https:'
const address = this.#serverAddress
const target = `${isIPv6(address) ? `[${address}]` : address}:${this.#serverPort}`
const headers = { host: target }
if (proxy.username !== '' || proxy.password !== '') {
const credentials = `${decodeURIComponent(proxy.username)}:${decodeURIComponent(proxy.password)}`
headers['proxy-authorization'] = `Basic ${Buffer.from(credentials).toString('base64')}`
}
const req = (isHttps ? httpsRequest : httpRequest)({
agent: false,
headers,
// URL keeps the brackets around IPv6 addresses
hostname: proxy.hostname.replace(/^\[(.*)\]$/, '$1'),
method: 'CONNECT',
path: target,
port: proxy.port,
rejectUnauthorized: this.#proxyRejectUnauthorized,
timeout: this.#proxyTimeout,
})
// `timeout` only emits an event, the request must be aborted explicitly
req.on('timeout', () => {
const error = new Error(`HTTP proxy did not answer the CONNECT to ${target} in time`)
error.code = 'NBD_PROXY_CONNECT_TIMEOUT'
req.destroy(error)
})
req.end()
// the NBD server may already have sent some data, it is in `head`
const [res, socket, head] = await once(req, 'connect')
// the tunnel is established, NBD has its own timeouts
socket.setTimeout(0)
if (res.statusCode !== 200) {
socket.destroy()
const error = new Error(`HTTP proxy refused to connect to ${target}: ${res.statusCode} ${res.statusMessage}`)
error.code = 'NBD_PROXY_CONNECT_FAILED'
throw error
}
if (head.length !== 0) {
socket.unshift(head)
}
return socket
}
// mandatory , at least to start the handshake: the connection always starts
// unsecured, and is upgraded to TLS during the handshake
async _openTransport() {
if (this.#httpProxy !== undefined) {
const socket = await this.#connectThroughHttpProxy()
return { readable: socket, writable: socket }
}
const socket = new Socket()
await new Promise((resolve, reject) => {
socket.connect(this.#serverPort, this.#serverAddress)
socket.once('error', reject)
socket.once('connect', () => {
socket.removeListener('error', reject)
resolve()
})
})
return { readable: socket, writable: socket }
}
async _secureTransport(transport) {
if (this.#serverCert === undefined) {
return transport
}
await this._sendOption(transport, NBD_OPT_STARTTLS)
const secured = await new Promise((resolve, reject) => {
const socket = connect({
socket: transport.writable,
rejectUnauthorized: false,
cert: this.#serverCert,
})
socket.once('error', reject)
socket.once('secureConnect', () => {
socket.removeListener('error', reject)
resolve(socket)
})
})
return { readable: secured, writable: secured }
}
/**
* @param {AbortSignal} [signal]
* @returns {Promise<{ offset: number, length: number, type: number }[]>}
*/
/* async */ getMap(signal) {
return new Promise((resolve, reject) => {
const process = spawn('nbdinfo', [
'--json',
'--map',
`nbd://${this.#serverAddress}:${this.#serverPort}/${encodeURIComponent(this.exportName)}`,
])
let text = ''
let errText = ''
process.stdout.on('data', data => (text += data))
process.stderr.on('data', data => (errText += data))
const onAbort = () => {
process.kill()
reject(signal.reason)
}
signal?.addEventListener('abort', onAbort, { once: true })
process.on('error', err => {
signal?.removeEventListener('abort', onAbort)
reject(err)
})
process.on('close', code => {
signal?.removeEventListener('abort', onAbort)
if (code !== 0) {
return reject(new Error(`Error during getMap (code: ${code}): ${errText}`))
}
try {
const json = JSON.parse(text)
resolve(json)
} catch (error) {
reject(new Error(`${error} ${text}`))
}
})
})
}
}

View File

@@ -0,0 +1,107 @@
import assert from 'node:assert/strict'
import { after, before, describe, it } from 'node:test'
import { createServer as createHttpServer } from 'node:http'
import { connect, createServer as createTcpServer } from 'node:net'
import { once } from 'node:events'
import NbdTcpClient from './NbdTcpClient.mjs'
import { serveNbd } from './tests/fake-nbd-server.mjs'
const BLOCK_SIZE = 64 * 1024
const DATA = Buffer.alloc(4 * BLOCK_SIZE)
for (let i = 0; i < DATA.length; i++) {
DATA[i] = i % 251
}
const CREDENTIALS = 'Basic ' + Buffer.from('user:p@ss').toString('base64')
describe('NbdTcpClient through an HTTP proxy', () => {
let nbdServer, proxyServer, nbdPort, proxyPort
const connectRequests = []
before(async () => {
nbdServer = createTcpServer(socket => {
serveNbd({ readable: socket, writable: socket, data: DATA }).catch(() => {})
})
nbdServer.listen(0, '127.0.0.1')
await once(nbdServer, 'listening')
nbdPort = nbdServer.address().port
// minimal CONNECT proxy, similar to the XO Proxy one
proxyServer = createHttpServer()
proxyServer.on('connect', (req, clientSocket, head) => {
connectRequests.push(req.url)
if (req.headers['proxy-authorization'] !== CREDENTIALS) {
clientSocket.end('HTTP/1.1 407 Proxy Authentication Required\r\n\r\n')
return
}
const { hostname, port } = new URL('http://' + req.url)
const serverSocket = connect(port, hostname, () => {
clientSocket.write('HTTP/1.1 200 Connection Established\r\n\r\n')
serverSocket.write(head)
serverSocket.pipe(clientSocket).pipe(serverSocket)
})
serverSocket.on('error', () => clientSocket.destroy())
clientSocket.on('error', () => serverSocket.destroy())
})
proxyServer.listen(0, '127.0.0.1')
await once(proxyServer, 'listening')
proxyPort = proxyServer.address().port
})
after(() => {
proxyServer.closeAllConnections()
proxyServer.close()
nbdServer.close()
})
it('tunnels the NBD connection with CONNECT', async () => {
const client = new NbdTcpClient({
address: '127.0.0.1',
port: nbdPort,
httpProxy: `http://user:p%40ss@127.0.0.1:${proxyPort}`,
})
await client.connect()
try {
assert.equal(client.exportSize, BigInt(DATA.length))
const block = await client.readBlock(2, BLOCK_SIZE)
assert.deepEqual(block, DATA.subarray(2 * BLOCK_SIZE, 3 * BLOCK_SIZE))
} finally {
await client.disconnect()
}
assert.equal(connectRequests.at(-1), `127.0.0.1:${nbdPort}`)
})
it('fails when the proxy refuses the connection', async () => {
const client = new NbdTcpClient({
address: '127.0.0.1',
port: nbdPort,
httpProxy: `http://user:wrong@127.0.0.1:${proxyPort}`,
})
await assert.rejects(client.connect(), { code: 'NBD_PROXY_CONNECT_FAILED' })
await client.disconnect().catch(() => {})
})
it('aborts the CONNECT when the proxy does not answer', async () => {
// accepts the TCP connection but never answers
const silentProxy = createTcpServer(() => {})
silentProxy.listen(0, '127.0.0.1')
await once(silentProxy, 'listening')
try {
const client = new NbdTcpClient({
address: '127.0.0.1',
port: nbdPort,
httpProxy: `http://127.0.0.1:${silentProxy.address().port}`,
proxyTimeout: 50,
})
await assert.rejects(client.connect(), { code: 'NBD_PROXY_CONNECT_TIMEOUT' })
} finally {
silentProxy.close()
}
})
it('rejects unsupported proxy protocols', () => {
assert.throws(() => new NbdTcpClient({ address: '127.0.0.1', httpProxy: 'socks5://127.0.0.1:1080' }), {
code: 'NBD_PROXY_UNSUPPORTED_PROTOCOL',
})
})
})

View File

@@ -14,9 +14,15 @@ npm install --save @vates/nbd-client
## Usage
### `new NdbClient({address, exportname, secure = true, port = 10809})`
This package provides a NBD client, in two flavors sharing the same protocol
implementation (`AbstractNbdClient`): over the network, or over the standard
streams of a NBD server run as a child process.
create a new nbd client
### `new NbdClient({address, exportname, cert, port = 10809})`
Create a new nbd client connecting to a server over TCP.
`NbdClient` is also exported as `NbdTcpClient`.
```js
import NbdClient from '@vates/nbd-client'
@@ -31,6 +37,58 @@ const block = await client.readBlock(blockIndex, BlockSize)
await client.disconnect()
```
### `new NbdStdioClient({command, args, exportname})`
Create a new nbd client talking to a server through its standard streams, like
`nbdkit --single` or any server reading its queries on stdin and writing its
answers on stdout.
The process is spawned by `connect()` and stopped by `disconnect()`.
```js
import { NbdStdioClient } from '@vates/nbd-client'
const client = new NbdStdioClient({
command: 'nbdkit',
args: ['--single', '--exit-with-parent', '--read-only', 'file', '/path/to/disk.raw'],
exportname: '', // optional, the default export of the server
})
await client.connect()
const block = await client.readBlock(blockIndex, BlockSize)
await client.disconnect()
```
There is no TLS here: the transport is a pair of pipes, there is nothing to
encrypt.
`readBlock()` retries on error, and a retry reconnects: with this client that
means killing the server and spawning a new one, so the command must be
restartable and must serve the same content on each run.
`getMap()` is not implemented, since `nbdinfo` needs either an URI or a server
supporting systemd socket activation: compute the map on the caller side and
pass it explicitly, for example `new NbdDisk(infos, blockSize, { dataMap })`.
The tail of the server stderr is available as `client.stderr` and attached to
the errors raised when the server dies unexpectedly.
### Writing another client
Subclass `AbstractNbdClient` and implement the transport:
- `_openTransport()`: **required**, returns `{ readable, writable }`, called on
every (re)connection.
- `_secureTransport(transport)`: optional, called during the handshake, the only
place where the transport can be upgraded (`NBD_OPT_STARTTLS`).
- `_closeTransport(transport, lastMessage)`: optional, must hand over
`lastMessage` (`NBD_CMD_DISC`) before closing.
- `_destroyTransport(transport)`: optional, forceful cleanup, must not throw.
- `getMap(signal)`: optional, the base implementation rejects with the code
`NBD_MAP_UNSUPPORTED`.
Any extra property set on the transport object is passed back to
`_closeTransport()` and `_destroyTransport()`.
## Contributions
Contributions are _very_ welcomed, either on the documentation or on

View File

@@ -1,408 +1,6 @@
import assert from 'node:assert'
import { Socket } from 'node:net'
import { connect } from 'node:tls'
import { pRetry, pDelay, pTimeout, pFromCallback } from 'promise-toolbox'
import { readChunkStrict } from '@vates/read-chunk'
import { createLogger } from '@xen-orchestra/log'
import {
INIT_PASSWD,
NBD_CMD_READ,
NBD_DEFAULT_BLOCK_SIZE,
NBD_DEFAULT_PORT,
NBD_FLAG_FIXED_NEWSTYLE,
NBD_FLAG_HAS_FLAGS,
NBD_OPT_EXPORT_NAME,
NBD_OPT_REPLY_MAGIC,
NBD_OPT_STARTTLS,
NBD_REPLY_ACK,
NBD_REPLY_MAGIC,
NBD_REQUEST_MAGIC,
OPTS_MAGIC,
NBD_CMD_DISC,
} from './constants.mjs'
import { spawn } from 'node:child_process'
// the TCP client is the historical default export of this package
export { default } from './NbdTcpClient.mjs'
const { warn } = createLogger('vates:nbd-client')
// documentation is here : https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md
export default class NbdClient {
#serverAddress
#serverCert
#serverPort
#serverSocket
#exportName
#exportSize
#waitBeforeReconnect
#readBlockRetries
#reconnectRetry
#connectTimeout
#messageTimeout
// AFAIK, there is no guaranty the server answers in the same order as the queries
// so we handle a backlog of command waiting for response and handle concurrency manually
#waitingForResponse // there is already a listener waiting for a response
#nextCommandQueryId = BigInt(0)
#commandQueryBacklog // map of command waiting for a response queryId => { size/*in byte*/, resolve, reject}
#connected = false
#reconnectingPromise
constructor(
{ address, port = NBD_DEFAULT_PORT, exportname, cert },
{
connectTimeout = 6e4,
messageTimeout = 6e4,
waitBeforeReconnect = 1e3,
readBlockRetries = 5,
reconnectRetry = 5,
} = {}
) {
this.#serverAddress = address
this.#serverPort = port
this.#exportName = exportname
this.#serverCert = cert
this.#waitBeforeReconnect = waitBeforeReconnect
this.#readBlockRetries = readBlockRetries
this.#reconnectRetry = reconnectRetry
this.#connectTimeout = connectTimeout
this.#messageTimeout = messageTimeout
}
get exportSize() {
return this.#exportSize
}
async #tlsConnect() {
return new Promise((resolve, reject) => {
this.#serverSocket = connect({
socket: this.#serverSocket,
rejectUnauthorized: false,
cert: this.#serverCert,
})
this.#serverSocket.once('error', reject)
this.#serverSocket.once('secureConnect', () => {
this.#serverSocket.removeListener('error', reject)
resolve()
})
})
}
// mandatory , at least to start the handshake
async #unsecureConnect() {
this.#serverSocket = new Socket()
return new Promise((resolve, reject) => {
this.#serverSocket.connect(this.#serverPort, this.#serverAddress)
this.#serverSocket.once('error', reject)
this.#serverSocket.once('connect', () => {
this.#serverSocket.removeListener('error', reject)
resolve()
})
})
}
async #connect() {
// first we connect to the server without tls, and then we upgrade the connection
// to tls during the handshake
await this.#unsecureConnect()
await this.#handshake()
this.#connected = true
// reset internal state if we reconnected a nbd client
this.#commandQueryBacklog = new Map()
this.#waitingForResponse = false
}
async connect() {
return pTimeout.call(this.#connect(), this.#connectTimeout)
}
async disconnect() {
if (!this.#connected) {
return
}
this.#connected = false
const socket = this.#serverSocket
const queryId = this.#nextCommandQueryId
this.#nextCommandQueryId++
const buffer = Buffer.alloc(28)
buffer.writeInt32BE(NBD_REQUEST_MAGIC, 0) // it is a nbd request
buffer.writeInt16BE(0, 4) // no command flags for a disconnect
buffer.writeInt16BE(NBD_CMD_DISC, 6) // we want to disconnect from nbd server
buffer.writeBigUInt64BE(queryId, 8)
buffer.writeBigUInt64BE(0n, 16)
buffer.writeInt32BE(0, 24)
const promise = pFromCallback(cb => {
socket.end(buffer, 'utf8', cb)
})
try {
await pTimeout.call(promise, this.#messageTimeout)
} catch (error) {
socket.destroy()
}
this.#serverSocket = undefined
this.#connected = false
}
#clearReconnectPromise = () => {
this.#reconnectingPromise = undefined
}
async #reconnect() {
await this.disconnect().catch(() => {})
await pDelay(this.#waitBeforeReconnect) // need to let the xapi clean things on its side
await this.connect()
}
async reconnect() {
// we need to ensure reconnections do not occur in parallel
if (this.#reconnectingPromise === undefined) {
this.#reconnectingPromise = pRetry(() => this.#reconnect(), {
tries: this.#reconnectRetry,
})
this.#reconnectingPromise.then(this.#clearReconnectPromise, this.#clearReconnectPromise)
}
return this.#reconnectingPromise
}
// we can use individual read/write from the socket here since there is no concurrency
async #sendOption(option, buffer = Buffer.alloc(0)) {
await this.#write(OPTS_MAGIC)
await this.#writeInt32(option)
await this.#writeInt32(buffer.length)
await this.#write(buffer)
assert.strictEqual(await this.#readInt64(), NBD_OPT_REPLY_MAGIC) // magic number everywhere
assert.strictEqual(await this.#readInt32(), option) // the option passed
assert.strictEqual(await this.#readInt32(), NBD_REPLY_ACK) // ACK
const length = await this.#readInt32()
assert.strictEqual(length, 0) // length
}
// we can use individual read/write from the socket here since there is only one handshake at once, no concurrency
async #handshake() {
assert((await this.#read(8)).equals(INIT_PASSWD))
assert((await this.#read(8)).equals(OPTS_MAGIC))
const flagsBuffer = await this.#read(2)
const flags = flagsBuffer.readInt16BE(0)
assert.strictEqual(flags & NBD_FLAG_FIXED_NEWSTYLE, NBD_FLAG_FIXED_NEWSTYLE) // only FIXED_NEWSTYLE one is supported from the server options
await this.#writeInt32(NBD_FLAG_FIXED_NEWSTYLE) // client also support NBD_FLAG_C_FIXED_NEWSTYLE
if (this.#serverCert !== undefined) {
// upgrade socket to TLS if needed
await this.#sendOption(NBD_OPT_STARTTLS)
await this.#tlsConnect()
}
// send export name we want to access.
// it's implicitly closing the negotiation phase.
await this.#write(OPTS_MAGIC)
await this.#writeInt32(NBD_OPT_EXPORT_NAME)
const exportNameBuffer = Buffer.from(this.#exportName)
await this.#writeInt32(exportNameBuffer.length)
await this.#write(exportNameBuffer)
// 8 (export size ) + 2 (flags) + 124 zero = 134
// must read all to ensure nothing stays in the buffer
const answer = await this.#read(134)
this.#exportSize = answer.readBigUInt64BE(0)
const transmissionFlags = answer.readInt16BE(8)
assert.strictEqual(transmissionFlags & NBD_FLAG_HAS_FLAGS, NBD_FLAG_HAS_FLAGS, 'NBD_FLAG_HAS_FLAGS') // must always be 1 by the norm
// note : xapi server always send NBD_FLAG_READ_ONLY (3) as a flag
}
#read(length) {
const promise = readChunkStrict(this.#serverSocket, length)
return pTimeout.call(promise, this.#messageTimeout)
}
#write(buffer) {
let timeout
const messageTimeout = this.#messageTimeout
const socket = this.#serverSocket
return Promise.race([
new Promise((resolve, reject) => {
timeout = setTimeout(() => {
reject(new Error('timeout'))
}, messageTimeout)
}),
new Promise(resolve =>
socket.write(buffer, () => {
clearTimeout(timeout)
resolve()
})
),
])
}
async #readInt32() {
const buffer = await this.#read(4)
return buffer.readInt32BE(0)
}
async #readInt64() {
const buffer = await this.#read(8)
return buffer.readBigUInt64BE(0)
}
#writeInt32(int) {
const buffer = Buffer.alloc(4)
buffer.writeInt32BE(int)
return this.#write(buffer)
}
// when one read fail ,stop everything
async #rejectAll(error) {
this.#commandQueryBacklog.forEach(({ reject }) => {
reject(error)
})
}
async #readBlockResponse() {
// ensure at most one read occur in parallel
if (this.#waitingForResponse) {
return
}
try {
this.#waitingForResponse = true
const buffer = await this.#read(16)
const magic = buffer.readInt32BE(0)
if (magic !== NBD_REPLY_MAGIC) {
throw new Error(`magic number for block answer is wrong : ${magic} ${NBD_REPLY_MAGIC}`)
}
const error = buffer.readInt32BE(4)
if (error !== 0) {
// @todo use error code from constants.mjs
throw new Error(`GOT ERROR CODE : ${error}`)
}
const blockQueryId = buffer.readBigUInt64BE(8)
const query = this.#commandQueryBacklog.get(blockQueryId)
if (!query) {
throw new Error(` no query associated with id ${blockQueryId}`)
}
this.#commandQueryBacklog.delete(blockQueryId)
const data = await this.#read(query.size)
query.resolve(data)
this.#waitingForResponse = false
if (this.#commandQueryBacklog.size > 0) {
// it doesn't throw directly but will throw all relevant promise on failure
this.#readBlockResponse()
}
} catch (error) {
// reject all the promises
// we don't need to call readBlockResponse on failure
// since we will empty the backlog
await this.#rejectAll(error)
}
}
async #readBlock(index, size) {
// we don't want to add anything in backlog while reconnecting
if (this.#reconnectingPromise) {
await this.#reconnectingPromise
}
const queryId = this.#nextCommandQueryId
this.#nextCommandQueryId++
// create and send command at once to ensure there is no concurrency issue
const buffer = Buffer.alloc(28)
buffer.writeInt32BE(NBD_REQUEST_MAGIC, 0) // it is a nbd request
buffer.writeInt16BE(0, 4) // no command flags for a simple block read
buffer.writeInt16BE(NBD_CMD_READ, 6) // we want to read a data block
buffer.writeBigUInt64BE(queryId, 8)
// byte offset in the raw disk
const offset = BigInt(index) * BigInt(size)
const remaining = this.#exportSize - offset
if (remaining < BigInt(size)) {
size = Number(remaining)
}
buffer.writeBigUInt64BE(offset, 16)
buffer.writeInt32BE(size, 24)
return new Promise((resolve, reject) => {
function decoratedReject(error) {
error.index = index
error.size = size
reject(error)
}
// this will handle one block response, but it can be another block
// since server does not guaranty to handle query in order
this.#commandQueryBacklog.set(queryId, {
size,
resolve,
reject: decoratedReject,
})
// really send the command to the server
this.#write(buffer).catch(decoratedReject)
// #readBlockResponse never throws directly
// but if it fails it will reject all the promises in the backlog
this.#readBlockResponse()
})
}
async readBlock(index, size = NBD_DEFAULT_BLOCK_SIZE) {
return pRetry(() => this.#readBlock(index, size), {
tries: this.#readBlockRetries,
when: error => error.code !== 'ERR_ABORTED',
onRetry: async err => {
warn('will retry reading block ', index, err)
await this.reconnect()
},
})
}
/**
* returns the map of the file with holes, zeros and data, useful to handle efficiently sparse source
* to implement this internally: use structure response if the server supports it, and then ask for BLOCK_STATUS *
*
* @returns {Promise<{ offset: number, length: number, type: number }[]>}
* A promise that resolves to an array where each object represents a segment:
* - `offset` — The byte offset from the start.
* - `length` — The size of the segment in bytes.
* - `type` — A numeric code indicating the segment type ( 0 means data).
*/
/* async */ getMap(signal) {
return new Promise((resolve, reject) => {
const process = spawn('nbdinfo', [
'--json',
'--map',
`nbd://${this.#serverAddress}:${this.#serverPort}/${encodeURIComponent(this.#exportName)}`,
])
let text = ''
let errText = ''
process.stdout.on('data', data => (text += data))
process.stderr.on('data', data => (errText += data))
const onAbort = () => {
process.kill()
reject(signal.reason)
}
signal?.addEventListener('abort', onAbort, { once: true })
process.on('error', err => {
signal?.removeEventListener('abort', onAbort)
reject(err)
})
process.on('close', code => {
signal?.removeEventListener('abort', onAbort)
if (code !== 0) {
return reject(new Error(`Error during getMap (code: ${code}): ${errText}`))
}
try {
const json = JSON.parse(text)
resolve(json)
} catch (error) {
reject(new Error(`${error} ${text}`))
}
})
})
}
}
export { default as AbstractNbdClient } from './AbstractNbdClient.mjs'
export { default as NbdStdioClient } from './NbdStdioClient.mjs'
export { default as NbdTcpClient } from './NbdTcpClient.mjs'

View File

@@ -7,6 +7,7 @@ const { warn } = createLogger('vates:nbd-client:multi')
export default class MultiNbdClient {
#clients = []
#nbdConcurrency
#nextClient = 0
#options
#readAhead
#settings
@@ -97,8 +98,31 @@ export default class MultiNbdClient {
* @returns {Promise<Buffer>}
*/
async readBlock(index, size = NBD_DEFAULT_BLOCK_SIZE) {
const clientId = index % this.#clients.length
return this.#clients[clientId].readBlock(index, size)
const clientId = this.#nextClient++ % this.#clients.length
const client = this.#clients[clientId]
try {
return await client.readBlock(index, size)
} catch (err) {
// client.readBlock() already exhausted its own retries/reconnects: this connection is dead.
// Evict it so future reads stop being routed to it, and retry this read on a surviving
// client, since the data is usually still reachable through the others.
this.#evict(client)
if (this.#clients.length === 0) {
throw err
}
warn(`evicted a dead nbd client, retrying block ${index} on a remaining client`, { err })
return this.readBlock(index, size)
}
}
// no-op if `client` was already evicted by a concurrent failed read on the same client
#evict(client) {
const i = this.#clients.indexOf(client)
if (i === -1) {
return
}
this.#clients.splice(i, 1)
client.disconnect().catch(() => {})
}
/**

View File

@@ -0,0 +1,130 @@
import { describe, it, mock } from 'node:test'
import assert from 'node:assert'
// MultiNbdClient instantiates NbdClient itself (`import NbdClient from './index.mjs'`), so the
// only way to control individual connections from a unit test is to replace that module with a
// fake before importing multi.mjs.
const created = []
class FakeNbdClient {
constructor(nbdInfo) {
this.nbdInfo = nbdInfo
this.disconnected = false
// overridden per-test after connect() to script this instance's behavior
this.readBlock = async () => {
throw new Error('readBlock not configured for this fake client')
}
created.push(this)
}
async connect() {}
async disconnect() {
this.disconnected = true
}
async getMap() {
return []
}
}
mock.module('./index.mjs', {
defaultExport: FakeNbdClient,
})
const { default: MultiNbdClient } = await import('./multi.mjs')
// a single settings entry is enough: connect() reuses candidates across concurrent slots, so
// nbdConcurrency still yields that many distinct FakeNbdClient instances
async function connectWithFakes(nbdConcurrency) {
created.length = 0
const client = new MultiNbdClient({ address: 'fake-host' }, { nbdConcurrency })
await client.connect()
return { client, fakes: created.slice() }
}
describe('MultiNbdClient.readBlock eviction', () => {
it('evicts a client whose readBlock rejects and retries on a surviving client', async () => {
const { client, fakes } = await connectWithFakes(2)
const [dead, alive] = fakes
dead.readBlock = async () => {
throw new Error('connection is dead')
}
alive.readBlock = async index => Buffer.from([index])
// index 0 initially routes to `dead` (0 % 2 === 0)
const data = await client.readBlock(0)
assert.deepStrictEqual(data, Buffer.from([0]))
assert.strictEqual(dead.disconnected, true, 'the dead client should have been disconnected')
// future reads must no longer be routed to the evicted client: with only `alive` left,
// every index should now succeed through it
alive.readBlock = async index => Buffer.from([index])
const data2 = await client.readBlock(1)
assert.deepStrictEqual(data2, Buffer.from([1]))
})
it('rejects once every client has been evicted, instead of retrying forever', async () => {
const { client, fakes } = await connectWithFakes(2)
const boom = new Error('connection is dead')
for (const fake of fakes) {
fake.readBlock = async () => {
throw boom
}
}
await assert.rejects(() => client.readBlock(0), boom)
assert.ok(
fakes.every(fake => fake.disconnected),
'every client should have been evicted (and disconnected) before giving up'
)
})
it('evicting the same dead client from concurrent failed reads is idempotent', async () => {
const { client, fakes } = await connectWithFakes(2)
const [dead, alive] = fakes
dead.readBlock = async () => {
throw new Error('connection is dead')
}
alive.readBlock = async index => Buffer.from([index])
let disconnectCalls = 0
dead.disconnect = async () => {
disconnectCalls++
dead.disconnected = true
}
// routing is round-robin by call order (0, 1, 0) over a 2-client pool: the 1st and 3rd
// calls both land on `dead` before either failure has been handled, so they race to evict
// the same client. The exact interleaving isn't guaranteed, only that eviction is safe
// either way and every read still completes.
const [data0, data1, data2] = await Promise.all([client.readBlock(10), client.readBlock(11), client.readBlock(12)])
assert.deepStrictEqual(data0, Buffer.from([10]))
assert.deepStrictEqual(data1, Buffer.from([11]))
assert.deepStrictEqual(data2, Buffer.from([12]))
assert.strictEqual(disconnectCalls, 1, 'the dead client must only be evicted/disconnected once')
})
})
describe('MultiNbdClient.readBlock routing', () => {
it('distributes reads round-robin by call order, not by index residue', async () => {
const { client, fakes } = await connectWithFakes(4)
const readsPerClient = fakes.map(() => 0)
fakes.forEach((fake, i) => {
fake.readBlock = async index => {
readsPerClient[i]++
return Buffer.from([index])
}
})
// a CBT-style changed-block set striding by exactly `nbdConcurrency`: under the old
// `index % clients.length` routing this would collapse entirely onto a single client
const stridedIndexes = Array.from({ length: 20 }, (_, i) => i * 4)
for (const index of stridedIndexes) {
await client.readBlock(index)
}
assert.deepStrictEqual(readsPerClient, [5, 5, 5, 5])
})
})

View File

@@ -13,7 +13,7 @@
"url": "https://vates.fr"
},
"license": "ISC",
"version": "3.5.0",
"version": "3.7.0",
"engines": {
"node": ">=14.0"
},
@@ -21,11 +21,11 @@
"dependencies": {
"@vates/async-each": "^1.0.3",
"@vates/read-chunk": "^1.2.1",
"@xen-orchestra/async-map": "^0.1.3",
"@xen-orchestra/disk-transform": "^1.3.3",
"@xen-orchestra/async-map": "^0.1.5",
"@xen-orchestra/disk-transform": "^1.3.4",
"@xen-orchestra/log": "^0.7.2",
"promise-toolbox": "^0.21.0",
"xen-api": "^4.8.0"
"xen-api": "^5.1.0"
},
"devDependencies": {
"tap": "^18.7.0",
@@ -33,6 +33,7 @@
},
"scripts": {
"postversion": "npm publish --access public",
"test-integration": "tap --allow-incomplete-coverage"
"test": "node --experimental-test-module-mocks --test",
"test-integration": "tap --allow-incomplete-coverage tests/*.integ.mjs"
}
}

View File

@@ -0,0 +1,132 @@
import assert from 'node:assert'
import { pFromCallback } from 'promise-toolbox'
import { readChunk, readChunkStrict } from '@vates/read-chunk'
import {
INIT_PASSWD,
NBD_CMD_DISC,
NBD_CMD_READ,
NBD_FLAG_FIXED_NEWSTYLE,
NBD_FLAG_HAS_FLAGS,
NBD_FLAG_READ_ONLY,
NBD_OPT_EXPORT_NAME,
NBD_OPT_REPLY_MAGIC,
NBD_REPLY_ACK,
NBD_REPLY_MAGIC,
NBD_REQUEST_MAGIC,
OPTS_MAGIC,
} from '../constants.mjs'
const NBD_EINVAL = 22
/**
* Minimal read-only newstyle NBD server, used by the tests only.
*
* It is transport agnostic on purpose: the unit tests drive it through two
* in-memory `PassThrough`, the integration tests through the standard streams
* of a real process.
*
* @param {object} options
* @param {import('node:stream').Readable} options.readable - where the client queries are read from
* @param {import('node:stream').Writable} options.writable - where the answers are written to
* @param {Buffer} options.data - the content of the export
* @param {string} [options.exportName] - the export name the client must ask for
* @param {boolean} [options.answerInReverse] - answer by pairs, in reverse order, to check the client handles out of order answers
* @param {(request: {offset: bigint, length: number, index: number}) => number} [options.errorCode] - non zero to answer an error to this read
* @returns {Promise<void>} resolves when the client disconnected or closed the connection
*/
export async function serveNbd({ readable, writable, data, exportName = '', answerInReverse = false, errorCode }) {
const write = buffer => pFromCallback(cb => writable.write(buffer, cb))
// handshake: server flags
await write(INIT_PASSWD)
await write(OPTS_MAGIC)
const serverFlags = Buffer.alloc(2)
serverFlags.writeInt16BE(NBD_FLAG_FIXED_NEWSTYLE)
await write(serverFlags)
// client flags
await readChunkStrict(readable, 4)
// options, until the client selects an export
let selected = false
do {
assert.ok((await readChunkStrict(readable, 8)).equals(OPTS_MAGIC), 'option magic')
const option = (await readChunkStrict(readable, 4)).readInt32BE(0)
const length = (await readChunkStrict(readable, 4)).readInt32BE(0)
const payload = length > 0 ? await readChunkStrict(readable, length) : Buffer.alloc(0)
if (option === NBD_OPT_EXPORT_NAME) {
assert.strictEqual(payload.toString(), exportName, 'export name')
// 8 (size) + 2 (transmission flags) + 124 zeroes
const answer = Buffer.alloc(134)
answer.writeBigUInt64BE(BigInt(data.length), 0)
answer.writeInt16BE(NBD_FLAG_HAS_FLAGS | NBD_FLAG_READ_ONLY, 8)
await write(answer)
selected = true
} else {
// ack everything else
const answer = Buffer.alloc(20)
answer.writeBigUInt64BE(NBD_OPT_REPLY_MAGIC, 0)
answer.writeInt32BE(option, 8)
answer.writeInt32BE(NBD_REPLY_ACK, 12)
answer.writeInt32BE(0, 16)
await write(answer)
}
} while (!selected)
// transmission
let index = 0
let pending = []
const answer = async ({ handle, offset, length, index }, forcedCode) => {
const code = forcedCode ?? errorCode?.({ handle, offset, length, index }) ?? 0
const header = Buffer.alloc(16)
header.writeInt32BE(NBD_REPLY_MAGIC, 0)
header.writeInt32BE(code, 4)
header.writeBigUInt64BE(handle, 8)
await write(header)
if (code === 0) {
await write(data.subarray(Number(offset), Number(offset) + length))
}
}
const flush = async () => {
const requests = pending
pending = []
for (const request of requests.reverse()) {
await answer(request)
}
}
while (true) {
const request = await readChunk(readable, 28)
if (request === null || request.length < 28) {
// the client closed the connection without a NBD_CMD_DISC
break
}
assert.strictEqual(request.readInt32BE(0), NBD_REQUEST_MAGIC, 'request magic')
const type = request.readInt16BE(6)
const handle = request.readBigUInt64BE(8)
if (type === NBD_CMD_DISC) {
await flush()
break
}
if (type !== NBD_CMD_READ) {
await answer({ handle, offset: 0n, length: 0, index: index++ }, NBD_EINVAL)
continue
}
const read = {
handle,
offset: request.readBigUInt64BE(16),
length: request.readInt32BE(24),
index: index++,
}
pending.push(read)
if (!answerInReverse || pending.length === 2) {
await flush()
}
}
}

View File

@@ -153,7 +153,13 @@ CYu1Xn/FVPx1HoRgWc7E8wFhDcA/P3SJtfIQWHB9FzSaBflKGR4t8WCE2eE8+cTB
nbdServer = await spawnNbdKit(path)
}
}
assert.rejects(() => client.readBlock(100, CHUNK_SIZE))
// reading past the end of the export must reject: a single client so the failure isn't
// multiplied by nbdConcurrency evicting and retrying on every other (equally healthy, equally
// unable to satisfy this out-of-range request) client
const singleClient = new MultiNbdClient(connectionSettings, { nbdConcurrency: 1, readAhead: 2 })
await singleClient.connect()
await assert.rejects(() => singleClient.readBlock(100, CHUNK_SIZE))
await singleClient.disconnect()
await client.disconnect()
// double disconnection shouldn't pose any problem

View File

@@ -0,0 +1,114 @@
import assert from 'node:assert'
import fs from 'node:fs/promises'
import { fileURLToPath } from 'node:url'
import { pFromCallback } from 'promise-toolbox'
import { test } from 'tap'
import tmp from 'tmp'
import NbdStdioClient from '../NbdStdioClient.mjs'
// nbdkit in single connection mode: the real thing this client targets
const NBDKIT = path => ({
command: 'nbdkit',
args: ['--single', '--exit-with-parent', '--read-only', 'file', path],
})
// a tiny NBD server we can ask to die in the middle of the transfer, which
// nbdkit won't do
const FLAKY_SERVER = fileURLToPath(new URL('./stdio-nbd-server.mjs', import.meta.url))
const CHUNK_SIZE = 64 * 1024
const FILE_SIZE = CHUNK_SIZE * 8 + 1232 // non aligned file size
async function createTempFile(size) {
const tmpPath = await pFromCallback(cb => tmp.file(cb))
const data = Buffer.alloc(size)
for (let i = 0; i < size; i += 4) {
data.writeUInt32BE(i, i)
}
await fs.writeFile(tmpPath, data)
return tmpPath
}
function checkBlock(tap, block, index) {
tap.equal(block.length, Math.min(CHUNK_SIZE, FILE_SIZE - index * CHUNK_SIZE), `block ${index} length`)
let ok = true
for (let i = 0; i < block.length; i += 4) {
ok = ok && block.readUInt32BE(i) === index * CHUNK_SIZE + i
}
tap.ok(ok, `block ${index} content`)
}
async function readWholeExport(tap, client) {
for (let index = 0; index < Math.ceil(FILE_SIZE / CHUNK_SIZE); index++) {
checkBlock(tap, await client.readBlock(index, CHUNK_SIZE), index)
}
}
const isAlive = pid => {
try {
process.kill(pid, 0)
return true
} catch (error) {
return false
}
}
test('it reads an export served by nbdkit on stdin/stdout', async tap => {
const path = await createTempFile(FILE_SIZE)
const client = new NbdStdioClient(NBDKIT(path))
await client.connect()
const { pid } = client
tap.equal(client.exportSize, BigInt(FILE_SIZE))
await readWholeExport(tap, client)
await client.disconnect()
tap.notOk(isAlive(pid), 'the server process is stopped after disconnect')
// double disconnection shouldn't pose any problem
await client.disconnect()
await fs.unlink(path)
})
test('it fails when the server does not serve the requested export', async tap => {
const path = await createTempFile(CHUNK_SIZE)
// the file plugin of nbdkit ignores the export name, so use our own server here
const client = new NbdStdioClient(
{
command: process.execPath,
args: [FLAKY_SERVER, path, '--export-name=MY_SECRET_EXPORT'],
exportname: 'NOPE',
},
{ connectTimeout: 5e3 }
)
await assert.rejects(client.connect())
tap.match(client.stderr, /NOPE/, 'the stderr of the server is captured')
await fs.unlink(path)
})
test('it recovers from a server dying in the middle of the transfer', async tap => {
const path = await createTempFile(FILE_SIZE)
const client = new NbdStdioClient(
{ command: process.execPath, args: [FLAKY_SERVER, path, '--die-after=3'] },
{ waitBeforeReconnect: 100 }
)
await client.connect()
await readWholeExport(tap, client)
await client.disconnect()
await fs.unlink(path)
})
test('it fails cleanly when the server cannot be spawned', async () => {
const client = new NbdStdioClient({ command: '/does/not/exist' })
await assert.rejects(client.connect(), ({ code, command }) => code === 'ENOENT' && command === '/does/not/exist')
// nothing to disconnect from
await client.disconnect()
})
test('it does not implement getMap', async () => {
const client = new NbdStdioClient({ command: 'nbdkit' })
await assert.rejects(client.getMap(), ({ code }) => code === 'NBD_MAP_UNSUPPORTED')
})

View File

@@ -0,0 +1,52 @@
#!/usr/bin/env node
// Read-only NBD server serving a file through its standard streams, à la
// `nbdkit -s`. Used by nbdstdio.integ.mjs, since nbdkit is not necessarily
// installed on the test machine.
//
// Usage: stdio-nbd-server.mjs <file> [--export-name=NAME] [--die-after=N]
//
// --die-after=N exits with a failure after having answered N read requests, to
// exercise the reconnection path of the client.
import fs from 'node:fs/promises'
import { serveNbd } from './fake-nbd-server.mjs'
const args = process.argv.slice(2)
const path = args.find(arg => !arg.startsWith('--'))
if (path === undefined) {
console.error('usage: stdio-nbd-server.mjs <file> [--export-name=NAME] [--die-after=N]')
process.exit(64)
}
const getOption = (name, defaultValue) => {
const prefix = `--${name}=`
const arg = args.find(arg => arg.startsWith(prefix))
return arg === undefined ? defaultValue : arg.slice(prefix.length)
}
const exportName = getOption('export-name', '')
const dieAfter = Number(getOption('die-after', Infinity))
const data = await fs.readFile(path)
let answered = 0
try {
await serveNbd({
readable: process.stdin,
writable: process.stdout,
data,
exportName,
errorCode: () => {
if (++answered > dieAfter) {
console.error(`stdio-nbd-server: dying after ${dieAfter} read(s)`)
process.exit(1)
}
return 0
},
})
} catch (error) {
console.error('stdio-nbd-server:', error)
process.exit(1)
}

View File

@@ -0,0 +1,102 @@
const XML_ENTITIES = {
'&amp;': '&',
'&apos;': "'",
'&gt;': '>',
'&lt;': '<',
'&quot;': '"',
}
const unescapeXml = text => text.replace(/&(?:amp|apos|gt|lt|quot);/g, entity => XML_ENTITIES[entity])
// node-soap wraps text nodes either as a plain string or as `{ $value }` depending on
// whether the element carries attributes
const valueOf = node => {
if (typeof node === 'string') {
return node
}
if (node !== null && typeof node === 'object' && typeof node.$value === 'string') {
return node.$value
}
}
const faultTypeOfElementName = name => name.replace(/^.*:/, '').replace(/Fault$/, '')
// SOAP 1.1 requires the children of `Fault` to be unqualified, and that is what ESXi and vCenter
// send, but the raw body is only read when node-soap failed to parse the response: it can then be
// anything, e.g. an error emitted by a reverse proxy in front of vCenter. Hence the tolerance to a
// namespace prefix and to attributes on the element
const textOfElement = name =>
new RegExp(`<(?:[\\w.-]+:)?${name}(?:\\s[^>]*)?>([\\s\\S]*?)</(?:[\\w.-]+:)?${name}>`, 'i')
const FAULTCODE_RE = textOfElement('faultcode')
const FAULTSTRING_RE = textOfElement('faultstring')
const LOCALIZED_MESSAGE_RE = textOfElement('localizedMessage')
const DETAIL_CHILD_RE = /<(?:[\w.-]+:)?detail(?:\s[^>]*)?>\s*<([^\s/>]+)/i
/**
* Extracts the vim25 fault of a failed SOAP call.
*
* The fault type (`FileLocked`, `InvalidDeviceSpec`, `ManagedObjectNotFound`, ...) is the only part
* a caller can branch on: `faultstring` and `localizedMessage` are meant for humans and change with
* the locale of the server.
*
* Two sources are used, in order of reliability:
* 1. `rawError.root`, the envelope parsed by node-soap ( see `wsdl.xmlToObject`, which throws on a
* fault and whose error is stored there by node-soap)
* 2. the raw response body, when the parsing itself failed
*
* @param {unknown} rawError - the error passed by node-soap to its callback
* @returns {{ code?: string, faultcode?: string, faultstring?: string, localizedMessage?: string, body?: string }}
*/
export function parseFault(rawError) {
const fault = rawError?.root?.Envelope?.Body?.Fault
const rawBody = rawError?.response?.data ?? rawError?.body
const body = typeof rawBody === 'string' && rawBody !== '<stream>' ? rawBody : undefined
let code, localizedMessage
let faultcode = valueOf(fault?.faultcode)
let faultstring = valueOf(fault?.faultstring)
const detail = fault?.detail
if (detail !== null && typeof detail === 'object') {
// the single child of `detail` names the fault, e.g.
// `<InvalidDeviceSpecFault xsi:type="InvalidDeviceSpec">`
for (const [name, value] of Object.entries(detail)) {
if (name === 'attributes') {
continue
}
code = value?.attributes?.['xsi:type'] ?? faultTypeOfElementName(name)
localizedMessage = valueOf(value?.localizedMessage)
break
}
}
if (body !== undefined) {
if (faultstring === undefined) {
const matches = body.match(FAULTSTRING_RE)
if (matches !== null) {
faultstring = unescapeXml(matches[1])
}
}
if (faultcode === undefined) {
const matches = body.match(FAULTCODE_RE)
if (matches !== null) {
faultcode = unescapeXml(matches[1])
}
}
if (code === undefined) {
const matches = body.match(DETAIL_CHILD_RE)
if (matches !== null) {
code = faultTypeOfElementName(matches[1])
}
}
if (localizedMessage === undefined) {
const matches = body.match(LOCALIZED_MESSAGE_RE)
if (matches !== null) {
localizedMessage = unescapeXml(matches[1])
}
}
}
return { code, faultcode, faultstring, localizedMessage, body }
}

View File

@@ -0,0 +1,104 @@
import assert from 'node:assert/strict'
import { describe, it } from 'node:test'
import { parseFault } from './_parseFault.mjs'
// captured from an ESXi 8 host
const MANAGED_OBJECT_NOT_FOUND = `<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring>The object &apos;vim.VirtualMachine:vm-42&apos; has already been deleted or has not been completely created</faultstring><detail><ManagedObjectNotFoundFault xmlns="urn:vim25" xsi:type="ManagedObjectNotFound"><obj type="VirtualMachine">vm-42</obj></ManagedObjectNotFoundFault></detail></soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>`
// not what a vSphere host sends — SOAP 1.1 requires these children to be unqualified — but a body
// which node-soap failed to parse can come from anything sitting in front of vCenter
const NAMESPACE_PREFIXED = `<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:vim25="urn:vim25" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<soapenv:Fault><soapenv:faultcode>ServerFaultCode</soapenv:faultcode><soapenv:faultstring>Unable to access file since it is locked</soapenv:faultstring><soapenv:detail xmlns="urn:vim25"><vim25:FileLockedFault xsi:type="FileLocked"><vim25:localizedMessage>Unable to access file [ds1] vm/vm.vmdk since it is locked</vim25:localizedMessage></vim25:FileLockedFault></soapenv:detail></soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>`
describe('parseFault', function () {
it('reads the fault from the envelope parsed by node-soap', function () {
const { code, faultcode, faultstring, localizedMessage } = parseFault({
root: {
Envelope: {
Body: {
Fault: {
faultcode: 'ServerFaultCode',
faultstring: 'Unable to access file since it is locked',
detail: {
FileLockedFault: {
attributes: { 'xsi:type': 'FileLocked' },
localizedMessage: 'Unable to access file [ds1] vm/vm.vmdk since it is locked',
},
},
},
},
},
},
})
assert.equal(code, 'FileLocked')
assert.equal(faultcode, 'ServerFaultCode')
assert.equal(faultstring, 'Unable to access file since it is locked')
assert.equal(localizedMessage, 'Unable to access file [ds1] vm/vm.vmdk since it is locked')
})
it('falls back to the element name when xsi:type is absent', function () {
const { code } = parseFault({
root: { Envelope: { Body: { Fault: { detail: { InvalidDeviceSpecFault: {} } } } } },
})
assert.equal(code, 'InvalidDeviceSpec')
})
it('falls back to the raw body when the envelope could not be parsed', function () {
const { code, faultcode, faultstring, body } = parseFault({
message: 'Request failed with status code 500',
response: { data: MANAGED_OBJECT_NOT_FOUND },
})
assert.equal(code, 'ManagedObjectNotFound')
assert.equal(faultcode, 'ServerFaultCode')
// entities are decoded
assert.match(faultstring, /^The object 'vim\.VirtualMachine:vm-42' has already been deleted/)
assert.equal(body, MANAGED_OBJECT_NOT_FOUND)
})
it('reads a namespace prefixed fault from the raw body', function () {
const { code, faultcode, faultstring, localizedMessage } = parseFault({
response: { data: NAMESPACE_PREFIXED },
})
assert.equal(code, 'FileLocked')
assert.equal(faultcode, 'ServerFaultCode')
assert.equal(faultstring, 'Unable to access file since it is locked')
assert.equal(localizedMessage, 'Unable to access file [ds1] vm/vm.vmdk since it is locked')
})
it('handles a `$value` wrapped text node', function () {
const { faultstring } = parseFault({
root: { Envelope: { Body: { Fault: { faultstring: { $value: 'boom' } } } } },
})
assert.equal(faultstring, 'boom')
})
it('does not throw on an error without any fault', function () {
assert.deepEqual(parseFault(new Error('socket hang up')), {
code: undefined,
faultcode: undefined,
faultstring: undefined,
localizedMessage: undefined,
body: undefined,
})
assert.deepEqual(parseFault(undefined).code, undefined)
})
it('ignores the placeholder body used by node-soap for streamed responses', function () {
assert.equal(parseFault({ body: '<stream>' }).body, undefined)
})
})

View File

@@ -19,16 +19,29 @@ import soap from 'soap'
import Cookie from 'soap-cookie' // required for session persistence
import { createLogger } from '@xen-orchestra/log'
import { parseFault } from './_parseFault.mjs'
// exposed for the consumers talking to the host outside of the WSDL, e.g. `Fetch`
export { parseFault } from './_parseFault.mjs'
const { warn } = createLogger('xo:node-vsphere-soap:client')
class VmwareError extends Error {
constructor(rawError) {
super(rawError.body)
const matches = rawError.body?.match(/<faultstring[^>]*>(.*)<\/faultstring>/im)
this.message = matches?.[1] ?? rawError.message
const { code, faultcode, faultstring, localizedMessage, body } = parseFault(rawError)
super(localizedMessage ?? faultstring ?? rawError?.message ?? 'unknown SOAP error')
this.name = 'VmwareError'
this.stack = rawError.stack
// not putting the cause since the error failed to be stringified correctly down the road
warn(rawError)
// the vim25 fault type, the only part of a fault a caller can branch on
this.code = code
this.faultcode = faultcode
this.faultstring = faultstring
this.localizedMessage = localizedMessage
this.body = body
if (rawError?.stack !== undefined) {
this.stack = rawError.stack
}
// legacy constraint: not putting the cause since the error failed to be stringified correctly
// down the road, the fault is exposed through own properties instead
warn('SOAP call failed', { code, faultcode, faultstring, localizedMessage })
}
}
// Client class
@@ -38,6 +51,7 @@ class VmwareError extends Error {
export function Client(vCenterHostname, username, password, sslVerify) {
this.status = 'disconnected'
this.reconnectCount = 0
this._exitHook = undefined
sslVerify = typeof sslVerify !== 'undefined' ? sslVerify : false
@@ -90,44 +104,49 @@ util.inherits(Client, EventEmitter)
Client.prototype.runCommand = function (command, args) {
const self = this
let cmdargs
if (!args || args === null) {
cmdargs = {}
} else {
cmdargs = args
}
const cmdargs = args ?? {}
const emitter = new EventEmitter()
const onResponse = function (err, result, raw, soapHeader) {
if (err) {
// an error and a result are mutually exclusive: emitting both makes every caller listening
// for 'result' believe the call succeeded, with `result` undefined
_soapErrorHandler(self, emitter, command, cmdargs, err)
return
}
if (command === 'Logout') {
self.status = 'disconnected'
self._unregisterExitHook()
}
emitter.emit('result', result, raw, soapHeader)
}
const send = function () {
self.client.VimService.VimPort[command](cmdargs, onResponse)
}
// check if client has successfully connected
if (self.status === 'ready' || self.status === 'connecting') {
self.client.VimService.VimPort[command](cmdargs, function (err, result, raw, soapHeader) {
if (err) {
_soapErrorHandler(self, emitter, command, cmdargs, err)
}
if (command === 'Logout') {
self.status = 'disconnected'
process.removeAllListeners('beforeExit')
}
emitter.emit('result', result, raw, soapHeader)
})
send()
} else {
// if connection not ready or connecting, reconnect to instance
if (self.status === 'disconnected') {
self.emit('connect')
}
self.once('ready', function () {
self.client.VimService.VimPort[command](cmdargs, function (err, result, raw, soapHeader) {
if (err) {
_soapErrorHandler(self, emitter, command, cmdargs, err)
}
if (command === 'Logout') {
self.status = 'disconnected'
process.removeAllListeners('beforeExit')
}
emitter.emit('result', result, raw, soapHeader)
})
})
// a failed connection would otherwise leave this listener registered for ever, and the next
// successful one would replay every command whose caller has long given up
const onConnectionError = error => {
self.off('ready', onReady)
emitter.emit('error', error)
}
const onReady = () => {
self.off('error', onConnectionError)
send()
}
self.once('ready', onReady)
self.once('error', onConnectionError)
}
return emitter
@@ -153,9 +172,11 @@ Client.prototype._connect = function () {
self.clientopts,
function (err, client) {
if (err) {
const vErr = new VmwareError(err)
self.emit('error', vErr)
throw vErr
self.status = 'disconnected'
// throwing here would be an uncaught exception: we are in an async callback, not in the
// stack of the caller
self.emit('error', new VmwareError(err))
return
}
self.client = client // save client for later use
@@ -186,7 +207,7 @@ Client.prototype._connect = function () {
self.status = 'ready'
self.emit('ready')
process.once('beforeExit', self._close)
self._registerExitHook()
})
.once('error', function (err) {
self.status = 'disconnected'
@@ -202,9 +223,36 @@ Client.prototype._connect = function () {
)
}
// Logs out on process exit, so that a session is not left open on the server until it expires.
//
// The hook must be stored: passing `self._close` directly would call it with `this === process`,
// and removing it with `process.removeAllListeners('beforeExit')` would drop the hooks of every
// other module of the process.
Client.prototype._registerExitHook = function () {
const self = this
if (self._exitHook === undefined) {
self._exitHook = function () {
self._close()
}
process.once('beforeExit', self._exitHook)
}
}
Client.prototype._unregisterExitHook = function () {
const self = this
if (self._exitHook !== undefined) {
process.off('beforeExit', self._exitHook)
self._exitHook = undefined
}
}
Client.prototype._close = function () {
const self = this
self._unregisterExitHook()
if (self.status === 'ready') {
self
.runCommand('Logout', { _this: self.sessionManager })
@@ -220,34 +268,64 @@ Client.prototype._close = function () {
}
}
function _soapErrorHandler(self, emitter, command, args, err) {
// Errors are only emitted on `emitter`, the emitter of the failed call: throwing from here would
// be an uncaught exception since we are always in an async callback, and emitting on the client
// would reject unrelated concurrent calls.
//
// Exported for the unit tests: building a `Client` connects to a host.
export function _soapErrorHandler(self, emitter, command, args, err) {
err = err || { body: 'general error' }
if (err.body?.match(/session is not authenticated/)) {
self.status = 'disconnected'
process.removeAllListeners('beforeExit')
const vErr = new VmwareError(err)
if (self.reconnectCount < 10) {
self.reconnectCount += 1
self
.runCommand(command, args)
.once('result', function (result, raw, soapHeader) {
emitter.emit('result', result, raw, soapHeader)
})
.once('error', function (err) {
const vErr = new VmwareError(err)
emitter.emit('error', vErr)
throw vErr
})
} else {
const vErr = new VmwareError(err)
emitter.emit('error', vErr)
throw vErr
}
} else {
const vErr = new VmwareError(err)
// `NotAuthenticated` is raised when the session expired: log in again and replay the call.
//
// The fault type is used instead of the previous `err.body?.match(...)`: node-soap only fills
// `body` for streamed responses, so the string test never matched and the session was never
// renewed
const isSessionExpired =
vErr.code === 'NotAuthenticated' || /session is not authenticated/i.test(vErr.faultstring ?? vErr.message)
if (!isSessionExpired || self.reconnectCount >= 10) {
emitter.emit('error', vErr)
throw vErr
return
}
// The call is replayed only once a new session is ready. Handing it straight to `runCommand`
// would send it right away while the status is 'connecting' — i.e. on the very session which
// just expired — and the failure of that replay would land back here.
const onConnectionError = function (error) {
self.off('ready', onReady)
emitter.emit('error', error)
}
const onReady = function () {
self.off('error', onConnectionError)
self
.runCommand(command, args)
.once('result', function (result, raw, soapHeader) {
emitter.emit('result', result, raw, soapHeader)
})
.once('error', function (retryError) {
emitter.emit('error', retryError)
})
}
self.once('ready', onReady)
self.once('error', onConnectionError)
// A whole wave of calls can fail on the same expired session, e.g. during an import. Only the
// first one to notice drops it and pays for a retry: the others ride the reconnect it started,
// instead of each burning one of the 10 and of resetting the status under a `_connect` already
// in flight — which used to start a second `soap.createClient` and replace `self.client` under
// the first one.
if (self.status === 'ready') {
self.status = 'disconnected'
self._unregisterExitHook()
self.reconnectCount += 1
}
if (self.status === 'disconnected') {
// no reconnect in flight: either this call is the first to notice, or the previous attempt
// already failed
self.emit('connect')
}
}

View File

@@ -0,0 +1,219 @@
import assert from 'node:assert/strict'
import { EventEmitter } from 'node:events'
import { describe, it } from 'node:test'
import { _soapErrorHandler } from './client.mjs'
// what a host answers to any call made on a session which expired
const NOT_AUTHENTICATED = {
root: {
Envelope: {
Body: {
Fault: {
faultcode: 'ServerFaultCode',
faultstring: 'The session is not authenticated.',
detail: { NotAuthenticatedFault: { attributes: { 'xsi:type': 'NotAuthenticated' } } },
},
},
},
},
}
const FILE_LOCKED = {
root: {
Envelope: {
Body: {
Fault: {
faultcode: 'ServerFaultCode',
faultstring: 'Unable to access file since it is locked',
detail: { FileLockedFault: { attributes: { 'xsi:type': 'FileLocked' } } },
},
},
},
},
}
/**
* A stand-in for a connected `Client`, reduced to what `_soapErrorHandler` touches: building a real
* one talks to a host.
*
* `connect` reproduces `_connect`, which only starts connecting from 'disconnected'.
*/
function fakeClient() {
const self = new EventEmitter()
self.status = 'ready'
self.reconnectCount = 0
self.unregisterExitHookCalls = 0
self._unregisterExitHook = () => {
self.unregisterExitHookCalls += 1
}
// the calls handed to `runCommand`, i.e. the ones actually sent to the host
self.sent = []
self.runCommand = (command, args) => {
const emitter = new EventEmitter()
self.sent.push({ args, command, emitter, statusWhenSent: self.status })
return emitter
}
self.connectCalls = 0
self.on('connect', () => {
self.connectCalls += 1
if (self.status === 'disconnected') {
self.status = 'connecting'
}
})
return self
}
// makes one in-flight call fail, and reports what its own emitter received
function failingCall(self, command, rawError = NOT_AUTHENTICATED) {
const emitter = new EventEmitter()
const outcome = {}
emitter.once('result', result => {
outcome.result = result
})
emitter.once('error', error => {
outcome.error = error
})
_soapErrorHandler(self, emitter, command, { _this: command }, rawError)
return outcome
}
describe('_soapErrorHandler', function () {
it('replays a call made on an expired session, once the new session is ready', function () {
const self = fakeClient()
const outcome = failingCall(self, 'RetrievePropertiesEx')
assert.equal(self.reconnectCount, 1)
assert.equal(self.status, 'connecting')
assert.equal(self.unregisterExitHookCalls, 1)
assert.deepEqual(self.sent, [])
self.status = 'ready'
self.emit('ready')
assert.equal(self.sent.length, 1)
assert.equal(self.sent[0].command, 'RetrievePropertiesEx')
self.sent[0].emitter.emit('result', 'replayed')
assert.equal(outcome.result, 'replayed')
})
it('starts a single reconnect for a whole wave of calls on the same session', function () {
const self = fakeClient()
const outcomes = [failingCall(self, 'A'), failingCall(self, 'B'), failingCall(self, 'C')]
// one retry for the wave, not one per call: they used to exhaust the 10 of a busy import
assert.equal(self.reconnectCount, 1)
assert.equal(self.connectCalls, 1)
assert.equal(self.unregisterExitHookCalls, 1)
// and none of them is sent while the reconnect is in flight, which would mean sending it on
// the session which just expired
assert.deepEqual(self.sent, [])
self.status = 'ready'
self.emit('ready')
assert.deepEqual(
self.sent.map(({ command, statusWhenSent }) => [command, statusWhenSent]),
[
['A', 'ready'],
['B', 'ready'],
['C', 'ready'],
]
)
// each replay answers its own caller
self.sent[1].emitter.emit('result', 'B replayed')
assert.equal(outcomes[1].result, 'B replayed')
assert.equal(outcomes[0].result, undefined)
assert.equal(outcomes[2].result, undefined)
})
it('restarts a reconnect which already failed', function () {
const self = fakeClient()
failingCall(self, 'A')
assert.equal(self.connectCalls, 1)
// the reconnect failed, nothing is in flight any more
self.status = 'disconnected'
failingCall(self, 'B')
assert.equal(self.connectCalls, 2)
// the session was already dropped by the first call, the second one must not pay for it again
assert.equal(self.reconnectCount, 1)
})
it('reports a failed reconnect to every waiting call, and stops listening', function () {
const self = fakeClient()
const outcomes = [failingCall(self, 'A'), failingCall(self, 'B')]
const connectionError = new Error('host unreachable')
self.status = 'disconnected'
self.emit('error', connectionError)
assert.equal(outcomes[0].error, connectionError)
assert.equal(outcomes[1].error, connectionError)
assert.deepEqual(self.sent, [])
// a listener left behind would replay a call whose caller has long given up
assert.equal(self.listenerCount('ready'), 0)
assert.equal(self.listenerCount('error'), 0)
})
it('reports the error of a replay which failed too', function () {
const self = fakeClient()
const outcome = failingCall(self, 'A')
self.status = 'ready'
self.emit('ready')
const replayError = new Error('failed again')
self.sent[0].emitter.emit('error', replayError)
assert.equal(outcome.error, replayError)
})
it('gives up after 10 reconnects', function () {
const self = fakeClient()
self.reconnectCount = 10
const outcome = failingCall(self, 'A')
assert.equal(outcome.error.code, 'NotAuthenticated')
assert.equal(self.connectCalls, 0)
assert.equal(self.status, 'ready')
assert.deepEqual(self.sent, [])
})
it('emits an error which is not a session expiry as is, without reconnecting', function () {
const self = fakeClient()
const outcome = failingCall(self, 'A', FILE_LOCKED)
assert.equal(outcome.error.code, 'FileLocked')
assert.equal(self.connectCalls, 0)
assert.equal(self.reconnectCount, 0)
assert.equal(self.status, 'ready')
assert.deepEqual(self.sent, [])
})
it('recognizes an expiry which only says so in its faultstring', function () {
const self = fakeClient()
// no `detail`, hence no fault type to branch on
failingCall(self, 'A', {
root: { Envelope: { Body: { Fault: { faultstring: 'The session is not authenticated.' } } } },
})
assert.equal(self.reconnectCount, 1)
assert.equal(self.connectCalls, 1)
})
})

View File

@@ -1,6 +1,6 @@
{
"name": "@vates/node-vsphere-soap",
"version": "2.1.3",
"version": "2.2.0",
"description": "interface to vSphere SOAP/WSDL from node for interfacing with vCenter or ESXi, forked from node-vsphere-soap",
"main": "lib/client.mjs",
"author": "reedog117",
@@ -31,6 +31,7 @@
"node": ">=14"
},
"scripts": {
"postversion": "npm publish --access public"
"postversion": "npm publish --access public",
"test": "node --test lib/*.test.mjs"
}
}

View File

@@ -1,7 +1,7 @@
// place your own credentials here for a vCenter or ESXi server
// this information will be used for connecting to a vCenter instance
// for module testing
// name the file config-test.js
// place your own credentials here for a vCenter or ESXi server: they are used by
// vsphere-soap.integ.mjs to connect to a real instance.
//
// copy this file next to it, as `config-test.mjs`. It is ignored by git, do not commit it.
export const vCenterTestCreds = {
vCenterIP: 'vcsa',

View File

@@ -1,138 +0,0 @@
/*
vsphere-soap.test.js
tests for the vCenterConnectionInstance class
*/
import assert from 'assert'
import { describe, it } from 'node:test'
import * as vc from '../lib/client.mjs'
// eslint-disable-next-line n/no-missing-import
import { vCenterTestCreds as TestCreds } from '../config-test.mjs'
const VItest = new vc.Client(TestCreds.vCenterIP, TestCreds.vCenterUser, TestCreds.vCenterPassword, false)
describe('Client object initialization:', function () {
it('provides a successful login', { timeout: 5000 }, function (t, done) {
VItest.once('ready', function () {
assert.notEqual(VItest.userName, null)
assert.notEqual(VItest.fullName, null)
assert.notEqual(VItest.serviceContent, null)
done()
}).once('error', function (err) {
console.error(err)
// this should fail if there's a problem
assert.notEqual(VItest.userName, null)
assert.notEqual(VItest.fullName, null)
assert.notEqual(VItest.serviceContent, null)
done()
})
})
})
describe('Client reconnection test:', function () {
it('can successfully reconnect', { timeout: 5000 }, function (t, done) {
VItest.runCommand('Logout', { _this: VItest.serviceContent.sessionManager })
.once('result', function (result) {
// now we're logged out, so let's try running a command to test automatic re-login
VItest.runCommand('CurrentTime', { _this: 'ServiceInstance' })
.once('result', function (result) {
assert(result.returnval instanceof Date)
done()
})
.once('error', function (err) {
console.error(err)
})
})
.once('error', function (err) {
console.error(err)
})
})
})
// these tests don't work yet
describe('Client tests - query commands:', function () {
it('retrieves current time', { timeout: 5000 }, function (t, done) {
VItest.runCommand('CurrentTime', { _this: 'ServiceInstance' }).once('result', function (result) {
assert(result.returnval instanceof Date)
done()
})
})
it('retrieves current time 2 (check for event clobbering)', { timeout: 5000 }, function (t, done) {
VItest.runCommand('CurrentTime', { _this: 'ServiceInstance' }).once('result', function (result) {
assert(result.returnval instanceof Date)
done()
})
})
it('can obtain the names of all Virtual Machines in the inventory', { timeout: 20000 }, function (t, done) {
// get property collector
const propertyCollector = VItest.serviceContent.propertyCollector
// get view manager
const viewManager = VItest.serviceContent.viewManager
// get root folder
const rootFolder = VItest.serviceContent.rootFolder
let containerView, objectSpec, traversalSpec, propertySpec, propertyFilterSpec
// this is the equivalent to
VItest.runCommand('CreateContainerView', {
_this: viewManager,
container: rootFolder,
type: ['VirtualMachine'],
recursive: true,
}).once('result', function (result) {
// build all the data structures needed to query all the vm names
containerView = result.returnval
objectSpec = {
attributes: { 'xsi:type': 'ObjectSpec' }, // setting attributes xsi:type is important or else the server may mis-recognize types!
obj: containerView,
skip: true,
}
traversalSpec = {
attributes: { 'xsi:type': 'TraversalSpec' },
name: 'traverseEntities',
type: 'ContainerView',
path: 'view',
skip: false,
}
objectSpec = { ...objectSpec, selectSet: [traversalSpec] }
propertySpec = {
attributes: { 'xsi:type': 'PropertySpec' },
type: 'VirtualMachine',
pathSet: ['name'],
}
propertyFilterSpec = {
attributes: { 'xsi:type': 'PropertyFilterSpec' },
propSet: [propertySpec],
objectSet: [objectSpec],
}
// TODO: research why it fails if propSet is declared after objectSet
VItest.runCommand('RetrievePropertiesEx', {
_this: propertyCollector,
specSet: [propertyFilterSpec],
options: { attributes: { type: 'RetrieveOptions' } },
})
.once('result', function (result, raw) {
assert.notEqual(result.returnval.objects, null)
if (Array.isArray(result.returnval.objects)) {
assert.strictEqual(result.returnval.objects[0].obj.attributes.type, 'VirtualMachine')
} else {
assert.strictEqual(result.returnval.objects.obj.attributes.type, 'VirtualMachine')
}
done()
})
.once('error', function (err) {
console.error('\n\nlast request : ' + VItest.client.lastRequest, err)
})
})
})
})

View File

@@ -1,7 +1,7 @@
{
"name": "@vates/types",
"private": false,
"version": "1.30.1",
"version": "2.5.0",
"main": "./dist/index.mjs",
"exports": {
".": {
@@ -11,6 +11,10 @@
"./xo": "./dist/xo.d.mts",
"./xo-app": "./dist/xo-app.d.mts",
"./xen-api": "./dist/xen-api.d.mts",
"./backup-expression-context": {
"default": "./dist/backup-expression-context.mjs",
"types": "./dist/backup-expression-context.d.mts"
},
"./common": {
"default": "./dist/common.mjs",
"types": "./dist/common.d.mts"
@@ -33,7 +37,8 @@
"@eslint/js": "^9.19.0",
"typescript": "~5.6.3",
"typescript-eslint": "^8.61.0",
"@types/express": "^5.0.0"
"@types/express": "^5.0.0",
"@xen-orchestra/disk-transform": "^1.3.4"
},
"scripts": {
"build": "tsc",

View File

@@ -0,0 +1,38 @@
export const BACKUP_EXPRESSION_CONTEXT = {
run: {
dayOfWeek: { type: 'number' },
dayOfMonth: { type: 'number' },
hour: { type: 'number' },
month: { type: 'number' },
year: { type: 'number' },
},
sr: {
name_label: { type: 'string' },
type: { type: 'string' },
description: { type: 'string' },
tags: { type: 'string[]' },
},
backupRepository: {
name: { type: 'string' },
type: { type: 'string' },
tags: { type: 'string[]' },
},
vm: {
name_label: { type: 'string' },
tags: { type: 'string[]' },
power_state: { type: 'string' },
},
chainLength: { type: 'number' },
} as const
export type BACKUP_EXPRESSION_CONTEXT = (typeof BACKUP_EXPRESSION_CONTEXT)[keyof typeof BACKUP_EXPRESSION_CONTEXT]
export const BACKUP_SETTING_CONTEXT = {
healthCheckVmsWithTags: ['vm', 'run'],
fullInterval: ['vm', 'run', 'chainLength'],
vmFilter: ['vm', 'run'],
srs: ['sr', 'run'],
remotes: ['backupRepository', 'run'],
} as const
export type BACKUP_SETTING_CONTEXT = (typeof BACKUP_SETTING_CONTEXT)[keyof typeof BACKUP_SETTING_CONTEXT]

View File

@@ -1,4 +1,4 @@
import type { XoHost, XoNetwork, XoVif } from './xo.mjs'
import type { XapiXoRecord, XoHost, XoNetwork, XoVif } from './xo.mjs'
declare const __brand: unique symbol
@@ -663,6 +663,19 @@ export const SUPPORTED_VDI_FORMAT = {
export type SUPPORTED_VDI_FORMAT = (typeof SUPPORTED_VDI_FORMAT)[keyof typeof SUPPORTED_VDI_FORMAT]
/**
* Formats a VDI can be exported to by XO: the XAPI ones, plus the ones XO generates
* itself from the blocks of the disk.
*/
export const SUPPORTED_VDI_EXPORT_FORMAT = {
raw: 'raw',
vhd: 'vhd',
qcow2: 'qcow2',
vmdk: 'vmdk',
} as const
export type SUPPORTED_VDI_EXPORT_FORMAT = (typeof SUPPORTED_VDI_EXPORT_FORMAT)[keyof typeof SUPPORTED_VDI_EXPORT_FORMAT]
// ----- XAPI Stats
type XapiStatsResponse<T> = {
@@ -755,10 +768,11 @@ export type TrafficRuleDirection = 'from' | 'to' | 'from/to'
export type RawTrafficRule = {
allow: boolean
protocol: string
protocol: TrafficRuleProtocol
ipRange: string
direction: string
direction: TrafficRuleDirection
port?: string
priority?: number
}
type BaseTrafficRule = RawTrafficRule & {
@@ -769,3 +783,38 @@ type BaseTrafficRule = RawTrafficRule & {
export type TrafficRule =
| (BaseTrafficRule & { type: XoVif['type']; sourceId: XoVif['id'] })
| (BaseTrafficRule & { type: XoNetwork['type']; sourceId: XoNetwork['id'] })
export const XAPI_TYPES: readonly string[] = [
'network',
'VIF',
'message',
'gpuGroup',
'host',
'PBD',
'PCI',
'PGPU',
'PIF',
'pool',
'SR',
'VBD',
'VDI',
'VDI-snapshot',
'VDI-unmanaged',
'vgpu',
'vgpuType',
'VM',
'VM-controller',
'VM-snapshot',
'VM-template',
'VTPM',
'SM',
] satisfies readonly XapiXoRecord['type'][]
/** steps of a rolling pool update on one host, in execution order */
export const RPU_RECOVERY_STEP_NAMES = ['evacuate', 'update', 'reboot', 'enable', 'restoreVms'] as const
export type RPU_RECOVERY_STEP_NAME = (typeof RPU_RECOVERY_STEP_NAMES)[number]
export const COMMON_TAG = {
NOBAK: 'xo:no-bak',
} as const

View File

@@ -4,5 +4,8 @@ export * from './xo.mjs'
export * from './xo-app.mjs'
export * from './lib/xen-orchestra-xapi.mjs'
export * from './lib/xen-orchestra-acl.mjs'
export * from './lib/xen-orchestra-plugins.mjs'
export * from './lib/xen-orchestra-v2v.mjs'
export * from './lib/complex-matcher.mjs'
export * from './lib/rest-api.mjs'
export * from './lib/xen-orchestra-vmware-explorer.mjs'

View File

@@ -4,40 +4,34 @@ import type { XoApp } from '../xo-app.mjs'
export type SecurityName = '*' | 'token' | 'basic' | 'none'
export type FieldDefinition =
export type FieldDefinition = { optional?: boolean; nullable?: boolean } & (
| {
type: 'string'
example?: string
optional?: boolean
}
| {
type: 'boolean'
example?: boolean
optional?: boolean
}
| {
type: 'number'
example?: number
optional?: boolean
}
| {
type: 'enum'
enum: string[]
example?: string
optional?: boolean
}
| {
type: 'object'
fields: Record<string, FieldDefinition>
optional?: boolean
}
| {
type: 'array'
items: FieldDefinition
example?: unknown[]
optional?: boolean
}
)
export type ParamFieldDefinition = Exclude<
FieldDefinition,
{ type: 'boolean' } | { type: 'object' } | { type: 'array' }

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,219 @@
import type { Branded } from '../common.mjs'
import type { XoNetwork, XoSr, XoVmTemplate } from '../xo.mjs'
// Stateless API: requests needing the source carry the connection in a POST body (never in URLs or access logs) and
// xo-server stores nothing it reads from the source.
export const V2V_PLATFORM = {
VMWARE: 'vmware',
} as const
export type V2V_PLATFORM = (typeof V2V_PLATFORM)[keyof typeof V2V_PLATFORM]
// 422 rather than 401 for invalid source credentials: a 401 would open the browser Basic authentication prompt
export const V2V_ERROR_CODE = {
INVALID_CREDENTIALS: 'V2V_INVALID_CREDENTIALS',
} as const
export type V2V_ERROR_CODE = (typeof V2V_ERROR_CODE)[keyof typeof V2V_ERROR_CODE]
// Clones, restores and replicas inherit `other_config`: the imported VM records its own UUID to be told apart from
// them, and the source identity because a moref is only unique within one vCenter. The source identity is the
// inventory `instanceUuid`, or `<lowercased host>:<port>` for a standalone ESXi which has none.
export const V2V_VM_OTHER_CONFIG = {
SOURCE_VM_ID: 'sourceVmId',
SOURCE_SNAPSHOT_ID: 'sourceSnapshotId',
IMPORTED_VM_UUID: 'xo:v2v:importedVmUuid',
SOURCE_ID: 'xo:v2v:sourceId',
SOURCE_VM_INSTANCE_UUID: 'xo:v2v:sourceVmInstanceUuid',
} as const
export type V2V_VM_OTHER_CONFIG = (typeof V2V_VM_OTHER_CONFIG)[keyof typeof V2V_VM_OTHER_CONFIG]
export type XoV2vVmOtherConfig = Partial<Record<V2V_VM_OTHER_CONFIG, string>>
// The disk of the source chain an imported VDI last received, one of the `chainPaths` of its source disk: the CID
// cannot tell a fresh snapshot delta from its parent, the path can
export const V2V_VDI_OTHER_CONFIG = {
DISK_PATH: 'esxi_diskPath',
} as const
export type V2V_VDI_OTHER_CONFIG = (typeof V2V_VDI_OTHER_CONFIG)[keyof typeof V2V_VDI_OTHER_CONFIG]
export const V2V_START_BLOCKED_REASON = 'Esxi migration in progress...'
export const V2V_PARTIAL_IMPORT_TAG = 'xo:no-bak=V2V partial import'
export const V2V_SNAPSHOT_NAME = {
AFTER_PARTIAL_IMPORT: 'after partial import from V2V',
AFTER_COMPLETE_IMPORT: 'after complete import from V2V',
COMPLETE_IMPORT: 'complete import from V2V',
} as const
export type V2V_SNAPSHOT_NAME = (typeof V2V_SNAPSHOT_NAME)[keyof typeof V2V_SNAPSHOT_NAME]
// Set on the task of a start so the front can tell which source VMs are being imported
export const V2V_TASK_PROPERTY = {
SOURCE_VM_IDS: 'sourceVmIds',
} as const
export type XoV2vConnection = {
host: string
port: number
username: string
password: string
skipSslCheck: boolean
}
export type XoV2vStatus = {
vectura: { status: 'success' } | { status: 'error'; error: string }
}
export type XoV2vInventoryPayload = {
platform: V2V_PLATFORM
connection: XoV2vConnection
}
export type XoV2vDatacenter = {
id: Branded<'v2v-datacenter'>
name: string
}
export type XoV2vCluster = {
id: Branded<'v2v-cluster'>
name: string
datacenterId: XoV2vDatacenter['id']
}
export type XoV2vHost = {
id: Branded<'v2v-host'>
name: string
clusterId?: XoV2vCluster['id']
datacenterId: XoV2vDatacenter['id']
}
// Sizes are in bytes, as in the rest of XO, not in the KB/MB VMware returns
export type XoV2vDatastore = {
id: Branded<'v2v-datastore'>
name: string
type: string
capacity: number
free: number
hostIds: XoV2vHost['id'][]
}
export type XoV2vPortGroup = {
id: Branded<'v2v-port-group'>
name: string
vlan?: number
}
export const V2V_DISK_MODE = {
PERSISTENT: 'persistent',
NON_PERSISTENT: 'nonpersistent',
INDEPENDENT_PERSISTENT: 'independent_persistent',
INDEPENDENT_NON_PERSISTENT: 'independent_nonpersistent',
UNDOABLE: 'undoable',
APPEND: 'append',
} as const
export type V2V_DISK_MODE = (typeof V2V_DISK_MODE)[keyof typeof V2V_DISK_MODE]
// `key` is the vmx device node (`scsi0:0`, `ethernet0`…), the identifier the import matches the start maps against
export type XoV2vDisk = {
key: Branded<'v2v-disk'>
label: string
datastoreId: XoV2vDatastore['id']
capacity: number
mode: V2V_DISK_MODE
isRdm: boolean
isEncrypted: boolean
// `dir/disk.vmdk` in its datastore (no `[datastore]` prefix) of the base disk, then of each snapshot delta down to
// the active disk: the chain the import walks for this vmx node
chainPaths: string[]
}
// No port group for opaque (NSX) or missing backings
export type XoV2vNic = {
key: Branded<'v2v-nic'>
label: string
portGroupId?: XoV2vPortGroup['id']
macAddress: string
}
export const V2V_VM_POWER_STATE = {
POWERED_ON: 'poweredOn',
POWERED_OFF: 'poweredOff',
SUSPENDED: 'suspended',
} as const
export type V2V_VM_POWER_STATE = (typeof V2V_VM_POWER_STATE)[keyof typeof V2V_VM_POWER_STATE]
export const V2V_FIRMWARE = {
BIOS: 'bios',
UEFI: 'uefi',
} as const
export type V2V_FIRMWARE = (typeof V2V_FIRMWARE)[keyof typeof V2V_FIRMWARE]
export type XoV2vVm = {
id: Branded<'v2v-vm'>
instanceUuid: string
name: string
description: string
powerState: V2V_VM_POWER_STATE
hostId: XoV2vHost['id']
nCpus: number
memory: number
firmware: V2V_FIRMWARE
secureBoot: boolean
datastoreIds: XoV2vDatastore['id'][]
snapshotCount: number
// Compared with `other_config.sourceSnapshotId` of a partial import: a delta is only possible while they match
currentSnapshotId?: string
disks: XoV2vDisk[]
nics: XoV2vNic[]
}
export type XoV2vInventory = {
product: string
version: string
// Absent on a standalone ESXi, identified by its connection address instead
instanceUuid?: string
datacenters: XoV2vDatacenter[]
clusters: XoV2vCluster[]
hosts: XoV2vHost[]
datastores: XoV2vDatastore[]
portGroups: XoV2vPortGroup[]
vms: XoV2vVm[]
}
export type XoV2vStartOptions = {
stopSource: boolean
startVm: boolean
cleanup: boolean
}
// A resumed import sends no disk or NIC map: the back end reads its mapping from the existing XCP-ng VM
export type XoV2vStartPayloadVm = Partial<XoV2vStartOptions> & {
id: XoV2vVm['id']
disks?: Record<XoV2vDisk['key'], XoSr['id']>
nics?: Record<XoV2vNic['key'], XoNetwork['id']>
template?: XoVmTemplate['id']
tags: string[]
keepMac: boolean
}
// Body of `POST /pools/{id}/actions/import_vms_from_vmware`: the pool is in the path. Checked against the body of the
// route when the REST API is built, which refuses any extra key
export type XoV2vStartPayload = {
connection: XoV2vConnection
defaults: XoV2vStartOptions & {
sr: XoSr['id']
network: XoNetwork['id']
template: XoVmTemplate['id']
}
vms: XoV2vStartPayloadVm[]
stopOnError: boolean
concurrency: number
}

View File

@@ -0,0 +1,103 @@
// shape of `Esxi#getInventory()` in `@xen-orchestra/vmware-explorer`
// the objects reference each other through their VMware managed object references (moref)
export type VmwareDatacenter = {
id: string
name: string
}
export type VmwareCluster = {
id: string
name: string
datacenter?: string
}
export type VmwareHost = {
id: string
name: string
/** a cluster, or the compute resource of a standalone host */
parent?: string
datacenter?: string
}
export type VmwareDatastore = {
id: string
name?: string
type?: string
capacity?: number
freeSpace?: number
/** the hosts which mount the datastore and can reach it */
hosts: string[]
}
export type VmwareVlan = number | { start: number; end: number }[]
export type VmwareStandardNetwork = {
id: string
name: string
distributed: false
}
export type VmwareDistributedPortgroup = {
id: string
name: string
distributed: true
vlan?: VmwareVlan
}
export type VmwareNetwork = VmwareStandardNetwork | VmwareDistributedPortgroup
export type VmwareVmDisk = {
/** node of the disk, as the vmx names it, e.g. `scsi0:0` */
key: string
label: string
capacity: number
datastore?: string
diskMode?: string
rdm: boolean
encrypted: boolean
/** `dir/disk.vmdk` in its datastore of the base disk, then of each snapshot delta down to the active disk */
chainPaths: string[]
}
export type VmwareVmNic = {
/** device key of the card */
key: string
label: string
macAddress?: string
addressType?: string
network?: string
}
export type VmwareVm = {
id: string
name?: string
annotation?: string
instanceUuid?: string
firmware?: string
secureBoot: boolean
isTemplate: boolean
cpus?: number
memory?: number
powerState?: 'poweredOff' | 'poweredOn' | 'suspended'
host?: string
datastores: string[]
currentSnapshot?: string
snapshotCount: number
disks: VmwareVmDisk[]
nics: VmwareVmNic[]
}
export type VmwareInventory = {
/** e.g. `VMware vCenter Server`, `VMware ESXi` */
product?: string
version?: string
/** only set by a vCenter */
instanceUuid?: string
datacenters: VmwareDatacenter[]
clusters: VmwareCluster[]
hosts: VmwareHost[]
datastores: VmwareDatastore[]
networks: VmwareNetwork[]
vms: VmwareVm[]
}

View File

@@ -24,6 +24,8 @@ import type {
VIF_LOCKING_MODE,
VM_OPERATIONS,
} from '../common.mjs'
import type { DiskPassthrough } from '@xen-orchestra/disk-transform'
import type { EventEmitter } from 'node:events'
import type { PassThrough, Readable } from 'node:stream'
import type {
XapiXoRecord,
@@ -128,6 +130,8 @@ export type XsPatches = {
uuid?: string
}
type WrappedRecordOfType<Type extends WrappedXenApiRecord['$type']> = Extract<WrappedXenApiRecord, { $type: Type }>
export interface Xapi {
status: string
pool?: { uuid: string }
@@ -145,6 +149,14 @@ export interface Xapi {
ref: T['$ref'],
field: K
): Promise<T[K]>
// the record is inferred from `type`, for a JS caller which cannot spell it out as a type argument
getField<Type extends WrappedXenApiRecord['$type'], Field extends keyof WrappedRecordOfType<Type>>(
type: Type,
ref: WrappedRecordOfType<Type>['$ref'],
field: Field
): Promise<WrappedRecordOfType<Type>[Field]>
/** Synchronous: the record as the object cache currently holds it. */
getObjectByRef<Ref extends WrappedXenApiRecord['$ref']>(ref: Ref): Extract<WrappedXenApiRecord, { $ref: Ref }>
getObject: <
XoRecord extends XapiXoRecord,
WrappedRecord extends WrappedXenApiRecord = Extract<WrappedXenApiRecord, { $type: XoRecord['type'] }>,
@@ -281,6 +293,42 @@ export interface Xapi {
}
): Promise<XenApiVdi['$ref']>
SR_reclaimSpace(ref: XenApiSr['$ref']): Promise<void>
VBD_create(params: {
bootable?: boolean
currently_attached?: boolean
device?: string
empty?: boolean
mode?: VBD_MODE
other_config?: Record<string, string>
qos_algorithm_params?: Record<string, string>
qos_algorithm_type?: string
/** a plug failure, only logged otherwise, is thrown: the VBD exists all the same */
throwVbdPlug?: boolean
type?: VBD_TYPE
unpluggable?: boolean
userdevice?: string
VDI?: XenApiVdi['$ref']
VM: XenApiVm['$ref']
}): Promise<XenApiVbd['$ref']>
/** unplugs the VBD first, if needed */
VBD_destroy(ref: XenApiVbd['$ref']): Promise<void>
VDI_create(
params: {
name_description?: string
name_label: string
other_config?: Record<string, string>
read_only?: boolean
sharable?: boolean
SR?: XenApiSr['$ref']
tags?: string[]
type?: VDI_TYPE
virtual_size: number
xenstore_data?: Record<string, string>
},
opts?: { sm_config?: Record<string, string> }
): Promise<XenApiVdi['$ref']>
/** a VDI which no longer exists counts as destroyed */
VDI_destroy(ref: XenApiVdi['$ref']): Promise<void>
editVm(
id: XoVm['id'],
props: EditVmProps,
@@ -469,6 +517,12 @@ export interface Xapi {
destroySr(id: XoSr['id']): Promise<void>
xostor_delete(ref: XenApiSr['$ref']): Promise<void>
objects: {
// emits `add`/`update`/`remove` only while the connection watches the pool events
allIndexes: {
type: {
getEventEmitterByType(type: WrappedXenApiRecord['$type']): EventEmitter
}
}
indexes: {
type: {
[XenApiRecord in WrappedXenApiRecord as XenApiRecord['$type']]: Record<XenApiRecord['uuid'], XenApiRecord>
@@ -476,3 +530,42 @@ export interface Xapi {
}
}
}
/**
* Parameters accepted by the `XapiDiskSource` constructor.
*
* `baseRef` turns the export into a differencing one: leaving it out exports the whole disk.
* `preferNbd` only expresses a preference, {@link XapiDiskSource} silently falls back to a plain
* stream export — and, when `baseRef` can't be used as a base, to a full one.
*/
export interface XapiDiskSourceOptions {
baseRef?: XenApiVdi['$ref']
blockSize?: number
nbdConcurrency?: number
/** When true, only `getBlockIndexes()` may be called: `readBlock()` throws. */
onlyListChangedBlocks?: boolean
preferNbd?: boolean
timeout?: number
vdiRef: XenApiVdi['$ref']
xapi: Xapi
}
/**
* Disk source handling the fallback logic of a VDI export: NBD + CBT, then NBD + stream export
* for the block list, then plain stream export.
*
* {@link XapiDiskSourceOptions.preferNbd} is a preference, not a guarantee: call
* {@link XapiDiskSource.useNbd} and {@link XapiDiskSource.useCbt} after `init()` to know what the
* export actually used.
*/
export interface XapiDiskSource extends DiskPassthrough {
useCbt(): boolean
useNbd(): boolean
}
/**
* Signature of `@xen-orchestra/xapi`'s `parseDateTime`.
*
* Returns a Unix timestamp in seconds, or `null` if the field is empty (as encoded by XAPI).
*/
export type ParseDateTime = (input: string | number | Date) => number | null

View File

@@ -824,7 +824,10 @@ export interface XenApiSr {
VDIs: XenApiVdi['$ref'][]
virtual_allocation: number
}
export type XenApiSrWrapped = WrapperXenApi<XenApiSr, 'SR', XenApiSrCallMethods>
export type XenApiSrWrapped = WrapperXenApi<XenApiSr, 'SR', XenApiSrCallMethods> & {
// resolved by xen-api from `PBDs`
$PBDs: XenApiPbdWrapped[]
}
export interface XenApiSrStat {
$ref: Branded<'sr_stat'>

View File

@@ -1,26 +1,38 @@
import { EventEmitter } from 'node:stream'
import type { EventEmitter, Readable } from 'node:stream'
import type {
AnyXoJob,
AnyXoLog,
NonXapiXoRecord,
XoBackupDiskPartition,
XapiXoRecord,
XoAuthenticationToken,
XoBackupRepository,
XoConfigBackupArchive,
XoGroup,
XoHost,
XoMetadataBackupJob,
XoMirrorBackupJob,
XoNetwork,
XoPlugin,
XoPool,
XoPoolBackupArchive,
XoProxy,
XoSchedule,
XoServer,
XoSr,
XoTask,
XoUser,
XoVif,
XoVm,
XoVmBackupArchive,
XoVmBackupJob,
XoVmTemplate,
} from './xo.mjs'
import { VatesTask } from './lib/vates-task.mjs'
import type { PluginRestRouteDefinition } from './lib/rest-api.mjs'
import type { VmwareInventory } from './lib/xen-orchestra-vmware-explorer.mjs'
import type { XoV2vStatus } from './lib/xen-orchestra-v2v.mjs'
import type { RPU_RECOVERY_STEP_NAME } from './common.mjs'
import {
Xapi,
XapiHostStats,
@@ -125,7 +137,89 @@ type License = {
bundleInfo?: { name: string; id: string }
}
export type XoApp = {
export type PoolRollingUpdateRecoveryStep = {
status: 'pending' | 'running' | 'observed-succeeded' | 'failed' | 'not-needed'
startedAt?: string
finishedAt?: string
}
/** error serialized by the recovery recorder: secret-looking keys are redacted */
export type PoolRollingUpdateRecoveryError = {
name?: string
message?: string
stack?: string
code?: string | number
[key: string]: unknown
}
export type PoolRollingUpdateRecoveryHost = {
status: 'pending' | 'running' | 'succeeded' | 'failed' | 'not-needed'
steps: Record<RPU_RECOVERY_STEP_NAME, PoolRollingUpdateRecoveryStep>
lastError: PoolRollingUpdateRecoveryError | null
}
export type PoolRollingUpdateRecoveryRun = {
runId: string
poolId: string
status: 'preparing' | 'running' | 'interrupted' | 'resuming' | 'failed' | 'cleaning' | 'succeeded'
startedAt: string
updatedAt: string
finishedAt?: string
interruptedAt?: string
taskId?: string
variant?: 'xcp' | 'xs-cdn'
hostOrder?: string[]
hosts: Record<string, PoolRollingUpdateRecoveryHost>
lastError: PoolRollingUpdateRecoveryError | null
/** VM UUID -> UUID of the host it must be started on */
haltedPinnedVms: Record<string, string>
}
/** record unreadable or of an unknown schema version: recovery needs a human */
export type PoolRollingUpdateRecoveryBlocked = {
poolId?: string
runId?: string
status: 'blocked'
blockedReason: string
}
export type PoolRollingUpdateRecovery = PoolRollingUpdateRecoveryRun | PoolRollingUpdateRecoveryBlocked
/** A disk of a backup archive currently served as an iSCSI LUN */
export type BackupArchiveDiskMount = {
/** Handle to pass to `unmountBackupArchiveDisk` */
id: string
/** UUID of the SR introduced on the host */
srUuid: string
/** UUID of the VDI exposing the backup disk, read-only unless it is cached */
vdiUuid: string
/** IQN of the target serving the disk */
iqn: string
/** Address of the portal, as advertised to the host */
address: string
/** Port of the portal (ephemeral, one target per mount) */
port: number
}
type XoStoreRangeOptions = {
gt?: string
gte?: string
lt?: string
lte?: string
limit?: number
reverse?: boolean
}
// `@xen-orchestra/vmware-explorer` is not typed
export type EsxiConnectionParams = {
/** `host` or `host:port` */
host: string
user: string
password: string
sslVerify: boolean
}
export type XoApp = EventEmitter & {
hooks: EventEmitter
_redis: {
get(key: string): Promise<string | null>
@@ -180,6 +274,7 @@ export type XoApp = {
user?: XoUser
permission?: XoUser['permission'] | 'none' | null
}
backupLogsEe: EventEmitter
// methods ------------
allocIpAddresses(vifId: XoVif['id'], addAddresses?: string[], removeAddresses?: string[]): Promise<void>
@@ -196,14 +291,51 @@ export type XoApp = {
}
) => () => void // eslint-disable-line @typescript-eslint/no-explicit-any
registerRestRoutes: (routes: PluginRestRouteDefinition[], base?: string) => () => void
/* a plugin's guard on the hosts VMs start on or move to (checkPlacement), returns its removal */
registerPlacementGuard: (guard: {
name: string
bypassRight: string
check: (context: {
operation: 'start' | 'resume' | 'migrate'
vm: XoVm
host: XoHost | undefined
pool: XoPool
vifNetworks: Record<XoVif['id'], XoNetwork['id']> | undefined
}) => Promise<{ host?: XoHost; reason: string } | undefined>
mayBypass: (context: { user: XoUser; pool: XoPool }) => Promise<boolean>
}) => () => void
authenticateUser: (
credentials: { token?: string; username?: string; password?: string },
userData?: { ip?: string },
opts?: { bypassOtp?: boolean; bypassTaskCreation?: boolean }
) => Promise<{ bypassOtp: boolean; expiration: number; user: XoUser }>
backupGuard(poolId: XoPool['id']): Promise<void>
backupGuard(objectId: XapiXoRecord['id'], opts?: { bypassBackupCheck?: boolean; operation: string }): Promise<void>
/* Throw if no authorization */
checkFeatureAuthorization(featureCode: FeatureCode): Promise<void>
/* whether this machine can run the tools of an import from VMware */
checkEsxiDependencies(): Promise<XoV2vStatus>
/* throw if the user may not migrate this VM, with its firewall group tags, to another pool */
checkMigration(user: XoUser | undefined, vm: XoVm, destinationPoolId: XoPool['id']): Promise<void>
/*
* throw if a placement guard (a plugin's: the firewall's) refuses to put the VM onto the host (none: the one XAPI
* picks in the VM's pool), unless `bypass` is set by a user the guard allows (placementRefused, forbiddenOperation)
*/
checkPlacement(params: {
operation: 'start' | 'resume' | 'migrate'
user: XoUser | undefined
vm: XoVm
host?: XoHost
bypass?: boolean
vifNetworks?: Record<XoVif['id'], XoNetwork['id']>
}): Promise<void>
/* throw if the user may not add or remove these tags on the object (tags reserved to the firewall) */
checkTagsChange(
user: XoUser | undefined,
object: XapiXoRecord,
change: { added?: string[]; removed?: string[] }
): Promise<void>
/* validate, apply and persist the configuration of a plugin */
configurePlugin(id: string, configuration: unknown, mergeWithExisting?: boolean): Promise<void>
/* connect a server (XCP-ng/XenServer) */
connectXenServer(id: XoServer['id']): Promise<void>
// TODO: replace all XoAclBasePrivilege with a more strict type. (discriminate union)
@@ -217,7 +349,29 @@ export type XoApp = {
id: XoAclRole['id'],
params?: { name?: XoAclRole['name']; description?: XoAclRole['description'] }
): Promise<XoAclRole['id']>
createAclV2Role(role: { name: XoAclRole['name']; description?: XoAclRole['description'] }): Promise<XoAclRole>
createAclV2Role(role: {
name: XoAclRole['name']
description?: XoAclRole['description']
privileges?: Omit<XoAclBasePrivilege, 'id' | 'roleId'>[]
}): Promise<XoAclRole>
createApiConnection(remoteAddress: string): EventEmitter & {
close(): void
get<T = unknown>(key: string, defaultValue: unknown): T
has(key: string): boolean
set(key: string, value: unknown): void
unset(key: string): void
}
createBackupNgJob<T extends XoVmBackupJob | XoMirrorBackupJob>(
type: T['type'],
props: Omit<T, 'id' | 'type'>,
/** keyed by temporary IDs, which can be used as keys in `props.settings` */
schedules?: Record<string, Pick<XoSchedule, 'cron' | 'enabled' | 'name' | 'timezone'>>
): Promise<T>
createMetadataBackupJob(
props: Omit<XoMetadataBackupJob, 'id' | 'type'>,
/** keyed by temporary IDs, which can be used as keys in `props.settings` */
schedules: Record<string, Pick<XoSchedule, 'cron' | 'enabled' | 'name' | 'timezone'>>
): Promise<XoMetadataBackupJob>
createAuthenticationToken(opts: {
client?: {
id?: string
@@ -233,7 +387,22 @@ export type XoApp = {
proxy?: XoProxy['id']
url: string
}): Promise<XoBackupRepository>
createUser(params: { name?: string; password?: string; [key: string]: unknown }): Promise<XoUser>
createSchedule(params: {
cron: XoSchedule['cron']
enabled: XoSchedule['enabled']
jobId: XoSchedule['jobId']
name?: XoSchedule['name']
timezone?: XoSchedule['timezone']
userId?: XoUser['id']
}): Promise<XoSchedule>
createUser(params: {
firstname?: string
lastname?: string
name?: string
password?: string
username?: string
[key: string]: unknown
}): Promise<XoUser>
deleteAclV2GroupRole(
groupId: XoGroup['id'],
roleId: XoAclRole['id'],
@@ -242,11 +411,16 @@ export type XoApp = {
deleteAclV2Privilege(privilegeId: XoAclBasePrivilege['id'], options?: { force?: boolean }): Promise<boolean>
deleteAclV2Role(roleId: XoAclRole['id'], options?: { force?: boolean }): Promise<boolean>
deleteGroup(id: XoGroup['id']): Promise<void>
deleteSchedule(id: XoSchedule['id']): Promise<void>
deleteUser(id: XoUser['id']): Promise<void>
detachHostFromPool(hostId: XoHost['id']): Promise<void>
/* tags added and removed when `oldTags` are replaced by `newTags` */
diffTags(oldTags: string[], newTags: string[]): { added: string[]; removed: string[] }
createGroup(params: { name: string; provider?: string; providerGroup?: string }): Promise<XoGroup>
disablePluginAutoload(id: XoPlugin['id']): Promise<void>
/* disconnect a server (XCP-ng/XenServer) */
disconnectXenServer(id: XoServer['id']): Promise<void>
enablePluginAutoload(id: XoPlugin['id']): Promise<void>
findEnabledScheduleSequenceFromSchedule(id: XoSchedule['id']): Promise<XoSchedule | undefined>
getAclV2Privilege(id: XoAclBasePrivilege['id']): Promise<XoAclBasePrivilege>
getAclV2Privileges(): Promise<XoAclBasePrivilege[]>
@@ -257,6 +431,7 @@ export type XoApp = {
roleId: XoAclRole['id'],
opts?: { bypassAuthorization?: boolean }
): Promise<boolean>
deleteSchedule(id: XoSchedule['id']): Promise<void>
getAclV2GroupRoles(
groupId: XoGroup['id'],
opts?: { bypassAuthorization?: boolean }
@@ -271,10 +446,14 @@ export type XoApp = {
getAllProxies(): Promise<XoProxy[]>
getAllJobs<T extends AnyXoJob['type']>(type: T): Promise<Extract<AnyXoJob, { type: T }>[]>
getAllJobs(type?: string): Promise<AnyXoJob[]>
getPlugin(id: XoPlugin['id']): Promise<XoPlugin>
getPlugins(): Promise<XoPlugin[]>
getProxy(id: XoProxy['id']): Promise<XoProxy>
getRemote(id: XoBackupRepository['id']): Promise<XoBackupRepository>
getAllRemotes(): Promise<XoBackupRepository[]>
removeRemote(id: XoBackupRepository['id']): Promise<void>
/** Also deletes the schedules of the job */
removeJob(id: AnyXoJob['id']): Promise<void>
getAllRemotesInfo(): Promise<
Record<
XoBackupRepository['id'],
@@ -291,11 +470,22 @@ export type XoApp = {
>
>
getAllSchedules(): Promise<XoSchedule[]>
getAllUsers(): Promise<XoUser[]>
getAllUsers(opts?: { obfuscatePassword?: boolean }): Promise<XoUser[]>
getAllXenServers(): Promise<XoServer[]>
getAuthenticationTokensForUser(userId: XoUser['id']): Promise<XoAuthenticationToken[]>
/** Archive/host a live-mounted disk belongs to, as recorded by `mountBackupArchiveDisk` */
getBackupArchiveDiskMountOwner(id: BackupArchiveDiskMount['id']): {
archiveId: XoVmBackupArchive['id']
hostId: XoHost['id']
/** set when the disk is served by a proxy instead of this appliance */
proxyId?: XoProxy['id']
}
getBackupNgLogs(): Promise<Record<string, AnyXoLog>>
getBackupNgLogs(id: AnyXoLog['id']): Promise<AnyXoLog>
/** Please use `REMOVE_CACHE_ENTRY` const to clear the cache */
getBackupNgLogs(removeCacheEntry: {}): void
/** Please use `REMOVE_CACHE_ENTRY` const to clear a cache entry */
getBackupNgLogs(removeCacheEntry: {}, id: AnyXoLog['id']): void
getBackupNgLogsSorted(opts: {
after?: number
before?: number
@@ -305,6 +495,7 @@ export type XoApp = {
getGroup(id: XoGroup['id']): Promise<XoGroup>
getHVSupportedVersions: undefined | (() => Promise<{ [key: XoHost['productBrand']]: string }>)
getJob<T extends AnyXoJob>(id: T['id']): Promise<T>
isJobSequence(job: AnyXoJob): boolean
getObject: <T extends XapiXoRecord>(id: T['id'], type?: T['type'] | T['type'][]) => T
getObjectsByType: <T extends XapiXoRecord>(
type: T['type'],
@@ -312,7 +503,21 @@ export type XoApp = {
) => Record<T['id'], T> | undefined
getTotalBackupSizeOnRemote(id: XoBackupRepository['id']): Promise<{ onDisk: number }>
getSchedule(id: XoSchedule['id']): Promise<XoSchedule>
getUser: (id: XoUser['id']) => Promise<XoUser>
getStore(namespace: string): Promise<{
/** rejects with an error having `notFound: true` when the key does not exist */
get(key: string): Promise<unknown>
has(key: string): Promise<boolean>
put(key: string, value: unknown): Promise<void>
del(key: string): Promise<void>
del(key: string, cb: (error?: Error | null) => void): void
/** object mode stream of `{ key, value }` */
createReadStream(opts?: XoStoreRangeOptions): Readable
/** object mode stream of keys */
createKeyStream(opts?: XoStoreRangeOptions): Readable
}>
getUser: (id: XoUser['id'], opts?: { obfuscatePassword?: boolean }) => Promise<XoUser>
getUserIdentityFields(): string[]
getEsxiInventory(params: EsxiConnectionParams): Promise<VmwareInventory>
getXapi(maybeId: XapiXoRecord['id'] | XapiXoRecord): Xapi
getXapiHostStats: (hostId: XoHost['id'], granularity?: XapiStatsGranularity) => Promise<XapiHostStats>
getXapiObject: <T extends XapiXoRecord>(
@@ -322,17 +527,90 @@ export type XoApp = {
getXapiPoolStats(poolId: XoPool['id'], granularity?: XapiStatsGranularity): Promise<XapiPoolStats>
getXapiVmStats: (vmId: XoVm['id'], granularity?: XapiStatsGranularity) => Promise<XapiVmStats>
getXenServer(id: XoServer['id']): Promise<XoServer>
/** the connection of the pool holding this object, which may not be connected */
getXenServerIdByObject<T extends XapiXoRecord>(maybeId: T['id'] | T, type?: T['type'] | T['type'][]): XoServer['id']
getXenServerWithCredentials(id: XoServer['id']): Promise<XoServer & { password: string }>
getXoEventEmitterByType(type: string): EventEmitter
hasFeatureAuthorization(featureCode: string): Promise<boolean>
/* may the user do this action of the RBAC `firewall` resource on the pool's firewall (XO admin, RBAC privilege on the pool, or the legacy ACL that counts as it); no user: xo-server's own work, allowed */
hasFirewallRight(
user: XoUser | undefined,
poolId: XoPool['id'],
action: 'read' | 'publish' | 'membership' | 'quarantine:add' | 'quarantine:remove'
): Promise<boolean>
hasObject<T extends XapiXoRecord>(id: T['id'], type: T['type']): boolean
listBackupNgDiskPartitions(remoteId: XoBackupRepository['id'], diskId: string): Promise<XoBackupDiskPartition[]>
listBackupNgPartitionFiles(
remoteId: XoBackupRepository['id'],
diskId: string,
partitionId: string | undefined,
path: string
// keys ending in "/" are directories; files carry size (bytes) and mtime (epoch ms).
// Values may be empty ({}) for directories or when served by an older proxy.
): Promise<Record<string, { size?: number; mtime?: number }>>
fetchBackupNgPartitionFiles(
remoteId: XoBackupRepository['id'],
diskId: string,
partitionId: string | undefined,
paths: string[],
format: string
): Promise<Readable>
loadPlugin(id: XoPlugin['id']): Promise<void>
listMetadataBackups(backupRepositoryIds: XoBackupRepository['id'][]): Promise<{
xo: Record<XoBackupRepository['id'], XoConfigBackupArchive[]>
pool: Record<XoBackupRepository['id'], Record<XoPool['id'], XoPoolBackupArchive[]>>
}>
/** `null` when the listing of a backup repository failed */
listVmBackupsNg(
backupRepositoryIds: XoBackupRepository['id'][],
opts?: { _forceRefresh?: boolean; vmId: XoVm['id'] }
): Promise<Record<XoBackupRepository['id'], Record<XoVm['id'], XoVmBackupArchive[]>>>
opts?: { _forceRefresh?: boolean; vmId?: XoVm['id'] }
): Promise<Record<XoBackupRepository['id'], Record<XoVm['id'], XoVmBackupArchive[]> | null>>
/** imports one VM from VMware, resuming a previous partial import when possible, returns the UUID of the XCP-ng VM */
migrationfromEsxi(
params: EsxiConnectionParams & {
sr: XoSr['id']
network: XoNetwork['id']
/** overrides `sr`, by disk node (e.g. `scsi0:0`) */
disks?: Record<string, XoSr['id']>
/** overrides `network`, by NIC device key (e.g. `4000`) */
nics?: Record<string, XoNetwork['id']>
template: XoVmTemplate['id']
stopSource: boolean
/** VMware moref of the VM */
vm: string
}
): Promise<XoVm['uuid']>
/**
* Serve one disk of a backup archive as an iSCSI LUN and attach it to `host`
* as an SR, read-only unless `cacheSrId` is set. Undone by `unmountBackupArchiveDisk`.
*
* The LUN is served by whoever can read the backup repository: this appliance, or the proxy the
* repository is linked to.
*/
mountBackupArchiveDisk(params: {
archiveId: XoVmBackupArchive['id']
/**
* SR of a VDI of the serving appliance, which the disk is materialized into as it is read and
* which holds the writes; unset, nothing is cached and the mount is read-only. On a local SR,
* the serving appliance must not be migrated while the mount lasts: the VDI cannot follow it
*/
cacheSrId?: XoSr['id']
/** One of the archive's `disks[].id` */
diskId: string
hostId: XoHost['id']
}): Promise<BackupArchiveDiskMount>
purgePluginConfiguration(id: XoPlugin['id']): Promise<void>
pingRemote(id: XoBackupRepository['id']): Promise<{ success: true }>
/**
* Record the live mounts a proxy created by itself, while running a restore: they never went
* through `mountBackupArchiveDisk`, so nothing else knows which proxy serves them.
*/
registerProxyBackupArchiveDiskMounts(params: {
archiveId: XoVmBackupArchive['id']
/** as reported by the restore, each with the host it is attached to */
mounts: { id: BackupArchiveDiskMount['id']; hostId: XoHost['id'] }[]
proxyId: XoProxy['id']
}): void
/** Allow to add a new server in the DB (XCP-ng/XenServer) */
registerXenServer(
body: Pick<XoServer, 'host' | 'httpProxy' | 'label' | 'username'> & {
@@ -341,36 +619,80 @@ export type XoApp = {
readOnly?: XoServer['readOnly']
}
): Promise<XoServer>
rollingPoolReboot(pool: XoPool, opts?: { parentTask?: VatesTask; shutdownPinnedVms?: boolean }): Promise<void>
rollingPoolReboot(
pool: XoPool,
opts?: { bypassBackupCheck?: boolean; parentTask?: VatesTask; shutdownPinnedVms?: boolean }
): Promise<void>
rollingPoolUpdate(
pool: XoPool,
opts?: { rebootVm?: boolean; parentTask?: VatesTask; shutdownPinnedVms?: boolean }
opts?: {
acceptCurrentStateAsBaseline?: boolean
bypassBackupCheck?: boolean
rebootVm?: boolean
parentTask?: VatesTask
shutdownPinnedVms?: boolean
}
): Promise<void>
getRollingUpdateRecovery(poolId: XoPool['id']): Promise<PoolRollingUpdateRecovery | undefined>
finalizeRollingUpdate(pool: XoPool, opts?: { force?: boolean; parentTask?: VatesTask }): Promise<void>
setVmResourceSet(vmId: XoVm['id'], resourceSetId: string | null, force?: boolean): Promise<void>
shareVmResourceSet(vmId: XoVm['id']): Promise<void>
/* remove from a new VM the firewall tags it inherited and the user may not add (never vdf:quarantine), before it can run; returns them */
stripInheritedReservedTags(user: XoUser | undefined, vm: XenApiVmWrapped, inheritedTags?: string[]): Promise<string[]>
/* stripInheritedReservedTags, destroying the new VM with its disks if the removal fails, then throwing its error */
stripInheritedReservedTagsOrDestroy(
user: XoUser | undefined,
vm: XenApiVmWrapped,
inheritedTags?: string[]
): Promise<string[]>
removeUserFromGroup(userId: XoUser['id'], id: XoGroup['id']): Promise<void>
runJob(job: AnyXoJob, schedule: XoSchedule): void
runWithApiContext: (user: XoUser | undefined, fn: () => void) => Promise<unknown>
testPlugin(id: XoPlugin['id'], data?: unknown): Promise<void>
testRemote(
id: XoBackupRepository['id']
): Promise<
| { success: true; readRate: number; writeRate: number }
| { success: false; step: string; file: string; error: unknown }
>
touchXoObject(type: string, id: NonXapiXoRecord['id']): Promise<void>
/** Detach a disk mounted by `mountBackupArchiveDisk` and stop serving it */
unmountBackupArchiveDisk(id: BackupArchiveDiskMount['id']): Promise<void>
unloadPlugin(id: XoPlugin['id']): Promise<void>
reclaimSpace(
remoteId: XoBackupRepository['id'],
opts?: { vmUuid?: XoVm['id']; merge?: boolean; remove?: boolean }
): Promise<{ vmUuid: string; success: boolean; merge?: boolean; size?: number; error?: string }[]>
/** Remove a server from the DB (XCP-ng/XenServer) */
unregisterXenServer(id: XoServer['id']): Promise<void>
/**
* Shallow merge of the properties into the existing job, a `null` value removes the property.
*/
updateJob(job: { id: AnyXoJob['id']; [key: string]: unknown }, merge?: boolean): Promise<void>
updateSchedule(params: {
id: XoSchedule['id']
cron?: XoSchedule['cron']
enabled?: XoSchedule['enabled']
jobId?: XoSchedule['jobId']
name?: XoSchedule['name']
timezone?: XoSchedule['timezone']
userId?: XoUser['id']
}): Promise<void>
updateUser(
id: XoUser['id'],
updates: {
email?: string
authProviders?: Record<string, string>
firstname?: string
lastname?: string
/**
* @deprecated
*/
email?: string
authProviders?: Record<string, string>
name?: string
password?: string
permission?: string
preferences?: Record<string, string>
username?: string
}
): Promise<void>
updateAclV2Privilege(

View File

@@ -19,8 +19,32 @@ import type {
VM_POWER_STATE,
} from './common.mjs'
import type * as CMType from './lib/complex-matcher.mjs'
import { XoAclPrivilege, XoAclRole, XoAclSupportedActionsByResource } from './lib/xen-orchestra-acl.mjs'
import { XoAclBasePrivilege, XoAclRole } from './lib/xen-orchestra-acl.mjs'
import type { XenApiHost, XenApiPool } from './xen-api.mjs'
import type {
XoAuditPlugin,
XoAuthGithubPlugin,
XoAuthGooglePlugin,
XoAuthLdapPlugin,
XoAuthOidcPlugin,
XoAuthSamlPlugin,
XoBackupReportsPlugin,
XoIpmiSensorsPlugin,
XoLoadBalancerPlugin,
XoNetboxPlugin,
XoOpenmetricsPlugin,
XoPerfAlertPlugin,
XoSdnControllerPlugin,
XoTestPluginPlugin,
XoTransportEmailPlugin,
XoTransportIcinga2Plugin,
XoTransportNagiosPlugin,
XoTransportSlackPlugin,
XoTransportXmppPlugin,
XoUsageReportPlugin,
XoVdfPlugin,
XoWebHooksPlugin,
} from './lib/xen-orchestra-plugins.mjs'
type BaseXapiXo = {
$pool: XoPool['id']
@@ -118,7 +142,7 @@ type BaseXoVm = BaseXapiXo & {
/**
* @deprecated use pvDriversVersion instead
*/
xentools?:
xenTools?:
| false
| {
major: null | number
@@ -140,6 +164,7 @@ export type XoAlarm = Omit<XoMessage, '$object' | 'body'> & {
}
// TODO: to be typed when Bastien.N has finished working on the XO task
// Once the [key:string]: unknown is removed, remove the `as XoMessage` in event.class.mts
type BaseXoLog = {
id: Branded<'xo-log'>
infos?: { data: unknown; message: string }[]
@@ -156,6 +181,15 @@ export type XoBackupLog = BaseXoLog & {
export type XoRestoreLog = BaseXoLog & {
message: 'restore'
}
export type XoBackupDiskPartition = {
id: Branded<'xo-backup-disk-partition'>
name?: string
size: number
start: number
/** number for MBR, string for GPT partition */
type: string | number
}
export type XoVmBackupArchive = {
id: Branded<'vm-backup-archive'>
type: 'xo-vm-backup'
@@ -200,15 +234,24 @@ export type XoPoolBackupArchive = XoMetadataBackupArchive & {
type: 'xo-pool-metadata-backup'
}
export type XoBackupRepositoryBenchmark = {
readRate: number
timestamp: number
writeRate: number
}
export type XoBackupRepository = {
benchmarks?: { readRate: number; timestamp: number; writeRate: number }[]
available?: number
benchmarks?: XoBackupRepositoryBenchmark[]
enabled: boolean
error?: Record<string, unknown>
id: Branded<'backup-repository'>
name: string
options?: string
proxy?: XoProxy['id']
size?: number
url: string
used?: number
}
export type XoGpuGroup = BaseXapiXo & {
@@ -450,6 +493,30 @@ export type XoPool = BaseXapiXo & {
zstdSupported: boolean
}
export type XoPlugin =
| XoAuditPlugin
| XoAuthGithubPlugin
| XoAuthGooglePlugin
| XoAuthLdapPlugin
| XoAuthOidcPlugin
| XoAuthSamlPlugin
| XoBackupReportsPlugin
| XoIpmiSensorsPlugin
| XoLoadBalancerPlugin
| XoNetboxPlugin
| XoOpenmetricsPlugin
| XoPerfAlertPlugin
| XoSdnControllerPlugin
| XoTestPluginPlugin
| XoTransportEmailPlugin
| XoTransportIcinga2Plugin
| XoTransportNagiosPlugin
| XoTransportSlackPlugin
| XoTransportXmppPlugin
| XoUsageReportPlugin
| XoVdfPlugin
| XoWebHooksPlugin
export type XoProxy = {
id: Branded<'proxy'>
url: string
@@ -474,15 +541,22 @@ type XoBackupJobGeneralSettings = {
[key: string]: unknown
}
/**
* Keeps the last backup of each of the last `retention` days/weeks/months/years, in addition to `exportRetention`.
*
* Requires the `timezone` setting.
*/
export type XoBackupJobLongTermRetention = {
daily?: { retention: number; settings: Record<string, unknown> }
weekly?: { retention: number; settings: Record<string, unknown> }
monthly?: { retention: number; settings: Record<string, unknown> }
yearly?: { retention: number; settings: Record<string, unknown> }
}
export type XoVmBackupJobGeneralSettings = XoBackupJobGeneralSettings & {
cbtDestroySnapshotData?: boolean
concurrency?: number
longTermRetention?: {
daily?: { retention: number; settings: Record<string, unknown> }
weekly?: { retention: number; settings: Record<string, unknown> }
monthly?: { retention: number; settings: Record<string, unknown> }
yearly?: { retention: number; settings: Record<string, unknown> }
}
longTermRetention?: XoBackupJobLongTermRetention
checkpointSnapshot?: boolean
offlineSnapshot?: boolean
maxExportRate?: number
@@ -496,8 +570,8 @@ export type XoVmBackupJobGeneralSettings = XoBackupJobGeneralSettings & {
}
export type XoVmBackupJobScheduleSettings = {
exportRetention?: number
healthCheckVmsWithTags?: XoVm['tags']
fullInterval?: number
healthCheckVmsWithTags?: XoVm['tags'] | string
fullInterval?: number | string
copyRetention?: number
snapshotRetention?: number
cbtDestroySnapshotData?: boolean
@@ -508,9 +582,9 @@ export type XoVmBackupJob = BaseXoJob & {
compression?: 'native' | 'zstd' | ''
proxy?: XoProxy['id']
mode: 'full' | 'delta'
remotes?: CMType.IdOr<XoBackupRepository['id']>
remotes?: CMType.IdOr<XoBackupRepository['id']> | string
vms: CMType.IdOr<XoVm['id']> | Record<string, unknown>
srs?: CMType.IdOr<XoSr['id']>
srs?: CMType.IdOr<XoSr['id']> | string
type: 'backup'
settings: {
'': XoVmBackupJobGeneralSettings
@@ -539,16 +613,17 @@ export type XoMetadataBackupJob = BaseXoJob & {
export type XoMirrorBackupGeneralSettings = XoBackupJobGeneralSettings & {
concurrency?: number
longTermRetention?: XoBackupJobLongTermRetention
nRetriesVmBackupFailures?: number
mergeBackupsSynchronously?: boolean
timeout?: number
timezone?: string
maxExportRate?: number
backupReportTpl?: 'compactMjml'
reportWhen: 'failure'
}
export type XoMirrorBackupScheduleSettings = {
exportRetention?: number
healthCheckVmsWithTags?: XoVm['tags']
healthCheckVmsWithTags?: XoVm['tags'] | string
vmFilter?: string
healthCheckSr?: XoSr['id']
[key: string]: unknown
}
@@ -578,6 +653,9 @@ export type XoJob = BaseXoJob & {
[key: string]: unknown
}[]
}
userId?: XoUser['id']
createdBy?: XoUser['id']
updatedBy?: XoUser['id']
}
export type XoSchedule = {
@@ -599,10 +677,11 @@ export type XoServer = {
label?: string
master?: XoHost['id']
poolId?: XoPool['id']
poolMembersAddresses?: string[]
poolNameDescription?: string
poolNameLabel?: string
readOnly: boolean
status: 'connected' | 'disconnected' | 'connecting'
status: 'connected' | 'disconnecting' | 'disconnected' | 'connecting'
username: string
}
@@ -618,6 +697,7 @@ export type XoSr = BaseXapiXo & {
current_operations: Record<string, STORAGE_OPERATIONS>
id: Branded<'SR'>
inMaintenanceMode: boolean
is_tools_sr: boolean
name_description: string
name_label: string
other_config: Record<string, string>
@@ -686,6 +766,7 @@ export type XoTask = {
| 'backup-log'
| 'backup-repository'
| 'group'
| 'plugin'
| 'proxy'
| 'restore'
| 'restore-log'
@@ -712,12 +793,18 @@ export type XoTask = {
export type XoUser = {
authProviders?: Record<string, string>
email: string
firstname?: string
groups: XoGroup['id'][]
id: Branded<'user'>
lastname?: string
/**
* @deprecated use username instead
*/
name?: string
permission: 'none' | 'admin'
pw_hash?: string
preferences: Record<string, string>
username?: string
}
export type XoAuthenticationToken = {
@@ -889,14 +976,13 @@ export type XapiXoRecord =
| XoVtpm
| XoSm
export type NonXapiXoRecord<
TActionsByResource extends XoAclSupportedActionsByResource = never,
TResource extends string = never,
> =
export type NonXapiXoRecord =
| AnyXoBackupArchive
| AnyXoJob
| AnyXoLog
| XoAuthenticationToken
| XoGroup
| XoPlugin
| XoProxy
| XoJob
| XoBackupRepository
@@ -905,12 +991,9 @@ export type NonXapiXoRecord<
| XoTask
| XoUser
| XoAclRole
| XoAclPrivilege<TActionsByResource, TResource>
| XoAclBasePrivilege
export type XoRecord<
TActionsByResource extends XoAclSupportedActionsByResource = never,
TResource extends string = never,
> = XapiXoRecord | NonXapiXoRecord<TActionsByResource, TResource>
export type XoRecord = XapiXoRecord | NonXapiXoRecord
export type AnyXoVm = XoVm | XoVmSnapshot | XoVmTemplate | XoVmController

View File

@@ -4,6 +4,7 @@
"declaration": true,
"strict": true,
"module": "ESNext",
"moduleResolution": "bundler",
"target": "ES2022",
"skipLibCheck": true,
"types": ["node"] // Only use types from Node.js and packages that are explicitly imported

View File

@@ -6,7 +6,7 @@
"main": "dist/index.mjs",
"name": "@xen-orchestra/acl",
"homepage": "https://github.com/vatesfr/xen-orchestra/tree/master/@xen-orchestra/acl",
"version": "1.5.0",
"version": "1.8.0",
"license": "AGPL-3.0-or-later",
"private": false,
"type": "module",
@@ -30,7 +30,7 @@
},
"devDependencies": {
"@eslint/js": "^9.19.0",
"@vates/types": "^1.30.1",
"@vates/types": "^2.5.0",
"rimraf": "^6.0.1",
"typescript": "~5.6",
"typescript-eslint": "^8.61.0"

View File

@@ -1,3 +1,5 @@
export default {
read: true,
export: true, // can export the content ( full or file by file )
read: true, // can read the metadata of an archive
mount: true, // can mount the archive for file level restore
}

View File

@@ -10,4 +10,5 @@ export default {
proxy: true,
url: true,
},
'reclaim-space': true,
}

View File

@@ -0,0 +1,19 @@
/**
* A pool's firewall (VDF). The object a privilege is evaluated against is the
* pool, so a selector selects pools (`id`, `name_label`, `tags`…) for every
* action.
*/
export default {
// firewall pages, status, history, rule log, trace
read: true,
// policies and groups, and the firewall's tags on the pool's networks
// (`vdf:net=…`, `vdf:unprotected`), which decide where policies apply
publish: true,
// the firewall's group tags (`vdf:…`) on the pool's VMs, except the quarantine
membership: true,
// the quarantine tag (`vdf:quarantine`) on the pool's VMs
quarantine: {
add: true,
remove: true,
},
}

View File

@@ -25,4 +25,14 @@ export default {
update: {
tags: true,
},
probe: {
nfs: true,
zfs: true,
hba: true,
iscsiiqn: true,
iscsilun: true,
'iscsi-exists': true,
'hba-exists': true,
'nfs-exists': true,
},
}

View File

@@ -5,6 +5,7 @@ import { default as backupArchive } from './backup-archive.mjs'
import { default as backupJob } from './backup-job.mjs'
import { default as backupLog } from './backup-log.mjs'
import { default as backupRepository } from './backup-repository.mjs'
import { default as firewall } from './firewall.mjs'
import { default as gpuGroup } from './gpuGroup.mjs'
import { default as group } from './group.mjs'
import { default as host } from './host.mjs'
@@ -14,6 +15,7 @@ import { default as pbd } from './pbd.mjs'
import { default as pci } from './pci.mjs'
import { default as pgpu } from './pgpu.mjs'
import { default as pif } from './pif.mjs'
import { default as plugin } from './plugin.mjs'
import { default as pool } from './pool.mjs'
import { default as proxy } from './proxy.mjs'
import { default as restoreLog } from './restore-log.mjs'
@@ -47,6 +49,7 @@ export const SUPPORTED_ACTIONS_BY_RESOURCE = {
'backup-job': backupJob,
'backup-log': backupLog,
'backup-repository': backupRepository,
firewall,
group,
gpuGroup,
host,
@@ -56,6 +59,7 @@ export const SUPPORTED_ACTIONS_BY_RESOURCE = {
pci,
pgpu,
pif,
plugin,
pool,
proxy,
'restore-log': restoreLog,

View File

@@ -0,0 +1,10 @@
export default {
load: true,
read: true,
test: true,
unload: true,
update: {
autoload: true,
configuration: true,
},
}

View File

@@ -3,9 +3,12 @@ export default {
delete: true,
read: true,
update: {
firstname: true,
lastname: true,
name: true,
password: true,
permission: true,
preferences: true,
username: true,
},
}

View File

@@ -1,4 +1,3 @@
import assert from 'node:assert'
import type { XoUser } from '@vates/types/xo'
import { SUPPORTED_ACTIONS_BY_RESOURCE } from './actions/index.mjs'
@@ -42,10 +41,21 @@ export function hasPrivilegeOn<T extends SupportedResource>({
}) {
// Function that will be called outside of the module
// We cannot be sure types are respected
assert.strictEqual(typeof user?.permission, 'string')
assert.strictEqual(typeof action, 'string')
assert.strictEqual(typeof resource, 'string')
assert.strictEqual(objects === undefined, false)
if (typeof user?.permission !== 'string') {
throw new TypeError('user.permission must be a string')
}
if (typeof action !== 'string') {
throw new TypeError('action must be a string')
}
if (typeof resource !== 'string') {
throw new TypeError('resource must be a string')
}
if (objects === undefined) {
throw new TypeError('objects must be defined')
}
CPrivilege.checkActionIsValid(resource, action as SupportedActions<typeof resource>)

View File

@@ -376,3 +376,84 @@ suite('getActionStrings', () => {
assert.ok(actions.includes('shutdown:hard'))
})
})
suite('firewall resource', () => {
const user = {
id: '2' as XoUser['id'],
groups: [],
permission: 'none',
email: 'user@user.net',
preferences: {},
} satisfies XoUser
const id = 'fake-priv-test-id' as Branded<'acl-v2-privilege'>
const roleId = 'fake-role-test-id' as Branded<'acl-v2-role'>
// its object is the pool
const prodPool = { id: 'pool-prod', name_label: 'prod', tags: ['dmz'] }
const labPool = { id: 'pool-lab', name_label: 'lab', tags: [] }
const privilege = (action: TPrivilege<'firewall'>['action'], selector?: string) =>
({ id, roleId, resource: 'firewall', action, effect: 'allow', selector }) satisfies TPrivilege<'firewall'>
const has = (
action: TPrivilege<'firewall'>['action'],
userPrivileges: TPrivilege<'firewall'>[],
pool: object = prodPool
) => hasPrivilegeOn({ user, resource: 'firewall', action, objects: pool, userPrivileges })
test('supports its actions', () => {
for (const action of [
'read',
'publish',
'membership',
'quarantine',
'quarantine:add',
'quarantine:remove',
] as const) {
assert.doesNotThrow(() => Privilege.checkActionIsValid('firewall', action))
}
// @ts-expect-error test invalid action
assert.throws(() => Privilege.checkActionIsValid('firewall', 'quarantine:lift'))
assert.deepStrictEqual([...new Set(getActionStrings('firewall'))].sort(), [
'*',
'membership',
'publish',
'quarantine',
'quarantine:add',
'quarantine:remove',
'read',
])
})
test('quarantine covers adding and removing it, each of them only itself', () => {
assert.strictEqual(has('quarantine:add', [privilege('quarantine')]), true)
assert.strictEqual(has('quarantine:remove', [privilege('quarantine')]), true)
assert.strictEqual(has('quarantine:add', [privilege('quarantine:add')]), true)
assert.strictEqual(has('quarantine:remove', [privilege('quarantine:add')]), false)
assert.strictEqual(has('membership', [privilege('quarantine')]), false)
assert.strictEqual(has('publish', [privilege('read')]), false)
})
test('* covers every action', () => {
for (const action of ['read', 'publish', 'membership', 'quarantine:add', 'quarantine:remove'] as const) {
assert.strictEqual(has(action, [privilege('*')]), true)
}
})
test('a selector selects pools', () => {
for (const selector of [`id:${prodPool.id}`, 'name_label:prod', 'tags:dmz']) {
const privileges = [privilege('membership', selector)]
assert.strictEqual(has('membership', privileges, prodPool), true, selector)
assert.strictEqual(has('membership', privileges, labPool), false, selector)
}
})
test('a deny on a pool wins over an allow on every pool', () => {
const privileges = [
privilege('*'),
{ ...privilege('quarantine:remove', 'name_label:prod'), effect: 'deny' },
] satisfies TPrivilege<'firewall'>[]
assert.strictEqual(has('quarantine:remove', privileges, prodPool), false)
assert.strictEqual(has('quarantine:add', privileges, prodPool), true)
assert.strictEqual(has('quarantine:remove', privileges, labPool), true)
})
})

View File

@@ -13,11 +13,11 @@ const wrapCall = (fn, arg, thisArg) => {
*
* WARNING: Does not handle plain objects
*
* @template Item,This
* @template Item, Result, [This=Iterable<Item>]
* @param {Iterable<Item>} iterable
* @param {(this: This, item: Item) => (Item | PromiseLike<Item>)} mapFn
* @param {This} [thisArg]
* @returns {Promise<Item[]>}
* @param {(this: This, item: Item) => (Result | PromiseLike<Result>)} mapFn
* @param {This} [thisArg] defaults to `iterable`
* @returns {Promise<Result[]>}
*/
exports.asyncMap = function asyncMap(iterable, mapFn, thisArg = iterable) {
return Promise.all(Array.from(iterable, mapFn, thisArg))
@@ -26,11 +26,11 @@ exports.asyncMap = function asyncMap(iterable, mapFn, thisArg = iterable) {
/**
* Like `asyncMap` but wait for all promises to settle before rejecting
*
* @template Item,This
* @template Item, Result, [This=Iterable<Item>]
* @param {Iterable<Item>} iterable
* @param {(this: This, item: Item) => (Item | PromiseLike<Item>)} mapFn
* @param {This} [thisArg]
* @returns {Promise<Item[]>}
* @param {(this: This, item: Item) => (Result | PromiseLike<Result>)} mapFn
* @param {This} [thisArg] defaults to `iterable`
* @returns {Promise<Result[]>}
*/
exports.asyncMapSettled = function asyncMapSettled(iterable, mapFn, thisArg = iterable) {
return new Promise((resolve, reject) => {

View File

@@ -1,7 +1,7 @@
{
"private": false,
"name": "@xen-orchestra/async-map",
"version": "0.1.3",
"version": "0.1.5",
"license": "ISC",
"description": "Promise.all + map for all iterables",
"keywords": [

View File

@@ -1,6 +1,6 @@
{
"name": "@xen-orchestra/backup-archive",
"version": "1.1.1",
"version": "1.2.0",
"description": "Helper to handle generic backup archives",
"license": "ISC",
"author": "",
@@ -21,25 +21,25 @@
"tar-stream": "^3.1.8",
"@types/tar-stream": "^3.1.4",
"tmp": "^0.2.1",
"@vates/types": "^1.30.1"
"@vates/types": "^2.5.0"
},
"dependencies": {
"@xen-orchestra/log": "^0.7.1",
"@xen-orchestra/fs": "^4.9.3",
"@xen-orchestra/disk-transform": "^1.3.3",
"@xen-orchestra/fs": "^4.10.0",
"@xen-orchestra/disk-transform": "^1.3.4",
"@vates/task": "0.7.0",
"golike-defer": "0.5.1",
"uuid": "^9.0.0",
"promise-toolbox": "^0.21.0",
"@vates/async-each": "^1.0.1",
"vhd-lib": "^4.16.1"
"vhd-lib": "^4.17.0"
},
"scripts": {
"build": "yarn run clean && tsc",
"dev": "tsc --watch",
"postversion": "npm publish --access public",
"test": "yarn run clean && tsc && node --test **/*.test.mjs",
"test-integration": "yarn run clean && tsc && node --test **/*.integ.mjs",
"test": "yarn run clean && tsc && node --test './dist/**/*.test.mjs'",
"test-integration": "tsc && node --test './src/**/*.integ.mjs'",
"clean": "rimraf dist/"
},
"private": false,

View File

@@ -0,0 +1,49 @@
import { RemoteHandlerAbstract } from '@xen-orchestra/fs'
import { promisify } from 'node:util'
import zlib from 'node:zlib'
const gzip = promisify(zlib.gzip)
const gunzip = promisify(zlib.gunzip)
type LogWarn = (message: any, opts?: object) => void
/**
* Reads a gzip-compressed JSON cache file, tolerant of a missing or corrupt file.
* Shared with the legacy @xen-orchestra/backups RemoteAdapter.
*/
export async function readBackupCache(
handler: RemoteHandlerAbstract,
path: string,
logWarn: LogWarn
): Promise<Record<string, unknown> | undefined> {
try {
return JSON.parse((await gunzip(await handler.readFile(path))).toString())
} catch (error) {
if (error?.code !== 'ENOENT') {
logWarn('failed to read cache', { error, path })
}
}
}
/**
* Writes data as a gzip-compressed JSON cache file.
* Shared with the legacy @xen-orchestra/backups RemoteAdapter.
*
* No-op on immutable remotes: the cache file may not be modifiable there,
* and callers should be regenerating/reading the data directly instead.
*/
export async function writeBackupCache(
handler: RemoteHandlerAbstract,
path: string,
data: Record<string, unknown>,
logWarn: LogWarn
): Promise<void> {
if (handler.isImmutable()) {
return
}
try {
await handler.writeFile(path, await gzip(JSON.stringify(data)), { flags: 'w' })
} catch (error) {
logWarn('failed to write cache', { error, path })
}
}

View File

@@ -0,0 +1,16 @@
import { RemoteHandlerAbstract } from '@xen-orchestra/fs'
const METADATA_BACKUP_ID_REGEXP =
/^xo-(config|pool-metadata)-backups\/\w{8}(-\w{4}){3}-\w{12}(\/\w{8}(-\w{4}){3}-\w{12})?\/\d{8}T\d{6}Z/
/**
* Deletes an XO config or pool metadata backup directory (not a VM backup).
* Shared with the legacy @xen-orchestra/backups RemoteAdapter.
*/
export async function deleteMetadataBackup(handler: RemoteHandlerAbstract, backupId: string): Promise<void> {
if (!METADATA_BACKUP_ID_REGEXP.test(backupId)) {
throw new Error(`The id (${backupId}) not correspond to a metadata folder`)
}
await handler.rmtree(backupId)
}

View File

@@ -42,12 +42,13 @@ export class RemoteDiskLineage {
#childOf: Map<string, string> = new Map()
// Disk paths declared active by their owning archives (accumulated across all referencing archives)
#activeDiskPaths: Set<string> = new Set()
// Interrupted merges: normalized parent path { stateFilePath, chain }
#interruptedMerges: Map<string, { stateFilePath: string; chain?: string[] }> = new Map()
// Interrupted merges: normalized parent path { stateFilePath, chain, step }
#interruptedMerges: Map<string, { stateFilePath: string; chain?: string[]; step?: 'mergeBlocks' | 'cleanup' }> =
new Map()
constructor(handler: RemoteHandlerAbstract, vdiDir: string, opts: ResolvedBackupCleanOptions) {
this.#handler = handler
this.#vdiDir = vdiDir
this.#vdiDir = normalize(vdiDir)
this.#opts = opts
}
@@ -266,8 +267,38 @@ export class RemoteDiskLineage {
return undefined
}
// Read-only walk toward the descendants: true if any of them is still referenced by a backup.
// Unlike getUsedChildChainOrDelete, it neither marks disks visited nor schedules deletions.
const hasUsedDescendant = (diskPath: string): boolean => {
// `seen` guards against a cycle in corrupted headers
const seen = new Set<string>()
for (let p: string | undefined = diskPath; p !== undefined && !seen.has(p); p = this.#childOf.get(p)) {
seen.add(p)
if (!orphanDisks.has(p)) {
return true
}
}
return false
}
// Keeps the orphan loop away from a chain that must not be touched this run. Pinned disks are
// differencing: every ancestor they read through must survive too, however deep the lineage is
// (e.g. after a retention change). The ancestors are merged normally once the chain is
// resumed or dropped.
const pin = (paths: readonly string[]): void => {
for (const path of paths) {
// `seen` guards against a cycle in corrupted headers
const seen = new Set<string>()
for (let p: string | undefined = path; p !== undefined && !seen.has(p); p = this.#parentOf.get(p)) {
seen.add(p)
visited.add(p)
toDelete.delete(p)
}
}
}
// Process interrupted merges first so their disks are protected from the orphan loop
for (const [parentPath, { stateFilePath, chain: stateChain }] of this.#interruptedMerges) {
for (const [parentPath, { stateFilePath, chain: stateChain, step }] of this.#interruptedMerges) {
if (!this.#diskPaths.has(parentPath)) {
this.#opts.logWarn('orphan merge state', { stateFilePath, missingDisk: parentPath })
if (remove) {
@@ -279,14 +310,85 @@ export class RemoteDiskLineage {
let chain: string[] | undefined
if (stateChain !== undefined) {
const missing = stateChain.filter(p => !this.#diskPaths.has(p))
// A merge only removes disks during its cleanup step, so outside of it every disk of the
// recorded chain must still be there. Merging what is left would fold the surviving
// children into the parent and then rename it onto the chain tip, silently dropping the
// blocks of the disks that went missing. Never resume such a chain.
if (missing.length > 0 && step !== 'cleanup') {
// Each surviving disk is walked on its own: the missing disk breaks the `childOf` links,
// so a walk from the parent alone would stop at the gap.
const survivors = stateChain.filter(p => this.#diskPaths.has(p))
if (survivors.some(hasUsedDescendant)) {
// A backup still references a disk downstream: pin the chain so the orphan loop
// neither merges it afresh nor deletes it. It is dropped once nothing uses it anymore.
this.#opts.logWarn('merge chain lost disks before its cleanup step, refusing to resume', {
stateFilePath,
parentPath,
step,
missing,
})
pin(survivors)
continue
}
// Nothing uses the chain anymore (descendants of the missing disk are broken, so never
// active): drop the state, the orphan loop below deletes the disks.
this.#opts.logWarn('unmergeable merge chain on a fully orphaned lineage', {
stateFilePath,
parentPath,
step,
missing,
})
if (remove) {
this.#opts.logInfo('deleting merge state of orphaned lineage', { stateFilePath })
await this.#handler.unlink(stateFilePath)
}
continue
}
const existing = stateChain.filter(p => this.#diskPaths.has(p))
if (existing.length >= 2) chain = existing
if (existing.length >= 2) {
chain = existing
} else {
// fewer than 2 disks of the recorded chain are left, there is nothing to resume and
// the state would otherwise stay on disk forever.
//
// only done when the chain was read successfully: an undefined `stateChain` also
// covers a state file that is merely unreadable right now, which must be kept
this.#opts.logWarn('merge state without a resumable chain', { stateFilePath, parentPath })
if (remove) {
this.#opts.logInfo('deleting unresumable merge state', { stateFilePath })
await this.#handler.unlink(stateFilePath)
}
continue
}
}
if (chain !== undefined) {
chain.forEach(p => visited.add(p))
toMerge.push({ chain, isResuming: true })
if (chain === undefined) {
// state file unreadable: leave it alone, the orphan loop below schedules a fresh merge
continue
}
// Nothing downstream of the merge parent is still referenced by a backup: merging would
// write into a lineage nothing points at, and pinning it below would exempt it from the
// orphan loop forever. The walk already marked the whole lineage for deletion, so only
// the state file is left to drop.
//
// It walks the live lineage rather than `stateChain`, which is frozen at the retention
// boundary of the run that created it and can be shorter than the real chain.
if (getUsedChildChainOrDelete(parentPath) === undefined) {
this.#opts.logWarn('merge state on a fully orphaned lineage', { stateFilePath, parentPath })
if (remove) {
this.#opts.logInfo('deleting merge state of orphaned lineage', { stateFilePath })
await this.#handler.unlink(stateFilePath)
}
continue
}
pin(chain)
toMerge.push({ chain, isResuming: true })
}
for (const orphan of orphanDisks) {

Some files were not shown because too many files have changed in this diff Show More