docs(nanokvm-go): address Voice Bridge review feedback
|
Before Width: | Height: | Size: 9.4 KiB |
|
After Width: | Height: | Size: 2.7 KiB |
|
Before Width: | Height: | Size: 21 KiB After Width: | Height: | Size: 21 KiB |
|
Before Width: | Height: | Size: 51 KiB After Width: | Height: | Size: 49 KiB |
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 20 KiB |
|
Before Width: | Height: | Size: 32 KiB After Width: | Height: | Size: 39 KiB |
|
Before Width: | Height: | Size: 50 KiB After Width: | Height: | Size: 48 KiB |
|
Before Width: | Height: | Size: 94 KiB After Width: | Height: | Size: 63 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 16 KiB |
|
Before Width: | Height: | Size: 14 KiB After Width: | Height: | Size: 14 KiB |
|
Before Width: | Height: | Size: 25 KiB After Width: | Height: | Size: 25 KiB |
|
Before Width: | Height: | Size: 24 KiB After Width: | Height: | Size: 24 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 44 KiB |
@@ -10,7 +10,7 @@ update:
|
||||
- Standardized the alignment of procedural screenshots
|
||||
- Fixed in-page navigation anchors
|
||||
- Corrected the manifest, lifecycle-script, and screen-recording examples
|
||||
- Synchronized ZIP installation screenshots and screenshot placeholders
|
||||
- Synchronized the ZIP installation screenshots
|
||||
- date: 2026-07-30
|
||||
version: v1.0
|
||||
author: taonyx
|
||||
@@ -138,7 +138,7 @@ To install the ZIP file:
|
||||
|
||||
3. after the web interface validates the ZIP file and `app.json`, complete any requested App configuration;
|
||||
|
||||

|
||||

|
||||
|
||||
4. click `Install` and wait for the `Installation log` window to report success;
|
||||
|
||||
@@ -373,8 +373,6 @@ Install `hello-world.zip` by following [Install an App from a ZIP File](#upload-
|
||||
- if installation fails, save the complete log and use the final error messages to correct the App or its configuration;
|
||||
- after installation succeeds, open the `Apps` page on the device touchscreen and select `Hello World`.
|
||||
|
||||
> 📷 **Screenshot needed:** The final `Hello World` display on the NanoKVM Go touchscreen, confirming that the example was installed and launched successfully.
|
||||
|
||||
The installation log records only the current operation. When troubleshooting, copy or capture at least the final lines before closing it.
|
||||
|
||||
<details>
|
||||
|
||||
@@ -77,7 +77,7 @@ Open the NanoKVM Go web interface, go to `Settings > Device > Virtual Audio`, an
|
||||
### Get Qwen Connection Information
|
||||
|
||||
1. Open the [Alibaba Cloud Model Studio console](https://modelstudio.console.alibabacloud.com/), register or sign in to an Alibaba Cloud account, and enable Model Studio as prompted.
|
||||
2. Select a region that supports realtime voice models, such as `China (Beijing)` or `Singapore`.
|
||||
2. Select `China (Beijing)`. The default `qwen-audio-3.0-realtime-flash` model used by Voice Bridge is currently available only in this region.
|
||||
|
||||

|
||||
|
||||
@@ -91,29 +91,31 @@ Open the NanoKVM Go web interface, go to `Settings > Device > Virtual Audio`, an
|
||||
|
||||

|
||||
|
||||
5. After the key is created, the page shows an `API Host` value. It may be shown as a hostname or as part of a full endpoint URL. `QWEN_WORKSPACE_ID` is the leading `ws-...` part of that host.
|
||||
5. After the key is created, the page shows an `API Host` value. It may be shown as a hostname or as part of a full endpoint URL. `QWEN_WORKSPACE_ID` is the leading `ws-...` part of that host, and `QWEN_REGION` is the region segment that follows it.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
API Host:
|
||||
https://ws-xxxxxxxxxxxxxxxx.ap-southeast-1.maas.aliyuncs.com/compatible-mode/v1
|
||||
https://ws-xxxxxxxxxxxxxxxx.cn-beijing.maas.aliyuncs.com
|
||||
```
|
||||
|
||||
In this case, enter the following value for `Qwen Workspace ID`:
|
||||
In this case, enter the following values:
|
||||
|
||||
```text
|
||||
ws-xxxxxxxxxxxxxxxx
|
||||
Qwen Workspace ID: ws-xxxxxxxxxxxxxxxx
|
||||
Qwen region: cn-beijing
|
||||
```
|
||||
|
||||
6. Confirm the realtime voice model name and region available to your account in the Model Studio console or the [Qwen-Omni-Realtime documentation](https://www.alibabacloud.com/help/en/model-studio/realtime).
|
||||
6. Confirm the realtime voice model name and supported region in the Model Studio console or the [Qwen-Audio Realtime documentation](https://www.alibabacloud.com/help/en/model-studio/qwen-audio-realtime-user-guides).
|
||||
|
||||
When configuring Voice Bridge later, the required Qwen fields are:
|
||||
When configuring Voice Bridge later, provide or verify these Qwen fields:
|
||||
|
||||
- `Qwen Workspace ID` (`QWEN_WORKSPACE_ID`);
|
||||
- `Qwen API key` (`QWEN_API_KEY`).
|
||||
- `Qwen API key` (`QWEN_API_KEY`);
|
||||
- `Qwen region` (`QWEN_REGION`), which must exactly match the region segment in the API Host.
|
||||
|
||||
`Qwen region` and `Qwen realtime model` can use the App form defaults first. Change them only if your account region or model access differs.
|
||||
For the Beijing setup above, `Qwen region` and `Qwen realtime model` can keep the App defaults. Always verify that the region matches the API Host before installation.
|
||||
|
||||
Available models, regions, quotas, and free usage may vary by account. Use the Model Studio console as the source of truth.
|
||||
|
||||
@@ -131,7 +133,7 @@ Voice Bridge can be installed directly from the built-in Sipeed official App Ser
|
||||

|
||||
|
||||
4. Wait for the installation configuration window to open.
|
||||
5. Fill in the automatically generated environment form. The web form shows user-facing field names, while the App saves them as environment variables internally. For a first test, you usually only need to fill in `NanoKVM MCP API key`, `Qwen Workspace ID`, and `Qwen API key`; keep the other fields at their defaults.
|
||||
5. Fill in the automatically generated environment form. The web form shows user-facing field names, while the App saves them as environment variables internally. For a first test, fill in `NanoKVM MCP API key`, `Qwen Workspace ID`, and `Qwen API key`. Confirm that `Qwen region` is `cn-beijing`, matching the API Host; the other fields can usually keep their defaults.
|
||||
|
||||
| Web form field | Environment variable | Description |
|
||||
| --- | --- | --- |
|
||||
@@ -141,7 +143,7 @@ Voice Bridge can be installed directly from the built-in Sipeed official App Ser
|
||||
| `Assistant instructions` | `QWEN_INSTRUCTIONS` | Has a default value. Sets the assistant identity, response style, and task requirements. |
|
||||
| `Base64 instructions` | `QWEN_INSTRUCTIONS_B64` | Optional. Base64-encoded UTF-8 instructions. If set, this overrides regular instructions. Leave it empty for the first test. |
|
||||
| `Qwen realtime model` | `QWEN_MODEL` | Has a default value. Change it only if your account requires a different available realtime voice model. |
|
||||
| `Qwen region` | `QWEN_REGION` | Has a default value. Change it only if your Qwen service and Workspace use a different region. |
|
||||
| `Qwen region` | `QWEN_REGION` | Has a default value. For this guide, keep `cn-beijing` and confirm that it matches the region segment in the API Host. |
|
||||
| `Session rotation interval` | `QWEN_SESSION_ROTATE_SECONDS` | Has a default value. Sets the interval for rotating the Qwen session. |
|
||||
| `Qwen voice` | `QWEN_VOICE` | Has a default value. Sets the voice used by model responses. |
|
||||
| `Qwen Workspace ID` | `QWEN_WORKSPACE_ID` | Get it from the `ws-...` prefix of the Model Studio `API Host`. |
|
||||
@@ -210,4 +212,4 @@ The official device-side example source code is in the [`voice-bridge/` director
|
||||
|
||||
If you need to replace Qwen, connect another realtime voice model, or add a knowledge base, Agent, business tools, or custom audio processing, continue with the [Realtime Voice Chat Technical Guide](./realtime_voice_bridge_technical.html).
|
||||
|
||||
References: [NanoKVM-Go-Apps](https://github.com/sipeed/NanoKVM-Go-Apps) · [Alibaba Cloud Model Studio console](https://modelstudio.console.alibabacloud.com/) · [Get an API key](https://www.alibabacloud.com/help/en/model-studio/apikey) · [Get a Workspace ID](https://help.aliyun.com/en/model-studio/obtain-the-app-id-and-workspace-id) · [Qwen-Omni-Realtime documentation](https://www.alibabacloud.com/help/en/model-studio/realtime)
|
||||
References: [NanoKVM-Go-Apps](https://github.com/sipeed/NanoKVM-Go-Apps) · [Alibaba Cloud Model Studio console](https://modelstudio.console.alibabacloud.com/) · [Get an API key](https://www.alibabacloud.com/help/en/model-studio/apikey) · [Get a Workspace ID](https://help.aliyun.com/en/model-studio/obtain-the-app-id-and-workspace-id) · [Qwen Audio Realtime documentation](https://www.alibabacloud.com/help/en/model-studio/qwen-audio-realtime-user-guides)
|
||||
|
||||
@@ -207,12 +207,20 @@ The Qwen WebSocket does not guarantee that deltas arrive every 20 ms. Network ch
|
||||
|
||||
## Create a Model Session
|
||||
|
||||
After connecting and before sending the first audio chunk, use `session.update` to configure the model, voice, input/output formats, instructions, and interaction mode. Common modes are `server_vad` and `smart_turn`.
|
||||
Select the model when opening the WebSocket by appending `?model=<model>` to the connection URL. The official App builds the endpoint as follows:
|
||||
|
||||
```text
|
||||
wss://<workspace>.<region>.maas.aliyuncs.com/api-ws/v1/realtime?model=<model>
|
||||
```
|
||||
|
||||
After the connection returns `session.created` and before sending the first audio chunk, use `session.update` to configure the voice, input/output formats, instructions, and interaction mode. Do not send `model` in `session.update`. Common interaction modes are `server_vad` and `smart_turn`.
|
||||
|
||||
Typical event flow:
|
||||
|
||||
```text
|
||||
client Qwen Realtime
|
||||
|-- WebSocket connect ----------->| ?model=<model>
|
||||
|<- session.created --------------|
|
||||
|-- session.update ------------->|
|
||||
|-- input_audio_buffer.append -->| 16 kHz mono S16LE
|
||||
|<- input_audio_buffer.speech_started
|
||||
@@ -224,7 +232,7 @@ client Qwen Realtime
|
||||
|
||||
PCM in WebSocket JSON is Base64 encoded and decoded. The upstream side can accumulate 100 ms / 3200 bytes before sending. Downstream deltas should enter the playback buffer immediately after arrival; do not wait for `response.done` and then play the whole response. The client should ignore and log unknown events at low frequency, instead of exiting when the server adds an event type.
|
||||
|
||||
When replacing the realtime voice model, keep the NanoKVM MCP, WebRTC, and audio media layers, and replace only the model adapter layer. The new adapter needs to reimplement authentication, session initialization, audio deltas, transcripts, completion, errors, and cancellation events, then normalize model output into the `48 kHz / stereo / 20 ms` audio frames required by NanoKVM.
|
||||
When replacing the realtime voice model, keep the NanoKVM MCP, WebRTC, and audio media layers, and replace only the model adapter layer. The new adapter needs to reimplement authentication, session initialization, audio deltas, transcripts, completion, errors, and cancellation events, then provide fixed-duration PCM frames in the format expected by the selected WebRTC send path. In the official Python App, provide `AudioFrame` objects to `aiortc`; for the current Qwen output, these remain `24 kHz / mono / 20 ms` frames, and `aiortc` handles resampling and Opus/RTP encoding. A low-level external Bridge instead needs to convert the model output to the negotiated NanoKVM WebRTC format and handle Opus/RTP itself.
|
||||
|
||||
## Realtime Buffering, Pacing, and Interruption
|
||||
|
||||
@@ -383,8 +391,10 @@ Downstream: model output PCM -> Base64 decode -> bounded queue -> 20 ms audio
|
||||
|
||||
If the model is Qwen Audio Realtime:
|
||||
|
||||
- Select the model while opening the WebSocket by appending `?model=<model>` to
|
||||
the connection URL. Do not include `model` in `session.update`.
|
||||
- Wait for session.created, then send session.update. Before sending the first
|
||||
audio chunk, configure model, voice, instructions, input/output format, and
|
||||
audio chunk, configure voice, instructions, input/output format, and
|
||||
interact_type, and confirm the server accepts the update.
|
||||
- Input PCM is S16LE/16000/mono. Sending 3200 bytes every 100 ms is
|
||||
recommended.
|
||||
@@ -395,9 +405,9 @@ If the model is Qwen Audio Realtime:
|
||||
environment variables.
|
||||
- Queue response.audio.delta immediately after it arrives. Do not wait for
|
||||
response.done and then play the whole response.
|
||||
- Handle complete User/Qwen transcripts, response.done/cancelled, and error.
|
||||
Ignore and log unknown events at low frequency; do not exit because the
|
||||
server adds an event type.
|
||||
- Handle complete User/Qwen transcripts, response.done (including
|
||||
status=cancelled), and error. Ignore and log unknown events at low frequency;
|
||||
do not exit because the server adds an event type.
|
||||
|
||||
## Replace With Another Realtime Voice Model
|
||||
|
||||
@@ -413,8 +423,12 @@ When replacing the model, ask the AI to focus on:
|
||||
rules.
|
||||
3. Actual audio delta, transcript, completion, error, and cancel event formats.
|
||||
4. VAD, interruption, idle timeout, rate limit, and session rotation behavior.
|
||||
5. Converting model output to NanoKVM `48 kHz / stereo / 20 ms` WebRTC audio
|
||||
frames.
|
||||
5. Converting model output into fixed-duration PCM frames expected by the
|
||||
selected WebRTC send path. With Python `aiortc`, provide `AudioFrame` objects
|
||||
and let `aiortc` handle resampling and Opus/RTP encoding; for the current Qwen
|
||||
output, use `24 kHz / mono / 20 ms` frames. Only a low-level external Bridge
|
||||
needs to convert to the negotiated NanoKVM WebRTC format and handle Opus/RTP
|
||||
itself.
|
||||
|
||||
Do not apply Qwen event names, field names, or 16/24 kHz audio parameters
|
||||
directly to another model. Provide the target model's official documentation
|
||||
@@ -543,13 +557,27 @@ redaction.
|
||||
|
||||
## Delivery Requirements
|
||||
|
||||
- Provide a runnable implementation, README, and a single-App ZIP installable
|
||||
from the NanoKVM Go web UI. Dependencies should be packaged in the App's own
|
||||
directory. Configuration should be declared in the `env` field of `app.json`,
|
||||
and the `Settings > Apps` page should generate the environment form.
|
||||
For device-side Python App mode:
|
||||
|
||||
- Provide a runnable implementation, README, `app.json`, and a single-App ZIP
|
||||
installable from the NanoKVM Go web UI. Package dependencies in the App's own
|
||||
directory, declare configuration in the `env` field of `app.json`, and let
|
||||
`Settings > Apps` generate the environment form.
|
||||
- Do not require users to use SSH, SCP, device-side pip, a standalone `.env`, or
|
||||
`kvmcomm` restart. The App should be dynamically scanned, and Launcher should
|
||||
a `kvmcomm` restart. The App should be dynamically scanned, and Launcher should
|
||||
handle startup, stop, and environment injection.
|
||||
|
||||
For external-host Bridge mode:
|
||||
|
||||
- Provide a runnable external executable or system service, together with its
|
||||
dependencies and deployment files.
|
||||
- Document the supported operating system and architecture, installation,
|
||||
configuration, startup, shutdown, restart, log access, and automatic-start
|
||||
procedure. Do not require an App ZIP, `app.json`, or installation through
|
||||
`Settings > Apps`.
|
||||
|
||||
For both modes:
|
||||
|
||||
- Configuration items should at least include URL, secret environment variables,
|
||||
voice, interact_type, TTL, queue/Pacer, Qwen session rotation, AEC, and debug
|
||||
switches.
|
||||
|
||||
@@ -10,7 +10,7 @@ update:
|
||||
- 统一操作截图的对齐方式
|
||||
- 修复章节内跳转锚点
|
||||
- 修正清单、生命周期脚本和录屏命令示例
|
||||
- 同步中英文 ZIP 安装截图和待补截图
|
||||
- 同步中英文 ZIP 安装截图
|
||||
- date: 2026-07-30
|
||||
version: v1.0
|
||||
author: taonyx
|
||||
@@ -374,8 +374,6 @@ zip -r hello-world.zip hello-world
|
||||
- 安装失败时,请保存完整日志,并根据末尾的错误信息修正 APP 或配置;
|
||||
- 安装成功后,回到设备触摸屏的 `Apps` 页面并选择 `Hello World`。
|
||||
|
||||
> 📷 **待补截图:** NanoKVM Go 触摸屏上运行 `Hello World` 的最终效果,用于确认示例已成功安装并启动。
|
||||
|
||||
安装日志只记录本次操作,遇到问题时建议复制或截图保存末尾几行。
|
||||
|
||||
<details>
|
||||
|
||||
@@ -12,7 +12,7 @@ update:
|
||||
version: v1.2
|
||||
author: Liang Ziyue
|
||||
content:
|
||||
- 重整快速体验流程并补充截图占位
|
||||
- 重整快速体验流程并补充操作截图
|
||||
- date: 2026-07-30
|
||||
version: v1.1
|
||||
author: taonyx
|
||||
@@ -87,7 +87,7 @@ Voice Bridge 是运行在 NanoKVM Go 上的实时语音桥接 APP。它在被控
|
||||
### 获取 Qwen 连接信息
|
||||
|
||||
1. 打开 [阿里云百炼控制台](https://bailian.console.aliyun.com/),注册或登录阿里云账号,并按页面提示开通百炼模型服务;
|
||||
2. 在控制台右上角选择支持实时语音模型的地域,例如 `华北2(北京)` 或 `新加坡`;
|
||||
2. 在控制台右上角选择 `华北2(北京)`。Voice Bridge 默认使用的 `qwen-audio-3.0-realtime-flash` 模型目前仅支持该地域;
|
||||
|
||||

|
||||
|
||||
@@ -99,7 +99,7 @@ Voice Bridge 是运行在 NanoKVM Go 上的实时语音桥接 APP。它在被控
|
||||
|
||||

|
||||
|
||||
5. 创建完成后,页面会显示 `OpenAI compatible` 和 `Anthropic` 两类兼容接口地址。`QWEN_WORKSPACE_ID` 就是接口地址域名最前面的 `ws-...` 这一段;
|
||||
5. 创建完成后,页面会显示 `OpenAI compatible` 和 `Anthropic` 两类兼容接口地址。`QWEN_WORKSPACE_ID` 是接口地址域名最前面的 `ws-...`,紧随其后的地域代码则是 `QWEN_REGION`;
|
||||
|
||||
例如接口地址为:
|
||||
|
||||
@@ -111,22 +111,24 @@ Anthropic:
|
||||
https://ws-xxxxxxxxxxxxxxxx.cn-beijing.maas.aliyuncs.com/apps/anthropic
|
||||
```
|
||||
|
||||
则 `QWEN_WORKSPACE_ID` 填写:
|
||||
则对应配置填写:
|
||||
|
||||
```text
|
||||
ws-xxxxxxxxxxxxxxxx
|
||||
Qwen Workspace ID: ws-xxxxxxxxxxxxxxxx
|
||||
Qwen region: cn-beijing
|
||||
```
|
||||
|
||||

|
||||
|
||||
6. 在百炼控制台或 [Qwen-Omni-Realtime 文档](https://help.aliyun.com/zh/model-studio/realtime)中确认账号当前可用的实时语音模型名称和地域。
|
||||
6. 在百炼控制台或 [Qwen Audio Realtime API 使用指南](https://help.aliyun.com/zh/model-studio/qwen-audio-realtime-user-guides)中确认账号当前可用的实时语音模型名称和支持地域。
|
||||
|
||||
后续配置 Voice Bridge 时,必须填写的 Qwen 字段为:
|
||||
后续配置 Voice Bridge 时,需要填写或确认以下 Qwen 字段:
|
||||
|
||||
- `Qwen Workspace ID`(对应 `QWEN_WORKSPACE_ID`);
|
||||
- `Qwen API key`(对应 `QWEN_API_KEY`)。
|
||||
- `Qwen API key`(对应 `QWEN_API_KEY`);
|
||||
- `Qwen region`(对应 `QWEN_REGION`),必须与 API Host 中的地域代码完全一致。
|
||||
|
||||
`Qwen region` 和 `Qwen realtime model` 可以先使用 APP 表单中的默认值;如果你的百炼账号、地域或模型权限不同,再按控制台信息修改。
|
||||
按照上面的北京地域配置时,`Qwen region` 和 `Qwen realtime model` 均可保留 APP 默认值。安装前仍需确认 region 与 API Host 中的地域代码一致。
|
||||
|
||||
不同账号的可用模型、地域和免费额度可能不同,请以百炼控制台显示的信息为准。
|
||||
|
||||
@@ -147,7 +149,7 @@ Voice Bridge 可以直接从 NanoKVM Go 内置的 Sipeed 官方 App Server 安
|
||||
|
||||
4. 等待安装配置窗口打开。
|
||||
|
||||
5. 在自动生成的环境变量表单中填写连接信息。网页表单显示的是配置项名称,APP 内部会把它们保存为对应的环境变量。首次体验时通常只需要填写 `NanoKVM MCP API key`、`Qwen Workspace ID` 和 `Qwen API key`,其他字段可以先保留默认值。
|
||||
5. 在自动生成的环境变量表单中填写连接信息。网页表单显示的是配置项名称,APP 内部会把它们保存为对应的环境变量。首次体验时填写 `NanoKVM MCP API key`、`Qwen Workspace ID` 和 `Qwen API key`,确认 `Qwen region` 为与 API Host 一致的 `cn-beijing`,其他字段通常可以保留默认值。
|
||||
|
||||
| 网页显示项 | 对应环境变量 | 说明 |
|
||||
| --- | --- | --- |
|
||||
@@ -157,7 +159,7 @@ Voice Bridge 可以直接从 NanoKVM Go 内置的 Sipeed 官方 App Server 安
|
||||
| `Assistant instructions` | `QWEN_INSTRUCTIONS` | 有默认值;用于设置模型的身份、回答方式和任务要求 |
|
||||
| `Base64 instructions` | `QWEN_INSTRUCTIONS_B64` | 可选;填写后会覆盖普通 instructions,首次体验建议留空 |
|
||||
| `Qwen realtime model` | `QWEN_MODEL` | 有默认值;如账号模型权限不同,再改为当前可用的实时语音模型 |
|
||||
| `Qwen region` | `QWEN_REGION` | 有默认值;如账号地域不同,再改为与 Qwen 服务和 Workspace 匹配的地域 |
|
||||
| `Qwen region` | `QWEN_REGION` | 有默认值;本文保留 `cn-beijing`,并确认它与 API Host 中的地域代码一致 |
|
||||
| `Session rotation interval` | `QWEN_SESSION_ROTATE_SECONDS` | 有默认值;用于设置 Qwen 会话轮换间隔 |
|
||||
| `Qwen voice` | `QWEN_VOICE` | 有默认值;用于设置模型回复使用的音色 |
|
||||
| `Qwen Workspace ID` | `QWEN_WORKSPACE_ID` | 从百炼兼容接口地址的 `ws-...` 前缀获取 |
|
||||
@@ -226,4 +228,4 @@ Voice Bridge 的安装脚本会自动安装编译依赖,并把固定版本的
|
||||
|
||||
如果你需要替换 Qwen、接入其他实时语音模型,或者加入知识库、Agent、业务工具和自定义音频处理,请继续阅读[实时语音二次开发](./realtime_voice_bridge_technical.html)。
|
||||
|
||||
参考:[NanoKVM-Go-Apps](https://github.com/sipeed/NanoKVM-Go-Apps) · [阿里云百炼控制台](https://bailian.console.aliyun.com/) · [获取 API Key](https://help.aliyun.com/zh/model-studio/get-api-key) · [获取 Workspace ID](https://help.aliyun.com/zh/model-studio/obtain-the-app-id-and-workspace-id) · [Qwen-Omni-Realtime 文档](https://help.aliyun.com/zh/model-studio/realtime)
|
||||
参考:[NanoKVM-Go-Apps](https://github.com/sipeed/NanoKVM-Go-Apps) · [阿里云百炼控制台](https://bailian.console.aliyun.com/) · [获取 API Key](https://help.aliyun.com/zh/model-studio/get-api-key) · [获取 Workspace ID](https://help.aliyun.com/zh/model-studio/obtain-the-app-id-and-workspace-id) · [Qwen Audio Realtime 文档](https://help.aliyun.com/zh/model-studio/qwen-audio-realtime-user-guides)
|
||||
|
||||
@@ -212,12 +212,20 @@ Qwen WebSocket 不保证按 20 ms 返回 delta;网络分片必须先进入缓
|
||||
|
||||
## 建立模型会话
|
||||
|
||||
连接后、发送第一段音频前,用 `session.update` 配置模型、音色、输入输出格式、instructions 和交互模式。常用模式为 `server_vad` 和 `smart_turn`。
|
||||
建立 WebSocket 连接时,通过 URL 中的 `?model=<model>` 查询参数选择模型。官方 APP 按以下格式构造连接地址:
|
||||
|
||||
```text
|
||||
wss://<workspace>.<region>.maas.aliyuncs.com/api-ws/v1/realtime?model=<model>
|
||||
```
|
||||
|
||||
连接返回 `session.created` 后、发送第一段音频前,用 `session.update` 配置音色、输入输出格式、instructions 和交互模式,不要在 `session.update` 中发送 `model` 字段。常用交互模式为 `server_vad` 和 `smart_turn`。
|
||||
|
||||
典型事件流:
|
||||
|
||||
```text
|
||||
client Qwen Realtime
|
||||
│── WebSocket connect ───────────→│ ?model=<model>
|
||||
│←─ session.created ──────────────│
|
||||
│── session.update ──────────────→│
|
||||
│── input_audio_buffer.append ───→│ 16 kHz mono S16LE
|
||||
│←─ input_audio_buffer.speech_started
|
||||
@@ -229,7 +237,7 @@ client Qwen Realtime
|
||||
|
||||
WebSocket JSON 中的 PCM 使用 Base64 编解码;上行可累计 100 ms / 3200 字节再发送;下行 delta 到达后立即进入播放缓冲,不要等 `response.done` 再整段播放。客户端应忽略并低频记录未知事件,不能因为服务端增加事件类型而退出。
|
||||
|
||||
替换其他实时语音模型时,可以保留 NanoKVM 的 MCP、WebRTC 和音频媒体层,只替换模型适配层。新的适配层需要重新实现鉴权、会话初始化、音频增量、transcript、结束、错误和取消事件,并将模型输出统一转换为 NanoKVM 所需的 `48 kHz / stereo / 20 ms` 音频帧。
|
||||
替换其他实时语音模型时,可以保留 NanoKVM 的 MCP、WebRTC 和音频媒体层,只替换模型适配层。新的适配层需要重新实现鉴权、会话初始化、音频增量、transcript、结束、错误和取消事件,再按所选 WebRTC 发送路径要求的格式提供固定时长 PCM 帧。官方 Python APP 应向 `aiortc` 提供 `AudioFrame`;对于当前 Qwen 输出,仍使用 `24 kHz / mono / 20 ms` 音频帧,由 `aiortc` 负责重采样和 Opus/RTP 编码。只有低层外部 Bridge 才需要把模型输出转换为 NanoKVM 协商得到的 WebRTC 格式,并自行处理 Opus/RTP。
|
||||
|
||||
## 实时缓冲、发送节拍与打断
|
||||
|
||||
@@ -365,15 +373,17 @@ AI 生成的代码同样必须以设备 MCP 返回值和模型官方文档为准
|
||||
|
||||
如果模型是 Qwen Audio Realtime:
|
||||
|
||||
- 等待 session.created,再发送 session.update;必须在第一段音频前完成模型、voice、
|
||||
- 建立 WebSocket 连接时,在 URL 中通过 `?model=<model>` 查询参数选择模型,
|
||||
不要在 `session.update` 中包含 `model` 字段;
|
||||
- 等待 session.created,再发送 session.update;必须在第一段音频前完成 voice、
|
||||
instructions、输入输出格式和 interact_type 配置,并确认服务端接受更新;
|
||||
- 输入 PCM 为 S16LE/16000/mono,建议每 100 ms 发送 3200 字节;
|
||||
- 输出 PCM 为 S16LE/24000/mono;
|
||||
- PCM 在 WebSocket JSON 中按 API 文档进行 Base64 编解码;
|
||||
- interact_type 支持 server_vad 和 smart_turn,并通过环境变量配置;
|
||||
- response.audio.delta 到达后立即入队,不能等 response.done 后整段播放;
|
||||
- 处理完整 User/Qwen transcript、response.done/cancelled 和 error;忽略并低频记录
|
||||
未知事件,不能因服务端增加事件类型而退出。
|
||||
- 处理完整 User/Qwen transcript、response.done(包括 status=cancelled)和 error;
|
||||
忽略并低频记录未知事件,不能因服务端增加事件类型而退出。
|
||||
|
||||
## 替换为其他实时语音模型
|
||||
|
||||
@@ -385,7 +395,10 @@ NanoKVM 的媒体层只负责 WebRTC、Opus 和 UAC2 音频,不依赖 Qwen。
|
||||
2. 输入/输出采样率、声道、PCM 编码、帧长和 Base64 规则;
|
||||
3. 音频增量、文本 transcript、结束、错误和取消事件的实际格式;
|
||||
4. VAD、打断、空闲超时、限流和会话轮换行为;
|
||||
5. 将模型输出统一转换到 NanoKVM 的 `48 kHz / stereo / 20 ms` WebRTC 音频轨道。
|
||||
5. 将模型输出转换为所选 WebRTC 发送路径要求的固定时长 PCM 帧。使用 Python
|
||||
`aiortc` 时提供 `AudioFrame`,由 `aiortc` 完成重采样和 Opus/RTP 编码;对于当前
|
||||
Qwen 输出,使用 `24 kHz / mono / 20 ms` 音频帧。只有低层外部 Bridge 才需要
|
||||
转换为 NanoKVM 协商得到的 WebRTC 格式,并自行处理 Opus/RTP。
|
||||
|
||||
不要把 Qwen 的事件名、字段名或 16/24 kHz 音频参数直接套到其他模型;先提供目标模型的官方文档,让 AI 重新完成适配和测试。
|
||||
|
||||
@@ -474,10 +487,22 @@ flush 屏障、Qwen-only 重连、媒体 TTL、进程退出清理和敏感信息
|
||||
|
||||
## 交付要求
|
||||
|
||||
- 提供可运行实现、README 和可由 NanoKVM Go 网页安装的单 App ZIP;依赖打包在 App
|
||||
自身目录,配置声明在 `app.json` 的 `env` 字段中,由 `Settings > Apps` 生成 environment 表单;
|
||||
- 不要要求用户使用 SSH、SCP、设备端 pip、独立 `.env` 或重启 `kvmcomm`;App 应能被
|
||||
动态扫描,并由 Launcher 负责启动、停止和注入环境变量;
|
||||
设备端 Python APP 模式:
|
||||
|
||||
- 提供可运行实现、README、`app.json` 和可由 NanoKVM Go 网页安装的单 APP ZIP;
|
||||
依赖打包在 APP 自身目录,配置声明在 `app.json` 的 `env` 字段中,由
|
||||
`Settings > Apps` 生成 environment 表单;
|
||||
- 不要要求用户使用 SSH、SCP、设备端 pip、独立 `.env` 或重启 `kvmcomm`;APP 应能被
|
||||
动态扫描,并由 Launcher 负责启动、停止和注入环境变量。
|
||||
|
||||
外部主机 Bridge 模式:
|
||||
|
||||
- 提供可运行的外部可执行程序或系统服务,以及所需的依赖和部署文件;
|
||||
- 说明支持的操作系统与架构,以及安装、配置、启动、停止、重启、日志查看和开机自启方法;
|
||||
不要求提供 APP ZIP、`app.json`,也不通过 `Settings > Apps` 安装。
|
||||
|
||||
两种模式共同要求:
|
||||
|
||||
- 配置项至少包含 URL、密钥环境变量、voice、interact_type、TTL、队列/Pacer、
|
||||
Qwen session 轮换、AEC 和 debug 开关;
|
||||
- 执行格式检查、单元测试和可完成的真实设备测试;
|
||||
|
||||