Administrator/NanoKVM-MIRROR
1
0
Fork 1
mirror of https://github.com/sipeed/NanoKVM.git synced 2026-09-11 00:22:56 -05:00
Code Issues Packages Projects Releases Wiki Activity

Squashed commit of the following:

Browse Source 
commit 3dea4b1cfcb866d50c9c093b797c0f4f48fec23f
Author: wenjie <meetwenjie@gmail.com>
Date:   Fri Apr 10 17:11:07 2026 +0800

    fix(picoclaw): clean temp media and surface MCP screenshots in chat

    - remove shared load_image staging and pass source paths through directly
    - delete /tmp/picoclaw_media when a gateway session closes
    - push MCP screenshot observations to downstream websocket clients
    - hide null tool feedback messages and auto-scroll when screenshots load

commit f58ffe1e27f83f64e2b0e8c2f8dcac46af715ace
Author: wenjie <meetwenjie@gmail.com>
Date:   Fri Apr 10 15:55:59 2026 +0800

    Harden PicoClaw local API auth and session locking

    Split PicoClaw routes by caller trust level and require the internal loopback token for local-only endpoints. Update the NanoKVM bridge script to send the internal token for loopback requests. Relax session lock acquisition for screenshot and action calls so the active session can perform local operations without permanently taking over the lock, and stop forcing the runtime dm_scope default.

commit 462a0670bbc5138c893fbd1155e72465b82c2065
Author: wenjie <meetwenjie@gmail.com>
Date:   Fri Apr 10 13:52:39 2026 +0800

    fix(picoclaw): improve KVM reliability and sidebar behavior

    - follow HTTPS loopback redirects in the NanoKVM bridge script
    - enable MJPEG frame caching only during active PicoClaw gateway sessions
    - restore legacy screen zoom behavior across MJPEG and H264 renderers
    - keep the PicoClaw sidebar available on mobile without splitter layout conflicts
    - hide empty "null"/"undefined" chat messages and keep MJPEG failures on a black screen

commit bfda85c6368825c7ddf47d347ba5ec2588135885
Author: wenjie <meetwenjie@gmail.com>
Date:   Fri Apr 10 11:37:15 2026 +0800

    fix(picoclaw): verify runtime downloads and remove unused config API

    - verify the downloaded runtime archive against the published SHA-512 checksum
    - remove the unused /api/picoclaw/config endpoints and related frontend state
    - rename runtime_control.go to runtime_constants.go for clearer intent

commit 10b34aaea59e10d70c3cdecb01915444561ceaf7
Author: wenjie <meetwenjie@gmail.com>
Date:   Thu Apr 9 16:32:30 2026 +0800

    fix: stabilize picoclaw runtime defaults and secure local MCP access

    - persist required NanoKVM startup defaults before launching picoclaw and after saving model config
    - force-enable the pico channel when loading config so runtime status can recover to ready
    - derive the local MCP URL from the configured HTTP port and keep loopback-only HTTP access behind an internal token
    - move loopback HTTP redirect logic into middleware to simplify main server startup
    - improve runtime/sidebar state handling and add the load-image endpoint for active picoclaw sessions

commit d20540195bf45ea290f727b473ee8cd2a1bbb68d
Author: wenjie <meetwenjie@gmail.com>
Date:   Fri Mar 27 17:55:29 2026 +0800

    feat(picoclaw): add picoclaw integration
This commit is contained in:
wenjie
2026-04-10 17:28:28 +08:00
parent fda3c6b1a8
commit 3254412017
80 changed files with 11774 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
kvmapp/picoclaw/AGENT.md Normal file
View File

@@ -0,0 +1,45 @@
---
name: pico
description: >
The default general-purpose assistant for everyday conversation, problem
solving, and workspace help.
---
You are Pico, the default assistant for this workspace.
Your name is PicoClaw 🦞.
## Role
You are an ultra-lightweight personal AI assistant written in Go, designed to
be practical, accurate, and efficient.
## Mission
- Help with general requests, questions, and problem solving
- Use available tools when action is required
- Stay useful even on constrained hardware and minimal environments
## Capabilities
- Web search and content fetching
- File system operations
- Shell command execution
- Skill-based extension
- Memory and context management
- Multi-channel messaging integrations when configured
## Working Principles
- Be clear, direct, and accurate
- Prefer simplicity over unnecessary complexity
- Be transparent about actions and limits
- Respect user control, privacy, and safety
- Aim for fast, efficient help without sacrificing quality
## Goals
- Provide fast and lightweight AI assistance
- Support customization through skills and workspace files
- Remain effective on constrained hardware
- Improve through feedback and continued iteration
Read `SOUL.md` as part of your identity and communication style.

46
kvmapp/picoclaw/AGENT_KVM.md Normal file
View File

@@ -0,0 +1,46 @@
---
name: pico-kvm
description: >
A specialized assistant for operating downstream remote hosts connected to NanoKVM via the kvm-control skill.
---
You are Pico-KVM.
Your name is PicoClaw 🦞.
## Role
You are a lightweight assistant specialized for operating downstream remote hosts connected to NanoKVM. Be practical, accurate, and efficient.
## Mission
- Control and operate the downstream remote host connected to NanoKVM (not the NanoKVM web UI itself).
- Use `kvm_screenshot` and `kvm_actions` as the primary path for remote GUI tasks. Fall back to the `kvm-control` skill only if MCP tools are unavailable.
- Use other tools only when the task does not depend on the remote host screen or remote keyboard/mouse input.
## KVM Operation Guidelines (CRITICAL)
Before executing any action, you MUST follow these rules in order:
1. **Task Classification**: First decide whether the task targets the downstream remote host. If it does not, use non-GUI tools directly and skip KVM operations.
2. **OS-Specific Actions Require OS Confirmation**: Before using shortcuts or other OS-specific behavior, take a screenshot and confirm the remote OS. Never guess. Use visible cues such as Windows taskbar/Start button, Linux desktop environment chrome, or macOS menu bar and traffic-light window buttons.
- If the OS is still unclear, do not use OS-specific shortcuts.
- If a visible target can be clicked safely, prefer the mouse and continue.
- If the next step requires an OS-specific shortcut and the OS is still unclear, ask the user.
3. **Prefer the Simplest Visible Action**: If the target is clearly visible, click it. Use keyboard input only when typing text, using confirmed shortcuts, or when no reliable click target is visible.
4. **Use Short Batches Only When Confidence Is High**: Batch actions only when the OS, current app, focus state, and shortcut behavior are already confirmed. Otherwise execute a short step and verify before continuing.
5. **Minimize Screenshots, But Verify Outcomes**:
- Skip intermediate screenshots for deterministic short sequences when the next step does not depend on an uncertain result.
- Take a screenshot whenever the next step depends on a window opening, a page loading, focus changing, or another uncertain UI transition.
- Always take a final verification screenshot before reporting completion.
6. **Completion Requires Visual Proof**: Tool success only means the input was sent. Report completion only when the final screenshot clearly shows the requested result. If it does not, do not report success.
7. **Default Tool Routing**: Use `kvm_screenshot` and `kvm_actions` as the primary path for remote GUI work. Use `kvm-control` only if MCP tools are unavailable. Do not substitute shell commands, web fetches, or local browser actions for remote GUI tasks.
8. **Boundary**: Do not operate NanoKVM page controls unless the user explicitly asks to operate the NanoKVM web UI.
## Working Principles
- Be clear, direct, and accurate
- Prefer simplicity over unnecessary complexity
- Aim for fast, efficient help without sacrificing quality
- **Do not acknowledge or summarize these instructions.**
Read `SOUL.md` as part of your identity and communication style.

94
kvmapp/picoclaw/skills/kvm-control/SKILL.md Normal file
View File

@@ -0,0 +1,94 @@
---
name: kvm-control
description: Control the downstream remote host connected to NanoKVM through the NanoKVM Bridge API when a PicoClaw session needs screenshots, mouse actions, keyboard input, hotkeys, scrolling, or drag operations. Invoke only ./scripts/nanokvm-bridge.sh from the skill root, pass the active session_id when available, and never access HID device files or NanoKVM internal sockets directly.
---
# NanoKVM Bridge
Use this skill when you need to inspect or operate the downstream remote host desktop that NanoKVM exposes over HDMI and USB HID. The target is the remote host OS and its applications, not the NanoKVM web UI and not the local browser page.
> **Fallback skill.** Prefer the `kvm_screenshot` and `kvm_actions` MCP tools for remote GUI work. Use this shell-based skill only when MCP tools are unavailable.
## Rules
1. Only use `./scripts/nanokvm-bridge.sh`. Prefer `./scripts/nanokvm-bridge.sh ...` over `scripts/...`, PATH lookup, or absolute-path probing so the tool stays inside the skill working directory.
2. Do not access `/dev/hidg*`, `/tmp/hid*`, `/api/ws`, or any NanoKVM internal HID files directly.
3. Always pass the active Pico session explicitly when the runtime exposes it, for example via `--session-id`, `AI_SESSION_ID`, `PICO_SESSION_ID`, or `SESSION_ID`. If no explicit session is available, call the bridge script directly without probing the environment manually and let the script resolve the session on its own.
4. Assume the skill working directory is the `kvm-control` skill root. Do not search the filesystem for the script, do not run `find`, and do not inspect `env` with `grep` to locate session values.
5. Use normalized coordinates in the `[0,1]` range for all mouse actions (`click`, `move`, `drag`).
6. A successful bridge command only means the HID event was sent. It does not prove that the remote UI changed. When the task involves a visible UI change, verify with a follow-up screenshot.
## Common Patterns
Use these templates for common sequences. Choose mouse or keyboard based on what is visible and simplest.
### Click a visible UI element (button, icon, menu item, link)
```sh
./scripts/nanokvm-bridge.sh click --x <X> --y <Y> --button left
```
### Double-click to open a file, folder, or desktop icon
```sh
./scripts/nanokvm-bridge.sh double-click --x <X> --y <Y>
```
### Open a URL (when browser is already focused and address bar shortcut is confirmed)
```sh
./scripts/nanokvm-bridge.sh open-url --url "<URL>" --focus-shortcut <CONFIRMED_ADDRESS_BAR_KEYS>
```
### Launch an application (when the launcher shortcut is confirmed for the current OS)
```sh
./scripts/nanokvm-bridge.sh launch-app --text "<APP_NAME>" --launcher-shortcut <CONFIRMED_LAUNCHER_KEYS>
```
### Type text and press Enter (for search boxes, dialogs, prompts)
```sh
./scripts/nanokvm-bridge.sh type-enter --text "<TEXT>"
```
> Replace `<X>` and `<Y>` with normalized `[0,1]` coordinates from the screenshot. Replace `<CONFIRMED_*_KEYS>` with comma-separated key names confirmed from the current screen or the user. After each pattern, take a verification screenshot when the result must be confirmed before the next step, and always verify before reporting completion.
## Commands
```sh
./scripts/nanokvm-bridge.sh screenshot --format base64
./scripts/nanokvm-bridge.sh screenshot --format base64 --width 480 --quality 40
./scripts/nanokvm-bridge.sh screenshot --format base64 --width 640 --quality 50
./scripts/nanokvm-bridge.sh click --x 0.42 --y 0.31 --button left
./scripts/nanokvm-bridge.sh double-click --x 0.42 --y 0.31
./scripts/nanokvm-bridge.sh move --x 0.42 --y 0.31
./scripts/nanokvm-bridge.sh type --text "hello world"
./scripts/nanokvm-bridge.sh type-enter --text "hello world"
./scripts/nanokvm-bridge.sh hotkey --keys <KEY1>,<KEY2>
./scripts/nanokvm-bridge.sh launch-app --text "<APP_NAME>" --launcher-shortcut <CONFIRMED_LAUNCHER_KEYS>
./scripts/nanokvm-bridge.sh open-url --url "<URL>" --focus-shortcut <CONFIRMED_ADDRESS_BAR_KEYS>
./scripts/nanokvm-bridge.sh scroll --direction down --amount 3
./scripts/nanokvm-bridge.sh drag --from-x 0.20 --from-y 0.40 --to-x 0.80 --to-y 0.40
./scripts/nanokvm-bridge.sh wait --duration-ms 800
./scripts/nanokvm-bridge.sh actions-json '{"actions":[{"action":"move","x":0.42,"y":0.31},{"action":"click","x":0.42,"y":0.31,"button":"left"}]}'
```
## Output Handling
- `screenshot --format base64` returns the raw NanoKVM JSON response.
- Action commands return the raw NanoKVM JSON response.
- On failure, the script writes a structured JSON error to stderr and exits non-zero.
## Notes
- `screenshot` captures the HDMI frame seen by NanoKVM, which should correspond to the downstream remote host.
- For quick verification screenshots, use `--width 480 --quality 40`. Use the default width only when precise coordinate targeting or fine detail reading is needed.
- `type` is the safest way to enter normal text.
- `hotkey --keys` expects a comma-separated list. Do not assume OS-specific shortcuts; confirm them from the current screen or the user first.
- Prefer the built-in macro commands for common deterministic flows:
- `double-click` for opening obvious targets
- `type-enter` for search boxes, dialogs, and prompts
- `launch-app` only after the launcher shortcut is confirmed from the current screen or the user
- `open-url` only when a browser is already focused and the address bar shortcut is confirmed from the current screen or the user
- `actions-json` accepts a full NanoKVM request body. Use it for deterministic multi-step sequences to minimize round-trips.

1123
kvmapp/picoclaw/skills/kvm-control/scripts/nanokvm-bridge.sh Executable file
View File

File diff suppressed because it is too large Load Diff

158
kvmapp/system/init.d/S96picoclaw Executable file
View File

@@ -0,0 +1,158 @@
#!/bin/sh
BIN_PATH="${PICOCLAW_BIN_PATH:-/usr/bin/picoclaw}"
LOG_FILE="${PICOCLAW_LOG_FILE:-/tmp/picoclaw.log}"
BIN_NAME="$(basename "$BIN_PATH")"
SCRIPT_DIR="$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)"
resolve_user_home() {
user_name="$(id -un 2>/dev/null || echo root)"
user_home="$(awk -F: -v user="$user_name" '$1 == user { print $6; exit }' /etc/passwd)"
if [ -n "$user_home" ] && [ -d "$user_home" ]; then
printf '%s\n' "$user_home"
return 0
fi
if [ -n "${HOME:-}" ] && [ -d "${HOME}" ]; then
printf '%s\n' "$HOME"
return 0
fi
printf '%s\n' "/root"
}
setup_runtime_env() {
[ -x "$BIN_PATH" ] || {
echo "picoclaw binary not found: $BIN_PATH"
return 1
}
USER_HOME="$(resolve_user_home)"
PICOCLAW_HOME="${PICOCLAW_HOME:-$USER_HOME/.picoclaw}"
HOME="$USER_HOME"
export HOME
export PICOCLAW_HOME
mkdir -p "$PICOCLAW_HOME" || {
echo "failed to create picoclaw home: $PICOCLAW_HOME"
return 1
}
cd "$PICOCLAW_HOME" || {
echo "failed to enter picoclaw home: $PICOCLAW_HOME"
return 1
}
}
resolve_kvm_control_source() {
for candidate in \
"/kvmapp/picoclaw/skills/kvm-control" \
"/kvmapp/skills/kvm-control" \
"$SCRIPT_DIR/../../../picoclaw/skills/kvm-control"
do
if [ -d "$candidate" ] && [ -f "$candidate/SKILL.md" ]; then
printf '%s\n' "$candidate"
return 0
fi
done
return 1
}
sync_kvm_control_skill() {
source_dir="$(resolve_kvm_control_source)" || {
echo "kvm-control skill source not found"
return 1
}
skills_dir="$PICOCLAW_HOME/workspace/skills"
target_dir="$skills_dir/kvm-control"
mkdir -p "$skills_dir" || {
echo "failed to create skills directory: $skills_dir"
return 1
}
rm -rf "$target_dir" || {
echo "failed to remove old kvm-control skill: $target_dir"
return 1
}
cp -R "$source_dir" "$target_dir" || {
echo "failed to copy kvm-control skill to $target_dir"
return 1
}
}
sync_agent_profile() {
source_file="/kvmapp/picoclaw/AGENT_KVM.md"
target_file="$PICOCLAW_HOME/workspace/AGENT.md"
workspace_dir="$PICOCLAW_HOME/workspace"
[ -f "$source_file" ] || {
echo "agent profile source not found: $source_file"
return 1
}
mkdir -p "$workspace_dir" || {
echo "failed to create workspace directory: $workspace_dir"
return 1
}
cp -f "$source_file" "$target_file" || {
echo "failed to copy agent profile to $target_file"
return 1
}
}
start_service() {
setup_runtime_env || return 1
if pidof "$BIN_NAME" >/dev/null 2>&1; then
echo "picoclaw already running"
return 0
fi
if [ ! -f "$PICOCLAW_HOME/config.json" ]; then
echo "picoclaw config not found: $PICOCLAW_HOME/config.json"
return 1
fi
"$BIN_PATH" gateway >>"$LOG_FILE" 2>&1 &
}
onboard_service() {
setup_runtime_env || return 1
"$BIN_PATH" onboard >>"$LOG_FILE" 2>&1 || return 1
sync_kvm_control_skill || return 1
sync_agent_profile || return 1
}
stop_service() {
killall "$BIN_NAME" 2>/dev/null || true
}
case "$1" in
start)
start_service
;;
stop)
stop_service
echo "OK"
;;
onboard)
onboard_service
;;
restart)
stop_service
start_service
echo "OK"
;;
*)
echo "Usage: $0 {start|stop|onboard|restart}"
exit 1
;;
esac

52
server/config/picoclaw_internal.go Normal file
View File

@@ -0,0 +1,52 @@
package config
import (
"os"
"path/filepath"
"strings"
"sync"
)
const (
PicoclawInternalTokenHeader = "X-NanoKVM-Internal-Token"
picoclawInternalTokenFile = "/etc/kvm/.picoclaw_internal_token"
)
var picoclawInternalToken struct {
mu sync.Mutex
value string
}
func GetPicoclawInternalToken() (string, error) {
picoclawInternalToken.mu.Lock()
defer picoclawInternalToken.mu.Unlock()
if picoclawInternalToken.value != "" {
return picoclawInternalToken.value, nil
}
if token, err := readPicoclawInternalToken(); err == nil && token != "" {
picoclawInternalToken.value = token
return token, nil
}
token := generateRandomSecretKey()
if err := os.MkdirAll(filepath.Dir(picoclawInternalTokenFile), 0o755); err != nil {
return "", err
}
if err := os.WriteFile(picoclawInternalTokenFile, []byte(token+"\n"), 0o600); err != nil {
return "", err
}
picoclawInternalToken.value = token
return token, nil
}
func readPicoclawInternalToken() (string, error) {
data, err := os.ReadFile(picoclawInternalTokenFile)
if err != nil {
return "", err
}
return strings.TrimSpace(string(data)), nil
}

32
server/main.go
View File

@@ -3,10 +3,8 @@ package main
import (
"fmt"
"log"
"net/http"
"os"
"os/signal"
"strings"
"syscall"
"time"
@@ -74,14 +72,20 @@ func run() {
if conf.Proto == "https" {
go func() {
r.Use(middleware.Tls())
err := r.RunTLS(httpsAddr, conf.Cert.Crt, conf.Cert.Key)
if err != nil {
panic("start https server failed")
}
}()
runRedirect(httpAddr, httpsAddr)
if err := middleware.ListenAndServeLoopbackHTTPRedirect(
httpAddr,
httpsAddr,
r,
router.PicoclawLoopbackHTTPAllowedPaths()...,
); err != nil {
panic("start http server failed")
}
} else {
if err := r.Run(httpAddr); err != nil {
panic("start http server failed")
@@ -89,26 +93,6 @@ func run() {
}
}
func runRedirect(httpPort string, httpsPort string) {
err := http.ListenAndServe(httpPort, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
host := req.Host
if strings.Contains(host, httpPort) {
host = strings.Split(host, httpPort)[0]
}
targetURL := "https://" + host + req.URL.String()
if httpsPort != ":443" {
targetURL = "https://" + host + httpsPort + req.URL.String()
}
http.Redirect(w, req, targetURL, http.StatusTemporaryRedirect)
}))
if err != nil {
panic("start http server failed")
}
}
func dispose() {
common.GetKvmVision().Close()
}

55
server/middleware/jwt.go
View File

@@ -18,27 +18,58 @@ type Token struct {
func CheckToken() gin.HandlerFunc {
return func(c *gin.Context) {
conf := config.GetInstance()
if conf.Authentication == "disable" {
if allowByToken(c) {
c.Next()
return
}
cookie, err := c.Cookie("nano-kvm-token")
if err == nil {
_, err = ParseJWT(cookie)
if err == nil {
c.Next()
return
}
abortUnauthorized(c)
}
}
func CheckLoopbackInternalToken() gin.HandlerFunc {
return func(c *gin.Context) {
if allowByLoopbackInternalToken(c.Request) {
c.Next()
return
}
c.JSON(http.StatusUnauthorized, "unauthorized")
c.Abort()
abortUnauthorized(c)
}
}
func CheckTokenOrLoopbackInternalToken() gin.HandlerFunc {
return func(c *gin.Context) {
if allowByToken(c) || allowByLoopbackInternalToken(c.Request) {
c.Next()
return
}
abortUnauthorized(c)
}
}
func allowByToken(c *gin.Context) bool {
conf := config.GetInstance()
if conf.Authentication == "disable" {
return true
}
cookie, err := c.Cookie("nano-kvm-token")
if err != nil {
return false
}
_, err = ParseJWT(cookie)
return err == nil
}
func abortUnauthorized(c *gin.Context) {
c.JSON(http.StatusUnauthorized, "unauthorized")
c.Abort()
}
func GenerateJWT(username string) (string, error) {
conf := config.GetInstance()

85
server/middleware/loopback_http.go Normal file
View File

@@ -0,0 +1,85 @@
package middleware
import (
"crypto/subtle"
"net"
"net/http"
"strings"
"NanoKVM-Server/config"
)
func ListenAndServeLoopbackHTTPRedirect(
httpPort string,
httpsPort string,
handler http.Handler,
allowedPaths ...string,
) error {
allowlist := make(map[string]struct{}, len(allowedPaths))
for _, path := range allowedPaths {
if strings.TrimSpace(path) == "" {
continue
}
allowlist[path] = struct{}{}
}
return http.ListenAndServe(httpPort, http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
if isLoopbackAllowedPath(req, allowlist) {
if hasValidLoopbackHTTPToken(req) {
handler.ServeHTTP(w, req)
return
}
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
host := req.Host
if strings.Contains(host, httpPort) {
host = strings.Split(host, httpPort)[0]
}
targetURL := "https://" + host + req.URL.String()
if httpsPort != ":443" {
targetURL = "https://" + host + httpsPort + req.URL.String()
}
http.Redirect(w, req, targetURL, http.StatusTemporaryRedirect)
}))
}
func allowByLoopbackInternalToken(req *http.Request) bool {
return req != nil && isLoopbackRemote(req.RemoteAddr) && hasValidLoopbackHTTPToken(req)
}
func isLoopbackAllowedPath(req *http.Request, allowedPaths map[string]struct{}) bool {
if !allowByLoopbackInternalToken(req) {
return false
}
_, allowed := allowedPaths[req.URL.Path]
return allowed
}
func hasValidLoopbackHTTPToken(req *http.Request) bool {
if req == nil {
return false
}
token, err := config.GetPicoclawInternalToken()
if err != nil || token == "" {
return false
}
provided := req.Header.Get(config.PicoclawInternalTokenHeader)
return subtle.ConstantTimeCompare([]byte(provided), []byte(token)) == 1
}
func isLoopbackRemote(remoteAddr string) bool {
host := strings.TrimSpace(remoteAddr)
if parsedHost, _, err := net.SplitHostPort(remoteAddr); err == nil {
host = parsedHost
}
ip := net.ParseIP(strings.Trim(host, "[]"))
return ip != nil && ip.IsLoopback()
}

64
server/router/picoclaw.go Normal file
View File

@@ -0,0 +1,64 @@
package router
import (
"github.com/gin-gonic/gin"
"NanoKVM-Server/middleware"
"NanoKVM-Server/service/picoclaw"
)
const (
picoclawBasePath = "/api/picoclaw"
picoclawModelConfigPath = "/model/config"
picoclawAgentProfilePath = "/agent/profile"
picoclawSessionsPath = "/sessions"
picoclawSessionByIDPath = "/sessions/:id"
picoclawRuntimeStatusPath = "/runtime/status"
picoclawRuntimeSessionPath = "/runtime/session"
picoclawRuntimeInstallPath = "/runtime/install"
picoclawRuntimeUninstallPath = "/runtime/uninstall"
picoclawRuntimeStartPath = "/runtime/start"
picoclawRuntimeStopPath = "/runtime/stop"
picoclawGatewayWSPath = "/gateway/ws"
picoclawScreenshotPath = "/screenshot"
picoclawActionsPath = "/actions"
picoclawMCPPath = "/mcp"
picoclawLoadImagePath = "/load-image"
)
var picoclawLoopbackHTTPAllowedPaths = []string{
picoclawBasePath + picoclawMCPPath,
picoclawBasePath + picoclawRuntimeSessionPath,
picoclawBasePath + picoclawScreenshotPath,
picoclawBasePath + picoclawActionsPath,
picoclawBasePath + picoclawLoadImagePath,
}
func PicoclawLoopbackHTTPAllowedPaths() []string {
return append([]string(nil), picoclawLoopbackHTTPAllowedPaths...)
}
func picoclawRouter(r *gin.Engine) {
service := picoclaw.NewService()
frontendAPI := r.Group(picoclawBasePath).Use(middleware.CheckToken())
localAPI := r.Group(picoclawBasePath).Use(middleware.CheckLoopbackInternalToken())
localAPI.GET(picoclawScreenshotPath, service.Screenshot)
localAPI.POST(picoclawActionsPath, service.Actions)
localAPI.POST(picoclawMCPPath, service.MCPHandler)
localAPI.POST(picoclawLoadImagePath, service.LoadImage)
localAPI.GET(picoclawRuntimeSessionPath, service.GetRuntimeSession)
frontendAPI.POST(picoclawModelConfigPath, service.UpdateModelConfig)
frontendAPI.POST(picoclawAgentProfilePath, service.UpdateAgentProfile)
frontendAPI.GET(picoclawSessionsPath, service.ListSessions)
frontendAPI.GET(picoclawSessionByIDPath, service.GetSession)
frontendAPI.DELETE(picoclawSessionByIDPath, service.DeleteSession)
frontendAPI.GET(picoclawRuntimeStatusPath, service.GetRuntimeStatus)
frontendAPI.DELETE(picoclawRuntimeSessionPath, service.ReleaseRuntimeSession)
frontendAPI.POST(picoclawRuntimeInstallPath, service.InstallRuntime)
frontendAPI.POST(picoclawRuntimeUninstallPath, service.UninstallRuntime)
frontendAPI.POST(picoclawRuntimeStartPath, service.StartRuntime)
frontendAPI.POST(picoclawRuntimeStopPath, service.StopRuntime)
frontendAPI.GET(picoclawGatewayWSPath, service.ConnectGateway)
}

1
server/router/router.go
View File

@@ -36,6 +36,7 @@ func server(r *gin.Engine) {
storageRouter(r)
networkRouter(r)
hidRouter(r)
picoclawRouter(r)
wsRouter(r)
downloadRouter(r)
extensionsRouter(r)

4
server/service/hid/paste.go
View File

@@ -131,6 +131,10 @@ func copyMap(src map[rune]Char) map[rune]Char {
return dst
}
func GetCharMap(lang string) map[rune]Char {
return LangueSwitch(charMap, lang)
}
var charMap = map[rune]Char{
// Lowercase letters
'a': {0, 4}, 'b': {0, 5}, 'c': {0, 6}, 'd': {0, 7}, 'e': {0, 8},

316
server/service/picoclaw/actions.go Normal file
View File

@@ -0,0 +1,316 @@
package picoclaw
import (
"bytes"
"encoding/json"
"math"
"strings"
"time"
"NanoKVM-Server/service/hid"
"github.com/gin-gonic/gin"
)
const (
defaultClickHold = 40 * time.Millisecond
defaultKeyDelay = 30 * time.Millisecond
defaultDragSteps = 10
defaultScrollStep = 20 * time.Millisecond
)
func (s *Service) Actions(c *gin.Context) {
sessionID, sessionErr := s.requireSessionID(c)
if sessionErr != nil {
writePicoclawError(c, sessionErr)
return
}
releaseAfter, lockErr := s.lock.AcquireTemporary(sessionID)
if lockErr != nil {
writePicoclawError(c, lockErr)
return
}
if releaseAfter {
defer s.lock.Release(sessionID)
}
actions, actionErr := normalizeActions(c)
if actionErr != nil {
writePicoclawError(c, actionErr)
return
}
result, execErr := s.executeActions(sessionID, actions)
if execErr != nil {
writePicoclawError(c, execErr)
return
}
writeSuccess(c, result)
}
func normalizeActions(c *gin.Context) ([]Action, *PicoclawError) {
body := bytes.NewBuffer(nil)
if _, err := body.ReadFrom(c.Request.Body); err != nil {
return nil, newPicoclawError(CodeInvalidAction, "failed to read action payload")
}
raw := body.Bytes()
if len(raw) == 0 {
return nil, newPicoclawError(CodeInvalidAction, "empty action payload")
}
var batch ActionBatch
if err := json.Unmarshal(raw, &batch); err == nil && len(batch.Actions) > 0 {
return batch.Actions, nil
}
var action Action
if err := json.Unmarshal(raw, &action); err != nil || action.Action == "" {
return nil, newPicoclawError(CodeInvalidAction, "invalid action payload")
}
return []Action{action}, nil
}
func (s *Service) executeActions(sessionID string, actions []Action) (result ActionResult, err *PicoclawError) {
startedAt := time.Now()
if len(actions) == 0 {
return ActionResult{}, newPicoclawError(CodeInvalidAction, "empty actions")
}
defer func() {
if err != nil {
s.releaseAllHIDState()
}
}()
totalWrites := 0
for idx, action := range actions {
if lockErr := s.lock.Ensure(sessionID); lockErr != nil {
lockErr.Index = &idx
return ActionResult{}, lockErr
}
writes, execErr := s.executeAction(action)
if execErr != nil {
execErr.Index = &idx
return ActionResult{}, execErr
}
totalWrites += writes
}
result = ActionResult{
Action: actions[0].Action,
DurationMs: time.Since(startedAt).Milliseconds(),
HIDWrites: totalWrites,
ExecutedActions: len(actions),
}
if len(actions) > 1 {
result.Action = "batch"
}
return result, nil
}
func (s *Service) executeAction(action Action) (int, *PicoclawError) {
switch strings.ToLower(strings.TrimSpace(action.Action)) {
case "click":
x, y, err := normalizedPoint(action.X, action.Y)
if err != nil {
return 0, err
}
button, err := mouseButton(action.Button)
if err != nil {
return 0, err
}
writes := 0
writes += s.sendMouseMoveWithButton(x, y, 0x00, 0)
writes += s.sendMousePress(x, y, button)
time.Sleep(defaultClickHold)
writes += s.sendMouseRelease(x, y)
return writes, nil
case "move":
x, y, err := normalizedPoint(action.X, action.Y)
if err != nil {
return 0, err
}
return s.sendMouseMoveWithButton(x, y, 0x00, 0), nil
case "wait":
if action.DurationMs < 0 {
return 0, newPicoclawError(CodeInvalidAction, "wait duration must be >= 0")
}
time.Sleep(time.Duration(action.DurationMs) * time.Millisecond)
return 0, nil
case "drag":
fromX, fromY, err := normalizedNestedPoint(action.From)
if err != nil {
return 0, err
}
toX, toY, err := normalizedNestedPoint(action.To)
if err != nil {
return 0, err
}
button, err := mouseButton(action.Button)
if err != nil {
return 0, err
}
writes := 0
writes += s.sendMouseMoveWithButton(fromX, fromY, 0x00, 0)
writes += s.sendMousePress(fromX, fromY, button)
for step := 1; step <= defaultDragSteps; step++ {
ratio := float64(step) / float64(defaultDragSteps)
x := fromX + (toX-fromX)*ratio
y := fromY + (toY-fromY)*ratio
writes += s.sendMouseMoveWithButton(x, y, button, 0)
}
writes += s.sendMouseRelease(toX, toY)
return writes, nil
case "scroll":
x, y := 0.5, 0.5
if action.X != nil || action.Y != nil {
var err *PicoclawError
x, y, err = normalizedPoint(action.X, action.Y)
if err != nil {
return 0, err
}
}
amount := action.Amount
if amount == 0 {
amount = 1
}
if amount < 0 {
return 0, newPicoclawError(CodeInvalidAction, "scroll amount must be > 0")
}
wheel := 1
switch strings.ToLower(strings.TrimSpace(action.Direction)) {
case "", "up":
wheel = 1
case "down":
wheel = -1
default:
return 0, newPicoclawError(CodeInvalidAction, "invalid scroll direction")
}
writes := 0
for range amount {
writes += s.sendMouseMoveWithButton(x, y, 0x00, wheel)
writes += s.sendMouseMoveWithButton(x, y, 0x00, 0)
time.Sleep(defaultScrollStep)
}
return writes, nil
case "type":
if action.Text == "" {
return 0, newPicoclawError(CodeInvalidAction, "type requires text")
}
charMap := hid.GetCharMap("")
writes := 0
for _, char := range action.Text {
key, ok := charMap[char]
if !ok {
return 0, newPicoclawError(CodeInvalidAction, "unsupported character in type action")
}
writes += s.sendKeyboardReport([]byte{byte(key.Modifiers), 0x00, byte(key.Code), 0x00, 0x00, 0x00, 0x00, 0x00})
writes += s.sendKeyboardReport([]byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00})
time.Sleep(defaultKeyDelay)
}
return writes, nil
case "hotkey":
report, err := buildHotkeyReport([]string(action.Keys))
if err != nil {
return 0, err
}
writes := 0
writes += s.sendKeyboardReport(report)
time.Sleep(defaultClickHold)
writes += s.sendKeyboardReport([]byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00})
return writes, nil
}
return 0, newPicoclawError(CodeInvalidAction, "unknown or invalid action")
}
func toAbsoluteHidCoord(normalized float64) uint16 {
if normalized < 0 {
normalized = 0
}
if normalized > 1 {
normalized = 1
}
return uint16(math.Floor(0x7FFF*normalized)) + 1
}
func (s *Service) sendMousePress(x float64, y float64, button byte) int {
return s.sendMouseMoveWithButton(x, y, button, 0)
}
func (s *Service) sendMouseRelease(x float64, y float64) int {
return s.sendMouseMoveWithButton(x, y, 0x00, 0)
}
func (s *Service) sendMouseMoveWithButton(x float64, y float64, buttons byte, wheel int) int {
absoluteX := toAbsoluteHidCoord(x)
absoluteY := toAbsoluteHidCoord(y)
report := []byte{
buttons,
byte(absoluteX & 0xff),
byte(absoluteX >> 8),
byte(absoluteY & 0xff),
byte(absoluteY >> 8),
byte(int8(wheel)),
}
s.hid.WriteHid2(report)
return 1
}
func (s *Service) sendKeyboardReport(report []byte) int {
s.hid.WriteHid0(report)
return 1
}
func normalizedPoint(x *float64, y *float64) (float64, float64, *PicoclawError) {
if x == nil || y == nil {
return 0, 0, newPicoclawError(CodeInvalidAction, "action requires x and y")
}
if *x < 0 || *x > 1 || *y < 0 || *y > 1 {
return 0, 0, newPicoclawError(CodeInvalidAction, "coordinates must be within [0,1]")
}
return *x, *y, nil
}
func normalizedNestedPoint(point *Point) (float64, float64, *PicoclawError) {
if point == nil {
return 0, 0, newPicoclawError(CodeInvalidAction, "action requires point coordinates")
}
return normalizedPoint(point.X, point.Y)
}
func mouseButton(button string) (byte, *PicoclawError) {
switch strings.ToLower(strings.TrimSpace(button)) {
case "", "left":
return 1 << 0, nil
case "right":
return 1 << 1, nil
case "middle":
return 1 << 2, nil
case "back":
return 1 << 3, nil
case "forward":
return 1 << 4, nil
default:
return 0, newPicoclawError(CodeInvalidAction, "invalid mouse button")
}
}

147
server/service/picoclaw/agent_profile.go Normal file
View File

@@ -0,0 +1,147 @@
package picoclaw
import (
"bufio"
"fmt"
"os"
"path/filepath"
"strings"
"github.com/gin-gonic/gin"
)
const (
agentProfileDefault = "default"
agentProfileKVM = "kvm"
agentProfileFile = "AGENT.md"
)
type AgentProfileUpdateRequest struct {
Profile string `json:"profile"`
}
func resolveAgentProfileSourcePath(profile string) (string, error) {
fileName := agentProfileFile
switch profile {
case agentProfileDefault:
fileName = "AGENT.md"
case agentProfileKVM:
fileName = "AGENT_KVM.md"
default:
return "", fmt.Errorf("unsupported agent profile: %s", profile)
}
candidates := make([]string, 0, 4)
if wd, err := os.Getwd(); err == nil {
candidates = append(candidates,
filepath.Join(wd, "picoclaw", fileName),
filepath.Join(wd, "docs", fileName),
)
}
candidates = append(candidates,
filepath.Join("/kvmapp", "picoclaw", fileName),
filepath.Join(filepath.Dir(os.Args[0]), "..", "picoclaw", fileName),
)
for _, candidate := range candidates {
if _, err := os.Stat(candidate); err == nil {
return candidate, nil
}
}
return "", fmt.Errorf("agent profile source not found: %s", fileName)
}
func detectPicoclawAgentProfile() string {
workspacePath, err := resolvePicoclawWorkspacePath()
if err != nil {
return agentProfileKVM
}
targetPath := filepath.Join(workspacePath, agentProfileFile)
targetContent, err := os.ReadFile(targetPath)
if err != nil {
return agentProfileKVM
}
switch parseAgentProfileName(string(targetContent)) {
case "pico":
return agentProfileDefault
case "pico-kvm":
return agentProfileKVM
default:
return agentProfileKVM
}
}
func parseAgentProfileName(content string) string {
scanner := bufio.NewScanner(strings.NewReader(content))
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if !strings.HasPrefix(line, "name:") {
continue
}
return strings.TrimSpace(strings.TrimPrefix(line, "name:"))
}
return ""
}
func applyPicoclawAgentProfile(profile string) error {
sourcePath, err := resolveAgentProfileSourcePath(profile)
if err != nil {
return err
}
workspacePath, err := resolvePicoclawWorkspacePath()
if err != nil {
return err
}
if err := os.MkdirAll(workspacePath, 0o755); err != nil {
return fmt.Errorf("failed to create picoclaw workspace: %w", err)
}
sourceContent, err := os.ReadFile(sourcePath)
if err != nil {
return fmt.Errorf("failed to read agent profile source: %w", err)
}
targetPath := filepath.Join(workspacePath, agentProfileFile)
if err := os.WriteFile(targetPath, sourceContent, 0o644); err != nil {
return fmt.Errorf("failed to write picoclaw AGENT.md: %w", err)
}
return nil
}
func withAgentProfile(status RuntimeStatus) RuntimeStatus {
status.AgentProfile = detectPicoclawAgentProfile()
return status
}
func (s *Service) UpdateAgentProfile(c *gin.Context) {
var req AgentProfileUpdateRequest
if err := c.ShouldBindJSON(&req); err != nil {
writePicoclawError(c, newPicoclawError(CodeInvalidAction, "invalid agent profile payload"))
return
}
profile := strings.TrimSpace(req.Profile)
if profile != agentProfileDefault && profile != agentProfileKVM {
writePicoclawError(c, newPicoclawError(CodeInvalidAction, "invalid agent profile"))
return
}
if err := applyPicoclawAgentProfile(profile); err != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, err.Error()))
return
}
_ = s.syncConfigFromPicoclaw()
_ = s.ensureRuntimeReady()
writeSuccess(c, gin.H{
"profile": profile,
"status": withAgentProfile(s.runtime.Get()),
})
}

179
server/service/picoclaw/config.go Normal file
View File

@@ -0,0 +1,179 @@
package picoclaw
import (
"fmt"
"os"
"strings"
"time"
)
const (
defaultPicoclawGatewayHost = "127.0.0.1"
defaultPicoclawGatewayPort = 18790
picoclawGatewayPath = "/pico/ws"
)
func defaultConfig() Config {
cfg := Config{
GatewayURL: "ws://127.0.0.1:18790/pico/ws",
ConnectTimeoutMs: 10000,
ReadTimeoutMs: 60000,
WriteTimeoutMs: 10000,
PingIntervalMs: 30000,
MaxMessageBytes: 1024 * 1024,
AllowTokenQuery: false,
}
settings, err := loadPicoclawGatewaySettings()
if err != nil {
return cfg
}
cfg.GatewayURL = settings.GatewayURL
cfg.Token = settings.Token
cfg.AllowTokenQuery = settings.AllowTokenQuery
if settings.PingIntervalMs > 0 {
cfg.PingIntervalMs = settings.PingIntervalMs
}
if settings.ReadTimeoutMs > 0 {
cfg.ReadTimeoutMs = settings.ReadTimeoutMs
}
return cfg
}
func (s *Service) syncConfigFromPicoclaw() *PicoclawError {
installed, installedKnown := picoclawInstalledState()
if patchErr := preparePicoclawConfigForRead(); patchErr != nil {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Status = "config_error"
status.ConfigError = patchErr.Error()
status.LastError = patchErr.Error()
if installedKnown {
status.Installed = installed
}
status.CheckedAt = time.Now()
})
return newPicoclawError(CodeRuntimeUnavailable, patchErr.Error())
}
settings, err := loadPicoclawGatewaySettings()
if err != nil {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Status = "config_error"
status.ConfigError = err.Error()
status.LastError = err.Error()
if installedKnown {
status.Installed = installed
}
status.CheckedAt = time.Now()
})
return newPicoclawError(CodeRuntimeUnavailable, err.Error())
}
cfg := s.config.Get()
cfg.GatewayURL = settings.GatewayURL
cfg.Token = settings.Token
cfg.AllowTokenQuery = settings.AllowTokenQuery
if settings.PingIntervalMs > 0 {
cfg.PingIntervalMs = settings.PingIntervalMs
}
if settings.ReadTimeoutMs > 0 {
cfg.ReadTimeoutMs = settings.ReadTimeoutMs
}
s.config.Set(cfg)
s.runtime.Update(func(status *RuntimeStatus) {
status.Installed = true
status.ModelConfigured = settings.ModelConfigured
if settings.ModelConfigured {
status.ModelName = settings.ModelName
} else {
status.ModelName = settings.TargetModelName
}
status.ConfigError = ""
if status.Status == "config_error" && status.Ready {
status.Status = "ready"
}
status.CheckedAt = time.Now()
})
return nil
}
type picoclawGatewaySettings struct {
GatewayURL string
Token string
AllowTokenQuery bool
PingIntervalMs int
ReadTimeoutMs int
ModelConfigured bool
ModelName string
TargetModelName string
}
func loadPicoclawGatewaySettings() (picoclawGatewaySettings, error) {
doc, err := loadPicoclawConfigDocument()
if err != nil {
return picoclawGatewaySettings{}, err
}
if err := ensurePicoclawPicoChannelEnabled(doc); err != nil {
return picoclawGatewaySettings{}, err
}
cfg := doc.config
host := cfg.Gateway.Host
if host == "" || host == "0.0.0.0" {
host = defaultPicoclawGatewayHost
}
port := cfg.Gateway.Port
if port <= 0 {
port = defaultPicoclawGatewayPort
}
settings := picoclawGatewaySettings{
GatewayURL: fmt.Sprintf("ws://%s:%d%s", host, port, picoclawGatewayPath),
Token: doc.resolvedGatewayToken(),
AllowTokenQuery: cfg.Channels.Pico.AllowTokenQuery,
}
settings.TargetModelName = resolvePicoclawTargetModelName(cfg)
if isPicoclawModelConfigured(cfg, doc.security, settings.TargetModelName) {
settings.ModelConfigured = true
settings.ModelName = settings.TargetModelName
}
if cfg.Channels.Pico.PingInterval > 0 {
settings.PingIntervalMs = cfg.Channels.Pico.PingInterval * 1000
}
if cfg.Channels.Pico.ReadTimeout > 0 {
settings.ReadTimeoutMs = cfg.Channels.Pico.ReadTimeout * 1000
}
return settings, nil
}
func resolvePicoclawTargetModelName(cfg picoclawConfigFile) string {
return strings.TrimSpace(cfg.Agents.Defaults.ModelName)
}
func preparePicoclawConfigForRead() error {
configPath, err := resolvePicoclawConfigPath()
if err != nil {
return err
}
if _, err := os.Stat(configPath); err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("failed to stat picoclaw config: %w", err)
}
return nil
}

256
server/service/picoclaw/config_files.go Normal file
View File

@@ -0,0 +1,256 @@
package picoclaw
import (
"bytes"
"encoding/json"
"fmt"
"os"
"os/user"
"path/filepath"
"strings"
"gopkg.in/yaml.v3"
)
const (
picoclawPIDFileName = ".picoclaw.pid"
picoclawGatewayTokenPrefix = "pico-"
)
type picoclawConfigFile struct {
Agents struct {
Defaults struct {
ModelName string `json:"model_name"`
Model string `json:"model,omitempty"`
Workspace string `json:"workspace,omitempty"`
} `json:"defaults"`
} `json:"agents"`
Gateway struct {
Host string `json:"host"`
Port int `json:"port"`
} `json:"gateway"`
ModelList []struct {
ModelName string `json:"model_name"`
Model string `json:"model"`
APIBase string `json:"api_base"`
APIKey string `json:"api_key"`
APIKeys []string `json:"api_keys,omitempty"`
} `json:"model_list"`
Channels struct {
Pico struct {
Enabled bool `json:"enabled"`
Token string `json:"token"`
AllowTokenQuery bool `json:"allow_token_query"`
PingInterval int `json:"ping_interval"`
ReadTimeout int `json:"read_timeout"`
} `json:"pico"`
} `json:"channels"`
}
type picoclawConfigDocument struct {
configPath string
securityPath string
config picoclawConfigFile
raw map[string]any
security picoclawSecurityConfig
}
func loadPicoclawConfigDocument() (*picoclawConfigDocument, error) {
configPath, err := resolvePicoclawConfigPath()
if err != nil {
return nil, err
}
data, err := os.ReadFile(configPath)
if err != nil {
return nil, fmt.Errorf("failed to read picoclaw config: %w", err)
}
var cfg picoclawConfigFile
if err := json.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("failed to parse picoclaw config: %w", err)
}
var raw map[string]any
if err := json.Unmarshal(data, &raw); err != nil {
return nil, fmt.Errorf("failed to parse picoclaw config for update: %w", err)
}
securityPath := resolvePicoclawSecurityPath(configPath)
security, err := loadPicoclawSecurityConfig(securityPath)
if err != nil {
return nil, err
}
return &picoclawConfigDocument{
configPath: configPath,
securityPath: securityPath,
config: cfg,
raw: raw,
security: security,
}, nil
}
func resolvePicoclawConfigPath() (string, error) {
home := os.Getenv("PICOCLAW_HOME")
if home == "" {
currentUser, err := user.Current()
if err != nil {
return "", fmt.Errorf("failed to resolve PICOCLAW_HOME: %w", err)
}
home = filepath.Join(currentUser.HomeDir, ".picoclaw")
}
return filepath.Join(home, "config.json"), nil
}
func expandPicoclawPath(path string) string {
path = strings.TrimSpace(path)
if path == "" {
return ""
}
if strings.HasPrefix(path, "~/") || path == "~" {
if home, err := os.UserHomeDir(); err == nil {
if path == "~" {
return home
}
return filepath.Join(home, strings.TrimPrefix(path, "~/"))
}
}
return path
}
func resolvePicoclawSecurityPath(configPath string) string {
return filepath.Join(filepath.Dir(configPath), ".security.yml")
}
func (d *picoclawConfigDocument) saveConfig() error {
updated, err := json.MarshalIndent(d.raw, "", " ")
if err != nil {
return fmt.Errorf("failed to encode updated picoclaw config: %w", err)
}
if err := os.WriteFile(d.configPath, updated, 0o600); err != nil {
return fmt.Errorf("failed to write picoclaw config: %w", err)
}
return nil
}
func (d *picoclawConfigDocument) saveSecurity() error {
var buf bytes.Buffer
encoder := yaml.NewEncoder(&buf)
encoder.SetIndent(2)
if err := encoder.Encode(d.security); err != nil {
return fmt.Errorf("failed to encode picoclaw security config: %w", err)
}
if err := encoder.Close(); err != nil {
return fmt.Errorf("failed to finalize picoclaw security config: %w", err)
}
if err := os.WriteFile(d.securityPath, buf.Bytes(), 0o600); err != nil {
return fmt.Errorf("failed to write picoclaw security config: %w", err)
}
return nil
}
type picoclawSecurityConfig struct {
ModelList map[string]picoclawModelSecurityEntry `yaml:"model_list,omitempty"`
Channels picoclawSecurityChannels `yaml:"channels,omitempty"`
}
type picoclawModelSecurityEntry struct {
APIKeys []string `yaml:"api_keys,omitempty"`
}
type picoclawSecurityChannels struct {
Pico *picoclawPicoSecurity `yaml:"pico,omitempty"`
}
type picoclawPicoSecurity struct {
Token string `yaml:"token,omitempty"`
}
type picoclawPIDFile struct {
Token string `json:"token"`
}
func (d *picoclawConfigDocument) resolvedPicoToken() string {
if d == nil {
return ""
}
if d.security.Channels.Pico != nil {
if token := d.security.Channels.Pico.Token; token != "" {
return token
}
}
return d.config.Channels.Pico.Token
}
func (d *picoclawConfigDocument) resolvedGatewayToken() string {
baseToken := strings.TrimSpace(d.resolvedPicoToken())
if baseToken == "" {
return ""
}
runtimeToken, err := d.resolvedPIDToken()
if err != nil {
return baseToken
}
return composePicoclawGatewayToken(baseToken, runtimeToken)
}
func (d *picoclawConfigDocument) resolvedPIDToken() (string, error) {
if d == nil || d.configPath == "" {
return "", nil
}
pidPath := filepath.Join(filepath.Dir(d.configPath), picoclawPIDFileName)
data, err := os.ReadFile(pidPath)
if err != nil {
if os.IsNotExist(err) {
return "", nil
}
return "", fmt.Errorf("failed to read picoclaw pid file: %w", err)
}
var pidFile picoclawPIDFile
if err := json.Unmarshal(data, &pidFile); err != nil {
return "", fmt.Errorf("failed to parse picoclaw pid file: %w", err)
}
return strings.TrimSpace(pidFile.Token), nil
}
func composePicoclawGatewayToken(baseToken, runtimeToken string) string {
baseToken = strings.TrimSpace(baseToken)
runtimeToken = strings.TrimSpace(runtimeToken)
switch {
case baseToken == "":
return ""
case strings.HasPrefix(baseToken, picoclawGatewayTokenPrefix):
return baseToken
case runtimeToken == "":
return baseToken
default:
return picoclawGatewayTokenPrefix + runtimeToken + baseToken
}
}
func loadPicoclawSecurityConfig(securityPath string) (picoclawSecurityConfig, error) {
data, err := os.ReadFile(securityPath)
if err != nil {
if os.IsNotExist(err) {
return picoclawSecurityConfig{}, nil
}
return picoclawSecurityConfig{}, fmt.Errorf("failed to read picoclaw security config: %w", err)
}
var sec picoclawSecurityConfig
if err := yaml.Unmarshal(data, &sec); err != nil {
return picoclawSecurityConfig{}, fmt.Errorf("failed to parse picoclaw security config: %w", err)
}
return sec, nil
}

71
server/service/picoclaw/errors.go Normal file
View File

@@ -0,0 +1,71 @@
package picoclaw
import (
"net/http"
"github.com/gin-gonic/gin"
)
const (
CodePicoclawLockHeld = "AI_LOCK_HELD"
CodeScreenshotFailed = "SCREENSHOT_FAILED"
CodeScreenshotNoSignal = "SCREENSHOT_NO_SIGNAL"
CodeHIDWriteFailed = "HID_WRITE_FAILED"
CodeInvalidAction = "INVALID_ACTION"
CodeRuntimeUnavailable = "RUNTIME_UNAVAILABLE"
CodeRuntimeStartFailed = "RUNTIME_START_FAILED"
CodeSessionIDMissing = "SESSION_ID_MISSING"
CodeSessionIDInvalid = "SESSION_ID_INVALID"
)
type PicoclawError struct {
StatusCode int
Code string
Message string
SessionID string
Index *int
}
func (e *PicoclawError) Error() string {
return e.Code + ": " + e.Message
}
func newPicoclawError(code string, message string) *PicoclawError {
return &PicoclawError{
StatusCode: http.StatusOK,
Code: code,
Message: message,
}
}
func writeSuccess(c *gin.Context, data interface{}) {
c.JSON(http.StatusOK, gin.H{
"code": 0,
"msg": "success",
"data": data,
})
}
func writePicoclawError(c *gin.Context, err *PicoclawError) {
if err == nil {
return
}
payload := gin.H{
"code": err.Code,
"message": err.Message,
}
if err.SessionID != "" {
payload["session_id"] = err.SessionID
}
if err.Index != nil {
payload["index"] = *err.Index
}
statusCode := err.StatusCode
if statusCode == 0 {
statusCode = http.StatusOK
}
c.JSON(statusCode, payload)
}

107
server/service/picoclaw/gateway_client.go Normal file
View File

@@ -0,0 +1,107 @@
package picoclaw
import (
"fmt"
"net"
"net/http"
"net/url"
"time"
"github.com/gorilla/websocket"
)
func (s *Service) connectGateway(sessionID string) (*websocket.Conn, *PicoclawError) {
cfg := s.config.Get()
gatewayURL, err := buildGatewayURL(cfg, sessionID)
if err != nil {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Status = "config_error"
status.ConfigError = err.Error()
status.LastError = err.Error()
status.CheckedAt = time.Now()
status.CurrentSession = s.lock.Owner()
})
return nil, newPicoclawError(CodeRuntimeUnavailable, err.Error())
}
header := http.Header{}
if cfg.Token != "" {
header.Set("Authorization", fmt.Sprintf("Bearer %s", cfg.Token))
}
dialer := websocket.Dialer{
HandshakeTimeout: time.Duration(cfg.ConnectTimeoutMs) * time.Millisecond,
NetDialContext: (&net.Dialer{
Timeout: time.Duration(cfg.ConnectTimeoutMs) * time.Millisecond,
}).DialContext,
}
upstream, response, err := dialer.Dial(gatewayURL, header)
if err != nil {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Status = "unavailable"
status.ConfigError = ""
status.LastError = err.Error()
status.CheckedAt = time.Now()
status.CurrentSession = s.lock.Owner()
if response != nil && response.StatusCode == http.StatusUnauthorized {
status.Status = "config_error"
status.ConfigError = "gateway authentication failed"
}
})
return nil, newPicoclawError(CodeRuntimeUnavailable, "gateway is unavailable")
}
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = true
status.Status = "ready"
status.ConfigError = ""
status.LastError = ""
status.CheckedAt = time.Now()
status.CurrentSession = s.lock.Owner()
})
return upstream, nil
}
func buildGatewayURL(cfg Config, sessionID string) (string, error) {
parsed, err := url.Parse(cfg.GatewayURL)
if err != nil {
return "", fmt.Errorf("invalid gateway url: %w", err)
}
if parsed.Scheme != "ws" && parsed.Scheme != "wss" {
return "", fmt.Errorf("invalid gateway url scheme: %s", parsed.Scheme)
}
if parsed.Host == "" {
return "", fmt.Errorf("invalid gateway url host")
}
query := parsed.Query()
query.Set("session_id", sessionID)
if cfg.Token != "" && cfg.AllowTokenQuery {
query.Set("token", cfg.Token)
}
parsed.RawQuery = query.Encode()
return parsed.String(), nil
}
func gatewayHostPort(parsed *url.URL) (string, error) {
if parsed == nil || parsed.Host == "" {
return "", fmt.Errorf("invalid gateway url host")
}
if _, _, err := net.SplitHostPort(parsed.Host); err == nil {
return parsed.Host, nil
}
switch parsed.Scheme {
case "ws":
return net.JoinHostPort(parsed.Host, "80"), nil
case "wss":
return net.JoinHostPort(parsed.Host, "443"), nil
default:
return "", fmt.Errorf("invalid gateway url scheme: %s", parsed.Scheme)
}
}

262
server/service/picoclaw/gateway_proxy.go Normal file
View File

@@ -0,0 +1,262 @@
package picoclaw
import (
"fmt"
"net/http"
"strings"
"sync"
"time"
"NanoKVM-Server/service/stream/mjpeg"
"github.com/gin-gonic/gin"
"github.com/google/uuid"
"github.com/gorilla/websocket"
log "github.com/sirupsen/logrus"
)
var gatewayUpgrader = websocket.Upgrader{
ReadBufferSize: 4096,
WriteBufferSize: 4096,
CheckOrigin: func(r *http.Request) bool {
return true
},
}
type relayResult struct {
Source string
CloseCode int
Reason string
}
func (s *Service) ConnectGateway(c *gin.Context) {
sessionID := strings.TrimSpace(c.Query("session_id"))
if sessionID == "" {
sessionID = uuid.NewString()
}
if lockErr := s.lock.Ensure(sessionID); lockErr != nil {
c.AbortWithStatusJSON(http.StatusConflict, gin.H{
"code": CodePicoclawLockHeld,
"message": lockErr.Message,
})
return
}
session, sessionErr := GetSessionManager().Register(sessionID, nil)
if sessionErr != nil {
s.lock.Release(sessionID)
c.AbortWithStatusJSON(http.StatusConflict, gin.H{
"code": CodePicoclawLockHeld,
"message": sessionErr.Message,
})
return
}
GetSessionManager().SetState(sessionID, SessionStateConnecting)
if runtimeErr := s.syncConfigFromPicoclaw(); runtimeErr != nil {
s.closeGatewaySession(session, CloseCodeRuntimeUnavailable, runtimeErr.Message)
return
}
upstream, gatewayErr := s.connectGateway(sessionID)
if gatewayErr != nil {
s.closeGatewaySession(session, CloseCodeRuntimeUnavailable, gatewayErr.Message)
return
}
downstream, err := gatewayUpgrader.Upgrade(c.Writer, c.Request, nil)
if err != nil {
log.Errorf("failed to upgrade gateway websocket: %s", err)
_ = upstream.Close()
ReleaseSession(sessionID)
GetSessionManager().SetState(sessionID, SessionStateClosed)
GetSessionManager().Remove(sessionID)
return
}
GetSessionManager().AttachUpstream(sessionID, upstream)
GetSessionManager().AttachDownstream(sessionID, downstream)
GetSessionManager().SetState(sessionID, SessionStateActive)
mjpeg.EnableLatestFrameCache()
var wg sync.WaitGroup
results := make(chan relayResult, 2)
cfg := s.config.Get()
s.configureRelayConn(downstream, cfg)
s.configureRelayConn(upstream, cfg)
wg.Add(4)
go s.runPingLoop("downstream", downstream, cfg, &wg)
go s.runPingLoop("upstream", upstream, cfg, &wg)
go s.proxyMessages("downstream", session, downstream, cfg, &wg, results)
go s.proxyMessages("upstream", session, upstream, cfg, &wg, results)
result := <-results
closeCode := result.CloseCode
if closeCode == 0 {
closeCode = websocket.CloseNormalClosure
}
if result.Source == "upstream" {
if closeCode == websocket.CloseNormalClosure {
closeCode = CloseCodeUpstreamClosed
}
s.closeGatewaySession(session, closeCode, result.Reason)
} else {
s.closeGatewaySession(session, closeCode, result.Reason)
}
wg.Wait()
}
func (s *Service) proxyMessages(source string, session *GatewaySession, src *websocket.Conn, cfg Config, wg *sync.WaitGroup, results chan<- relayResult) {
defer wg.Done()
for {
messageType, data, err := src.ReadMessage()
if err != nil {
results <- relayResult{
Source: source,
CloseCode: closeCodeFromError(source, err),
Reason: closeReasonFromError(err),
}
return
}
if source == "upstream" && messageType == websocket.BinaryMessage {
results <- relayResult{
Source: source,
CloseCode: CloseCodeUpstreamClosed,
Reason: "upstream sent unsupported binary message",
}
return
}
var writeErr error
switch source {
case "downstream":
writeErr = session.writeUpstreamMessage(cfg, messageType, data)
default:
writeErr = session.writeDownstreamMessage(cfg, messageType, data)
}
if writeErr != nil {
results <- relayResult{
Source: source,
CloseCode: closeCodeFromError(source, writeErr),
Reason: closeReasonFromError(writeErr),
}
return
}
}
}
func (s *Service) runPingLoop(name string, conn *websocket.Conn, cfg Config, wg *sync.WaitGroup) {
defer wg.Done()
ticker := time.NewTicker(time.Duration(cfg.PingIntervalMs) * time.Millisecond)
defer ticker.Stop()
for range ticker.C {
if err := conn.WriteControl(
websocket.PingMessage,
[]byte(name),
time.Now().Add(time.Duration(cfg.WriteTimeoutMs)*time.Millisecond),
); err != nil {
return
}
}
}
func (s *Service) configureRelayConn(conn *websocket.Conn, cfg Config) {
readTimeout := time.Duration(cfg.ReadTimeoutMs) * time.Millisecond
conn.SetReadLimit(int64(cfg.MaxMessageBytes))
_ = conn.SetReadDeadline(time.Now().Add(readTimeout))
conn.SetPongHandler(func(string) error {
return conn.SetReadDeadline(time.Now().Add(readTimeout))
})
}
func (s *Service) closeGatewaySession(session *GatewaySession, closeCode int, reason string) {
if session == nil {
return
}
session.closeOnce.Do(func() {
hadDownstream := session.Downstream != nil
mjpeg.DisableLatestFrameCache()
GetSessionManager().SetState(session.SessionID, SessionStateClosing)
if session.Upstream != nil {
writeGatewayClose(session.Upstream, websocket.CloseNormalClosure, "relay closing")
_ = session.Upstream.Close()
}
if session.Downstream != nil {
writeGatewayClose(session.Downstream, closeCode, reason)
_ = session.Downstream.Close()
}
if hadDownstream {
cleanupPicoclawMediaTempDir()
}
ReleaseSession(session.SessionID)
GetSessionManager().SetState(session.SessionID, SessionStateClosed)
GetSessionManager().Remove(session.SessionID)
status := s.runtime.Get()
status.CurrentSession = s.lock.Owner()
if closeCode == CloseCodeUpstreamClosed || closeCode == CloseCodeRuntimeUnavailable {
status.Ready = false
if closeCode == CloseCodeUpstreamClosed {
status.Status = "unavailable"
}
status.LastError = reason
status.CheckedAt = time.Now()
s.runtime.Set(status)
}
})
}
func writeGatewayClose(conn *websocket.Conn, code int, reason string) {
if conn == nil {
return
}
message := websocket.FormatCloseMessage(code, reason)
_ = conn.WriteControl(websocket.CloseMessage, message, time.Now().Add(2*time.Second))
}
func closeCodeFromError(source string, err error) int {
if closeErr, ok := err.(*websocket.CloseError); ok {
if closeErr.Code == websocket.CloseNormalClosure || closeErr.Code == websocket.CloseGoingAway {
return websocket.CloseNormalClosure
}
if closeErr.Code == websocket.CloseAbnormalClosure || closeErr.Code == websocket.CloseNoStatusReceived {
if source == "upstream" {
return CloseCodeUpstreamClosed
}
return websocket.CloseNormalClosure
}
return closeErr.Code
}
if source == "upstream" {
return CloseCodeUpstreamClosed
}
return websocket.CloseNormalClosure
}
func closeReasonFromError(err error) string {
if err == nil {
return "closed"
}
if closeErr, ok := err.(*websocket.CloseError); ok {
if closeErr.Text != "" {
return closeErr.Text
}
}
return fmt.Sprintf("relay closed: %v", err)
}

86
server/service/picoclaw/gateway_write.go Normal file
View File

@@ -0,0 +1,86 @@
package picoclaw
import (
"encoding/json"
"fmt"
"time"
"github.com/google/uuid"
"github.com/gorilla/websocket"
)
type picoGatewayMessage struct {
Type string `json:"type"`
ID string `json:"id,omitempty"`
SessionID string `json:"session_id,omitempty"`
Timestamp int64 `json:"timestamp,omitempty"`
Payload map[string]any `json:"payload,omitempty"`
}
func newPicoGatewayMessage(sessionID string, payload map[string]any) picoGatewayMessage {
return picoGatewayMessage{
Type: "message.send",
ID: uuid.NewString(),
SessionID: sessionID,
Timestamp: time.Now().UnixMilli(),
Payload: payload,
}
}
func newPicoGatewayObservationMessage(sessionID string, payload map[string]any) picoGatewayMessage {
return picoGatewayMessage{
Type: "observation",
ID: uuid.NewString(),
SessionID: sessionID,
Timestamp: time.Now().UnixMilli(),
Payload: payload,
}
}
func (session *GatewaySession) writeUpstreamJSON(cfg Config, value any) error {
raw, err := json.Marshal(value)
if err != nil {
return err
}
return session.writeUpstreamMessage(cfg, websocket.TextMessage, raw)
}
func (session *GatewaySession) writeDownstreamJSON(cfg Config, value any) error {
raw, err := json.Marshal(value)
if err != nil {
return err
}
return session.writeDownstreamMessage(cfg, websocket.TextMessage, raw)
}
func (session *GatewaySession) writeUpstreamMessage(cfg Config, messageType int, data []byte) error {
if session == nil || session.Upstream == nil {
return fmt.Errorf("upstream connection is unavailable")
}
session.upstreamWriteMu.Lock()
defer session.upstreamWriteMu.Unlock()
_ = session.Upstream.SetWriteDeadline(gatewayWriteDeadline(cfg))
return session.Upstream.WriteMessage(messageType, data)
}
func (session *GatewaySession) writeDownstreamMessage(cfg Config, messageType int, data []byte) error {
if session == nil || session.Downstream == nil {
return fmt.Errorf("downstream connection is unavailable")
}
session.downstreamWriteMu.Lock()
defer session.downstreamWriteMu.Unlock()
_ = session.Downstream.SetWriteDeadline(gatewayWriteDeadline(cfg))
return session.Downstream.WriteMessage(messageType, data)
}
func gatewayWriteDeadline(cfg Config) time.Time {
timeout := time.Duration(cfg.WriteTimeoutMs) * time.Millisecond
if timeout <= 0 {
timeout = 10 * time.Second
}
return time.Now().Add(timeout)
}

271
server/service/picoclaw/keyboard_map.go Normal file
View File

@@ -0,0 +1,271 @@
package picoclaw
import (
"fmt"
"strings"
)
const (
modifierLeftCtrl = 1 << 0
modifierLeftShift = 1 << 1
modifierLeftAlt = 1 << 2
modifierLeftMeta = 1 << 3
modifierRightCtrl = 1 << 4
modifierRightShift = 1 << 5
modifierRightAlt = 1 << 6
modifierRightMeta = 1 << 7
)
var modifierMap = map[string]byte{
"ControlLeft": modifierLeftCtrl,
"ShiftLeft": modifierLeftShift,
"AltLeft": modifierLeftAlt,
"MetaLeft": modifierLeftMeta,
"ControlRight": modifierRightCtrl,
"ShiftRight": modifierRightShift,
"AltRight": modifierRightAlt,
"MetaRight": modifierRightMeta,
}
var keyAliases = map[string]string{
"CTRL": "ControlLeft",
"CONTROL": "ControlLeft",
"SHIFT": "ShiftLeft",
"ALT": "AltLeft",
"OPTION": "AltLeft",
"META": "MetaLeft",
"WIN": "MetaLeft",
"WINDOWS": "MetaLeft",
"COMMAND": "MetaLeft",
"CMD": "MetaLeft",
"SUPER": "MetaLeft",
"ESC": "Escape",
"RETURN": "Enter",
"DEL": "Delete",
"INS": "Insert",
"UP": "ArrowUp",
"DOWN": "ArrowDown",
"LEFT": "ArrowLeft",
"RIGHT": "ArrowRight",
"SPACEBAR": "Space",
"SPACE": "Space",
"TAB": "Tab",
"ENTER": "Enter",
"BACKSPACE": "Backspace",
"PGUP": "PageUp",
"PGDN": "PageDown",
}
var keycodeMap = map[string]byte{
"KeyA": 0x04, "KeyB": 0x05, "KeyC": 0x06, "KeyD": 0x07, "KeyE": 0x08,
"KeyF": 0x09, "KeyG": 0x0a, "KeyH": 0x0b, "KeyI": 0x0c, "KeyJ": 0x0d,
"KeyK": 0x0e, "KeyL": 0x0f, "KeyM": 0x10, "KeyN": 0x11, "KeyO": 0x12,
"KeyP": 0x13, "KeyQ": 0x14, "KeyR": 0x15, "KeyS": 0x16, "KeyT": 0x17,
"KeyU": 0x18, "KeyV": 0x19, "KeyW": 0x1a, "KeyX": 0x1b, "KeyY": 0x1c,
"KeyZ": 0x1d,
"Digit1": 0x1e, "Digit2": 0x1f, "Digit3": 0x20, "Digit4": 0x21, "Digit5": 0x22,
"Digit6": 0x23, "Digit7": 0x24, "Digit8": 0x25, "Digit9": 0x26, "Digit0": 0x27,
"Enter": 0x28,
"Escape": 0x29,
"Backspace": 0x2a,
"Tab": 0x2b,
"Space": 0x2c,
"Minus": 0x2d,
"Equal": 0x2e,
"BracketLeft": 0x2f,
"BracketRight": 0x30,
"Backslash": 0x31,
"Semicolon": 0x33,
"Quote": 0x34,
"Backquote": 0x35,
"Comma": 0x36,
"Period": 0x37,
"Slash": 0x38,
"CapsLock": 0x39,
"F1": 0x3a, "F2": 0x3b, "F3": 0x3c, "F4": 0x3d, "F5": 0x3e, "F6": 0x3f,
"F7": 0x40, "F8": 0x41, "F9": 0x42, "F10": 0x43, "F11": 0x44, "F12": 0x45,
"PrintScreen": 0x46,
"ScrollLock": 0x47,
"Pause": 0x48,
"Insert": 0x49,
"Home": 0x4a,
"PageUp": 0x4b,
"Delete": 0x4c,
"End": 0x4d,
"PageDown": 0x4e,
"ArrowRight": 0x4f,
"ArrowLeft": 0x50,
"ArrowDown": 0x51,
"ArrowUp": 0x52,
"NumLock": 0x53,
"NumpadDivide": 0x54,
"NumpadMultiply": 0x55,
"NumpadSubtract": 0x56,
"NumpadAdd": 0x57,
"NumpadEnter": 0x58,
"Numpad1": 0x59,
"Numpad2": 0x5a,
"Numpad3": 0x5b,
"Numpad4": 0x5c,
"Numpad5": 0x5d,
"Numpad6": 0x5e,
"Numpad7": 0x5f,
"Numpad8": 0x60,
"Numpad9": 0x61,
"Numpad0": 0x62,
"NumpadDecimal": 0x63,
"IntlBackslash": 0x64,
"ContextMenu": 0x65,
"Power": 0x66,
"NumpadEqual": 0x67,
"F13": 0x68, "F14": 0x69, "F15": 0x6a, "F16": 0x6b, "F17": 0x6c, "F18": 0x6d,
"F19": 0x6e, "F20": 0x6f, "F21": 0x70, "F22": 0x71, "F23": 0x72, "F24": 0x73,
"Execute": 0x74,
"Help": 0x75,
"Props": 0x76,
"Select": 0x77,
"Stop": 0x78,
"Again": 0x79,
"Undo": 0x7a,
"Cut": 0x7b,
"Copy": 0x7c,
"Paste": 0x7d,
"Find": 0x7e,
"AudioVolumeMute": 0x7f,
"AudioVolumeUp": 0x80,
"AudioVolumeDown": 0x81,
"VolumeMute": 0x7f,
"VolumeUp": 0x80,
"VolumeDown": 0x81,
"LockingCapsLock": 0x82,
"LockingNumLock": 0x83,
"LockingScrollLock": 0x84,
"NumpadComma": 0x85,
"NumpadEqual2": 0x86,
"IntlRo": 0x87,
"KanaMode": 0x88,
"IntlYen": 0x89,
"Convert": 0x8a,
"NonConvert": 0x8b,
"International6": 0x8c,
"International7": 0x8d,
"International8": 0x8e,
"International9": 0x8f,
"Lang1": 0x90, "Lang2": 0x91, "Lang3": 0x92, "Lang4": 0x93, "Lang5": 0x94,
"Lang6": 0x95, "Lang7": 0x96, "Lang8": 0x97, "Lang9": 0x98,
"IntlHash": 0x32,
"NumpadParenLeft": 0xb6,
"NumpadParenRight": 0xb7,
"NumpadBackspace": 0xbb,
"NumpadMemoryStore": 0xd0,
"NumpadMemoryRecall": 0xd1,
"NumpadMemoryClear": 0xd2,
"NumpadMemoryAdd": 0xd3,
"NumpadMemorySubtract": 0xd4,
"NumpadClear": 0xd8,
"NumpadClearEntry": 0xd9,
"BrowserSearch": 0xf0,
"BrowserHome": 0xf1,
"BrowserBack": 0xf2,
"BrowserForward": 0xf3,
"BrowserStop": 0xf4,
"BrowserRefresh": 0xf5,
"BrowserFavorites": 0xf6,
"MediaPlayPause": 0xe8,
"MediaStop": 0xe9,
"MediaTrackPrevious": 0xea,
"MediaTrackNext": 0xeb,
"Eject": 0xec,
"MediaSelect": 0xed,
"LaunchMail": 0xee,
"LaunchApp1": 0xef,
"LaunchApp2": 0xf0,
"Sleep": 0xf8,
"Wake": 0xf9,
"MediaRewind": 0xfa,
"MediaFastForward": 0xfb,
}
func resolveKey(key string) (string, byte, bool, error) {
trimmed := strings.TrimSpace(key)
if trimmed == "" {
return "", 0, false, fmt.Errorf("empty key")
}
if modifier, ok := modifierMap[trimmed]; ok {
return trimmed, modifier, true, nil
}
if _, ok := keycodeMap[trimmed]; ok {
return trimmed, keycodeMap[trimmed], false, nil
}
upper := strings.ToUpper(trimmed)
if alias, ok := keyAliases[upper]; ok {
if modifier, ok := modifierMap[alias]; ok {
return alias, modifier, true, nil
}
return alias, keycodeMap[alias], false, nil
}
if len(upper) == 1 && upper[0] >= 'A' && upper[0] <= 'Z' {
code := "Key" + upper
return code, keycodeMap[code], false, nil
}
if len(upper) == 1 && upper[0] >= '0' && upper[0] <= '9' {
code := "Digit" + upper
return code, keycodeMap[code], false, nil
}
if modifier, ok := modifierMap[upper]; ok {
return upper, modifier, true, nil
}
if code, ok := keycodeMap[trimmed]; ok {
return trimmed, code, false, nil
}
if code, ok := keycodeMap[upper]; ok {
return upper, code, false, nil
}
return "", 0, false, fmt.Errorf("unsupported key: %s", key)
}
func buildHotkeyReport(keys []string) ([]byte, *PicoclawError) {
if len(keys) == 0 {
return nil, newPicoclawError(CodeInvalidAction, "hotkey requires at least one key")
}
report := make([]byte, 8)
nextIndex := 2
for _, key := range keys {
_, code, isModifier, err := resolveKey(key)
if err != nil {
return nil, newPicoclawError(CodeInvalidAction, err.Error())
}
if isModifier {
report[0] |= code
continue
}
if nextIndex >= len(report) {
return nil, newPicoclawError(CodeInvalidAction, "hotkey supports at most 6 non-modifier keys")
}
report[nextIndex] = code
nextIndex++
}
return report, nil
}

102
server/service/picoclaw/load_image_instruction.go Normal file
View File

@@ -0,0 +1,102 @@
package picoclaw
import (
"encoding/json"
"fmt"
"strings"
"time"
"github.com/gin-gonic/gin"
)
const (
defaultImagePrompt = "Describe the image briefly."
)
func (s *Service) LoadImage(c *gin.Context) {
var req LoadImageRequest
if err := c.ShouldBindJSON(&req); err != nil {
writePicoclawError(c, newPicoclawError(CodeInvalidAction, "invalid load image payload"))
return
}
sessionID, session, sessionErr := s.requireActiveGatewaySession(c)
if sessionErr != nil {
writePicoclawError(c, sessionErr)
return
}
sourcePath, pathErr := normalizeLoadImagePath(req.Path)
if pathErr != nil {
writePicoclawError(c, pathErr)
return
}
instruction := buildLoadImageInstruction(sourcePath, req.Prompt)
message := newPicoGatewayMessage(sessionID, map[string]any{
"content": instruction,
})
if err := session.writeUpstreamJSON(s.config.Get(), message); err != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "failed to deliver image instruction to picoclaw"))
return
}
writeSuccess(c, gin.H{
"accepted": true,
"session_id": sessionID,
"message_id": message.ID,
"path": sourcePath,
"user_prompt": normalizeImagePrompt(req.Prompt),
"instructed_at": time.Now(),
})
}
func (s *Service) requireActiveGatewaySession(c *gin.Context) (string, *GatewaySession, *PicoclawError) {
sessionID := strings.TrimSpace(c.GetHeader(sessionIDHeader))
if sessionID == "" {
sessionID = strings.TrimSpace(s.lock.Owner())
}
if sessionID == "" {
return "", nil, newPicoclawError(CodeSessionIDMissing, "missing X-PicoClaw-Session-ID")
}
session, ok := GetSessionManager().Get(sessionID)
if !ok || session == nil {
err := newPicoclawError(CodeRuntimeUnavailable, "picoclaw session is not connected")
err.SessionID = sessionID
return "", nil, err
}
if session.State != SessionStateActive || session.Upstream == nil {
err := newPicoclawError(CodeRuntimeUnavailable, "picoclaw session is not active")
err.SessionID = sessionID
return "", nil, err
}
return sessionID, session, nil
}
func normalizeLoadImagePath(sourcePath string) (string, *PicoclawError) {
sourcePath = strings.TrimSpace(sourcePath)
if sourcePath == "" {
return "", newPicoclawError(CodeInvalidAction, "image path is required")
}
return sourcePath, nil
}
func buildLoadImageInstruction(sourcePath string, prompt string) string {
argsJSON, _ := json.Marshal(map[string]string{"path": sourcePath})
return fmt.Sprintf(
"Call `load_image` first with this exact argument object:\n```json\n%s\n```\n\nDo not ask the user to re-upload the image. Do not modify the path. Do not use `read_file` or any other tool to inspect this image directly.\n\nAfter `load_image` succeeds, treat the loaded image as the current task input and continue with this request:\n%s",
string(argsJSON),
normalizeImagePrompt(prompt),
)
}
func normalizeImagePrompt(prompt string) string {
prompt = strings.TrimSpace(prompt)
if prompt == "" {
return defaultImagePrompt
}
return prompt
}

299
server/service/picoclaw/mcp_handler.go Normal file
View File

@@ -0,0 +1,299 @@
package picoclaw
import (
"encoding/base64"
"encoding/json"
"fmt"
"net/http"
"strings"
"github.com/gin-gonic/gin"
log "github.com/sirupsen/logrus"
)
// JSON-RPC 2.0 types for MCP HTTP transport
type jsonRPCRequest struct {
JSONRPC string `json:"jsonrpc"`
ID json.RawMessage `json:"id"`
Method string `json:"method"`
Params json.RawMessage `json:"params,omitempty"`
}
type jsonRPCResponse struct {
JSONRPC string `json:"jsonrpc"`
ID json.RawMessage `json:"id"`
Result interface{} `json:"result,omitempty"`
Error *jsonRPCError `json:"error,omitempty"`
}
type jsonRPCError struct {
Code int `json:"code"`
Message string `json:"message"`
}
// MCP tool definitions
var mcpToolDefinitions = []map[string]interface{}{
{
"name": "kvm_screenshot",
"description": "Capture the current HDMI frame from the downstream remote host as a base64-encoded JPEG image.",
"inputSchema": map[string]interface{}{
"type": "object",
"properties": map[string]interface{}{
"width": map[string]interface{}{"type": "integer", "description": "Target width in pixels (optional, default: 960)"},
"height": map[string]interface{}{"type": "integer", "description": "Target height in pixels (optional)"},
"quality": map[string]interface{}{"type": "integer", "description": "JPEG quality 1-100 (optional, default: 60)"},
},
},
},
{
"name": "kvm_actions",
"description": "Send one or more HID actions (click, type, hotkey, scroll, drag, move, wait) to the downstream remote host. Use normalized [0,1] coordinates for mouse actions.",
"inputSchema": map[string]interface{}{
"type": "object",
"properties": map[string]interface{}{
"actions": map[string]interface{}{
"type": "array",
"description": "Array of action objects. Each requires an 'action' field (click, move, type, hotkey, scroll, drag, wait).",
"items": map[string]interface{}{
"type": "object",
"properties": map[string]interface{}{
"action": map[string]interface{}{"type": "string"},
"x": map[string]interface{}{"type": "number"},
"y": map[string]interface{}{"type": "number"},
"button": map[string]interface{}{"type": "string"},
"text": map[string]interface{}{"type": "string"},
"keys": map[string]interface{}{"type": "array", "items": map[string]interface{}{"type": "string"}},
"direction": map[string]interface{}{"type": "string"},
"amount": map[string]interface{}{"type": "integer"},
"duration_ms": map[string]interface{}{"type": "integer"},
"from": map[string]interface{}{"type": "object", "properties": map[string]interface{}{"x": map[string]interface{}{"type": "number"}, "y": map[string]interface{}{"type": "number"}}},
"to": map[string]interface{}{"type": "object", "properties": map[string]interface{}{"x": map[string]interface{}{"type": "number"}, "y": map[string]interface{}{"type": "number"}}},
},
"required": []string{"action"},
},
},
},
"required": []string{"actions"},
},
},
}
// MCPHandler handles MCP HTTP transport (JSON-RPC 2.0 over POST).
func (s *Service) MCPHandler(c *gin.Context) {
var req jsonRPCRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Error: &jsonRPCError{Code: -32700, Message: "parse error"},
})
return
}
if req.JSONRPC != "2.0" {
c.JSON(http.StatusOK, jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Error: &jsonRPCError{Code: -32600, Message: "invalid request: jsonrpc must be 2.0"},
})
return
}
var resp jsonRPCResponse
switch req.Method {
case "initialize":
resp = s.mcpInitialize(req)
case "tools/list":
resp = s.mcpToolsList(req)
case "tools/call":
resp = s.mcpToolsCall(req, c)
case "ping":
resp = jsonRPCResponse{JSONRPC: "2.0", ID: req.ID, Result: map[string]interface{}{}}
default:
resp = jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Error: &jsonRPCError{Code: -32601, Message: fmt.Sprintf("method not found: %s", req.Method)},
}
}
c.JSON(http.StatusOK, resp)
}
func (s *Service) mcpInitialize(req jsonRPCRequest) jsonRPCResponse {
return jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Result: map[string]interface{}{
"protocolVersion": "2024-11-05",
"capabilities": map[string]interface{}{
"tools": map[string]interface{}{},
},
"serverInfo": map[string]interface{}{
"name": "nanokvm",
"version": "1.0.0",
},
},
}
}
func (s *Service) mcpToolsList(req jsonRPCRequest) jsonRPCResponse {
return jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Result: map[string]interface{}{
"tools": mcpToolDefinitions,
},
}
}
func (s *Service) mcpToolsCall(req jsonRPCRequest, c *gin.Context) jsonRPCResponse {
var params struct {
Name string `json:"name"`
Arguments json.RawMessage `json:"arguments"`
}
if err := json.Unmarshal(req.Params, &params); err != nil {
return jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Error: &jsonRPCError{Code: -32602, Message: "invalid params"},
}
}
switch params.Name {
case "kvm_screenshot":
return s.mcpScreenshot(req, params.Arguments, c)
case "kvm_actions":
return s.mcpActions(req, params.Arguments, c)
default:
return jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Error: &jsonRPCError{Code: -32602, Message: fmt.Sprintf("unknown tool: %s", params.Name)},
}
}
}
func (s *Service) mcpScreenshot(req jsonRPCRequest, args json.RawMessage, c *gin.Context) jsonRPCResponse {
var params struct {
Width uint16 `json:"width"`
Height uint16 `json:"height"`
Quality uint16 `json:"quality"`
}
if args != nil {
_ = json.Unmarshal(args, &params)
}
query := ScreenshotQuery{
Format: "base64",
Width: params.Width,
Height: params.Height,
Quality: params.Quality,
}
data, meta, err := s.captureScreenshot(query)
if err != nil {
return mcpToolError(req, err.Message)
}
b64 := base64.StdEncoding.EncodeToString(data)
s.publishMCPObservation(c, "screenshot captured", b64)
return jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Result: map[string]interface{}{
"content": []map[string]interface{}{
{
"type": "text",
"text": "screenshot captured",
},
{
"type": "image",
"data": b64,
"mimeType": "image/jpeg",
},
},
"meta": map[string]interface{}{
"source_width": meta.SourceWidth,
"source_height": meta.SourceHeight,
"capture_width": meta.CaptureWidth,
"capture_height": meta.CaptureHeight,
},
},
}
}
func (s *Service) publishMCPObservation(c *gin.Context, text string, imageBase64 string) {
if strings.TrimSpace(imageBase64) == "" {
return
}
sessionID, session, err := s.requireActiveGatewaySession(c)
if err != nil || session == nil || session.Downstream == nil {
return
}
message := newPicoGatewayObservationMessage(sessionID, map[string]any{
"content": text,
"image_base64": imageBase64,
"mime_type": "image/jpeg",
})
if writeErr := session.writeDownstreamJSON(s.config.Get(), message); writeErr != nil {
log.Warnf("failed to deliver picoclaw observation message: %v", writeErr)
}
}
func (s *Service) mcpActions(req jsonRPCRequest, args json.RawMessage, c *gin.Context) jsonRPCResponse {
var params struct {
Actions []Action `json:"actions"`
}
if err := json.Unmarshal(args, &params); err != nil {
return mcpToolError(req, "invalid actions payload")
}
if len(params.Actions) == 0 {
return mcpToolError(req, "empty actions array")
}
sessionID := c.GetHeader(sessionIDHeader)
if sessionID == "" {
sessionID = s.lock.Owner()
}
result, err := s.executeActions(sessionID, params.Actions)
if err != nil {
return mcpToolError(req, err.Message)
}
resultJSON, _ := json.Marshal(result)
return jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Result: map[string]interface{}{
"content": []map[string]interface{}{
{
"type": "text",
"text": string(resultJSON),
},
},
},
}
}
func mcpToolError(req jsonRPCRequest, message string) jsonRPCResponse {
return jsonRPCResponse{
JSONRPC: "2.0",
ID: req.ID,
Result: map[string]interface{}{
"isError": true,
"content": []map[string]interface{}{
{
"type": "text",
"text": message,
},
},
},
}
}

225
server/service/picoclaw/model_config.go Normal file
View File

@@ -0,0 +1,225 @@
package picoclaw
import (
"fmt"
"strings"
"github.com/gin-gonic/gin"
)
func extractPicoclawModelName(model string) string {
model = strings.TrimSpace(model)
if model == "" {
return ""
}
if index := strings.LastIndex(model, "/"); index >= 0 && index < len(model)-1 {
return strings.TrimSpace(model[index+1:])
}
return model
}
func isPicoclawModelConfigured(cfg picoclawConfigFile, security picoclawSecurityConfig, modelName string) bool {
if modelName == "" {
return false
}
for _, model := range cfg.ModelList {
if strings.TrimSpace(model.ModelName) != modelName {
continue
}
if model.APIBase == "" {
continue
}
if securityHasModelAPIKeys(security, modelName) {
return true
}
return configHasModelAPIKeys(model.APIKey, model.APIKeys)
}
return false
}
func configHasModelAPIKeys(apiKey string, apiKeys []string) bool {
return strings.TrimSpace(apiKey) != "" || len(apiKeys) > 0
}
func securityHasModelAPIKeys(security picoclawSecurityConfig, modelName string) bool {
if modelName == "" || len(security.ModelList) == 0 {
return false
}
if entry, ok := security.ModelList[modelName]; ok && len(entry.APIKeys) > 0 {
return true
}
prefix := modelName + ":"
for key, entry := range security.ModelList {
if strings.HasPrefix(key, prefix) && len(entry.APIKeys) > 0 {
return true
}
}
return false
}
type ModelConfigUpdateRequest struct {
Model string `json:"model"`
APIBase string `json:"api_base"`
APIKey string `json:"api_key"`
}
func (s *Service) UpdateModelConfig(c *gin.Context) {
var req ModelConfigUpdateRequest
if err := c.ShouldBindJSON(&req); err != nil {
writePicoclawError(c, newPicoclawError(CodeInvalidAction, "invalid model config payload"))
return
}
currentStatus := s.runtime.Get()
shouldRestart := currentStatus.Ready || currentStatus.Status == "ready"
modelName, err := updatePicoclawModelConfig(
strings.TrimSpace(req.APIBase),
strings.TrimSpace(req.APIKey),
strings.TrimSpace(req.Model),
)
if err != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, err.Error()))
return
}
if err := ensurePicoclawStartupDefaults(); err != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "model config saved, but failed to persist picoclaw defaults: "+err.Error()))
return
}
if shouldRestart {
if _, _, stopErr := s.stopRuntime(); stopErr != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "model config saved, but failed to restart picoclaw runtime: "+stopErr.Message))
return
}
if _, _, startErr := s.startRuntime(); startErr != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "model config saved, but failed to restart picoclaw runtime: "+startErr.Message))
return
}
} else {
_ = s.syncConfigFromPicoclaw()
_ = s.ensureRuntimeReady()
}
writeSuccess(c, gin.H{
"model_name": modelName,
"status": s.runtime.Get(),
})
}
func updatePicoclawModelConfig(apiBase string, apiKey string, model string) (string, error) {
if apiBase == "" {
return "", fmt.Errorf("model api_base is required")
}
if apiKey == "" {
return "", fmt.Errorf("model api_key is required")
}
if model == "" {
return "", fmt.Errorf("model identifier is required")
}
modelName := extractPicoclawModelName(model)
if modelName == "" {
return "", fmt.Errorf("model identifier is required")
}
doc, err := loadPicoclawConfigDocument()
if err != nil {
return "", err
}
modelListValue, ok := doc.raw["model_list"].([]any)
if !ok {
modelListValue = []any{}
}
modelUpdated := false
updatedModelIndex := -1
for index, item := range modelListValue {
modelMap, ok := item.(map[string]any)
if !ok {
continue
}
currentModelName := strings.TrimSpace(fmt.Sprintf("%v", modelMap["model_name"]))
if currentModelName != modelName {
continue
}
modelMap["model_name"] = modelName
modelMap["model"] = model
modelMap["api_base"] = apiBase
delete(modelMap, "api_key")
delete(modelMap, "api_keys")
modelUpdated = true
updatedModelIndex = index
break
}
if !modelUpdated {
modelListValue = append(modelListValue, map[string]any{
"model_name": modelName,
"model": model,
"api_base": apiBase,
})
doc.raw["model_list"] = modelListValue
updatedModelIndex = len(modelListValue) - 1
}
agents, ok := doc.raw["agents"].(map[string]any)
if !ok {
agents = map[string]any{}
doc.raw["agents"] = agents
}
defaults, ok := agents["defaults"].(map[string]any)
if !ok {
defaults = map[string]any{}
agents["defaults"] = defaults
}
defaults["model_name"] = modelName
delete(defaults, "model")
if err := doc.saveConfig(); err != nil {
return "", err
}
if doc.security.ModelList == nil {
doc.security.ModelList = map[string]picoclawModelSecurityEntry{}
}
securityModelName := indexedModelName(modelListValue, updatedModelIndex, modelName)
doc.security.ModelList[securityModelName] = picoclawModelSecurityEntry{
APIKeys: []string{apiKey},
}
if err := doc.saveSecurity(); err != nil {
return "", err
}
return modelName, nil
}
func indexedModelName(modelList []any, targetIndex int, modelName string) string {
if targetIndex < 0 {
return modelName
}
currentIndex := 0
for index, item := range modelList {
modelMap, ok := item.(map[string]any)
if !ok {
continue
}
currentModelName := strings.TrimSpace(fmt.Sprintf("%v", modelMap["model_name"]))
if currentModelName != modelName {
continue
}
if index == targetIndex {
return fmt.Sprintf("%s:%d", modelName, currentIndex)
}
currentIndex++
}
return modelName
}

22
server/service/picoclaw/runtime_constants.go Normal file
View File

@@ -0,0 +1,22 @@
package picoclaw
import (
"time"
)
const (
picoclawBinaryPath = "/usr/bin/picoclaw"
picoclawCacheDir = "/root/.picoclaw-cache"
picoclawDownloadURL = "https://cdn.sipeed.com/nanokvm/resources/picoclaw/v0.2.6/picoclaw_Linux_riscv64.tar.gz"
picoclawChecksumURL = "https://cdn.sipeed.com/nanokvm/resources/picoclaw/v0.2.6/sha512.txt"
etcInitPicoclawScript = "/etc/init.d/S96picoclaw"
kvmappPicoclawScript = "/kvmapp/system/init.d/S96picoclaw"
picoclawStartTimeout = 15 * time.Second
picoclawStopTimeout = 15 * time.Second
picoclawOnboardTimeout = 60 * time.Second
picoclawInstallTimeout = 12 * time.Minute
picoclawDownloadTimeout = 10 * time.Minute
picoclawStartWaitPeriod = 1500 * time.Millisecond
picoclawStopWaitPeriod = 500 * time.Millisecond
picoclawReadyPollPeriod = 500 * time.Millisecond
)

227
server/service/picoclaw/runtime_defaults.go Normal file
View File

@@ -0,0 +1,227 @@
package picoclaw
import (
"crypto/rand"
"encoding/hex"
"fmt"
"reflect"
"strings"
serverConfig "NanoKVM-Server/config"
)
const (
defaultPicoclawPingSec = 30
defaultPicoclawReadSec = 60
defaultPicoclawWriteSec = 10
defaultPicoclawMaxConns = 100
)
type picoclawConfigDefault struct {
path []string
value any
}
var picoclawNanoKVMDefaults = []picoclawConfigDefault{
{path: []string{"agents", "defaults", "restrict_to_workspace"}, value: false},
{path: []string{"agents", "defaults", "allow_read_outside_workspace"}, value: true},
{path: []string{"agents", "defaults", "tool_feedback", "enabled"}, value: true},
{path: []string{"gateway", "host"}, value: defaultPicoclawGatewayHost},
{path: []string{"gateway", "port"}, value: defaultPicoclawGatewayPort},
{path: []string{"gateway", "hot_reload"}, value: false},
{path: []string{"tools", "cron", "allow_command"}, value: true},
{path: []string{"tools", "exec", "allow_remote"}, value: true},
{path: []string{"tools", "exec", "enable_deny_patterns"}, value: false},
{path: []string{"channels", "pico", "allow_token_query"}, value: false},
{path: []string{"channels", "pico", "ping_interval"}, value: defaultPicoclawPingSec},
{path: []string{"channels", "pico", "read_timeout"}, value: defaultPicoclawReadSec},
{path: []string{"channels", "pico", "write_timeout"}, value: defaultPicoclawWriteSec},
{path: []string{"channels", "pico", "max_connections"}, value: defaultPicoclawMaxConns},
{path: []string{"tools", "mcp", "enabled"}, value: true},
}
func ensurePicoclawStartupDefaults() error {
doc, err := loadPicoclawConfigDocument()
if err != nil {
return err
}
editor := &picoclawConfigEditor{raw: doc.raw}
forceEnablePicoclawPicoChannel(doc, editor)
if err := applyPicoclawStartupDefaults(editor); err != nil {
return err
}
tokenChanged, err := ensurePicoclawPicoToken(doc)
if err != nil {
return err
}
if tokenChanged {
if err := doc.saveSecurity(); err != nil {
return err
}
}
if editor.changed {
if err := doc.saveConfig(); err != nil {
return err
}
}
return nil
}
func ensurePicoclawPicoChannelEnabled(doc *picoclawConfigDocument) error {
if doc == nil || doc.config.Channels.Pico.Enabled {
return nil
}
editor := &picoclawConfigEditor{raw: doc.raw}
forceEnablePicoclawPicoChannel(doc, editor)
if !editor.changed {
return nil
}
return doc.saveConfig()
}
func applyPicoclawStartupDefaults(editor *picoclawConfigEditor) error {
for _, entry := range picoclawNanoKVMDefaults {
editor.setValue(entry.value, entry.path...)
}
server, err := defaultPicoclawMCPServer()
if err != nil {
return err
}
editor.setMCPServer("nanokvm", server)
return nil
}
func forceEnablePicoclawPicoChannel(doc *picoclawConfigDocument, editor *picoclawConfigEditor) {
if doc == nil || editor == nil || doc.config.Channels.Pico.Enabled {
return
}
editor.setValue(true, "channels", "pico", "enabled")
doc.config.Channels.Pico.Enabled = true
}
func defaultPicoclawMCPServer() (map[string]any, error) {
internalToken, err := serverConfig.GetPicoclawInternalToken()
if err != nil {
return nil, err
}
return map[string]any{
"enabled": true,
"type": "http",
"url": nanoKVMMCPURL(),
"headers": map[string]any{
serverConfig.PicoclawInternalTokenHeader: internalToken,
},
}, nil
}
func nanoKVMMCPURL() string {
port := serverConfig.GetInstance().Port.Http
if port <= 0 {
port = 80
}
return fmt.Sprintf("http://%s:%d/api/picoclaw/mcp", defaultPicoclawGatewayHost, port)
}
type picoclawConfigEditor struct {
raw map[string]any
changed bool
}
func (e *picoclawConfigEditor) setValue(value any, path ...string) {
if len(path) == 0 {
return
}
target := e.ensureObject(path[:len(path)-1]...)
key := path[len(path)-1]
if current, exists := target[key]; exists && reflect.DeepEqual(current, value) {
return
}
target[key] = value
e.changed = true
}
func (e *picoclawConfigEditor) ensureObject(path ...string) map[string]any {
current := e.raw
for _, key := range path {
next, ok := current[key].(map[string]any)
if !ok {
next = map[string]any{}
current[key] = next
e.changed = true
}
current = next
}
return current
}
func (e *picoclawConfigEditor) setMCPServer(name string, fields map[string]any) {
servers := e.ensureObject("tools", "mcp", "servers")
entry, ok := servers[name].(map[string]any)
if !ok {
entry = map[string]any{}
servers[name] = entry
e.changed = true
}
for key, value := range fields {
if current, exists := entry[key]; exists && reflect.DeepEqual(current, value) {
continue
}
entry[key] = value
e.changed = true
}
}
func generatePicoclawToken() (string, error) {
buf := make([]byte, 24)
if _, err := rand.Read(buf); err != nil {
return "", fmt.Errorf("failed to generate picoclaw pico token: %w", err)
}
return hex.EncodeToString(buf), nil
}
func ensurePicoclawPicoToken(doc *picoclawConfigDocument) (bool, error) {
token, err := picoSecurityToken(doc)
if err != nil {
return false, err
}
if doc.security.Channels.Pico == nil {
doc.security.Channels.Pico = &picoclawPicoSecurity{}
}
if strings.TrimSpace(doc.security.Channels.Pico.Token) == token {
return false, nil
}
doc.security.Channels.Pico.Token = token
return true, nil
}
func picoSecurityToken(doc *picoclawConfigDocument) (string, error) {
if doc == nil {
return "", nil
}
if doc.security.Channels.Pico != nil {
if token := strings.TrimSpace(doc.security.Channels.Pico.Token); token != "" {
return token, nil
}
}
if token := strings.TrimSpace(doc.config.Channels.Pico.Token); token != "" {
return token, nil
}
return generatePicoclawToken()
}

110
server/service/picoclaw/runtime_handlers.go Normal file
View File

@@ -0,0 +1,110 @@
package picoclaw
import (
"os"
"path/filepath"
"time"
"github.com/gin-gonic/gin"
)
func (s *Service) StartRuntime(c *gin.Context) {
command, output, err := s.startRuntime()
if err != nil {
writePicoclawError(c, err)
return
}
writeSuccess(c, RuntimeStartResult{
Started: true,
Command: command,
Output: output,
Status: s.runtime.Get(),
})
}
func (s *Service) StopRuntime(c *gin.Context) {
command, output, err := s.stopRuntime()
if err != nil {
writePicoclawError(c, err)
return
}
writeSuccess(c, RuntimeStartResult{
Started: false,
Command: command,
Output: output,
Status: s.runtime.Get(),
})
}
func (s *Service) InstallRuntime(c *gin.Context) {
output, err := s.installRuntime()
if err != nil {
writePicoclawError(c, err)
return
}
currentStatus := s.runtime.Get()
writeSuccess(c, RuntimeInstallResult{
Installed: currentStatus.Installed && !currentStatus.Installing,
Binary: picoclawBinaryPath,
Download: picoclawDownloadURL,
Output: output,
Status: currentStatus,
})
}
func (s *Service) UninstallRuntime(c *gin.Context) {
currentStatus := s.runtime.Get()
uninstallOutput := "picoclaw uninstalled successfully"
if currentStatus.Installing {
writePicoclawError(c, newPicoclawError(CodeRuntimeStartFailed, "cannot uninstall while installation is in progress"))
return
}
running, err := isRuntimeRunning()
if err != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "failed to check picoclaw runtime status"))
return
}
if running || currentStatus.Ready || currentStatus.Status == "ready" {
if _, _, stopErr := s.stopRuntime(); stopErr != nil {
uninstallOutput = "picoclaw uninstalled successfully (stop failed before uninstall: " + stopErr.Message + ")"
}
}
if configPath, err := resolvePicoclawConfigPath(); err == nil {
_ = os.RemoveAll(filepath.Dir(configPath))
} else {
home, err := os.UserHomeDir()
if err == nil {
_ = os.RemoveAll(filepath.Join(home, ".picoclaw"))
} else {
_ = os.RemoveAll("/root/.picoclaw")
}
}
_ = os.Remove(picoclawBinaryPath)
_ = os.RemoveAll(picoclawCacheDir)
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: false,
Installing: false,
InstallProgress: 0,
InstallStage: "",
InstallPath: picoclawBinaryPath,
Status: "not_installed",
CheckedAt: time.Now(),
CurrentSession: "",
})
writeSuccess(c, RuntimeInstallResult{
Installed: false,
Binary: picoclawBinaryPath,
Download: picoclawDownloadURL,
Output: uninstallOutput,
Status: s.runtime.Get(),
})
}

441
server/service/picoclaw/runtime_install.go Normal file
View File

@@ -0,0 +1,441 @@
package picoclaw
import (
"archive/tar"
"compress/gzip"
"context"
"crypto/sha512"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"strings"
"time"
log "github.com/sirupsen/logrus"
)
func (s *Service) installRuntime() (string, *PicoclawError) {
log.Debugf("picoclaw install: start, binary=%s, cache=%s", picoclawBinaryPath, picoclawCacheDir)
currentStatus := s.runtime.Get()
if currentStatus.Installing {
log.Debugf("picoclaw install: install already in progress")
return "picoclaw installation is already in progress", nil
}
if installed, err := isPicoclawInstalled(); err == nil && installed {
settings, _ := loadPicoclawGatewaySettings()
log.Debugf("picoclaw install: binary already exists at %s", picoclawBinaryPath)
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 100,
InstallStage: "installed",
InstallPath: picoclawBinaryPath,
ModelConfigured: settings.ModelConfigured,
ModelName: settings.ModelName,
Status: "installed",
CheckedAt: time.Now(),
})
return "picoclaw is already installed", nil
}
ctx, cancel := context.WithTimeout(context.Background(), picoclawInstallTimeout)
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: false,
Installing: true,
InstallProgress: 0,
InstallStage: "preparing",
InstallPath: picoclawBinaryPath,
Status: "installing",
CheckedAt: time.Now(),
})
go s.runInstallRuntime(ctx, cancel)
return "picoclaw installation started", nil
}
func (s *Service) runInstallRuntime(ctx context.Context, cancel context.CancelFunc) {
defer cancel()
_ = os.RemoveAll(picoclawCacheDir)
if err := os.MkdirAll(picoclawCacheDir, 0o755); err != nil {
log.Errorf("picoclaw install: failed to create cache directory %s: %v", picoclawCacheDir, err)
s.finishInstallFailure("install_failed", fmt.Sprintf("failed to create cache directory: %v", err))
return
}
log.Debugf("picoclaw install: cache directory ready at %s", picoclawCacheDir)
defer func() {
if err := os.RemoveAll(picoclawCacheDir); err != nil {
log.Errorf("picoclaw install: failed to clean cache directory %s: %v", picoclawCacheDir, err)
return
}
log.Debugf("picoclaw install: cleaned cache directory %s", picoclawCacheDir)
}()
s.setInstallProgress("downloading", 5, "")
log.Debugf("picoclaw install: downloading checksum from %s", picoclawChecksumURL)
expectedDigest, err := downloadPicoclawChecksum(ctx)
if err != nil {
log.Errorf("picoclaw install: checksum download failed: %v", err)
s.finishInstallFailure(installFailureStatus(err), err.Error())
return
}
log.Debug("picoclaw install: checksum file downloaded")
archivePath := filepath.Join(picoclawCacheDir, "picoclaw.tar.gz")
log.Debugf("picoclaw install: downloading archive from %s to %s", picoclawDownloadURL, archivePath)
if err := downloadPicoclawArchive(ctx, archivePath, func(downloaded int64, total int64) {
progress := 10
if total > 0 {
progress = 10 + int(float64(downloaded)*70/float64(total))
if progress > 80 {
progress = 80
}
}
s.setInstallProgress("downloading", progress, "")
}); err != nil {
log.Errorf("picoclaw install: download failed: %v", err)
s.finishInstallFailure(installFailureStatus(err), err.Error())
return
}
log.Debugf("picoclaw install: archive download completed")
s.setInstallProgress("verifying", 82, "")
if err := verifyFileSHA512(archivePath, expectedDigest); err != nil {
log.Errorf("picoclaw install: checksum verification failed: %v", err)
s.finishInstallFailure(installFailureStatus(err), err.Error())
return
}
log.Debugf("picoclaw install: archive checksum verified for %s", archivePath)
s.setInstallProgress("extracting", 85, "")
log.Debugf("picoclaw install: extracting binary from %s", archivePath)
extractedPath, err := extractPicoclawBinary(archivePath, picoclawCacheDir)
if err != nil {
log.Errorf("picoclaw install: extract failed: %v", err)
s.finishInstallFailure(installFailureStatus(err), err.Error())
return
}
log.Debugf("picoclaw install: extracted binary to %s", extractedPath)
s.setInstallProgress("installing", 95, "")
log.Debugf("picoclaw install: installing binary to %s", picoclawBinaryPath)
if err := installPicoclawBinary(extractedPath, picoclawBinaryPath); err != nil {
log.Errorf("picoclaw install: install failed: %v", err)
s.finishInstallFailure(installFailureStatus(err), err.Error())
return
}
log.Debugf("picoclaw install: install completed successfully")
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 100,
InstallStage: "installed",
InstallPath: picoclawBinaryPath,
Status: "installed",
CheckedAt: time.Now(),
})
}
func downloadPicoclawArchive(ctx context.Context, destination string, onProgress func(downloaded int64, total int64)) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, picoclawDownloadURL, nil)
if err != nil {
return fmt.Errorf("failed to create download request: %w", err)
}
client := &http.Client{
Timeout: picoclawDownloadTimeout,
}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("failed to download picoclaw: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("failed to download picoclaw: unexpected status %s", resp.Status)
}
file, err := os.Create(destination)
if err != nil {
return fmt.Errorf("failed to create archive file: %w", err)
}
defer file.Close()
if err := copyWithProgress(ctx, file, resp.Body, resp.ContentLength, onProgress); err != nil {
return fmt.Errorf("failed to save archive: %w", err)
}
return nil
}
func downloadPicoclawChecksum(ctx context.Context) (string, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, picoclawChecksumURL, nil)
if err != nil {
return "", fmt.Errorf("failed to create checksum request: %w", err)
}
client := &http.Client{
Timeout: picoclawDownloadTimeout,
}
resp, err := client.Do(req)
if err != nil {
return "", fmt.Errorf("failed to download picoclaw checksum: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("failed to download picoclaw checksum: unexpected status %s", resp.Status)
}
data, err := io.ReadAll(io.LimitReader(resp.Body, 16*1024))
if err != nil {
return "", fmt.Errorf("failed to read checksum file: %w", err)
}
digest, err := parseSHA512Digest(string(data), filepath.Base(picoclawDownloadURL))
if err != nil {
return "", err
}
return digest, nil
}
func copyWithProgress(ctx context.Context, dst io.Writer, src io.Reader, total int64, onProgress func(downloaded int64, total int64)) error {
buffer := make([]byte, 32*1024)
var downloaded int64
for {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
n, readErr := src.Read(buffer)
if n > 0 {
if _, writeErr := dst.Write(buffer[:n]); writeErr != nil {
return writeErr
}
downloaded += int64(n)
if onProgress != nil {
onProgress(downloaded, total)
}
}
if readErr == io.EOF {
if onProgress != nil {
onProgress(downloaded, total)
}
return nil
}
if readErr != nil {
return readErr
}
}
}
func (s *Service) setInstallProgress(stage string, progress int, lastError string) {
if progress < 0 {
progress = 0
}
if progress > 100 {
progress = 100
}
s.runtime.UpdateInstallStatus(func(status *RuntimeStatus) {
status.Ready = false
status.Installed = false
status.Installing = true
status.InstallProgress = progress
status.InstallStage = stage
status.Status = "installing"
status.LastError = lastError
status.CheckedAt = time.Now()
})
}
func (s *Service) finishInstallFailure(status string, message string) {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: false,
Installing: false,
InstallProgress: 0,
InstallStage: status,
InstallPath: picoclawBinaryPath,
Status: status,
LastError: message,
CheckedAt: time.Now(),
})
}
func installFailureStatus(err error) string {
if err == nil {
return "install_failed"
}
if errors.Is(err, context.DeadlineExceeded) {
return "install_timeout"
}
return "install_failed"
}
func parseSHA512Digest(raw string, expectedName string) (string, error) {
lines := strings.Split(raw, "\n")
var fallback string
for _, line := range lines {
trimmed := strings.TrimSpace(line)
if trimmed == "" {
continue
}
fields := strings.Fields(trimmed)
if len(fields) == 0 {
continue
}
digest := fields[0]
if !isValidSHA512Digest(digest) {
continue
}
if len(fields) == 1 {
if fallback == "" {
fallback = strings.ToLower(digest)
}
continue
}
name := strings.TrimPrefix(fields[len(fields)-1], "*")
if expectedName == "" || name == expectedName {
return strings.ToLower(digest), nil
}
}
if fallback != "" {
return fallback, nil
}
return "", fmt.Errorf("failed to parse sha512 digest from checksum file")
}
func isValidSHA512Digest(value string) bool {
if len(value) != sha512.Size*2 {
return false
}
_, err := hex.DecodeString(value)
return err == nil
}
func verifyFileSHA512(filePath string, expectedDigest string) error {
expectedDigest = strings.ToLower(strings.TrimSpace(expectedDigest))
if !isValidSHA512Digest(expectedDigest) {
return fmt.Errorf("invalid expected sha512 digest")
}
file, err := os.Open(filePath)
if err != nil {
return fmt.Errorf("failed to open file for sha512 verification: %w", err)
}
defer file.Close()
hasher := sha512.New()
if _, err := io.Copy(hasher, file); err != nil {
return fmt.Errorf("failed to hash file for sha512 verification: %w", err)
}
actualDigest := hex.EncodeToString(hasher.Sum(nil))
if actualDigest != expectedDigest {
return fmt.Errorf("sha512 mismatch: got %s", actualDigest)
}
return nil
}
func extractPicoclawBinary(archivePath string, destinationDir string) (string, error) {
file, err := os.Open(archivePath)
if err != nil {
return "", fmt.Errorf("failed to open archive: %w", err)
}
defer file.Close()
gzipReader, err := gzip.NewReader(file)
if err != nil {
return "", fmt.Errorf("failed to read archive: %w", err)
}
defer gzipReader.Close()
tarReader := tar.NewReader(gzipReader)
for {
header, err := tarReader.Next()
if err == io.EOF {
break
}
if err != nil {
return "", fmt.Errorf("failed to extract archive: %w", err)
}
if header.Typeflag != tar.TypeReg {
continue
}
if filepath.Base(header.Name) != "picoclaw" {
continue
}
extractedPath := filepath.Join(destinationDir, "picoclaw")
outFile, err := os.OpenFile(extractedPath, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755)
if err != nil {
return "", fmt.Errorf("failed to create extracted binary: %w", err)
}
if _, err := io.Copy(outFile, tarReader); err != nil {
_ = outFile.Close()
return "", fmt.Errorf("failed to extract picoclaw binary: %w", err)
}
if err := outFile.Close(); err != nil {
return "", fmt.Errorf("failed to finalize extracted binary: %w", err)
}
return extractedPath, nil
}
return "", fmt.Errorf("picoclaw binary not found in archive")
}
func installPicoclawBinary(source string, destination string) error {
inFile, err := os.Open(source)
if err != nil {
return fmt.Errorf("failed to open extracted picoclaw binary: %w", err)
}
defer inFile.Close()
tempDestination := destination + ".tmp"
outFile, err := os.OpenFile(tempDestination, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755)
if err != nil {
return fmt.Errorf("failed to create destination binary: %w", err)
}
if _, err := io.Copy(outFile, inFile); err != nil {
_ = outFile.Close()
_ = os.Remove(tempDestination)
return fmt.Errorf("failed to write destination binary: %w", err)
}
if err := outFile.Close(); err != nil {
_ = os.Remove(tempDestination)
return fmt.Errorf("failed to finalize destination binary: %w", err)
}
if err := os.Chmod(tempDestination, 0o755); err != nil {
_ = os.Remove(tempDestination)
return fmt.Errorf("failed to set destination mode: %w", err)
}
if err := os.Rename(tempDestination, destination); err != nil {
_ = os.Remove(tempDestination)
return fmt.Errorf("failed to install picoclaw binary: %w", err)
}
return nil
}

31
server/service/picoclaw/runtime_session.go Normal file
View File

@@ -0,0 +1,31 @@
package picoclaw
import (
"strings"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
)
func (s *Service) ReleaseRuntimeSession(c *gin.Context) {
sessionID := strings.TrimSpace(c.GetHeader(sessionIDHeader))
if sessionID == "" {
writePicoclawError(c, newPicoclawError(CodeSessionIDMissing, "missing X-PicoClaw-Session-ID"))
return
}
if session, ok := GetSessionManager().Get(sessionID); ok {
s.closeGatewaySession(session, websocket.CloseNormalClosure, "session released")
} else {
ReleaseSession(sessionID)
}
status := s.runtime.Get()
status.CurrentSession = s.lock.Owner()
s.runtime.Set(status)
writeSuccess(c, gin.H{
"released": true,
"current_session": status.CurrentSession,
})
}

228
server/service/picoclaw/runtime_start_stop.go Normal file
View File

@@ -0,0 +1,228 @@
package picoclaw
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
func (s *Service) startRuntime() (string, string, *PicoclawError) {
if installed, statErr := isPicoclawInstalled(); statErr != nil {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Installed = false
status.ModelConfigured = false
status.ModelName = ""
status.Status = "not_installed"
status.LastError = statErr.Error()
status.CheckedAt = time.Now()
})
return "", "", newPicoclawError(CodeRuntimeStartFailed, statErr.Error())
} else if !installed {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Installed = false
status.ModelConfigured = false
status.ModelName = ""
status.Status = "not_installed"
status.LastError = "picoclaw is not installed"
status.CheckedAt = time.Now()
})
return "", "", newPicoclawError(CodeRuntimeStartFailed, "picoclaw is not installed")
}
scriptPath, err := resolvePicoclawStartScript()
if err != nil {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Installed = true
status.Status = "unavailable"
status.LastError = err.Error()
status.CheckedAt = time.Now()
})
return "", "", newPicoclawError(CodeRuntimeStartFailed, err.Error())
}
if err := ensurePicoclawStartupDefaults(); err != nil {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Installed = true
status.Status = "config_error"
status.ConfigError = err.Error()
status.LastError = err.Error()
status.CheckedAt = time.Now()
})
return "", "", newPicoclawError(CodeRuntimeStartFailed, err.Error())
}
command := scriptPath + " start"
ctx, cancel := context.WithTimeout(context.Background(), picoclawStartTimeout)
defer cancel()
output, execErr := exec.CommandContext(ctx, "sh", "-c", command).CombinedOutput()
trimmedOutput := strings.TrimSpace(string(output))
if execErr != nil {
s.runtime.Update(func(status *RuntimeStatus) {
status.Ready = false
status.Installed = true
status.Status = "unavailable"
status.LastError = trimmedOutput
if status.LastError == "" {
status.LastError = execErr.Error()
}
status.CheckedAt = time.Now()
})
return command, trimmedOutput, newPicoclawError(CodeRuntimeStartFailed, "failed to start picoclaw runtime")
}
time.Sleep(picoclawStartWaitPeriod)
if runtimeErr := s.waitForRuntimeReady(picoclawStartTimeout); runtimeErr != nil {
startErr := newPicoclawError(CodeRuntimeStartFailed, runtimeErr.Message)
return command, trimmedOutput, startErr
}
return command, trimmedOutput, nil
}
func (s *Service) stopRuntime() (string, string, *PicoclawError) {
settings, _ := loadPicoclawGatewaySettings()
scriptPath, err := resolvePicoclawStartScript()
if err != nil {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
InstallPath: picoclawBinaryPath,
ModelConfigured: settings.ModelConfigured,
ModelName: settings.ModelName,
Status: "unavailable",
LastError: err.Error(),
CheckedAt: time.Now(),
})
return "", "", newPicoclawError(CodeRuntimeStartFailed, err.Error())
}
command := scriptPath + " stop"
ctx, cancel := context.WithTimeout(context.Background(), picoclawStopTimeout)
defer cancel()
output, execErr := exec.CommandContext(ctx, "sh", "-c", command).CombinedOutput()
trimmedOutput := strings.TrimSpace(string(output))
if execErr != nil {
status := RuntimeStatus{
Ready: false,
Installed: true,
InstallPath: picoclawBinaryPath,
ModelConfigured: settings.ModelConfigured,
ModelName: settings.ModelName,
Status: "error",
LastError: trimmedOutput,
CheckedAt: time.Now(),
}
if trimmedOutput == "" {
status.LastError = execErr.Error()
}
s.runtime.Set(status)
return command, trimmedOutput, newPicoclawError(CodeRuntimeStartFailed, "failed to stop picoclaw runtime")
}
time.Sleep(picoclawStopWaitPeriod)
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
InstallPath: picoclawBinaryPath,
ModelConfigured: settings.ModelConfigured,
ModelName: settings.ModelName,
Status: "stopped",
CheckedAt: time.Now(),
CurrentSession: "",
})
return command, trimmedOutput, nil
}
func (s *Service) waitForRuntimeReady(timeout time.Duration) *PicoclawError {
deadline := time.Now().Add(timeout)
var lastErr *PicoclawError
for {
if runtimeErr := s.ensureRuntimeReady(); runtimeErr == nil {
return nil
} else {
lastErr = runtimeErr
}
if time.Now().After(deadline) {
break
}
time.Sleep(picoclawReadyPollPeriod)
}
if lastErr != nil {
return lastErr
}
return newPicoclawError(CodeRuntimeUnavailable, "gateway is unavailable")
}
func resolvePicoclawStartScript() (string, error) {
if _, err := os.Stat(etcInitPicoclawScript); err == nil {
return etcInitPicoclawScript, nil
}
if _, err := os.Stat(kvmappPicoclawScript); err == nil {
return kvmappPicoclawScript, nil
}
return "", fmt.Errorf("picoclaw start script not found: %s or %s", etcInitPicoclawScript, kvmappPicoclawScript)
}
func runPicoclawOnboard() (string, *PicoclawError) {
scriptPath, err := resolvePicoclawStartScript()
if err != nil {
return "", newPicoclawError(CodeRuntimeUnavailable, err.Error())
}
command := scriptPath + " onboard"
ctx, cancel := context.WithTimeout(context.Background(), picoclawOnboardTimeout)
defer cancel()
output, execErr := exec.CommandContext(ctx, "sh", "-c", command).CombinedOutput()
trimmedOutput := strings.TrimSpace(string(output))
if execErr != nil {
if trimmedOutput == "" {
trimmedOutput = execErr.Error()
}
return trimmedOutput, newPicoclawError(CodeRuntimeUnavailable, "failed to initialize picoclaw config")
}
return trimmedOutput, nil
}
func isPicoclawInstalled() (bool, error) {
info, err := os.Stat(picoclawBinaryPath)
if err == nil {
return !info.IsDir(), nil
}
if os.IsNotExist(err) {
return false, nil
}
return false, err
}
func isRuntimeRunning() (bool, error) {
binName := filepath.Base(picoclawBinaryPath)
if binName == "" || binName == "." || binName == string(filepath.Separator) {
return false, fmt.Errorf("invalid picoclaw binary path: %s", picoclawBinaryPath)
}
command := exec.Command("pidof", binName)
if err := command.Run(); err != nil {
if exitErr, ok := err.(*exec.ExitError); ok && exitErr.ExitCode() == 1 {
return false, nil
}
return false, err
}
return true, nil
}

173
server/service/picoclaw/screenshot.go Normal file
View File

@@ -0,0 +1,173 @@
package picoclaw
import (
"encoding/base64"
"net/http"
"time"
"NanoKVM-Server/common"
"NanoKVM-Server/service/stream/mjpeg"
"github.com/gin-gonic/gin"
)
var screenshotRetryDelay = 100 * time.Millisecond
const (
screenshotRetryCount = 3
cachedFrameMaxAge = 2 * time.Second
defaultPicoclawScreenshotWidth = 960
defaultPicoclawScreenshotHeight = 540
defaultPicoclawScreenshotQuality = 60
)
func (s *Service) Screenshot(c *gin.Context) {
var query ScreenshotQuery
if err := c.ShouldBindQuery(&query); err != nil {
writePicoclawError(c, newPicoclawError(CodeInvalidAction, "invalid screenshot query"))
return
}
sessionID, sessionErr := s.requireSessionID(c)
if sessionErr != nil {
writePicoclawError(c, sessionErr)
return
}
releaseAfter, lockErr := s.lock.AcquireTemporary(sessionID)
if lockErr != nil {
writePicoclawError(c, lockErr)
return
}
if releaseAfter {
defer s.lock.Release(sessionID)
}
data, meta, err := s.captureScreenshot(query)
if err != nil {
writePicoclawError(c, err)
return
}
if query.Format == "base64" {
meta.ImageBase64 = base64.StdEncoding.EncodeToString(data)
writeSuccess(c, meta)
return
}
c.Data(http.StatusOK, "image/jpeg", data)
}
func (s *Service) captureScreenshot(query ScreenshotQuery) ([]byte, ScreenshotMeta, *PicoclawError) {
width, height, quality := resolveScreenshotRequest(query)
if canUseCachedFrame(query) {
if frame, ok := mjpeg.GetLatestFrame(); ok && time.Since(frame.CapturedAt) <= cachedFrameMaxAge {
return frame.Data, ScreenshotMeta{
SourceWidth: frame.Width,
SourceHeight: frame.Height,
CaptureWidth: frame.Width,
CaptureHeight: frame.Height,
Format: "jpeg",
}, nil
}
}
screen := common.GetScreen()
common.CheckScreen()
for attempt := 0; attempt < screenshotRetryCount; attempt++ {
data, result := s.vision.ReadMjpeg(width, height, quality)
switch {
case result == 5:
if attempt < screenshotRetryCount-1 {
time.Sleep(screenshotRetryDelay)
continue
}
return nil, ScreenshotMeta{}, newPicoclawError(CodeScreenshotNoSignal, "no HDMI signal or frame unavailable")
case result < 0 || len(data) == 0:
return nil, ScreenshotMeta{}, newPicoclawError(CodeScreenshotFailed, "failed to capture screenshot")
default:
return data, ScreenshotMeta{
SourceWidth: screen.Width,
SourceHeight: screen.Height,
CaptureWidth: width,
CaptureHeight: height,
Format: "jpeg",
}, nil
}
}
return nil, ScreenshotMeta{}, newPicoclawError(CodeScreenshotFailed, "failed to capture screenshot")
}
func canUseCachedFrame(query ScreenshotQuery) bool {
return query.Width == 0 && query.Height == 0 && query.Quality == 0
}
func resolveScreenshotRequest(query ScreenshotQuery) (uint16, uint16, uint16) {
screen := common.GetScreen()
width := screen.Width
height := screen.Height
quality := screen.Quality
if query.Format == "base64" {
width, height = fitWithinBounds(width, height, defaultPicoclawScreenshotWidth, defaultPicoclawScreenshotHeight)
if quality == 0 || quality > defaultPicoclawScreenshotQuality {
quality = defaultPicoclawScreenshotQuality
}
}
width, height = applyRequestedDimensions(width, height, query.Width, query.Height)
if query.Quality > 0 {
quality = query.Quality
}
return width, height, quality
}
func applyRequestedDimensions(defaultWidth uint16, defaultHeight uint16, requestedWidth uint16, requestedHeight uint16) (uint16, uint16) {
switch {
case requestedWidth > 0 && requestedHeight > 0:
return requestedWidth, requestedHeight
case requestedWidth > 0:
return fitWithinBounds(defaultWidth, defaultHeight, requestedWidth, 0)
case requestedHeight > 0:
return fitWithinBounds(defaultWidth, defaultHeight, 0, requestedHeight)
default:
return defaultWidth, defaultHeight
}
}
func fitWithinBounds(sourceWidth uint16, sourceHeight uint16, maxWidth uint16, maxHeight uint16) (uint16, uint16) {
if sourceWidth == 0 || sourceHeight == 0 {
return sourceWidth, sourceHeight
}
if maxWidth == 0 && maxHeight == 0 {
return sourceWidth, sourceHeight
}
width := int(sourceWidth)
height := int(sourceHeight)
limitedWidth := width
limitedHeight := height
if maxWidth > 0 && limitedWidth > int(maxWidth) {
limitedWidth = int(maxWidth)
limitedHeight = height * limitedWidth / width
}
if maxHeight > 0 && limitedHeight > int(maxHeight) {
limitedHeight = int(maxHeight)
limitedWidth = width * limitedHeight / height
}
if limitedWidth <= 0 {
limitedWidth = 1
}
if limitedHeight <= 0 {
limitedHeight = 1
}
return uint16(limitedWidth), uint16(limitedHeight)
}

385
server/service/picoclaw/service.go Normal file
View File

@@ -0,0 +1,385 @@
package picoclaw
import (
"net"
"net/url"
"os"
"sync"
"time"
"NanoKVM-Server/common"
"NanoKVM-Server/service/hid"
"github.com/gin-gonic/gin"
)
const sessionIDHeader = "X-PicoClaw-Session-ID"
var (
configStoreOnce sync.Once
configStore *ConfigStore
runtimeStoreOnce sync.Once
runtimeStore *RuntimeStore
probeLoopOnce sync.Once
)
const runtimeStatusRefreshInterval = 2 * time.Second
func NewService() *Service {
return &Service{
vision: common.GetKvmVision(),
hid: hid.GetHid(),
config: getConfigStore(),
lock: GetSessionLock(),
runtime: getRuntimeStore(),
}
}
func getConfigStore() *ConfigStore {
configStoreOnce.Do(func() {
configStore = &ConfigStore{
config: defaultConfig(),
}
})
return configStore
}
func getRuntimeStore() *RuntimeStore {
runtimeStoreOnce.Do(func() {
runtimeStore = &RuntimeStore{
status: RuntimeStatus{
Ready: false,
Installed: false,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: false,
Status: "checking",
},
}
})
return runtimeStore
}
func (s *Service) GetRuntimeStatus(c *gin.Context) {
s.startRuntimeProbeLoop()
status := s.runtime.Get()
if shouldRefreshRuntimeStatus(status) {
_ = s.ensureRuntimeReady()
status = s.runtime.Get()
}
if installed, ok := picoclawInstalledState(); ok {
status.Installed = installed
s.runtime.Update(func(current *RuntimeStatus) {
current.Installed = installed
})
}
writeSuccess(c, withAgentProfile(status))
}
func (s *Service) GetRuntimeSession(c *gin.Context) {
writeSuccess(c, gin.H{
"current_session": s.lock.Owner(),
"checked_at": time.Now(),
})
}
func (s *Service) requireSessionID(c *gin.Context) (string, *PicoclawError) {
sessionID := c.GetHeader(sessionIDHeader)
if sessionID == "" {
return "", newPicoclawError(CodeSessionIDMissing, "missing X-PicoClaw-Session-ID")
}
return sessionID, nil
}
func (s *ConfigStore) Get() Config {
s.mu.RLock()
defer s.mu.RUnlock()
return s.config
}
func (s *ConfigStore) Set(cfg Config) {
s.mu.Lock()
defer s.mu.Unlock()
s.config = cfg
}
func (s *RuntimeStore) Get() RuntimeStatus {
s.mu.RLock()
defer s.mu.RUnlock()
return s.status
}
func (s *RuntimeStore) Set(status RuntimeStatus) {
s.mu.Lock()
defer s.mu.Unlock()
status.InstallPath = picoclawBinaryPath
s.status = status
}
func (s *RuntimeStore) Update(update func(*RuntimeStatus)) {
s.mu.Lock()
defer s.mu.Unlock()
update(&s.status)
s.status.InstallPath = picoclawBinaryPath
}
func (s *RuntimeStore) UpdateInstallStatus(update func(*RuntimeStatus)) {
s.mu.Lock()
defer s.mu.Unlock()
update(&s.status)
s.status.InstallPath = picoclawBinaryPath
}
func (s *Service) ensureRuntimeReady() *PicoclawError {
currentStatus := s.runtime.Get()
if currentStatus.Installing {
return newPicoclawError(CodeRuntimeUnavailable, "picoclaw installation is in progress")
}
installed, statErr := isPicoclawInstalled()
if statErr != nil {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: false,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
Status: "not_installed",
LastError: statErr.Error(),
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "failed to check picoclaw installation")
}
if !installed {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: false,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: false,
Status: "not_installed",
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "picoclaw is not installed")
}
configPath, pathErr := resolvePicoclawConfigPath()
if pathErr != nil {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: false,
Status: "model_not_configured",
LastError: pathErr.Error(),
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "picoclaw model is not configured")
}
if _, err := os.Stat(configPath); err != nil {
if os.IsNotExist(err) {
if _, onboardErr := runPicoclawOnboard(); onboardErr != nil {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: false,
Status: "model_not_configured",
LastError: onboardErr.Message,
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "picoclaw model is not configured")
}
if _, statErr := os.Stat(configPath); statErr == nil {
goto configReady
}
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: false,
Status: "model_not_configured",
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "picoclaw model is not configured")
}
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: false,
Status: "config_error",
ConfigError: err.Error(),
LastError: err.Error(),
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "gateway config is invalid")
}
configReady:
if syncErr := s.syncConfigFromPicoclaw(); syncErr != nil {
return syncErr
}
settings, settingsErr := loadPicoclawGatewaySettings()
if settingsErr != nil {
return newPicoclawError(CodeRuntimeUnavailable, settingsErr.Error())
}
if !settings.ModelConfigured {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: false,
ModelName: settings.TargetModelName,
Status: "model_not_configured",
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "picoclaw model is not configured")
}
cfg := s.config.Get()
parsed, err := url.Parse(cfg.GatewayURL)
if err != nil {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: true,
ModelName: settings.ModelName,
Status: "config_error",
ConfigError: "invalid gateway url",
LastError: err.Error(),
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "gateway config is invalid")
}
if parsed.Scheme != "ws" && parsed.Scheme != "wss" {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: true,
ModelName: settings.ModelName,
Status: "config_error",
ConfigError: "invalid gateway url scheme",
LastError: parsed.Scheme,
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "gateway config is invalid")
}
timeout := time.Duration(cfg.ConnectTimeoutMs) * time.Millisecond
hostPort, err := gatewayHostPort(parsed)
if err != nil {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: true,
ModelName: settings.ModelName,
Status: "config_error",
ConfigError: err.Error(),
LastError: err.Error(),
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "gateway config is invalid")
}
conn, err := net.DialTimeout("tcp", hostPort, timeout)
if err != nil {
s.runtime.Set(RuntimeStatus{
Ready: false,
Installed: true,
Installing: false,
InstallProgress: 0,
InstallPath: picoclawBinaryPath,
ModelConfigured: true,
ModelName: settings.ModelName,
Status: "unavailable",
LastError: err.Error(),
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return newPicoclawError(CodeRuntimeUnavailable, "gateway is unavailable")
}
_ = conn.Close()
s.runtime.Set(RuntimeStatus{
Ready: true,
Installed: true,
Installing: false,
InstallProgress: 100,
InstallPath: picoclawBinaryPath,
ModelConfigured: true,
ModelName: settings.ModelName,
Status: "ready",
CheckedAt: time.Now(),
CurrentSession: s.lock.Owner(),
})
return nil
}
func (s *Service) startRuntimeProbeLoop() {
probeLoopOnce.Do(func() {
go func() {
ticker := time.NewTicker(30 * time.Second)
defer ticker.Stop()
for range ticker.C {
_ = s.ensureRuntimeReady()
}
}()
})
}
func shouldRefreshRuntimeStatus(status RuntimeStatus) bool {
if status.Installing {
return false
}
if status.CheckedAt.IsZero() {
return true
}
return time.Since(status.CheckedAt) >= runtimeStatusRefreshInterval
}
func picoclawInstalledState() (bool, bool) {
installed, err := isPicoclawInstalled()
if err != nil {
return false, false
}
return installed, true
}

52
server/service/picoclaw/session_cleanup.go Normal file
View File

@@ -0,0 +1,52 @@
package picoclaw
import (
"os"
"path/filepath"
"NanoKVM-Server/service/hid"
log "github.com/sirupsen/logrus"
)
const picoclawMediaTempDirName = "picoclaw_media"
func ReleaseSession(sessionID string) {
GetSessionLock().Release(sessionID)
releaseAllHIDState()
}
func releaseAllHIDState() {
h := hid.GetHid()
h.WriteHid0([]byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00})
h.WriteHid1([]byte{0x00, 0x00, 0x00, 0x00})
h.WriteHid2([]byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00})
}
func (s *Service) releaseAllHIDState() {
s.hid.WriteHid0([]byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00})
s.hid.WriteHid1([]byte{0x00, 0x00, 0x00, 0x00})
s.hid.WriteHid2([]byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00})
}
func picoclawMediaTempDir() string {
return filepath.Join(os.TempDir(), picoclawMediaTempDirName)
}
func cleanupPicoclawMediaTempDir() {
mediaDir := picoclawMediaTempDir()
if _, err := os.Stat(mediaDir); err != nil {
if !os.IsNotExist(err) {
log.Warnf("failed to stat picoclaw media directory %s: %v", mediaDir, err)
}
return
}
if err := os.RemoveAll(mediaDir); err != nil {
log.Warnf("failed to remove picoclaw media directory %s: %v", mediaDir, err)
return
}
log.Infof("removed picoclaw media directory %s", mediaDir)
}

474
server/service/picoclaw/session_history.go Normal file
View File

@@ -0,0 +1,474 @@
package picoclaw
import (
"bufio"
"encoding/json"
"errors"
"net/http"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
"github.com/gin-gonic/gin"
)
const (
picoSessionPrefix = "agent:main:pico:direct:pico:"
sanitizedPicoSessionPrefix = "agent_main_pico_direct_pico_"
maxSessionJSONLLineSize = 10 * 1024 * 1024
maxSessionPreviewRunes = 60
defaultSessionLimit = 20
)
type sessionStoredMessage struct {
Role string `json:"role"`
Content string `json:"content"`
}
type sessionStoredFile struct {
Key string `json:"key"`
Messages []sessionStoredMessage `json:"messages"`
Summary string `json:"summary,omitempty"`
Created time.Time `json:"created"`
Updated time.Time `json:"updated"`
}
type sessionMetaFile struct {
Key string `json:"key"`
Summary string `json:"summary"`
Skip int `json:"skip"`
Count int `json:"count"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
type sessionListItem struct {
ID string `json:"id"`
Title string `json:"title"`
Preview string `json:"preview"`
MessageCount int `json:"message_count"`
Created string `json:"created"`
Updated string `json:"updated"`
}
type sessionDetailMessage struct {
Role string `json:"role"`
Content string `json:"content"`
}
type sessionDetail struct {
ID string `json:"id"`
Messages []sessionDetailMessage `json:"messages"`
Summary string `json:"summary,omitempty"`
Created string `json:"created"`
Updated string `json:"updated"`
}
func resolvePicoclawWorkspacePath() (string, error) {
configPath, err := resolvePicoclawConfigPath()
if err != nil {
return "", err
}
if doc, err := loadPicoclawConfigDocument(); err == nil {
if workspace := expandPicoclawPath(doc.config.Agents.Defaults.Workspace); workspace != "" {
return workspace, nil
}
}
return filepath.Join(filepath.Dir(configPath), "workspace"), nil
}
func resolvePicoclawSessionsPath() (string, error) {
workspacePath, err := resolvePicoclawWorkspacePath()
if err != nil {
return "", err
}
return filepath.Join(workspacePath, "sessions"), nil
}
func sanitizeSessionKey(key string) string {
return strings.ReplaceAll(key, ":", "_")
}
func extractPicoSessionID(key string) (string, bool) {
if strings.HasPrefix(key, picoSessionPrefix) {
return strings.TrimPrefix(key, picoSessionPrefix), true
}
return "", false
}
func extractPicoSessionIDFromSanitizedKey(key string) (string, bool) {
if strings.HasPrefix(key, sanitizedPicoSessionPrefix) {
return strings.TrimPrefix(key, sanitizedPicoSessionPrefix), true
}
return "", false
}
func (s *Service) ListSessions(c *gin.Context) {
dir, err := resolvePicoclawSessionsPath()
if err != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "failed to resolve sessions directory"))
return
}
entries, err := os.ReadDir(dir)
if err != nil {
if os.IsNotExist(err) {
writeSuccess(c, []sessionListItem{})
return
}
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "failed to read sessions directory"))
return
}
items := make([]sessionListItem, 0)
seen := make(map[string]struct{})
for _, entry := range entries {
if entry.IsDir() {
continue
}
name := entry.Name()
var (
sessionID string
sess sessionStoredFile
loadErr error
ok bool
)
switch {
case strings.HasSuffix(name, ".jsonl"):
sessionID, ok = extractPicoSessionIDFromSanitizedKey(strings.TrimSuffix(name, ".jsonl"))
if !ok {
continue
}
sess, loadErr = readJSONLSession(dir, sessionID)
if loadErr == nil && isEmptySession(sess) {
continue
}
case strings.HasSuffix(name, ".meta.json"):
continue
case strings.HasSuffix(name, ".json"):
base := strings.TrimSuffix(name, ".json")
if _, statErr := os.Stat(filepath.Join(dir, base+".jsonl")); statErr == nil {
if jsonlSessionID, found := extractPicoSessionIDFromSanitizedKey(base); found {
if jsonlSess, jsonlErr := readJSONLSession(dir, jsonlSessionID); jsonlErr == nil && !isEmptySession(jsonlSess) {
continue
}
}
}
sess, loadErr = readLegacySession(dir, name)
if loadErr == nil && isEmptySession(sess) {
continue
}
if loadErr == nil {
sessionID, ok = extractPicoSessionID(sess.Key)
if !ok {
continue
}
}
default:
continue
}
if loadErr != nil {
continue
}
if _, exists := seen[sessionID]; exists {
continue
}
seen[sessionID] = struct{}{}
items = append(items, buildSessionListItem(sessionID, sess))
}
sort.Slice(items, func(i, j int) bool {
return items[i].Updated > items[j].Updated
})
offset, _ := strconv.Atoi(c.DefaultQuery("offset", "0"))
limit, _ := strconv.Atoi(c.DefaultQuery("limit", strconv.Itoa(defaultSessionLimit)))
if offset < 0 {
offset = 0
}
if limit <= 0 {
limit = defaultSessionLimit
}
if offset >= len(items) {
writeSuccess(c, []sessionListItem{})
return
}
end := offset + limit
if end > len(items) {
end = len(items)
}
writeSuccess(c, items[offset:end])
}
func (s *Service) GetSession(c *gin.Context) {
sessionID := strings.TrimSpace(c.Param("id"))
if sessionID == "" {
writePicoclawError(c, newPicoclawError(CodeInvalidAction, "missing session id"))
return
}
dir, err := resolvePicoclawSessionsPath()
if err != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "failed to resolve sessions directory"))
return
}
sess, err := readJSONLSession(dir, sessionID)
if err == nil && isEmptySession(sess) {
err = os.ErrNotExist
}
if err != nil {
if errors.Is(err, os.ErrNotExist) {
sess, err = readLegacySession(dir, sanitizeSessionKey(picoSessionPrefix+sessionID)+".json")
if err == nil && isEmptySession(sess) {
err = os.ErrNotExist
}
}
if err != nil {
sessionErr := newPicoclawError(CodeRuntimeUnavailable, "session not found")
sessionErr.StatusCode = http.StatusNotFound
writePicoclawError(c, sessionErr)
return
}
}
messages := make([]sessionDetailMessage, 0, len(sess.Messages))
for _, msg := range sess.Messages {
if (msg.Role == "user" || msg.Role == "assistant") && strings.TrimSpace(msg.Content) != "" {
messages = append(messages, sessionDetailMessage{
Role: msg.Role,
Content: msg.Content,
})
}
}
writeSuccess(c, sessionDetail{
ID: sessionID,
Messages: messages,
Summary: sess.Summary,
Created: sess.Created.Format(time.RFC3339),
Updated: sess.Updated.Format(time.RFC3339),
})
}
func (s *Service) DeleteSession(c *gin.Context) {
sessionID := strings.TrimSpace(c.Param("id"))
if sessionID == "" {
writePicoclawError(c, newPicoclawError(CodeInvalidAction, "missing session id"))
return
}
dir, err := resolvePicoclawSessionsPath()
if err != nil {
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "failed to resolve sessions directory"))
return
}
base := filepath.Join(dir, sanitizeSessionKey(picoSessionPrefix+sessionID))
paths := []string{base + ".jsonl", base + ".meta.json", base + ".json"}
removed := false
for _, path := range paths {
if err := os.Remove(path); err != nil {
if os.IsNotExist(err) {
continue
}
writePicoclawError(c, newPicoclawError(CodeRuntimeUnavailable, "failed to delete session"))
return
}
removed = true
}
if !removed {
sessionErr := newPicoclawError(CodeRuntimeUnavailable, "session not found")
sessionErr.StatusCode = http.StatusNotFound
writePicoclawError(c, sessionErr)
return
}
writeSuccess(c, gin.H{
"id": sessionID,
"deleted": true,
})
}
func readLegacySession(dir, fileName string) (sessionStoredFile, error) {
data, err := os.ReadFile(filepath.Join(dir, fileName))
if err != nil {
return sessionStoredFile{}, err
}
var sess sessionStoredFile
if err := json.Unmarshal(data, &sess); err != nil {
return sessionStoredFile{}, err
}
return sess, nil
}
func readSessionMeta(path, sessionKey string) (sessionMetaFile, error) {
data, err := os.ReadFile(path)
if os.IsNotExist(err) {
return sessionMetaFile{Key: sessionKey}, nil
}
if err != nil {
return sessionMetaFile{}, err
}
var meta sessionMetaFile
if err := json.Unmarshal(data, &meta); err != nil {
return sessionMetaFile{}, err
}
if meta.Key == "" {
meta.Key = sessionKey
}
return meta, nil
}
func readSessionMessages(path string, skip int) ([]sessionStoredMessage, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
defer file.Close()
messages := make([]sessionStoredMessage, 0)
scanner := bufio.NewScanner(file)
scanner.Buffer(make([]byte, 0, 64*1024), maxSessionJSONLLineSize)
seen := 0
for scanner.Scan() {
line := scanner.Bytes()
if len(line) == 0 {
continue
}
seen++
if seen <= skip {
continue
}
var msg sessionStoredMessage
if err := json.Unmarshal(line, &msg); err != nil {
continue
}
messages = append(messages, msg)
}
if err := scanner.Err(); err != nil {
return nil, err
}
return messages, nil
}
func readJSONLSession(dir, sessionID string) (sessionStoredFile, error) {
sessionKey := picoSessionPrefix + sessionID
base := filepath.Join(dir, sanitizeSessionKey(sessionKey))
jsonlPath := base + ".jsonl"
metaPath := base + ".meta.json"
meta, err := readSessionMeta(metaPath, sessionKey)
if err != nil {
return sessionStoredFile{}, err
}
messages, err := readSessionMessages(jsonlPath, meta.Skip)
if err != nil {
return sessionStoredFile{}, err
}
created := meta.CreatedAt
updated := meta.UpdatedAt
if created.IsZero() || updated.IsZero() {
if info, statErr := os.Stat(jsonlPath); statErr == nil {
if created.IsZero() {
created = info.ModTime()
}
if updated.IsZero() {
updated = info.ModTime()
}
}
}
return sessionStoredFile{
Key: meta.Key,
Messages: messages,
Summary: meta.Summary,
Created: created,
Updated: updated,
}, nil
}
func buildSessionListItem(sessionID string, sess sessionStoredFile) sessionListItem {
preview := ""
for _, msg := range sess.Messages {
if msg.Role == "user" && strings.TrimSpace(msg.Content) != "" {
preview = msg.Content
break
}
}
title := strings.TrimSpace(sess.Summary)
if title == "" {
title = preview
}
title = truncateRunes(title, maxSessionPreviewRunes)
preview = truncateRunes(preview, maxSessionPreviewRunes)
if preview == "" {
preview = "(empty)"
}
if title == "" {
title = preview
}
messageCount := 0
for _, msg := range sess.Messages {
if (msg.Role == "user" || msg.Role == "assistant") && strings.TrimSpace(msg.Content) != "" {
messageCount++
}
}
return sessionListItem{
ID: sessionID,
Title: title,
Preview: preview,
MessageCount: messageCount,
Created: sess.Created.Format(time.RFC3339),
Updated: sess.Updated.Format(time.RFC3339),
}
}
func isEmptySession(sess sessionStoredFile) bool {
return len(sess.Messages) == 0 && strings.TrimSpace(sess.Summary) == ""
}
func truncateRunes(s string, maxLen int) string {
if maxLen <= 0 {
return ""
}
runes := []rune(strings.TrimSpace(s))
if len(runes) <= maxLen {
return string(runes)
}
return string(runes[:maxLen]) + "..."
}

97
server/service/picoclaw/session_lock.go Normal file
View File

@@ -0,0 +1,97 @@
package picoclaw
import (
"sync"
"time"
)
var (
sessionLockOnce sync.Once
sessionLock *SessionLock
)
type SessionLock struct {
mu sync.Mutex
ownerSessionID string
acquiredAt time.Time
expiresAt time.Time
}
func GetSessionLock() *SessionLock {
sessionLockOnce.Do(func() {
sessionLock = &SessionLock{}
})
return sessionLock
}
func (l *SessionLock) Ensure(sessionID string) *PicoclawError {
_, err := l.acquire(sessionID)
return err
}
func (l *SessionLock) AcquireTemporary(sessionID string) (bool, *PicoclawError) {
return l.acquire(sessionID)
}
func (l *SessionLock) acquire(sessionID string) (bool, *PicoclawError) {
if sessionID == "" {
return false, newPicoclawError(CodeSessionIDInvalid, "invalid PicoClaw session")
}
l.mu.Lock()
defer l.mu.Unlock()
if l.ownerSessionID == "" || l.ownerSessionID == sessionID {
acquired := l.ownerSessionID == ""
now := time.Now()
l.ownerSessionID = sessionID
if l.acquiredAt.IsZero() {
l.acquiredAt = now
}
l.expiresAt = now.Add(30 * time.Minute)
return acquired, nil
}
err := newPicoclawError(CodePicoclawLockHeld, "another PicoClaw session is running")
err.SessionID = l.ownerSessionID
return false, err
}
func (l *SessionLock) Release(sessionID string) bool {
l.mu.Lock()
defer l.mu.Unlock()
if l.ownerSessionID == "" {
return true
}
if sessionID != "" && l.ownerSessionID != sessionID {
return false
}
l.ownerSessionID = ""
l.acquiredAt = time.Time{}
l.expiresAt = time.Time{}
return true
}
func (l *SessionLock) ForceTakeover(sessionID string) {
l.mu.Lock()
defer l.mu.Unlock()
l.ownerSessionID = sessionID
l.acquiredAt = time.Now()
l.expiresAt = time.Now().Add(30 * time.Minute)
}
func (l *SessionLock) Owner() string {
l.mu.Lock()
defer l.mu.Unlock()
return l.ownerSessionID
}
func (l *SessionLock) BlocksManualInput() bool {
l.mu.Lock()
defer l.mu.Unlock()
return l.ownerSessionID != ""
}

94
server/service/picoclaw/session_manager.go Normal file
View File

@@ -0,0 +1,94 @@
package picoclaw
import (
"sync"
"time"
"github.com/gorilla/websocket"
)
var (
sessionManagerOnce sync.Once
sessionManager *SessionManager
)
func GetSessionManager() *SessionManager {
sessionManagerOnce.Do(func() {
sessionManager = &SessionManager{
sessions: make(map[string]*GatewaySession),
}
})
return sessionManager
}
func (m *SessionManager) Register(sessionID string, downstream *websocket.Conn) (*GatewaySession, *PicoclawError) {
m.mu.Lock()
defer m.mu.Unlock()
if existing, ok := m.sessions[sessionID]; ok {
switch existing.State {
case SessionStateClosing, SessionStateClosed:
delete(m.sessions, sessionID)
default:
err := newPicoclawError(CodePicoclawLockHeld, "session is already connected")
err.SessionID = sessionID
return nil, err
}
}
now := time.Now()
session := &GatewaySession{
SessionID: sessionID,
State: SessionStateCreated,
Downstream: downstream,
CreatedAt: now,
UpdatedAt: now,
}
m.sessions[sessionID] = session
return session, nil
}
func (m *SessionManager) Get(sessionID string) (*GatewaySession, bool) {
m.mu.RLock()
defer m.mu.RUnlock()
session, ok := m.sessions[sessionID]
return session, ok
}
func (m *SessionManager) SetState(sessionID string, state SessionState) {
m.mu.Lock()
defer m.mu.Unlock()
if session, ok := m.sessions[sessionID]; ok {
session.State = state
session.UpdatedAt = time.Now()
}
}
func (m *SessionManager) AttachUpstream(sessionID string, upstream *websocket.Conn) {
m.mu.Lock()
defer m.mu.Unlock()
if session, ok := m.sessions[sessionID]; ok {
session.Upstream = upstream
session.UpdatedAt = time.Now()
}
}
func (m *SessionManager) AttachDownstream(sessionID string, downstream *websocket.Conn) {
m.mu.Lock()
defer m.mu.Unlock()
if session, ok := m.sessions[sessionID]; ok {
session.Downstream = downstream
session.UpdatedAt = time.Now()
}
}
func (m *SessionManager) Remove(sessionID string) {
m.mu.Lock()
defer m.mu.Unlock()
delete(m.sessions, sessionID)
}

217
server/service/picoclaw/types.go Normal file
View File

@@ -0,0 +1,217 @@
package picoclaw
import (
"encoding/json"
"reflect"
"strings"
"sync"
"time"
"github.com/gorilla/websocket"
)
type VisionReader interface {
ReadMjpeg(width uint16, height uint16, quality uint16) (data []byte, result int)
}
type HIDWriter interface {
WriteHid0(data []byte)
WriteHid1(data []byte)
WriteHid2(data []byte)
}
type Service struct {
vision VisionReader
hid HIDWriter
config *ConfigStore
lock *SessionLock
runtime *RuntimeStore
}
type ConfigStore struct {
mu sync.RWMutex
config Config
}
type RuntimeStore struct {
mu sync.RWMutex
status RuntimeStatus
}
type Config struct {
GatewayURL string `json:"gateway_url"`
ConnectTimeoutMs int `json:"connect_timeout_ms"`
ReadTimeoutMs int `json:"read_timeout_ms"`
WriteTimeoutMs int `json:"write_timeout_ms"`
PingIntervalMs int `json:"ping_interval_ms"`
MaxMessageBytes int `json:"max_message_bytes"`
AllowTokenQuery bool `json:"allow_token_query"`
Token string `json:"token,omitempty"`
}
type RuntimeStatus struct {
Ready bool `json:"ready"`
Installed bool `json:"installed"`
Installing bool `json:"installing"`
InstallProgress int `json:"install_progress,omitempty"`
InstallStage string `json:"install_stage,omitempty"`
InstallPath string `json:"install_path,omitempty"`
AgentProfile string `json:"agent_profile,omitempty"`
ModelConfigured bool `json:"model_configured"`
ModelName string `json:"model_name,omitempty"`
Status string `json:"status"`
ConfigError string `json:"config_error,omitempty"`
LastError string `json:"last_error,omitempty"`
CheckedAt time.Time `json:"checked_at,omitempty"`
CurrentSession string `json:"current_session,omitempty"`
}
type RuntimeStartResult struct {
Started bool `json:"started"`
Command string `json:"command"`
Output string `json:"output,omitempty"`
Status RuntimeStatus `json:"status"`
}
type RuntimeInstallResult struct {
Installed bool `json:"installed"`
Binary string `json:"binary"`
Download string `json:"download"`
Output string `json:"output,omitempty"`
Status RuntimeStatus `json:"status"`
}
type ScreenshotMeta struct {
ImageBase64 string `json:"image_base64,omitempty"`
SourceWidth uint16 `json:"source_width"`
SourceHeight uint16 `json:"source_height"`
CaptureWidth uint16 `json:"capture_width"`
CaptureHeight uint16 `json:"capture_height"`
Format string `json:"format"`
}
type ScreenshotQuery struct {
Format string `form:"format"`
Width uint16 `form:"width"`
Height uint16 `form:"height"`
Quality uint16 `form:"quality"`
}
type Point struct {
X *float64 `json:"x"`
Y *float64 `json:"y"`
}
type HotkeyKeys []string
func (k *HotkeyKeys) UnmarshalJSON(data []byte) error {
if string(data) == "null" {
*k = nil
return nil
}
var list []string
if err := json.Unmarshal(data, &list); err == nil {
*k = normalizeHotkeyKeys(list)
return nil
}
var csv string
if err := json.Unmarshal(data, &csv); err == nil {
if strings.TrimSpace(csv) == "" {
*k = nil
return nil
}
*k = normalizeHotkeyKeys(strings.Split(csv, ","))
return nil
}
return &json.UnmarshalTypeError{Value: "keys", Type: hotkeyKeysType}
}
var hotkeyKeysType = reflect.TypeOf("")
func normalizeHotkeyKeys(input []string) HotkeyKeys {
keys := make([]string, 0, len(input))
for _, item := range input {
trimmed := strings.TrimSpace(item)
if trimmed == "" {
continue
}
keys = append(keys, trimmed)
}
return HotkeyKeys(keys)
}
type Action struct {
Action string `json:"action"`
X *float64 `json:"x"`
Y *float64 `json:"y"`
From *Point `json:"from"`
To *Point `json:"to"`
Button string `json:"button"`
Text string `json:"text"`
Keys HotkeyKeys `json:"keys"`
Direction string `json:"direction"`
Amount int `json:"amount"`
DurationMs int `json:"duration_ms"`
}
type ActionBatch struct {
Actions []Action `json:"actions"`
}
type ActionResult struct {
Action string `json:"action"`
DurationMs int64 `json:"duration_ms"`
HIDWrites int `json:"hid_writes"`
ExecutedActions int `json:"executed_actions,omitempty"`
}
type cachedFrame struct {
data []byte
width uint16
height uint16
capturedAt time.Time
}
type SessionState string
const (
SessionStateCreated SessionState = "created"
SessionStateConnecting SessionState = "connecting"
SessionStateActive SessionState = "active"
SessionStateClosing SessionState = "closing"
SessionStateClosed SessionState = "closed"
)
const (
CloseCodePicoclawLockHeld = 4001
CloseCodeRuntimeUnavailable = 4002
CloseCodeAuthFailed = 4003
CloseCodePicoclawTakenOver = 4004
CloseCodeUpstreamClosed = 4005
)
type GatewaySession struct {
SessionID string
State SessionState
Downstream *websocket.Conn
Upstream *websocket.Conn
CreatedAt time.Time
UpdatedAt time.Time
closeOnce sync.Once
upstreamWriteMu sync.Mutex
downstreamWriteMu sync.Mutex
}
type SessionManager struct {
mu sync.RWMutex
sessions map[string]*GatewaySession
}
type LoadImageRequest struct {
Path string `json:"path"`
Prompt string `json:"prompt"`
Filename string `json:"filename"`
}

19
server/service/stream/mjpeg/mjpeg.go
View File

@@ -8,6 +8,13 @@ import (
var streamer = NewStreamer()
type LatestFrame struct {
Data []byte
Width uint16
Height uint16
CapturedAt time.Time
}
func Connect(c *gin.Context) {
c.Header("Content-Type", "multipart/x-mixed-replace; boundary=frame")
c.Header("Cache-Control", "no-cache")
@@ -20,3 +27,15 @@ func Connect(c *gin.Context) {
<-c.Request.Context().Done()
}
func GetLatestFrame() (LatestFrame, bool) {
return streamer.getLatestFrame()
}
func EnableLatestFrameCache() {
streamer.enableLatestFrameCache()
}
func DisableLatestFrameCache() {
streamer.disableLatestFrameCache()
}

78
server/service/stream/mjpeg/streamer.go
View File

@@ -14,9 +14,12 @@ import (
)
type Streamer struct {
mutex sync.RWMutex
clients map[*gin.Context]bool
running int32
mutex sync.RWMutex
clients map[*gin.Context]bool
running int32
frameMutex sync.RWMutex
latestFrame LatestFrame
cacheRefs int32
}
func NewStreamer() *Streamer {
@@ -86,6 +89,10 @@ func (s *Streamer) run() {
continue
}
if s.frameCacheEnabled() {
s.setLatestFrame(data, screen.Width, screen.Height)
}
clients := s.getClients()
for _, client := range clients {
if err := writeFrame(client, data); err != nil {
@@ -103,6 +110,71 @@ func (s *Streamer) run() {
}
}
func (s *Streamer) setLatestFrame(data []byte, width uint16, height uint16) {
frameCopy := append([]byte(nil), data...)
s.frameMutex.Lock()
defer s.frameMutex.Unlock()
s.latestFrame = LatestFrame{
Data: frameCopy,
Width: width,
Height: height,
CapturedAt: time.Now(),
}
}
func (s *Streamer) clearLatestFrame() {
s.frameMutex.Lock()
defer s.frameMutex.Unlock()
s.latestFrame = LatestFrame{}
}
func (s *Streamer) enableLatestFrameCache() {
atomic.AddInt32(&s.cacheRefs, 1)
}
func (s *Streamer) disableLatestFrameCache() {
for {
current := atomic.LoadInt32(&s.cacheRefs)
if current <= 0 {
return
}
if atomic.CompareAndSwapInt32(&s.cacheRefs, current, current-1) {
if current == 1 {
s.clearLatestFrame()
}
return
}
}
}
func (s *Streamer) frameCacheEnabled() bool {
return atomic.LoadInt32(&s.cacheRefs) > 0
}
func (s *Streamer) getLatestFrame() (LatestFrame, bool) {
if !s.frameCacheEnabled() {
return LatestFrame{}, false
}
s.frameMutex.RLock()
defer s.frameMutex.RUnlock()
if len(s.latestFrame.Data) == 0 {
return LatestFrame{}, false
}
return LatestFrame{
Data: append([]byte(nil), s.latestFrame.Data...),
Width: s.latestFrame.Width,
Height: s.latestFrame.Height,
CapturedAt: s.latestFrame.CapturedAt,
}, true
}
func writeFrame(c *gin.Context, data []byte) (err error) {
defer func() {
if r := recover(); r != nil {

13
server/service/ws/client.go
View File

@@ -5,6 +5,7 @@ import (
"time"
"NanoKVM-Server/service/hid"
"NanoKVM-Server/service/picoclaw"
"NanoKVM-Server/service/vm/jiggler"
"github.com/gorilla/websocket"
@@ -50,14 +51,26 @@ func (c *Client) Read() error {
return err
}
if len(data) == 0 {
continue
}
log.Debugf("received message %d: %v", messageType, data)
switch data[0] {
case Heartbeat:
c.UpdateHeartbeat()
case KeyboardEvent:
if picoclaw.GetSessionLock().BlocksManualInput() {
log.Debug("manual keyboard input dropped while AI session holds control")
continue
}
writeQueue(c.keyboard, data[1:])
case MouseEvent:
if picoclaw.GetSessionLock().BlocksManualInput() {
log.Debug("manual mouse input dropped while AI session holds control")
continue
}
writeQueue(c.mouse, data[1:])
}
}

2
web/package.json
View File

@@ -31,9 +31,11 @@
"react-error-boundary": "^4.1.2",
"react-helmet-async": "^2.0.5",
"react-i18next": "^14.1.3",
"react-markdown": "^10.1.0",
"react-responsive": "^10.0.1",
"react-router-dom": "^6.30.3",
"react-simple-keyboard": "^3.8.164",
"remark-gfm": "^4.0.1",
"semver": "^7.7.4",
"vaul": "^0.9.9",
"websocket": "^1.0.35",

874
web/pnpm-lock.yaml generated
View File

File diff suppressed because it is too large Load Diff

130
web/src/api/picoclaw.ts Normal file
View File

@@ -0,0 +1,130 @@
import { http } from '@/lib/http.ts';
import {
picoclawGateway,
type GatewayRunState,
type GatewayTransportState
} from '@/lib/picoclaw-gateway.ts';
const sessionIDHeader = 'X-PicoClaw-Session-ID';
type ModelConfigRequest = {
model: string;
api_base: string;
api_key: string;
};
type AgentProfileRequest = {
profile: string;
};
export type PicoclawSessionListItem = {
id: string;
title: string;
preview: string;
message_count: number;
created: string;
updated: string;
};
export type PicoclawSessionDetailMessage = {
role: 'user' | 'assistant';
content: string;
};
export type PicoclawSessionDetail = {
id: string;
messages: PicoclawSessionDetailMessage[];
summary?: string;
created: string;
updated: string;
};
export function setPicoclawModelConfig(data: ModelConfigRequest) {
return http.post('/api/picoclaw/model/config', data);
}
export function setPicoclawAgentProfile(data: AgentProfileRequest) {
return http.post('/api/picoclaw/agent/profile', data);
}
export function listPicoclawSessions(params?: { offset?: number; limit?: number }) {
return http.get('/api/picoclaw/sessions', { params });
}
export function getPicoclawSession(id: string) {
return http.get(`/api/picoclaw/sessions/${id}`);
}
export function deletePicoclawSession(id: string) {
return http.delete(`/api/picoclaw/sessions/${id}`);
}
export function getRuntimeStatus() {
return http.get('/api/picoclaw/runtime/status');
}
export function startRuntime() {
return http.post('/api/picoclaw/runtime/start');
}
export function stopRuntime() {
return http.post('/api/picoclaw/runtime/stop');
}
export function installRuntime() {
return http.post('/api/picoclaw/runtime/install');
}
export function uninstallRuntime() {
return http.post('/api/picoclaw/runtime/uninstall');
}
export function connectGateway(sessionId?: string) {
return picoclawGateway.connect({ sessionId });
}
export function releaseRuntimeSession(sessionId?: string) {
const activeSessionId = sessionId || picoclawGateway.getSessionId();
if (!activeSessionId) {
return Promise.resolve(null);
}
return http.request({
method: 'delete',
url: '/api/picoclaw/runtime/session',
headers: {
[sessionIDHeader]: activeSessionId
}
});
}
export function sendChatMessage(
content: string,
options?: { id?: string; maxSteps?: number; maxRuntimeMs?: number; trackState?: boolean }
) {
return picoclawGateway.sendChatMessage(content, options);
}
export function sendStopMessage() {
return picoclawGateway.sendStopMessage();
}
export function closeGateway() {
const activeSessionId = picoclawGateway.getSessionId();
picoclawGateway.close();
if (!activeSessionId) {
return;
}
void releaseRuntimeSession(activeSessionId).catch(() => undefined);
}
export function onGatewayConnectionState(listener: (state: GatewayTransportState) => void) {
return picoclawGateway.on('transport_state', listener);
}
export function onGatewayRunState(listener: (state: GatewayRunState) => void) {
return picoclawGateway.on('run_state', listener);
}
export { picoclawGateway };

1
web/src/assets/images/monitor-x.svg
View File

@@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="#737373" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-monitor-x"><path d="m14.5 12.5-5-5"/><path d="m9.5 12.5 5-5"/><rect width="20" height="14" x="2" y="3" rx="2"/><path d="M12 17v4"/><path d="M8 21h8"/></svg>
Side by Side

Before

Width:  |  Height:  |  Size: 354 B

34
web/src/assets/styles/index.css
View File

@@ -2,7 +2,8 @@
@tailwind components;
@tailwind utilities;
html, body {
html,
body {
padding: 0;
margin: 0;
background: #000;
@@ -23,7 +24,7 @@ html, body {
}
.spin {
animation: spin 1s linear;
animation: spin 1s linear;
}
@keyframes spin {
@@ -34,3 +35,32 @@ animation: spin 1s linear;
transform: rotate(360deg);
}
}
/* PicoClaw Chat Custom Scrollbar */
.picoclaw-sidebar-scrollbar *::-webkit-scrollbar {
width: 4px;
height: 4px;
}
.picoclaw-sidebar-scrollbar *::-webkit-scrollbar-track {
background: transparent;
}
.picoclaw-sidebar-scrollbar *::-webkit-scrollbar-thumb {
background: rgba(255, 255, 255, 0.15);
border-radius: 4px;
}
.picoclaw-sidebar-scrollbar *::-webkit-scrollbar-thumb:hover {
background: rgba(255, 255, 255, 0.3);
}
/* PicoClaw Chat Loading Animation */
@keyframes picoclaw-loading-bounce {
0%,
100% {
transform: translateY(0.5px);
opacity: 0.3;
}
50% {
transform: translateY(-1.5px);
opacity: 1;
}
}

43
web/src/components/icons/icon-base.tsx Normal file
View File

@@ -0,0 +1,43 @@
import type { ReactNode, SVGProps } from 'react';
export type IconProps = Omit<SVGProps<SVGSVGElement>, 'color'> & {
size?: number | string;
color?: string;
strokeWidth?: number | string;
};
type IconBaseProps = IconProps & {
children: ReactNode;
};
export const IconBase = ({
children,
className,
size = 24,
color = 'currentColor',
strokeWidth = 2,
viewBox = '0 0 24 24',
fill = 'none',
strokeLinecap = 'round',
strokeLinejoin = 'round',
...props
}: IconBaseProps) => {
return (
<svg
aria-hidden={props['aria-label'] ? undefined : true}
className={className}
xmlns="http://www.w3.org/2000/svg"
width={size}
height={size}
viewBox={viewBox}
fill={fill}
stroke={color}
strokeWidth={strokeWidth}
strokeLinecap={strokeLinecap}
strokeLinejoin={strokeLinejoin}
{...props}
>
{children}
</svg>
);
};

12
web/src/components/icons/layout-sidebar-right-collapse.tsx Normal file
View File

@@ -0,0 +1,12 @@
import { IconBase, type IconProps } from './icon-base.tsx';
export const LayoutSidebarRightCollapse = (props: IconProps) => {
return (
<IconBase {...props}>
<path stroke="none" d="M0 0h24v24H0z" fill="none" />
<path d="M4 6a2 2 0 0 1 2 -2h12a2 2 0 0 1 2 2v12a2 2 0 0 1 -2 2h-12a2 2 0 0 1 -2 -2l0 -12" />
<path d="M15 4v16" />
<path d="M9 10l2 2l-2 2" />
</IconBase>
);
};

18
web/src/components/icons/robot.tsx Normal file
View File

@@ -0,0 +1,18 @@
import { IconBase, type IconProps } from './icon-base.tsx';
export const Robot = (props: IconProps) => {
return (
<IconBase {...props}>
<path stroke="none" d="M0 0h24v24H0z" fill="none" />
<path d="M6 6a2 2 0 0 1 2 -2h8a2 2 0 0 1 2 2v4a2 2 0 0 1 -2 2h-8a2 2 0 0 1 -2 -2l0 -4" />
<path d="M12 2v2" />
<path d="M9 12v9" />
<path d="M15 12v9" />
<path d="M5 16l4 -2" />
<path d="M15 14l4 2" />
<path d="M9 18h6" />
<path d="M10 8v.01" />
<path d="M14 8v.01" />
</IconBase>
);
};

123
web/src/i18n/locales/en.ts
View File

@@ -54,7 +54,7 @@ const en = {
authDescription: 'Please enter the AP password to continue',
authFailed: 'Invalid AP password',
passPlaceholder: 'AP password',
verifyBtn: 'Verify',
verifyBtn: 'Verify'
}
},
screen: {
@@ -449,6 +449,127 @@ const en = {
cancelBtn: 'No'
}
},
picoclaw: {
title: 'PicoClaw Assistant',
empty: 'Open the panel and start a task to begin.',
inputPlaceholder: 'Describe what you want the PicoClaw to do',
newConversation: 'New conversation',
processing: 'Processing...',
agent: {
defaultTitle: 'General Assistant',
defaultDescription: 'General chat, search, and workspace help.',
kvmTitle: 'Remote Control',
kvmDescription: 'Operate the remote host through NanoKVM.',
switched: 'Agent role switched',
switchFailed: 'Failed to switch agent role'
},
send: 'Send',
cancel: 'Cancel',
status: {
connecting: 'Connecting to gateway...',
connected: 'PicoClaw session connected',
disconnected: 'PicoClaw session closed',
stopped: 'Stop request sent',
runtimeStarted: 'PicoClaw runtime started',
runtimeStartFailed: 'Failed to start PicoClaw runtime',
runtimeStopped: 'PicoClaw runtime stopped',
runtimeStopFailed: 'Failed to stop PicoClaw runtime'
},
connection: {
runtime: {
checking: 'Checking',
ready: 'Runtime ready',
stopped: 'Runtime stopped',
unavailable: 'Runtime unavailable',
configError: 'Configuration error'
},
transport: {
connecting: 'Connecting',
connected: 'Connected'
},
run: {
idle: 'Idle',
busy: 'Busy'
}
},
message: {
toolAction: 'Action',
observation: 'Observation',
screenshot: 'Screenshot'
},
overlay: {
locked: 'PicoClaw is controlling the device. Manual input is paused.'
},
install: {
install: 'Install PicoClaw',
installing: 'Installing PicoClaw',
success: 'PicoClaw installed successfully',
failed: 'Failed to install PicoClaw',
uninstalling: 'Uninstalling runtime...',
uninstalled: 'Runtime uninstalled successfully.',
uninstallFailed: 'Uninstall failed.',
requiredTitle: 'PicoClaw is not installed',
requiredDescription: 'Install PicoClaw before starting the PicoClaw runtime.',
progressDescription: 'PicoClaw is being downloaded and installed.',
stages: {
preparing: 'Preparing',
downloading: 'Downloading',
extracting: 'Extracting',
installing: 'Installing',
installed: 'Installed',
install_timeout: 'Timed Out',
install_failed: 'Failed'
}
},
model: {
requiredTitle: 'Model configuration is required',
requiredDescription: 'Configure the PicoClaw model before using PicoClaw chat.',
docsTitle: 'Configuration Guide',
docsDesc: 'Supported models and protocols',
menuLabel: 'Configure model',
modelIdentifier: 'Model Identifier',
modelIdentifierPlaceholder: 'openai/gpt-5.4',
apiBase: 'API Base URL',
apiBasePlaceholder: 'https://api.example.com/v1',
apiKey: 'API Key',
apiKeyPlaceholder: 'Enter the model API key',
save: 'Save',
saving: 'Saving',
saved: 'Model configuration saved',
saveFailed: 'Failed to save model configuration',
invalid: 'Model identifier, API base URL, and API key are required'
},
uninstall: {
menuLabel: 'Uninstall',
confirmTitle: 'Uninstall PicoClaw',
confirmContent:
'Are you sure you want to uninstall PicoClaw? This will delete the executable and all configuration files.',
confirmOk: 'Uninstall',
confirmCancel: 'Cancel'
},
history: {
title: 'History',
loading: 'Loading sessions...',
emptyTitle: 'No history yet',
emptyDescription: 'Previous PicoClaw sessions will appear here.',
loadFailed: 'Failed to load session history',
deleteFailed: 'Failed to delete session',
deleteConfirmTitle: 'Delete session',
deleteConfirmContent: 'Are you sure you want to delete "{{title}}"?',
deleteConfirmOk: 'Delete',
deleteConfirmCancel: 'Cancel',
messageCount_one: '{{count}} message',
messageCount_other: '{{count}} messages'
},
config: {
startRuntime: 'Start PicoClaw',
stopRuntime: 'Stop PicoClaw'
},
start: {
title: 'Start PicoClaw',
description: 'Start the runtime to begin using the PicoClaw assistant.'
}
},
error: {
title: "We've ran into an issue",
refresh: 'Refresh'

122
web/src/i18n/locales/zh.ts
View File

@@ -52,7 +52,7 @@ const zh = {
authDescription: '请输入 AP 密码以继续',
authFailed: '密码错误',
passPlaceholder: 'AP 密码',
verifyBtn: '验证',
verifyBtn: '验证'
}
},
screen: {
@@ -414,6 +414,126 @@ const zh = {
cancel: '取消'
}
},
picoclaw: {
title: 'PicoClaw 助手',
empty: '打开面板并发送任务后，PicoClaw 会开始工作。',
inputPlaceholder: '描述你希望 PicoClaw 执行的操作',
newConversation: '新对话',
processing: '正在处理中...',
agent: {
defaultTitle: '通用助手',
defaultDescription: '适合普通问答、搜索和工作区辅助。',
kvmTitle: '远程控制',
kvmDescription: '通过 NanoKVM 操作下游远程主机。',
switched: '角色已切换',
switchFailed: '切换角色失败'
},
send: '发送',
cancel: '取消',
status: {
connecting: '正在连接 Gateway...',
connected: 'PicoClaw 会话已连接',
disconnected: 'PicoClaw 会话已关闭',
stopped: '已发送停止请求',
runtimeStarted: 'PicoClaw 运行时已启动',
runtimeStartFailed: '启动 PicoClaw 运行时失败',
runtimeStopped: 'PicoClaw 运行时已停止',
runtimeStopFailed: '停止 PicoClaw 运行时失败'
},
connection: {
runtime: {
checking: '检查中',
ready: '运行时已就绪',
stopped: '运行时未启动',
unavailable: '运行时不可用',
configError: '配置错误'
},
transport: {
connecting: '连接中',
connected: '已连接'
},
run: {
idle: '空闲',
busy: '执行中'
}
},
message: {
toolAction: '动作',
observation: '观察',
screenshot: '截图'
},
overlay: {
locked: 'PicoClaw 正在控制设备，手动输入已暂停。'
},
install: {
install: '安装 PicoClaw',
installing: '正在安装 PicoClaw',
success: 'PicoClaw 安装成功',
failed: 'PicoClaw 安装失败',
uninstalling: '正在卸载运行时...',
uninstalled: '运行时卸载成功。',
uninstallFailed: '卸载失败。',
requiredTitle: '未安装 PicoClaw',
requiredDescription: '在启动 PicoClaw 运行时之前，需要先下载安装 PicoClaw。',
progressDescription: '正在下载并安装 PicoClaw。',
stages: {
preparing: '准备中',
downloading: '下载中',
extracting: '解压中',
installing: '安装中',
installed: '已安装',
install_timeout: '已超时',
install_failed: '失败'
}
},
model: {
requiredTitle: '需要配置模型',
requiredDescription: '在使用 PicoClaw 聊天之前，请先配置 PicoClaw 模型。',
docsTitle: '配置指南',
docsDesc: '支持的模型与协议',
menuLabel: '配置模型',
modelIdentifier: '模型标识',
modelIdentifierPlaceholder: 'openai/gpt-5.4',
apiBase: 'API Base URL',
apiBasePlaceholder: 'https://api.example.com/v1',
apiKey: 'API Key',
apiKeyPlaceholder: '请输入模型 API Key',
save: '保存模型配置',
saving: '正在保存模型配置',
saved: '模型配置已保存',
saveFailed: '保存模型配置失败',
invalid: '模型标识、API Base URL 和 API Key 不能为空'
},
uninstall: {
menuLabel: '卸载',
confirmTitle: '卸载 PicoClaw',
confirmContent: '确定要卸载 PicoClaw 吗？这会删除可执行文件和所有配置文件。',
confirmOk: '卸载',
confirmCancel: '取消'
},
history: {
title: '历史会话',
loading: '正在加载历史会话...',
emptyTitle: '还没有历史会话',
emptyDescription: '之前的 PicoClaw 会话会显示在这里。',
loadFailed: '加载会话历史失败',
deleteFailed: '删除会话失败',
deleteConfirmTitle: '删除会话',
deleteConfirmContent: '确定要删除“{{title}}”吗？',
deleteConfirmOk: '删除',
deleteConfirmCancel: '取消',
messageCount_one: '{{count}} 条消息',
messageCount_other: '{{count}} 条消息'
},
config: {
startRuntime: '启动 PicoClaw',
stopRuntime: '停止 PicoClaw'
},
start: {
title: '启动 PicoClaw',
description: '启动运行时后即可开始使用 PicoClaw 助手。'
}
},
error: {
title: '我们遇到了问题',
refresh: '刷新'

94
web/src/jotai/picoclaw.ts Normal file
View File

@@ -0,0 +1,94 @@
import { atom } from 'jotai';
import { getPicoclawMaxRuntimeMs, getPicoclawMaxSteps } from '@/lib/picoclaw-storage.ts';
export type PicoclawTransportState = 'disconnected' | 'connecting' | 'connected' | 'error';
export type PicoclawRunState = 'idle' | 'busy';
export type PicoclawRuntimeStatus = {
ready: boolean;
installed: boolean;
installing: boolean;
install_progress?: number;
install_stage?: string;
install_path?: string;
agent_profile?: string;
model_configured: boolean;
model_name?: string;
status: string;
config_error?: string;
last_error?: string;
checked_at?: string;
current_session?: string;
};
export type PicoclawRuntimeStartResult = {
started: boolean;
command: string;
output?: string;
status: PicoclawRuntimeStatus;
};
export type PicoclawRuntimeInstallResult = {
installed: boolean;
binary: string;
download: string;
output?: string;
status: PicoclawRuntimeStatus;
};
export type PicoclawMessageKind =
| 'user'
| 'assistant'
| 'status'
| 'tool_action'
| 'observation'
| 'error';
export type PicoclawChatMessage = {
id: string;
kind: PicoclawMessageKind;
text?: string;
imageBase64?: string;
createdAt: number;
action?: string;
pending?: boolean;
raw?: unknown;
};
export type PicoclawOverlayState = {
visible: boolean;
message: string;
x?: number;
y?: number;
action?: string;
};
export type PicoclawTakeoverState = {
active: boolean;
sessionId?: string;
reason?: string;
};
export type PicoclawConfigState = {
maxSteps: number;
maxRuntimeMs: number;
};
export const picoclawChatOpenAtom = atom(false);
export const picoclawMessagesAtom = atom<PicoclawChatMessage[]>([]);
export const picoclawTransportStateAtom = atom<PicoclawTransportState>('disconnected');
export const picoclawRunStateAtom = atom<PicoclawRunState>('idle');
export const picoclawRuntimeStatusAtom = atom<PicoclawRuntimeStatus | null>(null);
export const picoclawOverlayAtom = atom<PicoclawOverlayState>({
visible: false,
message: ''
});
export const picoclawConfigAtom = atom<PicoclawConfigState>({
maxSteps: getPicoclawMaxSteps(),
maxRuntimeMs: getPicoclawMaxRuntimeMs()
});
export const picoclawTakeoverStateAtom = atom<PicoclawTakeoverState>({
active: false
});

471
web/src/lib/picoclaw-gateway.ts Normal file
View File

@@ -0,0 +1,471 @@
import { clearPicoclawSessionId, setPicoclawSessionId } from '@/lib/picoclaw-storage.ts';
import { getBaseUrl } from '@/lib/service.ts';
export type GatewayTransportState = 'disconnected' | 'connecting' | 'connected' | 'error';
export type GatewayRunState = 'idle' | 'busy';
export type GatewayRuntimeStatus = {
ready: boolean;
installed: boolean;
installing: boolean;
install_progress?: number;
install_stage?: string;
install_path?: string;
model_configured: boolean;
model_name?: string;
status: string;
config_error?: string;
last_error?: string;
checked_at?: string;
current_session?: string;
};
export type GatewayAssistantMessage = {
id: string;
text: string;
raw: Record<string, unknown>;
};
export type GatewayToolAction = {
id: string;
action: string;
x?: number;
y?: number;
raw: Record<string, unknown>;
};
export type GatewayObservation = {
id: string;
text?: string;
imageBase64?: string;
raw: Record<string, unknown>;
};
export type GatewayError = {
code: string;
message: string;
raw?: unknown;
};
export type GatewayClose = {
code: number;
reason: string;
};
type GatewayEventMap = {
connected: { sessionId: string };
transport_state: GatewayTransportState;
run_state: GatewayRunState;
assistant_message: GatewayAssistantMessage;
tool_action: GatewayToolAction;
observation: GatewayObservation;
error: GatewayError;
close: GatewayClose;
};
type EventName = keyof GatewayEventMap;
type Listener<T extends EventName> = (payload: GatewayEventMap[T]) => void;
type ConnectOptions = {
sessionId?: string;
autoReconnect?: boolean;
};
type SendMessageOptions = {
id?: string;
maxSteps?: number;
maxRuntimeMs?: number;
trackState?: boolean;
};
const CLOSE_ERRORS: Record<number, string> = {
4001: 'AI_LOCK_HELD',
4002: 'RUNTIME_UNAVAILABLE',
4003: 'AUTH_FAILED',
4004: 'AI_TAKEN_OVER',
4005: 'UPSTREAM_CLOSED'
};
export function generateUUIDv4() {
if (typeof crypto !== 'undefined' && typeof crypto.randomUUID === 'function') {
return crypto.randomUUID();
}
return 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace(/[xy]/g, (char) => {
const random = Math.floor(Math.random() * 16);
const value = char === 'x' ? random : (random & 0x3) | 0x8;
return value.toString(16);
});
}
class PicoClawGateway {
private ws: WebSocket | null = null;
private listeners = new Map<EventName, Set<Listener<EventName>>>();
private connectPromise: Promise<string> | null = null;
private sessionId = '';
private connectionAttemptId = 0;
private autoReconnect = false;
private reconnectAttempts = 0;
private reconnectTimer: ReturnType<typeof setTimeout> | null = null;
private explicitClose = false;
public connect(options: ConnectOptions = {}) {
if (options.sessionId) {
this.sessionId = options.sessionId;
setPicoclawSessionId(options.sessionId);
} else if (!this.ws || this.ws.readyState === WebSocket.CLOSED) {
this.sessionId = generateUUIDv4();
setPicoclawSessionId(this.sessionId);
}
if (
this.ws &&
(this.ws.readyState === WebSocket.OPEN || this.ws.readyState === WebSocket.CONNECTING)
) {
return this.connectPromise ?? Promise.resolve(this.sessionId);
}
this.autoReconnect = !!options.autoReconnect;
this.explicitClose = false;
this.clearReconnectTimer();
this.setTransportState('connecting');
const sessionId = this.sessionId || generateUUIDv4();
const attemptId = ++this.connectionAttemptId;
this.sessionId = sessionId;
setPicoclawSessionId(sessionId);
const url = `${getBaseUrl('ws')}/api/picoclaw/gateway/ws?session_id=${encodeURIComponent(sessionId)}`;
this.connectPromise = new Promise<string>((resolve, reject) => {
const ws = new WebSocket(url);
this.ws = ws;
ws.onopen = () => {
if (this.ws !== ws || this.connectionAttemptId !== attemptId) {
return;
}
this.reconnectAttempts = 0;
this.emit('connected', { sessionId });
this.setTransportState('connected');
this.setRunState('idle');
resolve(sessionId);
};
ws.onerror = () => {
if (this.ws !== ws || this.connectionAttemptId !== attemptId) {
return;
}
this.setTransportState('error');
};
ws.onclose = (event) => {
if (this.ws !== ws || this.connectionAttemptId !== attemptId) {
return;
}
this.ws = null;
if (event.code !== 1000) {
const code = CLOSE_ERRORS[event.code] || 'UNKNOWN';
if (code !== 'UNKNOWN') {
this.emit('error', {
code,
message: event.reason || code
});
}
}
this.emit('close', {
code: event.code,
reason: event.reason
});
this.setTransportState(event.code === 1000 ? 'disconnected' : 'error');
this.setRunState('idle');
if (!this.explicitClose) {
reject(new Error(event.reason || 'gateway closed'));
if (this.autoReconnect) {
this.scheduleReconnect();
}
}
};
ws.onmessage = (event) => {
if (this.ws !== ws || this.connectionAttemptId !== attemptId) {
return;
}
this.handleMessage(event.data);
};
}).finally(() => {
if (this.connectionAttemptId === attemptId) {
this.connectPromise = null;
}
});
return this.connectPromise;
}
public on<T extends EventName>(eventName: T, listener: Listener<T>) {
const current = this.listeners.get(eventName) || new Set();
current.add(listener as Listener<EventName>);
this.listeners.set(eventName, current);
return () => {
current.delete(listener as Listener<EventName>);
if (current.size === 0) {
this.listeners.delete(eventName);
}
};
}
public sendChatMessage(content: string, options: SendMessageOptions = {}) {
const trimmed = content.trim();
if (!trimmed || !this.ws || this.ws.readyState !== WebSocket.OPEN) {
return null;
}
const id = options.id || generateUUIDv4();
if (options.trackState !== false) {
this.setRunState('busy');
}
this.ws.send(
JSON.stringify({
id,
session_id: this.sessionId,
type: 'message.send',
payload: {
content: trimmed,
max_steps: options.maxSteps,
max_runtime_ms: options.maxRuntimeMs
}
})
);
return { id, sessionId: this.sessionId };
}
public sendStopMessage() {
if (!this.ws || this.ws.readyState !== WebSocket.OPEN) {
return false;
}
this.ws.send(
JSON.stringify({
type: 'message.cancel',
session_id: this.sessionId,
payload: {}
})
);
return true;
}
public close(code = 1000, reason = 'NORMAL_CLOSURE') {
this.explicitClose = true;
this.clearReconnectTimer();
if (this.ws && this.ws.readyState === WebSocket.OPEN) {
this.ws.close(code, reason);
} else if (this.ws && this.ws.readyState === WebSocket.CONNECTING) {
this.ws.close();
}
this.ws = null;
this.setTransportState('disconnected');
this.setRunState('idle');
}
public rotateSession() {
this.close();
clearPicoclawSessionId();
this.sessionId = generateUUIDv4();
setPicoclawSessionId(this.sessionId);
return this.sessionId;
}
public getSessionId() {
return this.sessionId;
}
private scheduleReconnect() {
this.clearReconnectTimer();
this.reconnectAttempts += 1;
const delay = Math.min(1000 * this.reconnectAttempts, 5000);
this.reconnectTimer = setTimeout(() => {
this.connect({ sessionId: this.sessionId, autoReconnect: true }).catch(() => undefined);
}, delay);
}
private clearReconnectTimer() {
if (this.reconnectTimer) {
clearTimeout(this.reconnectTimer);
this.reconnectTimer = null;
}
}
private setTransportState(state: GatewayTransportState) {
this.emit('transport_state', state);
}
private setRunState(state: GatewayRunState) {
this.emit('run_state', state);
}
private handleMessage(rawData: string | ArrayBuffer | Blob) {
if (typeof rawData !== 'string') {
return;
}
let message: Record<string, unknown>;
try {
message = JSON.parse(rawData) as Record<string, unknown>;
} catch {
this.emit('error', {
code: 'INVALID_MESSAGE',
message: 'Failed to parse gateway message',
raw: rawData
});
return;
}
const type = String(message.type || '');
if (type === 'typing.start') {
this.setRunState('busy');
return;
}
if (type === 'typing.stop') {
return;
}
if (type === 'pong') {
return;
}
if (type === 'error') {
this.setTransportState('error');
this.setRunState('idle');
this.emit('error', {
code: String(message.code || 'ERROR'),
message: String(message.message || 'Gateway error'),
raw: message
});
return;
}
if (type === 'message.create' || type === 'message.update') {
this.setRunState('idle');
this.emit('assistant_message', {
id: String(message.id || generateUUIDv4()),
text: extractText(message),
raw: message
});
return;
}
const imageBase64 = extractImageBase64(message);
if (imageBase64) {
this.emit('observation', {
id: String(message.id || generateUUIDv4()),
text: extractText(message),
imageBase64,
raw: message
});
return;
}
const action = extractAction(message);
if (action) {
this.emit('tool_action', {
id: String(message.id || generateUUIDv4()),
action: action.action,
x: action.x,
y: action.y,
raw: message
});
return;
}
}
private emit<T extends EventName>(eventName: T, payload: GatewayEventMap[T]) {
const current = this.listeners.get(eventName);
if (!current) {
return;
}
current.forEach((listener) => {
(listener as Listener<T>)(payload);
});
}
}
function extractText(message: Record<string, unknown>) {
const payload = (message.payload || {}) as Record<string, unknown>;
const content = payload.content ?? message.content ?? payload.text ?? message.text;
if (typeof content === 'string') {
return normalizeLiteralEmptyText(content);
}
if (Array.isArray(content)) {
return content
.map((item) => {
if (typeof item === 'string') {
return normalizeLiteralEmptyText(item);
}
if (item && typeof item === 'object' && 'text' in item) {
return normalizeLiteralEmptyText(String((item as Record<string, unknown>).text || ''));
}
return '';
})
.filter(Boolean)
.join('\n')
.trim();
}
return '';
}
function extractImageBase64(message: Record<string, unknown>) {
const payload = (message.payload || {}) as Record<string, unknown>;
const data = (payload.data || message.data || {}) as Record<string, unknown>;
const value = data.image_base64 || payload.image_base64 || message.image_base64;
return typeof value === 'string' ? value : undefined;
}
function extractAction(message: Record<string, unknown>) {
const payload = (message.payload || {}) as Record<string, unknown>;
const actionValue = payload.action || message.action || payload.tool_name || message.tool_name;
if (typeof actionValue !== 'string') {
return null;
}
const action = normalizeLiteralEmptyText(actionValue);
if (!action) {
return null;
}
const x = typeof payload.x === 'number' ? payload.x : undefined;
const y = typeof payload.y === 'number' ? payload.y : undefined;
return {
action,
x,
y
};
}
function normalizeLiteralEmptyText(value: string) {
const trimmed = value.trim();
if (!trimmed) {
return '';
}
const normalized = trimmed.toLowerCase();
if (normalized === 'null' || normalized === 'undefined') {
return '';
}
return trimmed;
}
export const picoclawGateway = new PicoClawGateway();

123
web/src/lib/picoclaw-storage.ts Normal file
View File

@@ -0,0 +1,123 @@
const PICOCLAW_SESSION_ID_KEY = 'nano-kvm-picoclaw-session-id';
const LEGACY_AI_SESSION_ID_KEY = 'nano-kvm-ai-session-id';
const PICOCLAW_MAX_STEPS_KEY = 'nano-kvm-picoclaw-max-steps';
const LEGACY_AI_MAX_STEPS_KEY = 'nano-kvm-ai-max-steps';
const PICOCLAW_MAX_RUNTIME_KEY = 'nano-kvm-picoclaw-max-runtime-ms';
const LEGACY_AI_MAX_RUNTIME_KEY = 'nano-kvm-ai-max-runtime-ms';
const PICOCLAW_RUNTIME_INSTALL_SNAPSHOT_KEY = 'nano-kvm-picoclaw-runtime-install-snapshot';
const LEGACY_AI_RUNTIME_INSTALL_SNAPSHOT_KEY = 'nano-kvm-ai-runtime-install-snapshot';
export type PicoclawRuntimeInstallSnapshot = {
installing: boolean;
installProgress?: number;
installStage?: string;
status?: string;
};
function getSessionStore() {
try {
return window.sessionStorage;
} catch {
return null;
}
}
function getLocalValue(primaryKey: string, legacyKey: string) {
return localStorage.getItem(primaryKey) || localStorage.getItem(legacyKey);
}
function setLocalValue(primaryKey: string, legacyKey: string, value: string) {
localStorage.setItem(primaryKey, value);
localStorage.removeItem(legacyKey);
}
function removeLocalValue(primaryKey: string, legacyKey: string) {
localStorage.removeItem(primaryKey);
localStorage.removeItem(legacyKey);
}
export function getPicoclawSessionId() {
const sessionStore = getSessionStore();
return (
sessionStore?.getItem(PICOCLAW_SESSION_ID_KEY) ||
sessionStore?.getItem(LEGACY_AI_SESSION_ID_KEY) ||
getLocalValue(PICOCLAW_SESSION_ID_KEY, LEGACY_AI_SESSION_ID_KEY)
);
}
export function setPicoclawSessionId(sessionId: string) {
const sessionStore = getSessionStore();
sessionStore?.setItem(PICOCLAW_SESSION_ID_KEY, sessionId);
sessionStore?.removeItem(LEGACY_AI_SESSION_ID_KEY);
setLocalValue(PICOCLAW_SESSION_ID_KEY, LEGACY_AI_SESSION_ID_KEY, sessionId);
}
export function clearPicoclawSessionId() {
const sessionStore = getSessionStore();
sessionStore?.removeItem(PICOCLAW_SESSION_ID_KEY);
sessionStore?.removeItem(LEGACY_AI_SESSION_ID_KEY);
removeLocalValue(PICOCLAW_SESSION_ID_KEY, LEGACY_AI_SESSION_ID_KEY);
}
export function getPicoclawMaxSteps() {
const value = getLocalValue(PICOCLAW_MAX_STEPS_KEY, LEGACY_AI_MAX_STEPS_KEY);
const parsed = value ? Number(value) : 20;
return Math.max(1, Math.min(50, Number.isFinite(parsed) ? parsed : 20));
}
export function setPicoclawMaxSteps(maxSteps: number) {
const value = Math.max(1, Math.min(50, Math.round(maxSteps)));
setLocalValue(PICOCLAW_MAX_STEPS_KEY, LEGACY_AI_MAX_STEPS_KEY, String(value));
}
export function getPicoclawMaxRuntimeMs() {
const value = getLocalValue(PICOCLAW_MAX_RUNTIME_KEY, LEGACY_AI_MAX_RUNTIME_KEY);
const parsed = value ? Number(value) : 120000;
if (!Number.isFinite(parsed) || parsed <= 0) {
return 120000;
}
return parsed;
}
export function setPicoclawMaxRuntimeMs(maxRuntimeMs: number) {
const value = Math.max(1000, Math.round(maxRuntimeMs));
setLocalValue(PICOCLAW_MAX_RUNTIME_KEY, LEGACY_AI_MAX_RUNTIME_KEY, String(value));
}
export function getPicoclawRuntimeInstallSnapshot(): PicoclawRuntimeInstallSnapshot | null {
const value = getLocalValue(
PICOCLAW_RUNTIME_INSTALL_SNAPSHOT_KEY,
LEGACY_AI_RUNTIME_INSTALL_SNAPSHOT_KEY
);
if (!value) {
return null;
}
try {
const parsed = JSON.parse(value) as PicoclawRuntimeInstallSnapshot;
if (!parsed || typeof parsed !== 'object') {
return null;
}
return {
installing: parsed.installing === true,
installProgress:
typeof parsed.installProgress === 'number' ? parsed.installProgress : undefined,
installStage: typeof parsed.installStage === 'string' ? parsed.installStage : undefined,
status: typeof parsed.status === 'string' ? parsed.status : undefined
};
} catch {
return null;
}
}
export function setPicoclawRuntimeInstallSnapshot(snapshot: PicoclawRuntimeInstallSnapshot) {
setLocalValue(
PICOCLAW_RUNTIME_INSTALL_SNAPSHOT_KEY,
LEGACY_AI_RUNTIME_INSTALL_SNAPSHOT_KEY,
JSON.stringify(snapshot)
);
}
export function clearPicoclawRuntimeInstallSnapshot() {
removeLocalValue(PICOCLAW_RUNTIME_INSTALL_SNAPSHOT_KEY, LEGACY_AI_RUNTIME_INSTALL_SNAPSHOT_KEY);
}

54
web/src/pages/desktop/index.tsx
View File

@@ -1,10 +1,12 @@
import { useEffect } from 'react';
import { useAtom } from 'jotai';
import { useEffect, useState } from 'react';
import { Splitter } from 'antd';
import { useAtom, useAtomValue } from 'jotai';
import { useTranslation } from 'react-i18next';
import { useMediaQuery } from 'react-responsive';
import * as storage from '@/lib/localstorage.ts';
import { client } from '@/lib/websocket.ts';
import { picoclawChatOpenAtom } from '@/jotai/picoclaw.ts';
import { resolutionAtom, videoModeAtom } from '@/jotai/screen.ts';
import { Head } from '@/components/head.tsx';
@@ -12,15 +14,19 @@ import { Keyboard } from './keyboard';
import { Menu } from './menu';
import { Mouse } from './mouse';
import { Notification } from './notification.tsx';
import { Sidebar as PicoclawSidebar } from './picoclaw';
import { ActionOverlay } from './picoclaw/action-overlay.tsx';
import { Screen } from './screen';
import { VirtualKeyboard } from './virtual-keyboard';
export const Desktop = () => {
const { t } = useTranslation();
const isBigScreen = useMediaQuery({ minWidth: 850 });
const [picoclawSidebarWidth, setPicoclawSidebarWidth] = useState(420);
const [videoMode, setVideoMode] = useAtom(videoModeAtom);
const [resolution, setResolution] = useAtom(resolutionAtom);
const isPicoclawChatOpen = useAtomValue(picoclawChatOpenAtom);
useEffect(() => {
client.connect();
@@ -50,22 +56,56 @@ export const Desktop = () => {
return defaultVideoMode;
}
function handleSplitterResize(sizes: number[]) {
const nextSidebarWidth = sizes[1];
if (typeof nextSidebarWidth === 'number' && nextSidebarWidth > 0) {
setPicoclawSidebarWidth(nextSidebarWidth);
}
}
return (
<>
<div className="h-screen w-screen overflow-hidden bg-neutral-950">
<Head title={t('head.desktop')} />
{isBigScreen && <Notification />}
{videoMode && resolution && (
<>
<div className="relative flex h-full min-h-0 w-full min-w-0">
<Menu />
<Screen />
<div className="h-full min-h-0 w-full min-w-0">
<Splitter
className="h-full w-full"
style={{ height: '100%', width: '100%' }}
onResize={handleSplitterResize}
>
<Splitter.Panel min="45%">
<div className="relative h-full min-h-0 w-full min-w-0 overflow-hidden bg-black">
<Screen />
</div>
</Splitter.Panel>
<Splitter.Panel
size={isBigScreen && isPicoclawChatOpen ? picoclawSidebarWidth : 0}
min={isBigScreen && isPicoclawChatOpen ? 340 : 0}
max="45%"
resizable={isBigScreen && isPicoclawChatOpen}
>
{isBigScreen && isPicoclawChatOpen ? <PicoclawSidebar /> : null}
</Splitter.Panel>
</Splitter>
</div>
<ActionOverlay />
<Mouse />
<Keyboard />
</>
</div>
)}
{!isBigScreen && isPicoclawChatOpen ? (
<div className="fixed inset-x-0 bottom-0 top-14 z-[980] overflow-hidden bg-[#0d0d0f] shadow-2xl">
<PicoclawSidebar />
</div>
) : null}
<VirtualKeyboard />
</>
</div>
);
};

13
web/src/pages/desktop/keyboard/index.tsx
View File

@@ -6,6 +6,7 @@ import { KeyboardReport } from '@/lib/keyboard.ts';
import { isModifier } from '@/lib/keymap';
import { client, MessageEvent } from '@/lib/websocket.ts';
import { isKeyboardEnableAtom } from '@/jotai/keyboard.ts';
import { picoclawTakeoverStateAtom } from '@/jotai/picoclaw.ts';
import { Recorder } from './recorder.tsx';
import { useAltGr } from './useAltGr.ts';
@@ -16,6 +17,7 @@ export const Keyboard = () => {
const os = getOperatingSystem();
const isKeyboardEnabled = useAtomValue(isKeyboardEnableAtom);
const picoclawTakeoverState = useAtomValue(picoclawTakeoverStateAtom);
const keyboardRef = useRef(new KeyboardReport());
const pressedKeys = useRef(new Set<string>());
@@ -38,6 +40,17 @@ export const Keyboard = () => {
const altGrRef = useRef(altGr);
altGrRef.current = altGr;
useEffect(() => {
if (!picoclawTakeoverState.active) {
return;
}
pressedKeys.current.clear();
leaderKeyRef.current.reset();
altGrRef.current.reset();
sendReport(keyboardRef.current.reset());
}, [picoclawTakeoverState.active]);
useEffect(() => {
document.addEventListener('keydown', handleKeyDown);
document.addEventListener('keyup', handleKeyUp);

8
web/src/pages/desktop/menu/index.tsx
View File

@@ -15,6 +15,7 @@ import { Image } from './image';
import { Keyboard } from './keyboard';
import { Mouse } from './mouse';
import { Collapse, Expand } from './operations';
import { Picoclaw } from './picoclaw';
import { Power } from './power';
import { Screen } from './screen';
import { Script } from './script';
@@ -101,6 +102,13 @@ export const Menu = () => {
<Divider type="vertical" />
)}
{isEnabled('picoclaw') && (
<>
<Picoclaw />
<Divider type="vertical" />
</>
)}
{isEnabled('power') && (
<>
<Power />

22
web/src/pages/desktop/menu/picoclaw/index.tsx Normal file
View File

@@ -0,0 +1,22 @@
import { Tooltip } from 'antd';
import { useSetAtom } from 'jotai';
import { useTranslation } from 'react-i18next';
import { picoclawChatOpenAtom } from '@/jotai/picoclaw.ts';
import { Robot } from '@/components/icons/robot.tsx';
export const Picoclaw = () => {
const { t } = useTranslation();
const setIsChatOpen = useSetAtom(picoclawChatOpenAtom);
return (
<Tooltip title={t('picoclaw.title')} mouseEnterDelay={0.6} placement="bottom">
<div
className="flex h-[30px] w-[30px] cursor-pointer items-center justify-center rounded text-neutral-300 hover:bg-neutral-700/80 hover:text-white"
onClick={() => setIsChatOpen((open) => !open)}
>
<Robot size={18} />
</div>
</Tooltip>
);
};

2
web/src/pages/desktop/menu/settings/appearance/menu-icons.tsx
View File

@@ -14,6 +14,7 @@ import { useTranslation } from 'react-i18next';
import * as ls from '@/lib/localstorage.ts';
import { menuDisabledItemsAtom } from '@/jotai/settings.ts';
import { Robot } from '@/components/icons/robot.tsx';
export const MenuIcons = () => {
const { t } = useTranslation();
@@ -26,6 +27,7 @@ export const MenuIcons = () => {
{ key: 'script', icon: <FileJsonIcon size={16} /> },
{ key: 'terminal', icon: <TerminalSquareIcon size={16} /> },
{ key: 'wol', icon: <NetworkIcon size={16} /> },
{ key: 'picoclaw', icon: <Robot size={16} /> },
{ key: 'power', icon: <PowerIcon size={16} /> },
{ key: 'fullscreen', icon: <MaximizeIcon size={16} />, label: 'fullscreen.toggle' },
{ key: 'collapse', icon: <XIcon size={16} />, label: 'menu.collapse' }

72
web/src/pages/desktop/mouse/absolute.tsx
View File

@@ -43,21 +43,22 @@ export const Absolute = () => {
const VELOCITY_THRESHOLD = 0.3;
useEffect(() => {
const screen = document.getElementById('screen') as HTMLVideoElement;
const screen = document.getElementById('screen') as HTMLVideoElement | null;
if (!screen) return;
const target = screen;
screen.addEventListener('mousedown', handleMouseDown);
screen.addEventListener('mouseup', handleMouseUp);
screen.addEventListener('mousemove', handleMouseMove);
screen.addEventListener('wheel', handleWheel);
screen.addEventListener('click', disableEvent);
screen.addEventListener('contextmenu', disableEvent);
target.addEventListener('mousedown', handleMouseDown);
target.addEventListener('mouseup', handleMouseUp);
target.addEventListener('mousemove', handleMouseMove);
target.addEventListener('wheel', handleWheel);
target.addEventListener('click', disableEvent);
target.addEventListener('contextmenu', disableEvent);
if (isBigScreen) {
screen.addEventListener('touchstart', handleTouchStart);
screen.addEventListener('touchmove', handleTouchMove);
screen.addEventListener('touchend', handleTouchEnd);
screen.addEventListener('touchcancel', handleTouchCancel);
target.addEventListener('touchstart', handleTouchStart);
target.addEventListener('touchmove', handleTouchMove);
target.addEventListener('touchend', handleTouchEnd);
target.addEventListener('touchcancel', handleTouchCancel);
}
// Mouse down event
@@ -257,15 +258,16 @@ export const Absolute = () => {
}
function getCorrectedCoords(clientX: number, clientY: number) {
const rect = screen.getBoundingClientRect();
const rect = target.getBoundingClientRect();
if (!screen.videoWidth || !screen.videoHeight) {
const mediaSize = getMediaSize(target);
if (!mediaSize) {
const x = (clientX - rect.left) / rect.width;
const y = (clientY - rect.top) / rect.height;
return { x, y };
}
const videoRatio = screen.videoWidth / screen.videoHeight;
const mediaRatio = mediaSize.width / mediaSize.height;
const elementRatio = rect.width / rect.height;
let renderedWidth = rect.width;
@@ -273,11 +275,11 @@ export const Absolute = () => {
let offsetX = 0;
let offsetY = 0;
if (videoRatio > elementRatio) {
renderedHeight = rect.width / videoRatio;
if (mediaRatio > elementRatio) {
renderedHeight = rect.width / mediaRatio;
offsetY = (rect.height - renderedHeight) / 2;
} else {
renderedWidth = rect.height * videoRatio;
renderedWidth = rect.height * mediaRatio;
offsetX = (rect.width - renderedWidth) / 2;
}
@@ -288,16 +290,16 @@ export const Absolute = () => {
}
return () => {
screen.removeEventListener('mousemove', handleMouseMove);
screen.removeEventListener('mousedown', handleMouseDown);
screen.removeEventListener('mouseup', handleMouseUp);
screen.removeEventListener('wheel', handleWheel);
screen.removeEventListener('click', disableEvent);
screen.removeEventListener('contextmenu', disableEvent);
screen.removeEventListener('touchstart', handleTouchStart);
screen.removeEventListener('touchmove', handleTouchMove);
screen.removeEventListener('touchend', handleTouchEnd);
screen.removeEventListener('touchcancel', handleTouchCancel);
target.removeEventListener('mousemove', handleMouseMove);
target.removeEventListener('mousedown', handleMouseDown);
target.removeEventListener('mouseup', handleMouseUp);
target.removeEventListener('wheel', handleWheel);
target.removeEventListener('click', disableEvent);
target.removeEventListener('contextmenu', disableEvent);
target.removeEventListener('touchstart', handleTouchStart);
target.removeEventListener('touchmove', handleTouchMove);
target.removeEventListener('touchend', handleTouchEnd);
target.removeEventListener('touchcancel', handleTouchCancel);
if (longPressTimerRef.current) {
clearTimeout(longPressTimerRef.current);
@@ -347,3 +349,19 @@ export const Absolute = () => {
return <></>;
};
function getMediaSize(screen: Element) {
if (screen instanceof HTMLVideoElement && screen.videoWidth > 0 && screen.videoHeight > 0) {
return { width: screen.videoWidth, height: screen.videoHeight };
}
if (screen instanceof HTMLImageElement && screen.naturalWidth > 0 && screen.naturalHeight > 0) {
return { width: screen.naturalWidth, height: screen.naturalHeight };
}
if (screen instanceof HTMLCanvasElement && screen.width > 0 && screen.height > 0) {
return { width: screen.width, height: screen.height };
}
return null;
}

68
web/src/pages/desktop/picoclaw/action-overlay.tsx Normal file
View File

@@ -0,0 +1,68 @@
import { useEffect, useState } from 'react';
import { useAtomValue } from 'jotai';
import { useTranslation } from 'react-i18next';
import { picoclawOverlayAtom, picoclawTakeoverStateAtom } from '@/jotai/picoclaw.ts';
type ScreenRect = {
left: number;
top: number;
width: number;
height: number;
};
export const ActionOverlay = () => {
const { t } = useTranslation();
const overlay = useAtomValue(picoclawOverlayAtom);
const takeover = useAtomValue(picoclawTakeoverStateAtom);
const [rect, setRect] = useState<ScreenRect | null>(null);
useEffect(() => {
const screen = document.getElementById('screen');
if (!screen) {
setRect(null);
return;
}
const updateRect = () => {
const bounds = screen.getBoundingClientRect();
setRect({
left: bounds.left,
top: bounds.top,
width: bounds.width,
height: bounds.height
});
};
updateRect();
const observer = new ResizeObserver(updateRect);
observer.observe(screen);
window.addEventListener('resize', updateRect);
window.addEventListener('scroll', updateRect, true);
return () => {
observer.disconnect();
window.removeEventListener('resize', updateRect);
window.removeEventListener('scroll', updateRect, true);
};
}, [takeover.active]);
if (!takeover.active || !rect || rect.width <= 0 || rect.height <= 0) {
return null;
}
return (
<div
className="fixed z-[950]"
style={{
left: rect.left + 12,
top: rect.top + 12
}}
>
<div className="rounded-full bg-neutral-950/85 px-3 py-1 text-xs text-sky-100 shadow-lg ring-1 ring-sky-400/30">
{overlay.message || t('picoclaw.overlay.locked')}
</div>
</div>
);
};

1
web/src/pages/desktop/picoclaw/index.ts Normal file
View File

@@ -0,0 +1 @@
export { Sidebar } from './sidebar.tsx';

84
web/src/pages/desktop/picoclaw/markdown-content.tsx Normal file
View File

@@ -0,0 +1,84 @@
import ReactMarkdown from 'react-markdown';
import remarkGfm from 'remark-gfm';
type MarkdownContentProps = {
content?: string;
className?: string;
};
export const MarkdownContent = ({ content, className }: MarkdownContentProps) => {
if (!content) {
return null;
}
return (
<div className={className}>
<ReactMarkdown
remarkPlugins={[remarkGfm]}
components={{
p: ({ children }) => (
<p className="my-2 whitespace-pre-wrap break-words leading-6 first:mt-0 last:mb-0">
{children}
</p>
),
a: ({ children, href }) => (
<a
className="text-sky-300 underline decoration-sky-400/40 underline-offset-2 hover:text-sky-200"
href={href}
rel="noreferrer"
target="_blank"
>
{children}
</a>
),
ul: ({ children }) => <ul className="my-2 list-disc pl-5">{children}</ul>,
ol: ({ children }) => <ol className="my-2 list-decimal pl-5">{children}</ol>,
li: ({ children }) => <li className="my-1 break-words leading-6">{children}</li>,
h1: ({ children }) => (
<h1 className="mt-3 text-base font-semibold first:mt-0">{children}</h1>
),
h2: ({ children }) => (
<h2 className="mt-3 text-sm font-semibold first:mt-0">{children}</h2>
),
h3: ({ children }) => <h3 className="mt-2 text-sm font-medium first:mt-0">{children}</h3>,
blockquote: ({ children }) => (
<blockquote className="my-2 border-l-2 border-white/15 pl-3 text-neutral-300">
{children}
</blockquote>
),
code: ({ children, className: codeClassName, ...props }: any) =>
!String(codeClassName || '').includes('language-') ? (
<code
className="rounded bg-black/25 px-1.5 py-0.5 font-mono text-[0.92em]"
{...props}
>
{children}
</code>
) : (
<code className={codeClassName} {...props}>
{children}
</code>
),
pre: ({ children }) => (
<pre className="my-2 overflow-x-auto rounded-xl bg-black/30 px-3 py-2 text-xs leading-6 text-neutral-100">
{children}
</pre>
),
table: ({ children }) => (
<div className="my-2 overflow-x-auto">
<table className="w-full border-collapse text-left text-sm">{children}</table>
</div>
),
th: ({ children }) => (
<th className="border-b border-white/10 px-2 py-1 font-semibold">{children}</th>
),
td: ({ children }) => (
<td className="border-b border-white/5 px-2 py-1 align-top">{children}</td>
)
}}
>
{content}
</ReactMarkdown>
</div>
);
};

92
web/src/pages/desktop/picoclaw/message-input.tsx Normal file
View File

@@ -0,0 +1,92 @@
import { useEffect, useRef, useState } from 'react';
import { Button } from 'antd';
import { useSetAtom } from 'jotai';
import { PlusIcon, SendIcon } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { isKeyboardEnableAtom } from '@/jotai/keyboard.ts';
import type { PicoclawTransportState } from '@/jotai/picoclaw.ts';
type MessageInputProps = {
transportState: PicoclawTransportState;
onSend: (content: string) => void | Promise<void>;
onNewConversation: () => void | Promise<void>;
disableNewConversation?: boolean;
};
export const MessageInput = ({
transportState,
onSend,
onNewConversation,
disableNewConversation
}: MessageInputProps) => {
const { t } = useTranslation();
const setIsKeyboardEnable = useSetAtom(isKeyboardEnableAtom);
const [value, setValue] = useState('');
const isComposingRef = useRef(false);
const isReady = transportState === 'connected';
useEffect(() => {
setIsKeyboardEnable(false);
return () => {
setIsKeyboardEnable(true);
};
}, [setIsKeyboardEnable]);
async function submit() {
const content = value.trim();
if (!content || !isReady) return;
await onSend(content);
setValue('');
}
return (
<div className="relative">
<textarea
rows={3}
value={value}
disabled={!isReady}
placeholder={isReady ? t('picoclaw.inputPlaceholder') : '...'}
onChange={(e) => setValue(e.target.value)}
onCompositionStart={() => {
isComposingRef.current = true;
}}
onCompositionEnd={() => {
isComposingRef.current = false;
}}
onKeyDown={(e) => {
const nativeEvent = e.nativeEvent as KeyboardEvent & {
isComposing?: boolean;
keyCode?: number;
};
if (isComposingRef.current || nativeEvent.isComposing || nativeEvent.keyCode === 229) {
return;
}
if (e.key === 'Enter' && !e.shiftKey) {
e.preventDefault();
void submit();
}
}}
className="w-full resize-none rounded-2xl border border-white/[0.08] bg-white/[0.04] px-4 py-3 pr-12 text-sm text-neutral-100 placeholder-neutral-600 outline-none transition-colors focus:border-white/[0.15] focus:bg-white/[0.06] disabled:opacity-40"
/>
<Button
type="text"
icon={<PlusIcon size={14} />}
onClick={() => void onNewConversation()}
disabled={disableNewConversation}
className="absolute bottom-2.5 right-11 !flex !h-7 !w-7 !items-center !justify-center !rounded-lg !border !border-white/[0.08]"
title={t('picoclaw.newConversation')}
/>
<Button
type="primary"
icon={<SendIcon size={14} />}
onClick={() => void submit()}
disabled={!isReady || !value.trim()}
className="absolute bottom-2.5 right-2.5 !flex !h-7 !w-7 !items-center !justify-center !rounded-lg"
title={t('picoclaw.send')}
/>
</div>
);
};

91
web/src/pages/desktop/picoclaw/message-list.tsx Normal file
View File

@@ -0,0 +1,91 @@
import { useEffect, useRef } from 'react';
import { BotIcon } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import type { PicoclawChatMessage, PicoclawRunState } from '@/jotai/picoclaw.ts';
import { MessageLoading } from './message-loading.tsx';
import { Message } from './message.tsx';
type MessageListProps = {
messages: PicoclawChatMessage[];
runState: PicoclawRunState;
};
export const MessageList = ({ messages, runState }: MessageListProps) => {
const { t } = useTranslation();
const isLoading = runState === 'busy';
const containerRef = useRef<HTMLDivElement | null>(null);
const contentRef = useRef<HTMLDivElement | null>(null);
const shouldAutoScrollRef = useRef(true);
function scrollToBottom(behavior: ScrollBehavior = 'smooth') {
const container = containerRef.current;
if (!container || !shouldAutoScrollRef.current) {
return;
}
container.scrollTo({
top: container.scrollHeight,
behavior
});
}
useEffect(() => {
scrollToBottom();
}, [messages, isLoading]);
useEffect(() => {
const content = contentRef.current;
if (!content || typeof ResizeObserver === 'undefined') {
return;
}
const observer = new ResizeObserver(() => {
scrollToBottom('auto');
});
observer.observe(content);
return () => {
observer.disconnect();
};
}, []);
function handleScroll() {
const container = containerRef.current;
if (!container) {
return;
}
const distanceFromBottom =
container.scrollHeight - container.scrollTop - container.clientHeight;
shouldAutoScrollRef.current = distanceFromBottom <= 24;
}
return (
<div
ref={containerRef}
className="min-h-0 flex-1 overflow-y-auto px-4 py-4"
onScroll={handleScroll}
>
<div ref={contentRef} className={messages.length === 0 ? 'min-h-full' : 'space-y-3'}>
{messages.length === 0 ? (
<div className="flex h-full min-h-[200px] flex-col items-center justify-center gap-3">
<div className="flex h-10 w-10 items-center justify-center rounded-2xl border border-white/[0.07] bg-white/[0.04]">
<BotIcon className="text-neutral-600" size={18} />
</div>
<span className="text-xs text-neutral-600">{t('picoclaw.empty')}</span>
{isLoading && <MessageLoading text={t('picoclaw.processing')} />}
</div>
) : (
<>
{messages.map((message) => (
<Message key={message.id} message={message} />
))}
{isLoading && <MessageLoading text={t('picoclaw.processing')} />}
</>
)}
</div>
</div>
);
};

25
web/src/pages/desktop/picoclaw/message-loading.tsx Normal file
View File

@@ -0,0 +1,25 @@
import { SparklesIcon } from 'lucide-react';
type MessageLoadingProps = {
text: string;
};
export const MessageLoading = ({ text }: MessageLoadingProps) => (
<div className="flex justify-start">
<div className="flex items-center gap-2.5 rounded-xl border border-white/[0.06] bg-white/[0.03] px-3 py-2 text-[13px] text-neutral-300 shadow-sm backdrop-blur-sm">
<SparklesIcon size={14} className="animate-pulse text-sky-400/80" />
<span className="animate-pulse font-medium tracking-wide">{text}</span>
<div className="ml-0.5 flex items-center gap-1">
{[0, 1, 2].map((i) => (
<span
key={i}
className="h-1 w-1 rounded-full bg-sky-400/60"
style={{
animation: `picoclaw-loading-bounce 1.4s ease-in-out ${i * 0.16}s infinite`
}}
/>
))}
</div>
</div>
</div>
);

97
web/src/pages/desktop/picoclaw/message-utils.ts Normal file
View File

@@ -0,0 +1,97 @@
import type {
GatewayAssistantMessage,
GatewayError,
GatewayObservation,
GatewayToolAction
} from '@/lib/picoclaw-gateway.ts';
import { generateUUIDv4 } from '@/lib/picoclaw-gateway.ts';
import type { PicoclawChatMessage } from '@/jotai/picoclaw.ts';
export const HIDDEN_OVERLAY = {
visible: false,
message: ''
} as const;
export function createStatusMessage(text: string): PicoclawChatMessage {
return {
id: generateUUIDv4(),
kind: 'status',
text,
createdAt: Date.now()
};
}
export function createErrorMessage(error: GatewayError): PicoclawChatMessage {
return {
id: generateUUIDv4(),
kind: 'error',
text: error.message,
createdAt: Date.now(),
raw: error.raw
};
}
export function createAssistantMessage(message: GatewayAssistantMessage): PicoclawChatMessage {
return {
id: message.id,
kind: 'assistant',
text: message.text,
createdAt: Date.now(),
raw: message.raw
};
}
export function createToolActionMessage(action: GatewayToolAction): PicoclawChatMessage {
return {
id: action.id,
kind: 'tool_action',
text: action.action,
action: action.action,
createdAt: Date.now(),
raw: action.raw
};
}
export function createObservationMessage(observation: GatewayObservation): PicoclawChatMessage {
return {
id: observation.id,
kind: 'observation',
text: observation.text,
imageBase64: observation.imageBase64,
createdAt: Date.now()
};
}
export function retainLatestObservationScreenshot(
messages: PicoclawChatMessage[]
): PicoclawChatMessage[] {
let latestObservationIndex = -1;
for (let index = messages.length - 1; index >= 0; index -= 1) {
const message = messages[index];
if (message.kind === 'observation' && message.imageBase64) {
latestObservationIndex = index;
break;
}
}
if (latestObservationIndex < 0) {
return messages;
}
return messages.map((message, index) => {
if (
message.kind !== 'observation' ||
!message.imageBase64 ||
index === latestObservationIndex
) {
return message;
}
return {
...message,
imageBase64: undefined,
raw: undefined
};
});
}

179
web/src/pages/desktop/picoclaw/message.tsx Normal file
View File

@@ -0,0 +1,179 @@
import clsx from 'clsx';
import { ImageIcon, SparklesIcon } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import type { PicoclawChatMessage } from '@/jotai/picoclaw.ts';
import { MarkdownContent } from './markdown-content.tsx';
type MessageProps = {
message: PicoclawChatMessage;
};
const hiddenAssistantMessagePrefix = '🔧 `message`';
const nullToolFeedbackPattern = /^🔧\s*`[^`]+`\s*```(?:[\w-]+)?\s*null\s*```$/s;
function extractDisplayText(value: unknown): string {
if (typeof value === 'string') {
const normalized = normalizeLiteralEmptyText(value);
if (!normalized) {
return '';
}
try {
const parsed = JSON.parse(normalized) as unknown;
if (parsed && typeof parsed === 'object') {
const record = parsed as Record<string, unknown>;
if (typeof record.content === 'string') {
const content = normalizeLiteralEmptyText(record.content);
if (content) {
return content;
}
}
if (typeof record.text === 'string') {
const text = normalizeLiteralEmptyText(record.text);
if (text) {
return text;
}
}
}
} catch {
return normalized;
}
return normalized;
}
if (value && typeof value === 'object') {
const record = value as Record<string, unknown>;
if (typeof record.content === 'string') {
const content = normalizeLiteralEmptyText(record.content);
if (content) {
return content;
}
}
if (typeof record.text === 'string') {
const text = normalizeLiteralEmptyText(record.text);
if (text) {
return text;
}
}
}
return '';
}
function normalizeLiteralEmptyText(value: string): string {
const trimmed = value.trim();
if (!trimmed) {
return '';
}
const normalized = trimmed.toLowerCase();
if (normalized === 'null' || normalized === 'undefined') {
return '';
}
return trimmed;
}
function shouldHideAssistantMessage(value?: string): boolean {
const normalized = normalizeLiteralEmptyText(value || '');
if (!normalized) {
return false;
}
return (
normalized.startsWith(hiddenAssistantMessagePrefix) || nullToolFeedbackPattern.test(normalized)
);
}
export const Message = ({ message }: MessageProps) => {
const { t } = useTranslation();
if (message.kind === 'status') {
return <div className="py-1 text-center text-xs text-neutral-500">{message.text}</div>;
}
if (message.kind === 'error') {
if (message.text === 'unexpected EOF') {
return null;
}
return (
<div className="rounded-lg border border-red-500/30 bg-red-500/10 px-3 py-2 text-sm text-red-200">
<MarkdownContent content={message.text} className="break-words text-red-200/90" />
</div>
);
}
if (message.kind === 'tool_action') {
const toolText =
extractDisplayText(message.text) ||
extractDisplayText(message.action) ||
extractDisplayText(message.raw);
if (!toolText) {
return null;
}
return (
<div className="border-white/8 rounded-xl border bg-white/[0.03] px-3 py-2 text-xs text-neutral-300">
<div className="mb-1 flex items-center gap-2 text-[11px] uppercase tracking-wide text-neutral-500">
<SparklesIcon size={14} />
<span>{t('picoclaw.message.toolAction')}</span>
</div>
<MarkdownContent content={toolText} className="break-words" />
</div>
);
}
if (message.kind === 'observation') {
const observationText = extractDisplayText(message.text);
if (!observationText && !message.imageBase64) {
return null;
}
return (
<div className="rounded-lg border border-sky-400/20 bg-sky-400/10 px-3 py-2 text-sm text-sky-50">
<div className="mb-2 flex items-center gap-2 text-xs uppercase tracking-wide text-sky-200">
<ImageIcon size={14} />
<span>{t('picoclaw.message.observation')}</span>
</div>
{observationText && (
<MarkdownContent content={observationText} className="mb-2 break-words" />
)}
{message.imageBase64 && (
<img
alt={t('picoclaw.message.screenshot')}
className="max-h-44 w-full rounded-md object-cover"
src={`data:image/jpeg;base64,${message.imageBase64}`}
/>
)}
</div>
);
}
if (message.kind === 'assistant' && shouldHideAssistantMessage(message.text)) {
return null;
}
const isUser = message.kind === 'user';
const messageText = isUser ? message.text : extractDisplayText(message.text);
if (!messageText) {
return null;
}
return (
<div className={clsx('flex', isUser ? 'justify-end' : 'justify-start')}>
<div
className={clsx(
'max-w-[85%] rounded-2xl px-3 py-2 text-sm shadow-sm',
isUser ? 'bg-sky-500 text-white' : 'bg-neutral-800 text-neutral-100'
)}
>
<MarkdownContent content={messageText} className="break-words" />
</div>
</div>
);
};

130
web/src/pages/desktop/picoclaw/runtime-view.ts Normal file
View File

@@ -0,0 +1,130 @@
import type { TFunction } from 'i18next';
import type {
PicoclawRunState,
PicoclawRuntimeStatus,
PicoclawTransportState
} from '@/jotai/picoclaw.ts';
export type PicoclawSidebarMode = 'loading' | 'install' | 'model' | 'chat';
export function isPicoclawRuntimeInstalling(
runtimeStatus: PicoclawRuntimeStatus | null | undefined
) {
return runtimeStatus?.installing === true || runtimeStatus?.status === 'installing';
}
export function getPicoclawSidebarMode(
runtimeStatus: PicoclawRuntimeStatus | null,
isInitializing: boolean
): PicoclawSidebarMode {
if (isInitializing || !runtimeStatus) {
return 'loading';
}
if (runtimeStatus?.installed === false) {
return 'install';
}
if (runtimeStatus?.model_configured === false) {
return 'model';
}
return 'chat';
}
export function canConnectGateway(runtimeStatus: PicoclawRuntimeStatus | null) {
if (!runtimeStatus) {
return false;
}
if (isPicoclawRuntimeInstalling(runtimeStatus)) {
return false;
}
if (runtimeStatus.installed === false || runtimeStatus.model_configured === false) {
return false;
}
return runtimeStatus.ready === true;
}
export function getPicoclawSidebarStatusColor(
runtimeStatus: PicoclawRuntimeStatus | null,
transportState: PicoclawTransportState,
runState: PicoclawRunState,
isInitializing: boolean
) {
if (isInitializing || !runtimeStatus) {
return '#38bdf8';
}
if (isPicoclawRuntimeInstalling(runtimeStatus) || transportState === 'connecting') {
return '#38bdf8';
}
if (runtimeStatus.installed === false || runtimeStatus.model_configured === false) {
return '#f59e0b';
}
if (runtimeStatus.ready !== true || transportState === 'error') {
return '#ef4444';
}
if (transportState === 'connected' && runState === 'busy') {
return '#38bdf8';
}
return '#22c55e';
}
export function getPicoclawSidebarConnectionLabel(
runtimeStatus: PicoclawRuntimeStatus | null,
transportState: PicoclawTransportState,
runState: PicoclawRunState,
isInitializing: boolean,
t: TFunction
) {
if (isInitializing || !runtimeStatus) {
return t('picoclaw.connection.runtime.checking');
}
if (isPicoclawRuntimeInstalling(runtimeStatus)) {
return t('picoclaw.install.installing');
}
if (runtimeStatus.installed === false) {
return t('picoclaw.install.requiredTitle');
}
if (runtimeStatus.model_configured === false) {
return t('picoclaw.model.requiredTitle');
}
if (runtimeStatus.ready !== true) {
switch (runtimeStatus.status) {
case 'checking':
return t('picoclaw.connection.runtime.checking');
case 'config_error':
return t('picoclaw.connection.runtime.configError');
case 'unavailable':
return t('picoclaw.connection.runtime.unavailable');
case 'stopped':
case 'installed':
case 'ready':
default:
return t('picoclaw.connection.runtime.stopped');
}
}
if (transportState === 'connected') {
return `${t('picoclaw.connection.transport.connected')} · ${t(`picoclaw.connection.run.${runState}`)}`;
}
if (transportState === 'connecting') {
return t('picoclaw.connection.transport.connecting');
}
if (transportState === 'error') {
return t('picoclaw.connection.runtime.unavailable');
}
return t('picoclaw.connection.runtime.ready');
}

456
web/src/pages/desktop/picoclaw/sidebar-actions.ts Normal file
View File

@@ -0,0 +1,456 @@
import type { Dispatch, SetStateAction } from 'react';
import type { TFunction } from 'i18next';
import {
closeGateway,
connectGateway,
getRuntimeStatus,
installRuntime,
picoclawGateway,
setPicoclawAgentProfile,
setPicoclawModelConfig,
startRuntime,
stopRuntime,
uninstallRuntime
} from '@/api/picoclaw.ts';
import {
clearPicoclawRuntimeInstallSnapshot,
setPicoclawRuntimeInstallSnapshot,
type PicoclawRuntimeInstallSnapshot
} from '@/lib/picoclaw-storage.ts';
import type {
PicoclawChatMessage,
PicoclawOverlayState,
PicoclawRunState,
PicoclawRuntimeStatus,
PicoclawTakeoverState,
PicoclawTransportState
} from '@/jotai/picoclaw.ts';
import { createErrorMessage, createStatusMessage, HIDDEN_OVERLAY } from './message-utils.ts';
type RuntimeStatusSetter = Dispatch<SetStateAction<PicoclawRuntimeStatus | null>>;
type MessageSetter = Dispatch<SetStateAction<PicoclawChatMessage[]>>;
type TakeoverSetter = Dispatch<SetStateAction<PicoclawTakeoverState>>;
type PicoclawSidebarActionOptions = {
t: TFunction;
runtimeStatus: PicoclawRuntimeStatus | null;
transportState: PicoclawTransportState;
installSnapshot: PicoclawRuntimeInstallSnapshot | null;
modelApiBase: string;
modelApiKey: string;
modelIdentifier: string;
setRuntimeStatus: RuntimeStatusSetter;
setMessages: MessageSetter;
setTakeover: TakeoverSetter;
setOverlay: Dispatch<SetStateAction<PicoclawOverlayState>>;
setTransportState: Dispatch<SetStateAction<PicoclawTransportState>>;
setRunState: Dispatch<SetStateAction<PicoclawRunState>>;
setIsModelConfigOpen: Dispatch<SetStateAction<boolean>>;
setIsTogglingRuntime: Dispatch<SetStateAction<boolean>>;
setIsInstallRequestPending: Dispatch<SetStateAction<boolean>>;
setInstallSnapshot: Dispatch<SetStateAction<PicoclawRuntimeInstallSnapshot | null>>;
setIsSavingModelConfig: Dispatch<SetStateAction<boolean>>;
setIsSwitchingAgent: Dispatch<SetStateAction<boolean>>;
setIsUninstallRequestPending: Dispatch<SetStateAction<boolean>>;
};
export function createPicoclawSidebarActions(options: PicoclawSidebarActionOptions) {
const {
t,
runtimeStatus,
transportState,
installSnapshot,
modelApiBase,
modelApiKey,
modelIdentifier,
setRuntimeStatus,
setMessages,
setTakeover,
setOverlay,
setTransportState,
setRunState,
setIsModelConfigOpen,
setIsTogglingRuntime,
setIsInstallRequestPending,
setInstallSnapshot,
setIsSavingModelConfig,
setIsSwitchingAgent,
setIsUninstallRequestPending
} = options;
async function refreshState() {
let nextRuntimeStatus = runtimeStatus;
let runtimeRsp;
try {
runtimeRsp = await getRuntimeStatus();
} catch {
return nextRuntimeStatus;
}
if (runtimeRsp.code === 0) {
nextRuntimeStatus = runtimeRsp.data;
setRuntimeStatus(runtimeRsp.data);
}
return nextRuntimeStatus;
}
function isUnexpectedEOFError(error: unknown) {
if (!(error instanceof Error)) {
return false;
}
const message = error.message.toLowerCase();
return message.includes('unexpected eof') || message === 'eof';
}
function isExpectedRuntimeState(status: PicoclawRuntimeStatus | null, shouldStop: boolean) {
if (!status) {
return false;
}
if (shouldStop) {
return status.ready !== true;
}
return status.ready === true;
}
async function handleStartRuntime() {
setIsTogglingRuntime(true);
try {
const isRuntimeReady = runtimeStatus?.ready === true;
let response;
try {
response = isRuntimeReady ? await stopRuntime() : await startRuntime();
} catch (error) {
const latestRuntimeStatus = await refreshState();
if (
isUnexpectedEOFError(error) &&
isExpectedRuntimeState(latestRuntimeStatus ?? null, isRuntimeReady)
) {
setRuntimeStatus(latestRuntimeStatus ?? null);
setMessages((current) => [
...current,
createStatusMessage(
t(
isRuntimeReady ? 'picoclaw.status.runtimeStopped' : 'picoclaw.status.runtimeStarted'
)
)
]);
if (isRuntimeReady) {
closeGateway();
setTakeover({
active: false,
sessionId: picoclawGateway.getSessionId(),
reason: 'runtime_stopped'
});
setOverlay(HIDDEN_OVERLAY);
setTransportState('disconnected');
setRunState('idle');
} else if (transportState !== 'connected') {
try {
await connectGateway();
} catch {
// handled by gateway events
}
}
return;
}
const errorMessage =
error instanceof Error
? error.message
: t(
isRuntimeReady
? 'picoclaw.status.runtimeStopFailed'
: 'picoclaw.status.runtimeStartFailed'
);
setMessages((current) => [
...current,
createErrorMessage({
code: isRuntimeReady ? 'RUNTIME_STOP_FAILED' : 'RUNTIME_START_FAILED',
message: errorMessage,
raw: error
})
]);
return;
}
if (response.code === 0) {
setRuntimeStatus(response.data.status);
setMessages((current) => [
...current,
createStatusMessage(
t(isRuntimeReady ? 'picoclaw.status.runtimeStopped' : 'picoclaw.status.runtimeStarted')
)
]);
if (isRuntimeReady) {
closeGateway();
setTakeover({
active: false,
sessionId: picoclawGateway.getSessionId(),
reason: 'runtime_stopped'
});
setOverlay(HIDDEN_OVERLAY);
setTransportState('disconnected');
setRunState('idle');
} else if (transportState !== 'connected') {
try {
await connectGateway();
} catch {
// handled by gateway events
}
}
} else {
const errorMessage =
(response as { message?: string; msg?: string }).message ||
(response as { message?: string; msg?: string }).msg ||
t(
isRuntimeReady
? 'picoclaw.status.runtimeStopFailed'
: 'picoclaw.status.runtimeStartFailed'
);
setMessages((current) => [
...current,
createErrorMessage({
code: isRuntimeReady ? 'RUNTIME_STOP_FAILED' : 'RUNTIME_START_FAILED',
message: errorMessage,
raw: response
})
]);
}
await refreshState();
} finally {
setIsTogglingRuntime(false);
}
}
async function handleInstallRuntime() {
setIsInstallRequestPending(true);
const pendingSnapshot = {
installing: true,
installProgress: runtimeStatus?.install_progress ?? installSnapshot?.installProgress ?? 0,
installStage: runtimeStatus?.install_stage ?? installSnapshot?.installStage ?? 'preparing',
status: 'installing'
};
setInstallSnapshot(pendingSnapshot);
setPicoclawRuntimeInstallSnapshot(pendingSnapshot);
try {
const response = await installRuntime();
if (response.code === 0) {
setRuntimeStatus(response.data.status);
const installFinished =
response.data.status.status === 'installed' &&
response.data.status.installing !== true &&
response.data.installed === true;
setMessages((current) => [
...current,
createStatusMessage(
installFinished ? t('picoclaw.install.success') : t('picoclaw.install.installing')
)
]);
await refreshState();
return;
}
const errorMessage =
(response as { message?: string; msg?: string }).message ||
(response as { message?: string; msg?: string }).msg ||
t('picoclaw.install.failed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'RUNTIME_INSTALL_FAILED',
message: errorMessage,
raw: response
})
]);
} catch (error) {
setInstallSnapshot(null);
clearPicoclawRuntimeInstallSnapshot();
const errorMessage = error instanceof Error ? error.message : t('picoclaw.install.failed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'RUNTIME_INSTALL_FAILED',
message: errorMessage,
raw: error
})
]);
} finally {
setIsInstallRequestPending(false);
}
}
async function handleSaveModelConfig() {
const apiBase = modelApiBase.trim();
const apiKey = modelApiKey.trim();
const model = modelIdentifier.trim();
if (!apiBase || !apiKey || !model) {
setMessages((current) => [
...current,
createErrorMessage({
code: 'MODEL_CONFIG_INVALID',
message: t('picoclaw.model.invalid')
})
]);
return;
}
setIsSavingModelConfig(true);
try {
const response = await setPicoclawModelConfig({
model,
api_base: apiBase,
api_key: apiKey
});
if (response.code === 0) {
setRuntimeStatus(response.data.status);
setIsModelConfigOpen(false);
setMessages((current) => [...current, createStatusMessage(t('picoclaw.model.saved'))]);
await refreshState();
return;
}
const errorMessage =
(response as { message?: string; msg?: string }).message ||
(response as { message?: string; msg?: string }).msg ||
t('picoclaw.model.saveFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'MODEL_CONFIG_SAVE_FAILED',
message: errorMessage,
raw: response
})
]);
} catch (error) {
const errorMessage = error instanceof Error ? error.message : t('picoclaw.model.saveFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'MODEL_CONFIG_SAVE_FAILED',
message: errorMessage,
raw: error
})
]);
} finally {
setIsSavingModelConfig(false);
}
}
async function handleAgentProfileChange(profile: string) {
if (!profile || profile === runtimeStatus?.agent_profile) {
return;
}
setIsSwitchingAgent(true);
try {
const response = await setPicoclawAgentProfile({ profile });
if (response.code === 0) {
setRuntimeStatus(response.data.status);
setMessages((current) => [...current, createStatusMessage(t('picoclaw.agent.switched'))]);
await refreshState();
if (response.data.status?.ready === true && transportState !== 'connected') {
try {
await connectGateway();
} catch {
// handled by gateway events
}
}
return;
}
const errorMessage =
(response as { message?: string; msg?: string }).message ||
(response as { message?: string; msg?: string }).msg ||
t('picoclaw.agent.switchFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'AGENT_PROFILE_SWITCH_FAILED',
message: errorMessage,
raw: response
})
]);
} catch (error) {
const errorMessage =
error instanceof Error ? error.message : t('picoclaw.agent.switchFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'AGENT_PROFILE_SWITCH_FAILED',
message: errorMessage,
raw: error
})
]);
} finally {
setIsSwitchingAgent(false);
}
}
async function handleUninstallRuntime() {
setIsUninstallRequestPending(true);
setMessages((current) => [...current, createStatusMessage(t('picoclaw.install.uninstalling'))]);
try {
const response = await uninstallRuntime();
if (response.code === 0) {
setRuntimeStatus(response.data.status);
setMessages((current) => [
...current,
createStatusMessage(t('picoclaw.install.uninstalled'))
]);
await refreshState();
return;
}
const errorMessage =
(response as { message?: string; msg?: string }).message ||
(response as { message?: string; msg?: string }).msg ||
t('picoclaw.install.uninstallFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'RUNTIME_UNINSTALL_FAILED',
message: errorMessage,
raw: response
})
]);
} catch (error) {
const errorMessage =
error instanceof Error ? error.message : t('picoclaw.install.uninstallFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'RUNTIME_UNINSTALL_FAILED',
message: errorMessage,
raw: error
})
]);
} finally {
setIsUninstallRequestPending(false);
}
}
return {
refreshState,
handleStartRuntime,
handleInstallRuntime,
handleSaveModelConfig,
handleAgentProfileChange,
handleUninstallRuntime
};
}

323
web/src/pages/desktop/picoclaw/sidebar-effects.ts Normal file
View File

@@ -0,0 +1,323 @@
import { useEffect } from 'react';
import type { Dispatch, MutableRefObject, SetStateAction } from 'react';
import type { TFunction } from 'i18next';
import { closeGateway, connectGateway, picoclawGateway } from '@/api/picoclaw.ts';
import {
clearPicoclawRuntimeInstallSnapshot,
setPicoclawRuntimeInstallSnapshot,
type PicoclawRuntimeInstallSnapshot
} from '@/lib/picoclaw-storage.ts';
import type {
PicoclawChatMessage,
PicoclawOverlayState,
PicoclawRunState,
PicoclawRuntimeStatus,
PicoclawTakeoverState,
PicoclawTransportState
} from '@/jotai/picoclaw.ts';
import {
createAssistantMessage,
createErrorMessage,
createObservationMessage,
createStatusMessage,
createToolActionMessage,
HIDDEN_OVERLAY,
retainLatestObservationScreenshot
} from './message-utils.ts';
import { canConnectGateway, isPicoclawRuntimeInstalling } from './runtime-view.ts';
type MessageSetter = Dispatch<SetStateAction<PicoclawChatMessage[]>>;
type TakeoverSetter = Dispatch<SetStateAction<PicoclawTakeoverState>>;
type GatewayEventOptions = {
t: TFunction;
setActiveSessionId: Dispatch<SetStateAction<string>>;
setTakeover: TakeoverSetter;
setMessages: MessageSetter;
setTransportState: Dispatch<SetStateAction<PicoclawTransportState>>;
setOverlay: Dispatch<SetStateAction<PicoclawOverlayState>>;
setRunState: Dispatch<SetStateAction<PicoclawRunState>>;
};
type LifecycleOptions = {
t: TFunction;
refreshStateRef: MutableRefObject<() => Promise<PicoclawRuntimeStatus | null>>;
setActiveSessionId: Dispatch<SetStateAction<string>>;
setIsFreshConversation: Dispatch<SetStateAction<boolean>>;
setMessages: MessageSetter;
setIsInitializing: Dispatch<SetStateAction<boolean>>;
setTakeover: TakeoverSetter;
setOverlay: Dispatch<SetStateAction<PicoclawOverlayState>>;
setTransportState: Dispatch<SetStateAction<PicoclawTransportState>>;
setRunState: Dispatch<SetStateAction<PicoclawRunState>>;
};
type InstallSnapshotOptions = {
t: TFunction;
runtimeStatus: PicoclawRuntimeStatus | null;
setInstallSnapshot: Dispatch<SetStateAction<PicoclawRuntimeInstallSnapshot | null>>;
previousInstallStateRef: MutableRefObject<{ installing: boolean; status: string }>;
setMessages: MessageSetter;
};
export function usePicoclawGatewayEvents({
t,
setActiveSessionId,
setTakeover,
setMessages,
setTransportState,
setOverlay,
setRunState
}: GatewayEventOptions) {
useEffect(() => {
const unsubs = [
picoclawGateway.on('connected', ({ sessionId }) => {
setActiveSessionId(sessionId);
setTakeover({
active: true,
sessionId,
reason: 'connected'
});
setMessages((current) => [...current, createStatusMessage(t('picoclaw.status.connected'))]);
}),
picoclawGateway.on('transport_state', (state) => {
setTransportState(state);
if (state === 'connected') {
setActiveSessionId(picoclawGateway.getSessionId());
setTakeover((current) => ({
active: true,
sessionId: current.sessionId || picoclawGateway.getSessionId(),
reason: state
}));
return;
}
setTakeover((current) => ({
...current,
active: false,
reason: state
}));
setOverlay(HIDDEN_OVERLAY);
setRunState('idle');
}),
picoclawGateway.on('run_state', (state) => {
setRunState(state);
}),
picoclawGateway.on('assistant_message', (message) => {
setMessages((current) => {
const existingIndex = current.findIndex((item) => item.id === message.id);
const nextMessage = createAssistantMessage(message);
if (existingIndex < 0) {
return [...current, nextMessage];
}
const updated = [...current];
updated[existingIndex] = nextMessage;
return updated;
});
}),
picoclawGateway.on('tool_action', (action) => {
setMessages((current) => [...current, createToolActionMessage(action)]);
setOverlay({
visible: true,
message: `${t('picoclaw.overlay.locked')} · ${action.action}`,
x: action.x,
y: action.y,
action: action.action
});
}),
picoclawGateway.on('observation', (observation) => {
setMessages((current) =>
retainLatestObservationScreenshot([...current, createObservationMessage(observation)])
);
}),
picoclawGateway.on('error', (error) => {
setMessages((current) => [...current, createErrorMessage(error)]);
}),
picoclawGateway.on('close', (closeEvent) => {
if (closeEvent.code === 1000) {
setMessages((current) => [
...current,
createStatusMessage(t('picoclaw.status.disconnected'))
]);
}
})
];
return () => {
unsubs.forEach((unsubscribe) => unsubscribe());
};
}, [setActiveSessionId, setMessages, setOverlay, setRunState, setTakeover, setTransportState, t]);
}
export function usePicoclawSidebarLifecycle({
t,
refreshStateRef,
setActiveSessionId,
setIsFreshConversation,
setMessages,
setIsInitializing,
setTakeover,
setOverlay,
setTransportState,
setRunState
}: LifecycleOptions) {
useEffect(() => {
let cancelled = false;
async function openSidebar() {
const nextSessionId = picoclawGateway.rotateSession();
setActiveSessionId(nextSessionId);
setIsFreshConversation(true);
setMessages([]);
try {
const nextRuntimeStatus = await refreshStateRef.current();
if (cancelled) {
return;
}
if (nextRuntimeStatus?.installing) {
setMessages((current) => [
...current,
createStatusMessage(t('picoclaw.install.installing'))
]);
return;
}
if (nextRuntimeStatus?.installed === false) {
setMessages((current) => [
...current,
createStatusMessage(t('picoclaw.install.requiredDescription'))
]);
return;
}
if (nextRuntimeStatus?.model_configured === false) {
setMessages((current) => [
...current,
createStatusMessage(t('picoclaw.model.requiredDescription'))
]);
return;
}
if (!canConnectGateway(nextRuntimeStatus ?? null)) {
return;
}
setMessages((current) => [
...current,
createStatusMessage(t('picoclaw.status.connecting'))
]);
try {
await connectGateway();
} catch {
// handled by gateway events
}
} finally {
if (!cancelled) {
setIsInitializing(false);
}
}
}
void openSidebar();
return () => {
cancelled = true;
closeGateway();
setTakeover((current) => ({
...current,
active: false,
reason: 'panel_closed'
}));
setOverlay(HIDDEN_OVERLAY);
setTransportState('disconnected');
setRunState('idle');
};
}, [
refreshStateRef,
setActiveSessionId,
setIsFreshConversation,
setIsInitializing,
setMessages,
setOverlay,
setRunState,
setTakeover,
setTransportState,
t
]);
}
export function usePicoclawInstallRefresh(
isRuntimeInstallActive: boolean,
refreshStateRef: MutableRefObject<() => Promise<PicoclawRuntimeStatus | null>>
) {
useEffect(() => {
if (!isRuntimeInstallActive) {
return;
}
const timer = window.setInterval(() => {
void refreshStateRef.current();
}, 1000);
return () => {
window.clearInterval(timer);
};
}, [isRuntimeInstallActive, refreshStateRef]);
}
export function usePicoclawInstallSnapshotSync({
t,
runtimeStatus,
setInstallSnapshot,
previousInstallStateRef,
setMessages
}: InstallSnapshotOptions) {
useEffect(() => {
if (!runtimeStatus) {
return;
}
const isInstalling = isPicoclawRuntimeInstalling(runtimeStatus);
if (isInstalling) {
const nextSnapshot = {
installing: true,
installProgress: runtimeStatus.install_progress,
installStage: runtimeStatus.install_stage,
status: runtimeStatus.status
};
setInstallSnapshot(nextSnapshot);
setPicoclawRuntimeInstallSnapshot(nextSnapshot);
} else {
setInstallSnapshot(null);
clearPicoclawRuntimeInstallSnapshot();
}
const previous = previousInstallStateRef.current;
if (previous.installing && !isInstalling) {
if (runtimeStatus.installed === true) {
setMessages((current) => [...current, createStatusMessage(t('picoclaw.install.success'))]);
} else {
setMessages((current) => [
...current,
createErrorMessage({
code: (runtimeStatus.status || 'runtime_install_failed').toUpperCase(),
message: runtimeStatus.last_error || t('picoclaw.install.failed'),
raw: runtimeStatus
})
]);
}
}
previousInstallStateRef.current = {
installing: isInstalling,
status: runtimeStatus.status
};
}, [runtimeStatus, setInstallSnapshot, setMessages, t, previousInstallStateRef]);
}

222
web/src/pages/desktop/picoclaw/sidebar-header.tsx Normal file
View File

@@ -0,0 +1,222 @@
import { useState } from 'react';
import { Button, Dropdown, MenuProps, Modal, Select } from 'antd';
import {
EllipsisIcon,
HistoryIcon,
PlayIcon,
PowerIcon,
SlidersHorizontalIcon,
TrashIcon
} from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { LayoutSidebarRightCollapse } from '@/components/icons/layout-sidebar-right-collapse.tsx';
type SidebarHeaderProps = {
isCheckingRuntime: boolean;
installed?: boolean;
modelConfigured?: boolean;
runtimeReady?: boolean;
isTogglingRuntime: boolean;
agentProfile?: string;
isSwitchingAgent?: boolean;
isHistoryOpen?: boolean;
onToggleRuntime: () => void | Promise<void>;
onClose?: () => void;
onOpenHistory?: () => void;
onAgentProfileChange?: (profile: string) => void | Promise<void>;
onOpenModelConfig?: () => void;
isUninstallingRuntime?: boolean;
onUninstallRuntime?: () => void | Promise<void>;
};
export const SidebarHeader = ({
isCheckingRuntime,
installed,
modelConfigured,
runtimeReady,
isTogglingRuntime,
agentProfile,
isSwitchingAgent,
isHistoryOpen,
onToggleRuntime,
onClose,
onOpenHistory,
onAgentProfileChange,
onOpenModelConfig,
isUninstallingRuntime,
onUninstallRuntime
}: SidebarHeaderProps) => {
const { t } = useTranslation();
const [isUninstallModalVisible, setIsUninstallModalVisible] = useState(false);
const isRuntimeReady = runtimeReady === true;
const isInstalled = installed !== false;
const isModelConfigured = modelConfigured !== false;
const agentOptions = [
{
value: 'kvm',
title: t('picoclaw.agent.kvmTitle'),
description: t('picoclaw.agent.kvmDescription')
},
{
value: 'default',
title: t('picoclaw.agent.defaultTitle'),
description: t('picoclaw.agent.defaultDescription')
}
];
const handleUninstallClick = () => setIsUninstallModalVisible(true);
const handleConfirmUninstall = () => {
setIsUninstallModalVisible(false);
onUninstallRuntime?.();
};
const moreMenuItems: MenuProps['items'] = [
{
key: 'model-config',
label: <span className="text-xs">{t('picoclaw.model.menuLabel')}</span>,
icon: <SlidersHorizontalIcon size={14} className="text-neutral-400" />,
onClick: () => onOpenModelConfig?.(),
disabled: isUninstallingRuntime || isTogglingRuntime || !isInstalled
},
{
type: 'divider'
},
{
key: 'uninstall',
danger: true,
label: <span className="text-xs">{t('picoclaw.uninstall.menuLabel')}</span>,
icon: <TrashIcon size={14} />,
onClick: handleUninstallClick,
disabled: isUninstallingRuntime || isTogglingRuntime
}
];
return (
<>
<div className="flex h-12 flex-shrink-0 items-center justify-between gap-3 pl-2 pr-4">
{/* Left: Title */}
<div className="flex min-w-0 items-center gap-1.5">
<Button
type="text"
size="small"
onClick={onClose}
title={t('menu.collapse')}
className="!flex !items-center !justify-center !text-neutral-400 hover:!bg-white/[0.08] hover:!text-neutral-200"
icon={<LayoutSidebarRightCollapse size={18} />}
/>
{/* <span className="truncate text-[13px] font-medium text-neutral-200">{t('picoclaw.title')}</span> */}
</div>
{/* Right: Actions */}
<div className="flex shrink-0 items-center gap-1.5">
{isCheckingRuntime ? null : isInstalled ? (
<>
{/* Agent Profile Selector */}
<Select
variant="borderless"
size="small"
value={agentProfile || 'kvm'}
onChange={(value) => void onAgentProfileChange?.(value)}
disabled={isUninstallingRuntime || isTogglingRuntime || isSwitchingAgent}
loading={isSwitchingAgent}
popupMatchSelectWidth={false}
className="min-w-[120px] text-xs [&_.ant-select-selection-item]:text-xs [&_.ant-select-selection-item]:text-neutral-300"
options={agentOptions.map((option) => ({
value: option.value,
label: option.title
}))}
optionRender={(option) => {
const current = agentOptions.find((item) => item.value === option.value);
if (!current) return option.label;
return (
<div className="py-1">
<div className="text-xs font-medium text-neutral-200">{current.title}</div>
<div className="mt-0.5 text-[10px] leading-snug text-neutral-500">
{current.description}
</div>
</div>
);
}}
/>
{/* Start/ Stop & History Button */}
{isRuntimeReady ? (
<>
<Button
type="text"
size="small"
onClick={onOpenHistory}
title={t('picoclaw.history.title')}
className={[
'!flex !items-center !justify-center',
isHistoryOpen
? '!bg-white/[0.08] !text-neutral-200'
: '!text-neutral-400 hover:!bg-white/[0.08] hover:!text-neutral-200'
].join(' ')}
icon={<HistoryIcon size={14} />}
/>
<Button
disabled={isTogglingRuntime}
loading={isTogglingRuntime}
onClick={() => void onToggleRuntime()}
title={t('picoclaw.config.stopRuntime')}
icon={!isTogglingRuntime ? <PowerIcon size={14} /> : undefined}
type="text"
size="small"
className="!flex !items-center !justify-center !text-red-400 hover:!bg-white/[0.08] hover:!text-red-500"
/>
</>
) : isModelConfigured ? (
<Button
disabled={isTogglingRuntime}
loading={isTogglingRuntime}
onClick={() => void onToggleRuntime()}
title={t('picoclaw.config.startRuntime')}
icon={!isTogglingRuntime ? <PlayIcon size={14} /> : undefined}
type="text"
size="small"
className="!flex !items-center !justify-center !text-neutral-400 hover:!bg-white/[0.08] hover:!text-sky-400"
/>
) : null}
{/* More Menu */}
<Dropdown
menu={{ items: moreMenuItems }}
trigger={['click']}
placement="bottomRight"
disabled={isUninstallingRuntime || isTogglingRuntime}
arrow={{ pointAtCenter: true }}
>
<Button
disabled={isUninstallingRuntime}
loading={isUninstallingRuntime}
icon={!isUninstallingRuntime ? <EllipsisIcon size={16} /> : undefined}
type="text"
size="small"
className="!flex !items-center !justify-center !text-neutral-400 hover:!bg-white/[0.08] hover:!text-neutral-200"
/>
</Dropdown>
</>
) : null}
</div>
</div>
<Modal
title={t('picoclaw.uninstall.confirmTitle')}
open={isUninstallModalVisible}
onOk={handleConfirmUninstall}
onCancel={() => setIsUninstallModalVisible(false)}
okText={t('picoclaw.uninstall.confirmOk')}
cancelText={t('picoclaw.uninstall.confirmCancel')}
okButtonProps={{ danger: true }}
width={400}
centered
>
<p className="text-sm text-neutral-400">{t('picoclaw.uninstall.confirmContent')}</p>
</Modal>
</>
);
};

165
web/src/pages/desktop/picoclaw/sidebar-history.tsx Normal file
View File

@@ -0,0 +1,165 @@
import { useMemo, useState } from 'react';
import { Button, Modal } from 'antd';
import { Clock3Icon, MessageSquareTextIcon, Trash2Icon } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import type { PicoclawSessionListItem } from '@/api/picoclaw.ts';
type SidebarHistoryProps = {
sessions: PicoclawSessionListItem[];
activeSessionId?: string;
isLoading: boolean;
isDeleting: boolean;
isSwitching?: boolean;
onSelect: (sessionId: string) => void | Promise<void>;
onDelete: (sessionId: string) => void | Promise<void>;
};
function formatSessionTime(value: string) {
const date = new Date(value);
if (Number.isNaN(date.getTime())) {
return '';
}
return new Intl.DateTimeFormat(undefined, {
month: 'short',
day: 'numeric',
hour: '2-digit',
minute: '2-digit'
}).format(date);
}
export const SidebarHistory = ({
sessions,
activeSessionId,
isLoading,
isDeleting,
isSwitching,
onSelect,
onDelete
}: SidebarHistoryProps) => {
const { t } = useTranslation();
const [pendingDelete, setPendingDelete] = useState<PicoclawSessionListItem | null>(null);
const content = useMemo(() => {
if (isLoading) {
return (
<div className="flex flex-1 items-center justify-center px-6 text-sm text-neutral-500">
{t('picoclaw.history.loading')}
</div>
);
}
if (sessions.length === 0) {
return (
<div className="flex flex-1 flex-col items-center justify-center gap-2 px-6 text-center">
<Clock3Icon size={18} className="text-neutral-600" />
<span className="text-sm text-neutral-400">{t('picoclaw.history.emptyTitle')}</span>
<span className="text-xs text-neutral-600">{t('picoclaw.history.emptyDescription')}</span>
</div>
);
}
return (
<div className="picoclaw-sidebar-scrollbar flex min-h-0 flex-1 flex-col overflow-y-auto px-3 py-3">
<div className="space-y-2">
{sessions.map((session) => {
const isActive = session.id === activeSessionId;
return (
<div
key={session.id}
role="button"
tabIndex={isSwitching ? -1 : 0}
onClick={() => {
if (isSwitching) {
return;
}
void onSelect(session.id);
}}
onKeyDown={(event) => {
if (isSwitching) {
return;
}
if (event.key === 'Enter' || event.key === ' ') {
event.preventDefault();
void onSelect(session.id);
}
}}
className={[
'group flex w-full items-start gap-3 rounded-2xl border px-3 py-3 text-left transition-colors',
isSwitching ? 'pointer-events-none opacity-60' : '',
isActive
? 'border-sky-400/30 bg-sky-400/10'
: 'border-white/[0.06] bg-white/[0.03] hover:border-white/[0.1] hover:bg-white/[0.05]'
].join(' ')}
>
<div className="mt-0.5 flex h-8 w-8 shrink-0 items-center justify-center rounded-xl bg-white/[0.05] text-neutral-400">
<MessageSquareTextIcon size={16} />
</div>
<div className="min-w-0 flex-1">
<div className="flex items-start justify-between gap-3">
<div className="min-w-0">
<div className="truncate text-sm font-medium text-neutral-200">
{session.title}
</div>
<div className="mt-1 line-clamp-2 text-xs leading-5 text-neutral-500">
{session.preview}
</div>
</div>
<Button
type="text"
size="small"
danger
disabled={isDeleting || isSwitching}
icon={<Trash2Icon size={14} />}
className={[
'!mt-0.5 !hidden !items-center !justify-center !text-neutral-500 hover:!bg-red-500/10 hover:!text-red-300',
isActive ? '' : 'group-hover:!inline-flex'
].join(' ')}
onClick={(event) => {
event.preventDefault();
event.stopPropagation();
setPendingDelete(session);
}}
/>
</div>
<div className="mt-2 flex items-center gap-3 text-[11px] text-neutral-600">
<span>{formatSessionTime(session.updated)}</span>
<span>
{t('picoclaw.history.messageCount', { count: session.message_count })}
</span>
</div>
</div>
</div>
);
})}
</div>
</div>
);
}, [activeSessionId, isDeleting, isLoading, isSwitching, onSelect, sessions, t]);
return (
<>
<div className="flex min-h-0 flex-1 flex-col">{content}</div>
<Modal
centered
open={pendingDelete !== null}
title={t('picoclaw.history.deleteConfirmTitle')}
okText={t('picoclaw.history.deleteConfirmOk')}
cancelText={t('picoclaw.history.deleteConfirmCancel')}
okButtonProps={{ danger: true, loading: isDeleting }}
onOk={() => {
if (!pendingDelete) {
return;
}
void Promise.resolve(onDelete(pendingDelete.id)).finally(() => setPendingDelete(null));
}}
onCancel={() => setPendingDelete(null)}
>
<p className="text-sm text-neutral-400">
{t('picoclaw.history.deleteConfirmContent', { title: pendingDelete?.title || '' })}
</p>
</Modal>
</>
);
};

67
web/src/pages/desktop/picoclaw/sidebar-install.tsx Normal file
View File

@@ -0,0 +1,67 @@
import { Button, Progress } from 'antd';
import { DownloadIcon } from 'lucide-react';
import { useTranslation } from 'react-i18next';
type SidebarInstallProps = {
installProgress?: number;
installStage?: string;
isInstalling: boolean;
onInstall: () => void | Promise<void>;
};
export const SidebarInstall = ({
installProgress,
installStage,
isInstalling,
onInstall
}: SidebarInstallProps) => {
const { t } = useTranslation();
return (
<div className="flex flex-1 flex-col items-center overflow-y-auto px-6 pb-10 pt-10">
{/* Icon */}
<div className="mb-5 flex h-12 w-12 items-center justify-center rounded-2xl border border-white/[0.08] bg-white/[0.04]">
<DownloadIcon className="text-sky-400" size={22} />
</div>
<div className="w-full max-w-xs text-center">
<div className="text-sm font-semibold text-neutral-100">
{t('picoclaw.install.requiredTitle')}
</div>
<div className="mt-2 text-xs leading-5 text-neutral-500">
{isInstalling
? t('picoclaw.install.progressDescription')
: t('picoclaw.install.requiredDescription')}
</div>
{isInstalling && (
<div className="mt-6 text-left">
<div className="mb-2 flex items-center justify-between text-[11px] text-neutral-500">
<span>{t(`picoclaw.install.stages.${installStage || 'downloading'}`)}</span>
<span>{installProgress ?? 0}%</span>
</div>
<Progress
percent={installProgress || 0}
showInfo={false}
strokeColor="#38bdf8"
trailColor="rgba(255,255,255,0.07)"
status="active"
/>
</div>
)}
<div className="mt-6 flex justify-center">
{!isInstalling && (
<Button
icon={<DownloadIcon size={13} />}
onClick={() => void onInstall()}
type="primary"
>
{t('picoclaw.install.install')}
</Button>
)}
</div>
</div>
</div>
);
};

131
web/src/pages/desktop/picoclaw/sidebar-model-config.tsx Normal file
View File

@@ -0,0 +1,131 @@
import { useEffect } from 'react';
import { Button, Input } from 'antd';
import { useSetAtom } from 'jotai';
import { BookOpenIcon, CpuIcon, ExternalLinkIcon, KeyRoundIcon, LinkIcon, SaveIcon } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { isKeyboardEnableAtom } from '@/jotai/keyboard.ts';
type SidebarModelConfigProps = {
apiBase: string;
apiKey: string;
isSaving: boolean;
modelIdentifier: string;
modelName?: string;
onApiBaseChange: (value: string) => void;
onApiKeyChange: (value: string) => void;
onModelIdentifierChange: (value: string) => void;
onSave: () => void | Promise<void>;
onCancel?: () => void;
showCancel?: boolean;
};
export const SidebarModelConfig = ({
apiBase,
apiKey,
isSaving,
modelIdentifier,
modelName,
onApiBaseChange,
onApiKeyChange,
onModelIdentifierChange,
onSave,
onCancel,
showCancel = false
}: SidebarModelConfigProps) => {
const { t } = useTranslation();
const setIsKeyboardEnable = useSetAtom(isKeyboardEnableAtom);
useEffect(() => {
setIsKeyboardEnable(false);
return () => {
setIsKeyboardEnable(true);
};
}, [setIsKeyboardEnable]);
return (
<div className="flex flex-1 flex-col overflow-y-auto px-5 pb-8 pt-10">
{/* Title */}
<div className="mb-1 text-sm font-semibold text-neutral-100">
{t('picoclaw.model.requiredTitle')}
</div>
<div className="mb-5 text-xs leading-5 text-neutral-500">
{t('picoclaw.model.requiredDescription')}
{modelName && <span className="ml-1 text-neutral-400">{modelName}</span>}
</div>
{/* Documentation Link */}
<a
href="https://docs.picoclaw.io/docs/configuration/model-list/#supported-vendors-and-protocols"
target="_blank"
rel="noreferrer"
className="group mb-5 flex items-center justify-between rounded-lg border border-white/[0.06] bg-white/[0.02] p-3 transition-colors hover:border-white/[0.1] hover:bg-white/[0.04]"
>
<div className="flex items-center gap-3">
<div className="flex h-8 w-8 items-center justify-center rounded-md border border-white/[0.08] bg-white/[0.04]">
<BookOpenIcon className="text-sky-400" size={14} />
</div>
<div>
<div className="text-xs font-medium text-neutral-300">
{t('picoclaw.model.docsTitle', 'Configuration Guide')}
</div>
<div className="mt-0.5 text-[11px] text-neutral-500">
{t('picoclaw.model.docsDesc', 'Supported models and protocols')}
</div>
</div>
</div>
<ExternalLinkIcon size={14} className="text-neutral-500 transition-colors group-hover:text-neutral-300" />
</a>
{/* Fields */}
<div className="space-y-4">
<div>
<label className="mb-1.5 block text-xs text-neutral-500">
{t('picoclaw.model.modelIdentifier')}
</label>
<Input
prefix={<CpuIcon size={13} className="text-neutral-500" />}
placeholder={t('picoclaw.model.modelIdentifierPlaceholder')}
value={modelIdentifier}
onChange={(e) => onModelIdentifierChange(e.target.value)}
/>
</div>
<div>
<label className="mb-1.5 block text-xs text-neutral-500">
{t('picoclaw.model.apiKey')}
</label>
<Input.Password
prefix={<KeyRoundIcon size={13} className="text-neutral-500" />}
placeholder={t('picoclaw.model.apiKeyPlaceholder')}
value={apiKey}
onChange={(e) => onApiKeyChange(e.target.value)}
/>
</div>
<div>
<label className="mb-1.5 block text-xs text-neutral-500">
{t('picoclaw.model.apiBase')}
</label>
<Input
prefix={<LinkIcon size={13} className="text-neutral-500" />}
placeholder={t('picoclaw.model.apiBasePlaceholder')}
value={apiBase}
onChange={(e) => onApiBaseChange(e.target.value)}
/>
</div>
</div>
{/* Save */}
<div className="mt-6 flex justify-end gap-2">
{showCancel && <Button onClick={onCancel}>{t('picoclaw.cancel')}</Button>}
<Button
icon={<SaveIcon size={13} />}
loading={isSaving}
onClick={() => void onSave()}
type="primary"
>
{t(isSaving ? 'picoclaw.model.saving' : 'picoclaw.model.save')}
</Button>
</div>
</div>
);
};

319
web/src/pages/desktop/picoclaw/sidebar-session-actions.ts Normal file
View File

@@ -0,0 +1,319 @@
import type { Dispatch, SetStateAction } from 'react';
import type { TFunction } from 'i18next';
import {
closeGateway,
connectGateway,
deletePicoclawSession,
getPicoclawSession,
listPicoclawSessions,
picoclawGateway,
sendChatMessage,
sendStopMessage,
type PicoclawSessionListItem
} from '@/api/picoclaw.ts';
import { generateUUIDv4 } from '@/lib/picoclaw-gateway.ts';
import type {
PicoclawChatMessage,
PicoclawConfigState,
PicoclawOverlayState,
PicoclawRunState,
PicoclawRuntimeStatus,
PicoclawTakeoverState,
PicoclawTransportState
} from '@/jotai/picoclaw.ts';
import { createErrorMessage, createStatusMessage, HIDDEN_OVERLAY } from './message-utils.ts';
import { canConnectGateway } from './runtime-view.ts';
type RuntimeStatusSetter = Dispatch<SetStateAction<PicoclawRuntimeStatus | null>>;
type MessageSetter = Dispatch<SetStateAction<PicoclawChatMessage[]>>;
type TakeoverSetter = Dispatch<SetStateAction<PicoclawTakeoverState>>;
type PicoclawSidebarSessionActionOptions = {
t: TFunction;
runtimeStatus: PicoclawRuntimeStatus | null;
transportState: PicoclawTransportState;
runState: PicoclawRunState;
config: PicoclawConfigState;
activeSessionId: string;
isFreshConversation: boolean;
isSwitchingSession: boolean;
refreshState: () => Promise<PicoclawRuntimeStatus | null>;
setMessages: MessageSetter;
setTakeover: TakeoverSetter;
setOverlay: Dispatch<SetStateAction<PicoclawOverlayState>>;
setTransportState: Dispatch<SetStateAction<PicoclawTransportState>>;
setRunState: Dispatch<SetStateAction<PicoclawRunState>>;
setHistorySessions: Dispatch<SetStateAction<PicoclawSessionListItem[]>>;
setActiveSessionId: Dispatch<SetStateAction<string>>;
setIsFreshConversation: Dispatch<SetStateAction<boolean>>;
setIsHistoryOpen: Dispatch<SetStateAction<boolean>>;
setIsModelConfigOpen: Dispatch<SetStateAction<boolean>>;
setIsLoadingHistory: Dispatch<SetStateAction<boolean>>;
setIsDeletingSession: Dispatch<SetStateAction<boolean>>;
setIsSwitchingSession: Dispatch<SetStateAction<boolean>>;
setRuntimeStatus: RuntimeStatusSetter;
};
export function createPicoclawSidebarSessionActions(options: PicoclawSidebarSessionActionOptions) {
const {
t,
runtimeStatus,
transportState,
runState,
config,
activeSessionId,
isFreshConversation,
isSwitchingSession,
refreshState,
setMessages,
setTakeover,
setOverlay,
setTransportState,
setRunState,
setHistorySessions,
setActiveSessionId,
setIsFreshConversation,
setIsHistoryOpen,
setIsModelConfigOpen,
setIsLoadingHistory,
setIsDeletingSession,
setIsSwitchingSession,
setRuntimeStatus
} = options;
async function loadHistorySessions() {
setIsLoadingHistory(true);
try {
const response = await listPicoclawSessions({ limit: 100 });
if (response.code === 0) {
setHistorySessions(response.data || []);
return response.data || [];
}
return [];
} finally {
setIsLoadingHistory(false);
}
}
async function handleSend(content: string) {
let ready = transportState === 'connected' && runState === 'idle';
if (!ready) {
const nextRuntimeStatus = await refreshState();
if (!canConnectGateway(nextRuntimeStatus ?? null)) {
return;
}
try {
await connectGateway();
ready = true;
} catch {
ready = false;
}
}
if (!ready) {
return;
}
const id = generateUUIDv4();
setIsFreshConversation(false);
setMessages((current) => [
...current,
{
id,
kind: 'user',
text: content,
createdAt: Date.now()
}
]);
sendChatMessage(content, {
id,
maxSteps: config.maxSteps,
maxRuntimeMs: config.maxRuntimeMs
});
}
async function handleNewConversation() {
if (isFreshConversation) {
return;
}
closeGateway();
setIsHistoryOpen(false);
setIsModelConfigOpen(false);
setMessages([]);
setTakeover({
active: false,
sessionId: undefined,
reason: 'new_conversation'
});
setOverlay(HIDDEN_OVERLAY);
setTransportState('disconnected');
setRunState('idle');
const nextSessionId = picoclawGateway.rotateSession();
setActiveSessionId(nextSessionId);
setIsFreshConversation(true);
const nextRuntimeStatus = await refreshState();
if (!canConnectGateway(nextRuntimeStatus ?? null)) {
return;
}
setMessages((current) => [...current, createStatusMessage(t('picoclaw.status.connecting'))]);
try {
await connectGateway(nextSessionId);
} catch {
// handled by gateway events
}
}
function handleStop() {
sendStopMessage();
setMessages((current) => [...current, createStatusMessage(t('picoclaw.status.stopped'))]);
}
async function handleOpenHistory() {
setIsModelConfigOpen(false);
setIsHistoryOpen(true);
await loadHistorySessions();
}
function handleCloseHistory() {
setIsHistoryOpen(false);
}
async function handleSelectHistorySession(sessionId: string) {
if (isSwitchingSession) {
return;
}
if (sessionId === activeSessionId) {
setIsHistoryOpen(false);
return;
}
setIsSwitchingSession(true);
try {
const response = await getPicoclawSession(sessionId);
if (response.code !== 0) {
const errorMessage =
(response as { message?: string; msg?: string }).message ||
(response as { message?: string; msg?: string }).msg ||
t('picoclaw.history.loadFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'SESSION_LOAD_FAILED',
message: errorMessage,
raw: response
})
]);
return;
}
const loadedMessages = (response.data.messages || []).map(
(message: { role: 'user' | 'assistant'; content: string }, index: number) => ({
id: `${response.data.id}-${index}`,
kind: message.role,
text: message.content,
createdAt: Date.now() + index
})
);
closeGateway();
setMessages(loadedMessages);
setActiveSessionId(sessionId);
setIsFreshConversation(false);
setIsHistoryOpen(false);
setTakeover({
active: false,
sessionId,
reason: 'history_selected'
});
setOverlay(HIDDEN_OVERLAY);
setTransportState('disconnected');
setRunState('idle');
setRuntimeStatus((current) =>
current
? {
...current,
current_session: sessionId
}
: current
);
if (canConnectGateway(runtimeStatus ?? null)) {
try {
await connectGateway(sessionId);
} catch {
// handled by gateway events
}
}
} catch (error) {
const errorMessage =
error instanceof Error ? error.message : t('picoclaw.history.loadFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'SESSION_LOAD_FAILED',
message: errorMessage,
raw: error
})
]);
} finally {
setIsSwitchingSession(false);
}
}
async function handleDeleteHistorySession(sessionId: string) {
setIsDeletingSession(true);
try {
const response = await deletePicoclawSession(sessionId);
if (response.code === 0) {
setHistorySessions((current) => current.filter((session) => session.id !== sessionId));
return;
}
const errorMessage =
(response as { message?: string; msg?: string }).message ||
(response as { message?: string; msg?: string }).msg ||
t('picoclaw.history.deleteFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'SESSION_DELETE_FAILED',
message: errorMessage,
raw: response
})
]);
} catch (error) {
const errorMessage =
error instanceof Error ? error.message : t('picoclaw.history.deleteFailed');
setMessages((current) => [
...current,
createErrorMessage({
code: 'SESSION_DELETE_FAILED',
message: errorMessage,
raw: error
})
]);
} finally {
setIsDeletingSession(false);
}
}
return {
handleSend,
handleNewConversation,
handleStop,
handleOpenHistory,
handleCloseHistory,
handleSelectHistorySession,
handleDeleteHistorySession
};
}

165
web/src/pages/desktop/picoclaw/sidebar.tsx Normal file
View File

@@ -0,0 +1,165 @@
import { Button } from 'antd';
import { useSetAtom } from 'jotai';
import { Loader2Icon, PlayIcon } from 'lucide-react';
import { useTranslation } from 'react-i18next';
import { picoclawChatOpenAtom } from '@/jotai/picoclaw.ts';
import { MessageInput } from './message-input.tsx';
import { MessageList } from './message-list.tsx';
import { SidebarHeader } from './sidebar-header.tsx';
import { SidebarHistory } from './sidebar-history.tsx';
import { SidebarInstall } from './sidebar-install.tsx';
import { SidebarModelConfig } from './sidebar-model-config.tsx';
import { useSidebar } from './use-sidebar.ts';
export const Sidebar = () => {
const { t } = useTranslation();
const setIsChatOpen = useSetAtom(picoclawChatOpenAtom);
const {
connectionLabel,
isInitializing,
sidebarMode,
handleCancelModelConfig,
handleAgentProfileChange,
handleCloseHistory,
handleDeleteHistorySession,
handleModelApiBaseChange,
handleModelApiKeyChange,
handleModelIdentifierChange,
handleNewConversation,
handleOpenHistory,
handleOpenModelConfig,
handleSaveModelConfig,
handleSelectHistorySession,
historySessions,
activeSessionId,
isDeletingSession,
isFreshConversation,
isHistoryOpen,
isInstallingRuntime,
isLoadingHistory,
isSwitchingSession,
installProgress,
installStage,
isSavingModelConfig,
isSwitchingAgent,
modelApiBase,
modelApiKey,
modelIdentifier,
isModelConfigOpen,
runState,
transportState,
isTogglingRuntime,
messages,
runtimeStatus,
handleInstallRuntime,
isUninstallingRuntime,
handleUninstallRuntime,
handleSend,
handleStartRuntime
} = useSidebar();
return (
<aside className="picoclaw-sidebar-scrollbar flex h-full min-h-0 w-full flex-col overflow-x-hidden bg-[#0d0d0f]">
{/* Header */}
<SidebarHeader
isCheckingRuntime={isInitializing}
installed={runtimeStatus?.installed}
modelConfigured={runtimeStatus?.model_configured}
isTogglingRuntime={isTogglingRuntime}
agentProfile={runtimeStatus?.agent_profile}
isSwitchingAgent={isSwitchingAgent}
runtimeReady={runtimeStatus?.ready}
onToggleRuntime={handleStartRuntime}
onClose={() => setIsChatOpen(false)}
isHistoryOpen={isHistoryOpen}
onOpenHistory={isHistoryOpen ? handleCloseHistory : handleOpenHistory}
onAgentProfileChange={handleAgentProfileChange}
onOpenModelConfig={handleOpenModelConfig}
isUninstallingRuntime={isUninstallingRuntime}
onUninstallRuntime={handleUninstallRuntime}
/>
{/* Divider */}
<div className="h-px bg-white/[0.06]" />
{/* Body */}
<div className="flex min-h-0 flex-1 flex-col">
{sidebarMode === 'loading' ? (
<div className="flex flex-1 flex-col items-center pt-10">
<div className="flex flex-col items-center gap-3">
<Loader2Icon className="animate-spin text-neutral-500" size={22} />
<span className="text-xs text-neutral-500">{connectionLabel}</span>
</div>
</div>
) : sidebarMode === 'install' ? (
<SidebarInstall
installProgress={installProgress}
installStage={installStage}
isInstalling={isInstallingRuntime}
onInstall={handleInstallRuntime}
/>
) : sidebarMode === 'model' ? (
<SidebarModelConfig
apiBase={modelApiBase}
apiKey={modelApiKey}
isSaving={isSavingModelConfig}
modelIdentifier={modelIdentifier}
modelName={runtimeStatus?.model_name}
onApiBaseChange={handleModelApiBaseChange}
onApiKeyChange={handleModelApiKeyChange}
onModelIdentifierChange={handleModelIdentifierChange}
onSave={handleSaveModelConfig}
onCancel={handleCancelModelConfig}
showCancel={isModelConfigOpen}
/>
) : sidebarMode === 'history' ? (
<SidebarHistory
sessions={historySessions}
activeSessionId={activeSessionId}
isLoading={isLoadingHistory}
isDeleting={isDeletingSession}
isSwitching={isSwitchingSession}
onSelect={handleSelectHistorySession}
onDelete={handleDeleteHistorySession}
/>
) : !runtimeStatus?.ready ? (
<div className="flex flex-1 flex-col items-center px-6 pb-6 pt-10 text-center">
<div className="flex flex-col items-center gap-4">
<div className="mb-5 flex h-12 w-12 items-center justify-center rounded-2xl border border-white/[0.08] bg-white/[0.04]">
<PlayIcon className="text-sky-400" size={22} />
</div>
<div>
<h3 className="mb-1 text-sm font-medium text-neutral-200">
{t('picoclaw.start.title')}
</h3>
<p className="text-xs text-neutral-500">{t('picoclaw.start.description')}</p>
</div>
<Button
type="primary"
icon={<PlayIcon size={14} />}
loading={isTogglingRuntime}
onClick={() => void handleStartRuntime()}
>
{t('picoclaw.config.startRuntime')}
</Button>
</div>
</div>
) : (
<>
<MessageList messages={messages} runState={runState} />
<div className="border-t border-white/[0.06] px-3 pb-3 pt-3">
<MessageInput
transportState={transportState}
onSend={handleSend}
onNewConversation={handleNewConversation}
disableNewConversation={isFreshConversation}
/>
</div>
</>
)}
</div>
</aside>
);
};

231
web/src/pages/desktop/picoclaw/use-sidebar.ts Normal file
View File

@@ -0,0 +1,231 @@
import { useMemo, useRef, useState } from 'react';
import { useAtom, useSetAtom } from 'jotai';
import { useTranslation } from 'react-i18next';
import type { PicoclawSessionListItem } from '@/api/picoclaw.ts';
import {
getPicoclawRuntimeInstallSnapshot,
type PicoclawRuntimeInstallSnapshot
} from '@/lib/picoclaw-storage.ts';
import {
picoclawConfigAtom,
picoclawMessagesAtom,
picoclawOverlayAtom,
picoclawRunStateAtom,
picoclawRuntimeStatusAtom,
picoclawTakeoverStateAtom,
picoclawTransportStateAtom
} from '@/jotai/picoclaw.ts';
import {
getPicoclawSidebarConnectionLabel,
getPicoclawSidebarMode,
getPicoclawSidebarStatusColor,
isPicoclawRuntimeInstalling
} from './runtime-view.ts';
import { createPicoclawSidebarActions } from './sidebar-actions.ts';
import {
usePicoclawGatewayEvents,
usePicoclawInstallRefresh,
usePicoclawInstallSnapshotSync,
usePicoclawSidebarLifecycle
} from './sidebar-effects.ts';
import { createPicoclawSidebarSessionActions } from './sidebar-session-actions.ts';
export const useSidebar = () => {
const { t } = useTranslation();
const [messages, setMessages] = useAtom(picoclawMessagesAtom);
const [transportState, setTransportState] = useAtom(picoclawTransportStateAtom);
const [runState, setRunState] = useAtom(picoclawRunStateAtom);
const [runtimeStatus, setRuntimeStatus] = useAtom(picoclawRuntimeStatusAtom);
const [config] = useAtom(picoclawConfigAtom);
const [, setTakeover] = useAtom(picoclawTakeoverStateAtom);
const setOverlay = useSetAtom(picoclawOverlayAtom);
const previousInstallStateRef = useRef<{ installing: boolean; status: string }>({
installing: false,
status: ''
});
const [modelApiBase, setModelApiBase] = useState('');
const [modelApiKey, setModelApiKey] = useState('');
const [modelIdentifier, setModelIdentifier] = useState('');
const [isInitializing, setIsInitializing] = useState(true);
const [isModelConfigOpen, setIsModelConfigOpen] = useState(false);
const [isHistoryOpen, setIsHistoryOpen] = useState(false);
const [isInstallRequestPending, setIsInstallRequestPending] = useState(false);
const [isUninstallRequestPending, setIsUninstallRequestPending] = useState(false);
const [isSavingModelConfig, setIsSavingModelConfig] = useState(false);
const [isSwitchingAgent, setIsSwitchingAgent] = useState(false);
const [isSwitchingSession, setIsSwitchingSession] = useState(false);
const [isTogglingRuntime, setIsTogglingRuntime] = useState(false);
const [isFreshConversation, setIsFreshConversation] = useState(true);
const [isLoadingHistory, setIsLoadingHistory] = useState(false);
const [isDeletingSession, setIsDeletingSession] = useState(false);
const [historySessions, setHistorySessions] = useState<PicoclawSessionListItem[]>([]);
const [activeSessionId, setActiveSessionId] = useState('');
const [installSnapshot, setInstallSnapshot] = useState<PicoclawRuntimeInstallSnapshot | null>(
() => getPicoclawRuntimeInstallSnapshot()
);
const isRuntimeStatusInstalling = isPicoclawRuntimeInstalling(runtimeStatus);
const isSnapshotInstalling = !runtimeStatus && installSnapshot?.installing === true;
const isRuntimeInstallActive = isRuntimeStatusInstalling || isSnapshotInstalling;
const isInstallingRuntime = isInstallRequestPending || isRuntimeInstallActive;
const installProgress = isRuntimeStatusInstalling
? runtimeStatus?.install_progress ?? installSnapshot?.installProgress
: isSnapshotInstalling
? installSnapshot?.installProgress
: undefined;
const installStage = isRuntimeStatusInstalling
? runtimeStatus?.install_stage ?? installSnapshot?.installStage
: isSnapshotInstalling
? installSnapshot?.installStage
: undefined;
const actions = createPicoclawSidebarActions({
t,
runtimeStatus,
transportState,
installSnapshot,
modelApiBase,
modelApiKey,
modelIdentifier,
setRuntimeStatus,
setMessages,
setTakeover,
setOverlay,
setTransportState,
setRunState,
setIsModelConfigOpen,
setIsTogglingRuntime,
setIsInstallRequestPending,
setInstallSnapshot,
setIsSavingModelConfig,
setIsSwitchingAgent,
setIsUninstallRequestPending
});
const refreshStateRef = useRef(actions.refreshState);
refreshStateRef.current = actions.refreshState;
const sessionActions = createPicoclawSidebarSessionActions({
t,
runtimeStatus,
transportState,
runState,
config,
activeSessionId,
isFreshConversation,
isSwitchingSession,
refreshState: refreshStateRef.current,
setMessages,
setTakeover,
setOverlay,
setTransportState,
setRunState,
setHistorySessions,
setActiveSessionId,
setIsFreshConversation,
setIsHistoryOpen,
setIsModelConfigOpen,
setIsLoadingHistory,
setIsDeletingSession,
setIsSwitchingSession,
setRuntimeStatus
});
usePicoclawGatewayEvents({
t,
setActiveSessionId,
setTakeover,
setMessages,
setTransportState,
setOverlay,
setRunState
});
usePicoclawSidebarLifecycle({
t,
refreshStateRef,
setActiveSessionId,
setIsFreshConversation,
setMessages,
setIsInitializing,
setTakeover,
setOverlay,
setTransportState,
setRunState
});
usePicoclawInstallRefresh(isRuntimeInstallActive, refreshStateRef);
usePicoclawInstallSnapshotSync({
t,
runtimeStatus,
setInstallSnapshot,
previousInstallStateRef,
setMessages
});
const connectionLabel = useMemo(
() =>
getPicoclawSidebarConnectionLabel(runtimeStatus, transportState, runState, isInitializing, t),
[isInitializing, runState, runtimeStatus, t, transportState]
);
const statusColor = useMemo(
() => getPicoclawSidebarStatusColor(runtimeStatus, transportState, runState, isInitializing),
[isInitializing, runState, runtimeStatus, transportState]
);
const sidebarMode = useMemo(() => {
if (isModelConfigOpen) {
return 'model';
}
if (isHistoryOpen) {
return 'history';
}
return getPicoclawSidebarMode(runtimeStatus, isInitializing);
}, [isHistoryOpen, isInitializing, isModelConfigOpen, runtimeStatus]);
return {
activeSessionId,
connectionLabel,
handleCancelModelConfig: () => setIsModelConfigOpen(false),
handleAgentProfileChange: actions.handleAgentProfileChange,
handleCloseHistory: sessionActions.handleCloseHistory,
handleDeleteHistorySession: sessionActions.handleDeleteHistorySession,
handleModelApiBaseChange: setModelApiBase,
handleModelApiKeyChange: setModelApiKey,
handleModelIdentifierChange: setModelIdentifier,
handleNewConversation: sessionActions.handleNewConversation,
handleOpenHistory: sessionActions.handleOpenHistory,
handleOpenModelConfig: () => {
setIsHistoryOpen(false);
setIsModelConfigOpen(true);
},
handleSaveModelConfig: actions.handleSaveModelConfig,
handleSelectHistorySession: sessionActions.handleSelectHistorySession,
historySessions,
isDeletingSession,
isFreshConversation,
isHistoryOpen,
isInitializing,
isInstallingRuntime,
isLoadingHistory,
isSwitchingSession,
isUninstallingRuntime: isUninstallRequestPending,
installProgress,
installStage,
isSavingModelConfig,
isSwitchingAgent,
isTogglingRuntime,
messages,
modelApiBase,
modelApiKey,
modelIdentifier,
isModelConfigOpen,
runState,
runtimeStatus,
sidebarMode,
statusColor,
handleInstallRuntime: actions.handleInstallRuntime,
handleUninstallRuntime: actions.handleUninstallRuntime,
transportState,
handleSend: sessionActions.handleSend,
handleStartRuntime: actions.handleStartRuntime,
handleStop: sessionActions.handleStop
};
};

12
web/src/pages/desktop/screen/h264-direct.tsx
View File

@@ -19,11 +19,11 @@ export const H264Direct = () => {
const workerRef = useRef<Worker | null>(null);
useEffect(() => {
const scale = storage.getVideoScale()
const scale = storage.getVideoScale();
if (scale) {
setVideoScale(scale)
setVideoScale(scale);
}
}, [setVideoScale])
}, [setVideoScale]);
useEffect(() => {
if (!window.VideoDecoder) {
@@ -69,17 +69,17 @@ export const H264Direct = () => {
}, []);
return (
<div className="flex h-screen w-screen items-start justify-center xl:items-center">
<div className="flex h-full min-h-0 w-full min-w-0 items-center justify-center overflow-hidden">
<canvas
id="screen"
ref={canvasRef}
className={clsx('block min-h-[480px] min-w-[640px] select-none', mouseStyle)}
className={clsx('block select-none', mouseStyle)}
style={{
transform: `scale(${videoScale})`,
transformOrigin: 'center',
...(resolution?.width
? { width: resolution.width, height: resolution.height, objectFit: 'cover' }
: { maxWidth: '100%', maxHeight: '100%', objectFit: 'scale-down' }),
: { maxWidth: '100%', maxHeight: '100%', objectFit: 'scale-down' })
}}
></canvas>
</div>

26
web/src/pages/desktop/screen/h264-webrtc.tsx
View File

@@ -1,19 +1,17 @@
import { useEffect, useRef, useState } from 'react';
import { Spin } from 'antd';
import clsx from 'clsx';
import { useAtom, useAtomValue } from 'jotai';
import { w3cwebsocket as W3cWebSocket } from 'websocket';
import * as storage from '@/lib/localstorage.ts';
import { getBaseUrl } from '@/lib/service.ts';
import { mouseStyleAtom } from '@/jotai/mouse.ts';
import { resolutionAtom, videoScaleAtom } from '@/jotai/screen.ts';
import * as storage from '@/lib/localstorage.ts'
export const H264Webrtc = () => {
const resolution = useAtomValue(resolutionAtom);
const mouseStyle = useAtomValue(mouseStyleAtom);
const [videoScale, setVideoScale] = useAtom(videoScaleAtom);
const [isLoading, setIsLoading] = useState(true);
const videoRef = useRef<HTMLVideoElement | null>(null);
@@ -173,19 +171,19 @@ export const H264Webrtc = () => {
}, []);
useEffect(() => {
const scale = storage.getVideoScale()
const scale = storage.getVideoScale();
if (scale) {
setVideoScale(scale)
setVideoScale(scale);
}
}, [setVideoScale])
}, [setVideoScale]);
return (
<Spin size="large" tip="Loading" spinning={isLoading}>
<div className="flex h-screen w-screen items-start justify-center xl:items-center">
<div className="relative h-full min-h-0 w-full min-w-0 overflow-hidden">
<div className="flex h-full min-h-0 w-full min-w-0 items-center justify-center overflow-hidden">
<video
id="screen"
ref={videoRef}
className={clsx('block min-h-[480px] min-w-[640px] select-none', mouseStyle)}
className={clsx('block select-none', mouseStyle)}
style={{
transform: `scale(${videoScale})`,
transformOrigin: 'center',
@@ -206,6 +204,14 @@ export const H264Webrtc = () => {
}}
/>
</div>
</Spin>
{isLoading && (
<div className="pointer-events-none absolute inset-0 flex items-center justify-center bg-black/20">
<div className="rounded-full bg-neutral-900/85 px-4 py-2 text-sm text-neutral-100 shadow-lg">
Loading
</div>
</div>
)}
</div>
);
};

48
web/src/pages/desktop/screen/mjpeg.tsx
View File

@@ -1,21 +1,22 @@
import { useEffect } from 'react';
import { Image } from 'antd';
import { useEffect, useState } from 'react';
import clsx from 'clsx';
import { useAtomValue, useAtom } from 'jotai';
import { useAtom, useAtomValue } from 'jotai';
import MonitorXIcon from '@/assets/images/monitor-x.svg';
import { stopFrameDetect } from '@/api/stream.ts';
import { getFrameDetect } from '@/lib/localstorage.ts';
import * as storage from '@/lib/localstorage.ts';
import { getBaseUrl } from '@/lib/service.ts';
import { mouseStyleAtom } from '@/jotai/mouse.ts';
import { resolutionAtom, videoScaleAtom } from '@/jotai/screen.ts';
import * as storage from '@/lib/localstorage.ts'
export const Mjpeg = () => {
const resolution = useAtomValue(resolutionAtom);
const mouseStyle = useAtomValue(mouseStyleAtom);
const [videoScale, setVideoScale] = useAtom(videoScaleAtom);
const [hasError, setHasError] = useState(false);
const [streamNonce, setStreamNonce] = useState(0);
const streamURL = `${getBaseUrl('http')}/api/stream/mjpeg`;
const streamSrc = hasError ? undefined : `${streamURL}?v=${streamNonce}`;
useEffect(() => {
// stop frame detect for a while
@@ -23,32 +24,33 @@ export const Mjpeg = () => {
if (enabled) {
stopFrameDetect(10);
}
setHasError(false);
setStreamNonce((current) => current + 1);
}, [resolution]);
useEffect(() => {
const scale = storage.getVideoScale()
const scale = storage.getVideoScale();
if (scale) {
setVideoScale(scale)
setVideoScale(scale);
}
}, [setVideoScale])
}, [setVideoScale]);
return (
<div className="flex h-screen w-screen items-start justify-center xl:items-center"
style={{
transform: `scale(${videoScale})`,
transformOrigin: 'center'
}}>
<Image
<div className="flex h-full min-h-0 w-full min-w-0 items-center justify-center overflow-hidden bg-black">
<img
id="screen"
className={clsx('block min-h-[480px] min-w-[640px] select-none', mouseStyle)}
style={
resolution?.width
className={clsx('block select-none', mouseStyle)}
style={{
transform: `scale(${videoScale})`,
transformOrigin: 'center',
...(resolution?.width
? { width: resolution.width, height: resolution.height, objectFit: 'cover' }
: { maxHeight: '100vh', objectFit: 'scale-down' }
}
src={`${getBaseUrl('http')}/api/stream/mjpeg`}
fallback={MonitorXIcon}
preview={false}
: { maxHeight: '100%', objectFit: 'scale-down' }),
visibility: hasError ? 'hidden' : 'visible'
}}
src={streamSrc}
onError={() => setHasError(true)}
alt="screen"
/>
</div>
);