Linux/Proxmox Transition Runbook

Where to start

This runbook is for operators preparing the move from Windows development to Linux/Proxmox. It records nested-lab evidence, deployment decisions, and the checks needed for the eventual physical server. Nested-lab success is not physical-host acceptance.

• Check current progress: Read purpose and status and the Linux blockers.
• Configure lab inventory: See read-only VM inventory.
• Plan the physical transition: Review the recommended sequence, including recovery and compatibility checks before making host changes.
• Verify the target: Use the Linux acceptance checklist alongside the detailed procedures and their stated validation scope.

Purpose and status

Windows remains the active build and prototype environment for shared application and API behavior. An additional deployment/integration lab is running Proxmox VE 9.1-1 as a nested Hyper-V guest on the Windows PC. The dashboard and backend are deployed in that guest with loopback-only service ports, and NGINX provides verified HTTPS access from LAN clients. This runbook identifies the platform-specific changes, decisions, and target-host checks needed to carry the implementation to Linux/Proxmox. Nested-lab results do not constitute physical-host acceptance.

The current guest release is 20261106-updater-verify. Rootless Podman/Quadlet backend and dashboard units publish only loopback ports; boot auto-start and clean/forced guest recovery were verified before this release. The latest switch left both containers healthy, with the dashboard root returning 200 and /docs returning 404. NGINX/TLS access was verified from Windows and iPhone before this release. A signed Stage 1 test bundle was verified through the dashboard and Discard appeared to work. Release-to-release rollback, some updater negative/large-upload checks, and reboot recovery after the latest release remain unverified. A dedicated non-root ai-deploy SSH account is available for deployment work; it has no sudo access. Physical-host acceptance remains separate.

Read-only VM inventory lab integration

The dashboard can query Proxmox VM and container inventory through GET /api/proxmox/vms. Configure this integration only on the backend:

• Set PROXMOX_API_URL to the Proxmox HTTPS origin using a hostname covered by the API certificate, for example https://<pve-hostname>:8006.
• Create a dedicated Proxmox user and API token for inventory. Use a privilege-separated token and grant only the audit/read privileges required for cluster resource inventory (normally Sys.Audit and VM.Audit) at the narrowest path that returns the intended guests. Do not grant VM modify, power, allocation, console, or storage privileges for this feature.
• Set PROXMOX_INVENTORY_TOKEN_ID to user@realm!token-name and PROXMOX_INVENTORY_TOKEN_SECRET to the token secret. Store the secret only in /opt/ai-server/.env, owned by ai-server and mode 0640.
• If the Proxmox certificate is not trusted by the system, set PROXMOX_API_CA_BUNDLE to a readable PEM CA certificate. Keep hostname and certificate verification enabled; do not use an insecure TLS bypass.
• The nested guest's Proxmox CA lacks the X.509 keyUsage extension and Python OpenSSL strict verification rejects it. For this isolated lab only, PROXMOX_API_ALLOW_MISSING_CA_KEY_USAGE=true may be set alongside the explicit CA bundle. It disables OpenSSL's strict X.509 extension-policy checks for Proxmox connections while retaining trust-chain, validity, and hostname verification. Do not carry this compatibility setting into physical/production deployment; use a properly issued CA/certificate there.

Restart the backend after configuring settings, then open Virtual Machines and refresh. A successful response verifies this token can read the intended inventory; an HTTP 403 indicates that the token's ACL scope or audit privileges need review. The route makes only a read-only cluster resource request and returns only guest name, ID, node, type, and status. It does not expose token credentials, cached/sample inventory, or VM controls.

Nested-lab verification — 2026-10-16

On the nested Proxmox VE 9.1.1 guest (kernel 6.17.2-1-pve, Python 3.13.5), the installed inventory client ran as ai-server, loaded its protected environment file without displaying credentials, and successfully retrieved one guest. The returned record had only name, node, status, type, and vmid, confirming the configured token can read the requested cluster inventory. The lab-only missing-CA-keyUsage compatibility option was enabled with the explicit CA bundle; TLS chain and hostname verification were retained.

The backend and static dashboard were also started in temporary, non-persistent systemd-run units as ai-server; the static page used Python's standard library HTTP server. Both reached active state with successful exit status, bound only to 127.0.0.1, and returned HTTP 200 for the dashboard root and backend OpenAPI document. The unauthenticated inventory route returned HTTP 401, and Windows-side probes could not connect to ports 8000/8001. Both temporary units were stopped and the ports were confirmed closed.

Podman 5.4.2 was subsequently installed for rootless ai-server operation. Digest-pinned backend and static-dashboard images and Quadlet definitions were built and started. Both containers passed health checks; the dashboard and OpenAPI returned HTTP 200, the anonymous inventory request returned HTTP 401, and host ports remained loopback-only. The units have bounded restart behavior, but no boot target is configured. A direct host-side inventory read previously succeeded. Container DNS and CA-file readability were confirmed, but the container's initial connection to the guest LAN address on port 8006 was refused. A throwaway container test verified that --add-host=pve.mshome.net:host-gateway reaches the API through Podman's host.containers.internal gateway (169.254.1.2). This mapping has been added to the backend Quadlet and deployed. After restarting the backend, the live inventory client inside the container returned one guest with only the expected name, node, status, type, and vmid fields. The backend and dashboard remained active and healthy with loopback-only host ports. The configured API hostname and TLS hostname verification were retained.

On 2026-10-17, explicit stop/start lifecycle checks passed: both services became inactive/dead, their containers exited and listeners closed, then both services returned active and healthy. Dashboard HTTP returned 200, OpenAPI returned 200, and unauthenticated inventory returned 401. For a single controlled failure test, the backend container was killed with Podman; systemd restarted the backend once (NRestarts=1). The backend returned healthy, OpenAPI returned 200, and the in-container inventory request again returned one guest with the expected five fields. The dashboard remained healthy and ports stayed loopback-only. A subsequent boundary test killed the backend four times within 60 seconds: the first three failures triggered automatic restarts and the fourth left the unit failed at the configured limit (NRestarts=4). reset-failed followed by manual start restored the backend to healthy; OpenAPI returned 200, anonymous inventory returned 401, and both ports remained loopback-only. The start-limit and manual recovery behavior passed in this lab.

Deployment rollback was then tested using a retained copy of the backend Quadlet and a localhost/ai-server:rollback-known-good-20261017 tag for the current image. A temporary candidate deliberately referenced a nonexistent local image with Pull=never; after systemd reached the start limit, the saved Quadlet was restored, the failed unit reset, and the service manually started. The backend returned healthy, OpenAPI returned 200, anonymous inventory returned 401, and live Proxmox inventory returned one guest with the expected five fields. The dashboard remained healthy and both ports stayed loopback-only. This proves recovery from an unavailable candidate to the pre-test snapshot, not rollback between distinct application releases; the known-good image tag and config snapshot remain on the guest under /var/lib/ai-server/deploy-backups/rollback-test-20261017/.

These results are nested-lab evidence only; physical-host acceptance remains separate. At this point in the 2026-10-17 timeline, boot auto-start, reboot recovery, NGINX/TLS, and LAN application exposure had not yet been validated; later nested-lab checks are recorded below.

The current target is a Proxmox VE 9.x host with one AMD RX 6800 XT, with services managed using Podman and Quadlet. The RX 6800 XT is currently in the Windows development machine and is planned to move to the server after a replacement GPU is acquired. The proposed single-GPU runtime keeps a Qwen2.5 1.5B Utility AI resident alongside either a Qwen3 8B Household model or a Qwen2.5 Coder 7B Maintenance model, selected by operating profile. Persistent Utility residency, profile routing, and graceful transitions are planned only; the current model manager runs one llama.cpp model and restarts it to switch. GPU passthrough is not planned; the GPU will be used directly by the Proxmox host and its AI runtime. A second GPU is not selected. This is a target design, not an installed or compatibility-tested stack.

Intended Linux/Proxmox stack

The planned components are:

• GPU compute: AMD ROCm and HIP.
• ROCm placement: install the ROCm/HIP stack on the Proxmox host; GPU driver, kernel, and device setup therefore require host-specific validation.
• LLM inference: llama.cpp built with ROCm support, exposing llama-server to the local backend.
• Vision inference: ONNX Runtime with its ROCm Execution Provider.
• Image generation: ComfyUI with a ROCm-compatible PyTorch/runtime setup.
• Service packaging and lifecycle: Podman containers managed by Quadlet systemd units where containerization is appropriate.
• Web entry point: NGINX serving the dashboard and reverse-proxying the protected backend endpoints.
• Vector storage: Qdrant, managed as a service/container.

The exact point release, kernel, ROCm/HIP and application versions, container boundaries, persistent storage paths, network bindings, and installation sources remain to be recorded and verified before generating version-pinned installation steps. Treat ROCm/HIP on the planned RX 6800 XT + Proxmox host as experimental and community-supported only: AMD's current support matrix does not list this GPU/OS combination. Community reports cover related gfx1030 cards on other Linux distributions, but do not prove native ROCm operation on this host. The GPU is intended for direct host use; assigning it to a VM via passthrough is not part of this design. See the experimental community evidence and limitations. Do not assume the Windows Vulkan results establish ROCm, HIP, or ROCm telemetry compatibility.

Ubuntu and Proxmox share Linux-kernel ecosystem and Debian-related foundations, which make testing a reasonable hypothesis, but this alone does not establish low compatibility risk or AMD support for Proxmox. Their packaging, kernel integration, and supported release combinations differ. Verify the exact Proxmox release/kernel, AMD driver, ROCm version, container device access, and the intended inference workload on the physical host before treating the combination as usable.

Provisional host hardware and storage plan

This planning outline is not a finalized bill of materials. The planned platform is a Z590 motherboard with an Intel Core i9-11900KF; the exact board model and delivered hardware still need confirmation. Validate firmware/boot configuration and the CPU's available resources against the intended VM count and concurrent workloads, including Proxmox overhead and host services. Start with 32 GB system RAM, with a possible upgrade to 64 GB after measuring actual headroom. System RAM does not replace or add to the RX 6800 XT's 16 GB VRAM.

The initial storage target is a 2 TB NVMe drive for Proxmox/application storage, active or write-intensive VM disks, databases, and frequently used models. Additional NVMe devices may be added later. Larger HDDs are intended for bulk file data and lower-I/O VMs such as DHCP and DNS. A file-hosting VM can keep its OS and applications on NVMe while storing bulk files on an HDD pool. Expect HDD-backed VMs to have slower boots and disk-intensive operations.

The model library may be tiered between HDD capacity and NVMe: keep frequently used or pinned models on NVMe and less-used models on HDD. The eventual model manager must verify copies and avoid moving models while loaded or in use. This tiering chiefly reduces model access and load time; it does not make inference faster once the model is loaded.

An HDD RAID/redundant pool is being considered, but the drive count, exact layout, controller mode, usable capacity, filesystem, and expansion method have not been selected. Storage redundancy does not replace separate backups; document and test backup restoration. Finalize storage ownership and pool design before creating production VM disks or file shares.

What is portable now

• The dashboard is static HTML, CSS, and JavaScript.
• The API, chat client, model inventory/switching logic, and web-research integration are Python and use the standard library plus the packages in ai-backend/requirements.txt.
• CPU and system RAM percentages are read through psutil.
• Chat and research call local HTTP services. Their URLs are configurable using LLAMA_CPP_BASE_URL and SEARXNG_BASE_URL.
• The model manager already has POSIX process handling and defaults to a Linux llama.cpp device name of ROCm0. This is not proof that the eventual llama.cpp build exposes that device name or that ROCm supports the chosen OS/kernel/GPU combination.
• Backend/dashboard Containerfile and Quadlet source contracts have portable tests for loopback-only port publication, fixed non-root identity, read-only secret/CA mounts, persistent backend state, health probes, bounded restart settings, and a digest-pinned base image. The HTTP health probe also has success/failure tests. These tests do not execute Podman, systemd, Quadlet generation, or reboot recovery.

The Windows build is the working prototype for shared application behavior; these portability points are not merely mocks. However, Windows verification does not establish Proxmox runtime behavior. Validate OS-, hardware-, network-, and service-specific components using the staged host checks and end-to-end acceptance below.

Linux blockers to close

1. Record the operating-system target. Services are intended to run on the Proxmox VE 9.x host, managed through Podman/Quadlet. Record the exact Proxmox point release, kernel, boot mode, GPU ownership, storage locations, and recovery access before driver work. The GPU is for direct host use; moving services into a guest or assigning the GPU via passthrough would change the architecture and is not planned.
2. Investigate the experimental GPU software stack for the exact target. Check the current official AMD ROCm support matrix for the selected Proxmox release/kernel, RX 6800 XT (and any additional GPUs), and ROCm/HIP version. The current combination is not listed as officially supported; community reports are leads, not a support guarantee. Only proceed with a reviewed recovery plan and explicit acceptance of the experimental risk. Then build llama.cpp with the ROCm backend and record its pinned revision, build flags, device name, and smoke-test results. Windows Vulkan test results do not certify this build.
3. Validate the Linux telemetry provider. A Linux AMD provider is wired through the platform dispatcher and reads amdgpu DRM utilization and VRAM counters, with optional hwmon sensors. Fixture tests cover valid, missing, malformed, and ambiguous readings. Validate actual adapter identity, permissions, and idle/inference readings against an independent host tool; this does not assert ROCm/HIP support.
4. Decide and verify how browsers reach the API. js/app.js uses http://127.0.0.1:8000 when opened as a local file, uses the dashboard host's port 8000 when the local dashboard is served on port 8001, and uses the dashboard's own origin otherwise. A deployment may set window.AI_SERVER_API_BASE before the app loads. Choose a server-reachable API origin or same-origin reverse proxy and test from a separate LAN client. Do not expose the llama.cpp port to the LAN.
5. Validate deployment security before LAN use. Dashboard-owned local accounts, explicitly mapped Proxmox identities, role-protected REST and WebSocket access, session cookies, CSRF checks, and origin restrictions are implemented and tested in the Windows environment. This does not validate real Proxmox ticket behavior or the production proxy. Do not expose the API, model-switch endpoint, or llama.cpp to an untrusted network. Before remote/LAN access, validate Proxmox login and role mappings as applicable, HTTPS and secure-cookie behavior through the chosen proxy, firewall rules, browser access from a separate client, and the model-switch policy.
6. Create Linux service management. The .bat scripts only support Windows. Prepare Podman/Quadlet units for containerized services and systemd units or documented host environments for components that need direct host/GPU access. Cover llama-server, backend, NGINX, Qdrant, ComfyUI, and any ONNX Runtime service actually integrated. Use a dedicated service account, explicit environment files, controlled startup ordering, persistent volumes/logs, health checks, and a least-privilege GPU device access policy. Ensure model switching can inspect and stop only its own llama-server process.
7. Set Linux paths and permissions. Configure AI_SERVER_MODEL_DIR and LLAMA_SERVER_EXECUTABLE; ensure the service account can read model files, write its logs, and access the selected GPU device. Keep models and llama.cpp build artifacts outside the source checkout and back them up or document how to reacquire them.
8. Install Linux-specific AI packages separately. Windows DirectML packages in ai-backend/requirements-windows.txt and ai-discord-bot/requirements-windows.txt are not the Linux install path. Install only the common requirements.txt dependencies first. Pin and validate the selected ONNX Runtime ROCm Execution Provider and ComfyUI ROCm-compatible dependency/model versions; neither is currently a production integration.
9. Configure optional services and secrets. Deploy SearXNG and enable its JSON response format before enabling web research. Set its URL through SEARXNG_BASE_URL. If the Discord bot is enabled, set a real token outside source control and set AI_SERVER_API_BASE to a reachable API address.
10. Integrate and verify the existing application as a whole. Run the backend unit tests on Linux; start each service under its intended supervisor with persistent paths; verify backend-to-llama.cpp model discovery, switching, chat and streaming; verify dashboard access and WebSocket traffic through the selected reverse proxy; verify logs and the observe-only diagnostics API/dashboard; then test optional SearXNG research and Discord commands/heartbeat when configured. Exercise service restart and host reboot recovery. Record failures and exact tested versions; do not mark Windows-only, mocked, or planned features as integrated.

Deployment configuration inventory

Setting	Current default/behavior	Linux deployment action
AI_SERVER_MODEL_DIR	Windows %LOCALAPPDATA%\AI-Server\models; otherwise ~/AI-Server/models	Set an explicit persistent model directory.
LLAMA_SERVER_EXECUTABLE	Windows build path, or llama-server on PATH when LOCALAPPDATA is absent	Set the absolute Linux executable path.
LLAMA_CPP_BASE_URL	http://127.0.0.1:8080	Keep loopback when API and runtime share the host.
LLAMA_CPP_DEVICE	Vulkan0 on Windows; ROCm0 otherwise	Set to the exact device identifier reported by the deployed build.
LLAMA_CPP_GPU_LAYERS	99	Confirm available VRAM and inference behavior; tune if needed.
LLAMA_CPP_CTX_SIZE	4096	Confirm memory use and desired context length.
SEARXNG_BASE_URL	http://127.0.0.1:8888	Set the reachable SearXNG endpoint and enable JSON output.
AI_SERVER_API_BASE	Discord bot defaults to http://127.0.0.1:8000	Set it if the bot runs on another host or guest.
DISCORD_TOKEN	Placeholder default	Supply through a protected environment file only if using the bot.

Container image tags, Quadlet unit names, Qdrant data volume, ComfyUI model directories, and NGINX hostnames/certificates must be selected in the deployment-specific configuration; do not put secret values in unit files tracked in the repository.

Host preparation script

scripts/proxmox/prepare-ai-host.sh is an initial, deliberately conservative bootstrap:

• Run bash scripts/proxmox/prepare-ai-host.sh --check for a read-only inventory of the installed distribution, Proxmox release, running kernel, architecture, and detected AMD display device. The script checks AMD's live ROCm compatibility matrix and fails closed unless the RX 6800 XT, Proxmox VE 9, and installed kernel family are explicitly listed.
• Run bash scripts/proxmox/prepare-ai-host.sh --report to create ai-server-host-report.md in the current directory, or pass a new output path with --report /path/to/report.md. This read-only snapshot records platform/kernel, CPU and memory, storage capacity/models, display device and driver details, GPU kernel-module and Secure Boot status, relevant installed package versions, selected service states, and the AMD compatibility-check result. The report is created with owner-only permissions and the script refuses to overwrite an existing destination. It excludes hostnames, usernames, network addresses, serial numbers, credentials, environment variables, configuration contents, and logs. It is not uploaded; the report mode does make the same read-only request to AMD's public compatibility matrix as --check. Review it before sharing. It supports implementation planning but does not replace live integration tests or prove that host features work. Nested-lab status: --check and --report ran on the Proxmox VE 9.1.1 guest on 2026-11-05 as a non-root user without changing the system; --check exited 2 (gate blocked, expected without the GPU). Physical-host runs are still pending.
• Run sudo bash scripts/proxmox/prepare-ai-host.sh --install-base only on the server, after reviewing its simulated apt plan. It installs common build/runtime and host tools and adds headers for the exact running kernel only when an apt candidate exists. It refuses a transaction that proposes replacing or removing a kernel package.
• Run sudo bash scripts/proxmox/prepare-ai-host.sh --install-searxng on the server to install SearXNG (the web-research backend). It runs as the unprivileged searxng user under systemd, bound to 127.0.0.1:8888, and is independent of the GPU gate. The planned Podman/Quadlet layout may replace this systemd service later; the loopback address and JSON format stay the same, so SEARXNG_BASE_URL does not change. As of 2026-10-12, AMD's current compatibility matrix does not list the RX 6800 XT or Proxmox VE 9 as officially supported for ROCm. Therefore the script must not be treated as permission to install ROCm on this target, and the GPU compute stack remains gated. This is an explicit limitation, not a claim that community-enabled ROCm configurations cannot work.

The script does not yet install ROCm/HIP, llama.cpp, ONNX Runtime ROCm EP, ComfyUI, Qdrant, or production Quadlet/NGINX configuration. Those steps need version pins and an install method verified for the actual host; do not silently install floating latest releases on a Proxmox hypervisor.

Application deployment installer — planned, not implemented

Keep host preparation separate from application deployment. The existing prepare-ai-host.sh is for host inventory, carefully reviewed prerequisites, and the separately gated SearXNG setup; do not turn it into an all-components installer. The nested lab currently uses the manual release procedure below; an automatic application deployment/update installer for the backend and dashboard image and their Podman/Quadlet units has not been built.

The dashboard unit runs ai-backend/dashboard_server.py, which adds a Content-Security-Policy and framing/sniffing protections and serves only the dashboard assets. The deployed lab dashboard has been verified with CSP and frame protections; NGINX/TLS is installed and was verified from Windows and iPhone. Production/physical-host exposure remains gated.

Lab release procedure (manual; repeatedly executed on the nested lab)

The 2026-10-19 deployment attempt left the backend unreachable and the cause was never isolated. The Containerfile has since been found to omit the backend api/ and api/routes/ packages: an image built from it fails at startup with ModuleNotFoundError: No module named 'api'. The Containerfile now copies both packages, and a contract test fails if any backend package directory is not copied. That is the most likely cause, but it was reproduced only by simulating the image contents on Windows; it was not confirmed against the failed image.

Also, the Windows working tree uses CRLF while the guest needs LF; a CRLF Containerfile or Quadlet unit breaks parsing. Never copy the working tree directly. .gitattributes now pins LF for these files, and scripts/proxmox/build-lab-bundle.py builds an allow-listed, LF-normalized bundle with no .env files, tests, databases, or logs.

Stage (from Windows, as the non-root ai-deploy account; no root needed):

python scripts\proxmox\build-lab-bundle.py $env:TEMP\ai-server-RELEASE.tar.gz   # prints sha256
ssh ai-deploy@GUEST "mkdir -p ~/releases/RELEASE"
scp $env:TEMP\ai-server-RELEASE.tar.gz ai-deploy@GUEST:releases/RELEASE/
ssh ai-deploy@GUEST "cd ~/releases/RELEASE && echo 'SHA256  ai-server-RELEASE.tar.gz' | sha256sum -c - && mkdir -p src && tar -xzf ai-server-RELEASE.tar.gz -C src"

Deploy (operator, as root on the guest; take a guest checkpoint first). The import gate runs in the new image before any service is switched, and every step after the backup can be reversed with the rollback block:

cd /   # rootless podman warns "cannot chdir to /root" when run from root's home
set -euo pipefail
ID=RELEASE
AS() { runuser -u ai-server -- env XDG_RUNTIME_DIR=/run/user/999 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/999/bus HOME=/var/lib/ai-server "$@"; }
Q=/var/lib/ai-server/.config/containers/systemd   # confirm this path exists first
B=/var/lib/ai-server/deploy-backups/$ID
[ -d "$Q" ] && [ -d /home/ai-deploy/releases/$ID/src ]

# 1. Stop, then retain the current image, units, and data.
AS systemctl --user stop ai-server-dashboard ai-server-backend
install -d -o ai-server -g ai-server -m 700 "$B" "$B/data"
AS podman tag localhost/ai-server:lab localhost/ai-server:rollback-$ID
cp -a "$Q" "$B/quadlet"
find /var/lib/ai-server -maxdepth 3 -name '*.sqlite3' -not -path '*/deploy-backups/*' \
  -exec cp -a --parents {} "$B/data" \; 2>/dev/null || true

# 2. Build the new image from the staged bundle.
cp -r /home/ai-deploy/releases/$ID/src /var/lib/ai-server/build-$ID
chown -R ai-server:ai-server /var/lib/ai-server/build-$ID
LOG=/var/lib/ai-server/build-$ID.log   # owned by ai-server so the housekeeping job can prune it later
rc=0
AS podman build -t localhost/ai-server:$ID -f /var/lib/ai-server/build-$ID/Containerfile /var/lib/ai-server/build-$ID > "$LOG" 2>&1 || rc=$?
chown ai-server:ai-server "$LOG"
echo "build exit=$rc"; tail -n 5 "$LOG"; [ "$rc" -eq 0 ]

# 3. Gate: the image must import the app with the docs routes disabled. The app
# creates and chmods 0700 the parent directory of its databases, so they must
# live in a subdirectory, not directly in the root-owned /tmp.
AS podman run --rm --read-only --tmpfs /tmp --userns=keep-id --user 999:990 \
  -e AI_SERVER_AUTH_DB_PATH=/tmp/gate/a.db -e AI_SERVER_MEMORY_DB_PATH=/tmp/gate/m.db -e AI_SERVER_LOG_DIR=/tmp/gate/logs \
  localhost/ai-server:$ID python -c "import main; assert main.app.openapi_url is None; print('import ok')"

# 4. Switch: retag, install the new units, restart.
AS podman tag localhost/ai-server:$ID localhost/ai-server:lab
install -o ai-server -g ai-server -m 644 /var/lib/ai-server/build-$ID/deploy/quadlet/*.container "$Q"/
AS systemctl --user daemon-reload
AS systemctl --user reset-failed ai-server-backend ai-server-dashboard || true
AS systemctl --user start ai-server-backend ai-server-dashboard

Verify from the guest: both services report healthy in podman ps (or podman inspect --format '{{.State.Health.Status}}'), GET / returns 200, /openapi.json and /docs return 404, anonymous API routes return 401, and the dashboard response has a Content-Security-Policy header. Do not use podman healthcheck run or podman exec through runuser: they fail with a cgroup permission error (see the container health record below) even when the containers are healthy. If any step or check fails, roll back:

AS systemctl --user stop ai-server-dashboard ai-server-backend
AS podman tag localhost/ai-server:rollback-$ID localhost/ai-server:lab
cp -a "$B/quadlet/." "$Q"/ && chown -R ai-server:ai-server "$Q"
AS systemctl --user daemon-reload
AS systemctl --user reset-failed ai-server-backend ai-server-dashboard || true
AS systemctl --user start ai-server-backend ai-server-dashboard

Rollback never touches /var/lib/ai-server application data; the SQLite copies in the backup directory are for manual recovery only. Boot auto-start is not part of this procedure; see the next section.

The first run of this procedure on 2026-11-05 (nested lab only) built the image, passed the import gate, and switched both services to healthy. Record the result of every later run, including any failure and its cause, before treating deployment as verified.

A frontend-only release (20261106-settings, tabbed Settings page) was deployed with this procedure on the nested lab on 2026-11-05: build, import gate and switch succeeded; the new dashboard was served over the HTTPS proxy with the docs routes still 404 and the backend ports still loopback-only. In the browser (Windows and iPhone), the Setup / Deployment tab appeared for an admin and not for a viewer, and the mobile layout was acceptable. Container health was not re-checked after this switch.

A combined release (20261106-frontend) followed the same procedure on the nested lab on 2026-11-05 with no step failing. It carried the frontend polish (self-hosted fonts, manifest and icons, offline banner, live CPU values, visibility-aware polling, mobile readability), the Account sign-out, the JSON login throttle response, AI_SERVER_TRUST_PROXY_HEADERS support, and the updated NGINX configuration. Verified over HTTPS afterwards: /, the manifest, icons, fonts, and js/visibility.js return 200 with correct content types; the CSP header shows style-src 'self', font-src 'self', and manifest-src 'self' with no Google origins; /docs and /openapi.json return 404 and /api/memory returns 401. The backend answers on loopback. The shared-address lockout test then passed: failed sign-ins with a made-up username on an iPhone produced the readable "Too many login attempts" message, and labadmin still signed in on the desktop, so the client address behind the proxy is now seen correctly. iPhone home-screen install then worked, with one caveat: until full trust was enabled for the lab CA (Settings → General → About → Certificate Trust Settings), iOS showed a plain letter instead of the icon because it fetches the touch icon separately from the page. Enabling that trust fixed it.

Container health (nested lab, release 20261106-nonblocking, 2026-11-05). Both containers showed healthy with a failing streak of 0 after 31 minutes up, both Quadlet units were active with NRestarts=0, and the containers' own periodic probes (30 s interval) passed repeatedly. The probe URLs (http://127.0.0.1:8000/ and :8001/) returned 200 in about 1-2 ms from the host. Running podman healthcheck run and podman exec as ai-server through runuser fails (crun: write file …/cgroup.procs: Permission denied) because that shell has no systemd user session; the probe never starts, and the container is briefly marked unhealthy until the next periodic probe passes. That is an artifact of the invocation, not an application fault. Use podman ps or podman inspect for health. Open: running the check from a proper user session (for example machinectl shell ai-server@, unverified), and health after a guest reboot.

A follow-up release (20261106-nonblocking) fixed /api/system blocking the backend's event loop. With two dashboards open, the backend stalled long enough for sign-in to hang on "Signing in…" and for requests to time out. The release used the same procedure and no step failed. Afterwards, 30 consecutive requests to the backend's auth status endpoint on loopback returned 200 in 1-3 ms. /api/system itself was not probed directly (it needs a session); the open dashboard's polling kept the backend responsive. and post-boot cleanup (opt-in; lab procedure)

Auto-start is deliberately not in the base Quadlet units. It is an explicit Quadlet drop-in (deploy/boot-autostart/*.container.d/10-boot-autostart.conf, WantedBy=default.target), so removing the drop-in returns the guest to manual start. The ai-server account already has lingering enabled in the nested lab (/var/lib/systemd/linger/ai-server), so its user manager starts at boot without a login. Check that before relying on this on another host.

A separate, optional user timer prunes old lab build directories and build logs. deploy/systemd/ai-server-housekeeping.timer runs 15 minutes after the user manager starts and then daily. The root-owned script scripts/proxmox/ai-server-housekeeping.sh exits without deleting anything unless both containers report healthy. It only removes build-YYYYMMDD-<id> directories and build-YYYYMMDD-<id>.log files directly under /var/lib/ai-server that are older than 7 days, always keeps the newest build, never follows symlinks, and never touches deploy-backups, SQLite files, application logs, or images. Use --dry-run to list what it would remove. Logs written to /tmp by older runs are root-owned and are not covered; remove them manually.

Install (operator, as root on the guest; take a guest checkpoint first). Stage a release bundle first so the files are under ~ai-deploy/releases/$ID/src:

cd /
ID=RELEASE
S=/home/ai-deploy/releases/$ID/src
AS() { runuser -u ai-server -- env XDG_RUNTIME_DIR=/run/user/999 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/999/bus HOME=/var/lib/ai-server "$@"; }
Q=/var/lib/ai-server/.config/containers/systemd
U=/var/lib/ai-server/.config/systemd/user
test -f /var/lib/systemd/linger/ai-server && echo "linger on"

install -d -o root -g root -m 755 /usr/local/libexec/ai-server
install -o root -g root -m 755 $S/scripts/proxmox/ai-server-housekeeping.sh /usr/local/libexec/ai-server/
for n in backend dashboard; do
  install -d -o ai-server -g ai-server -m 755 "$Q/ai-server-$n.container.d"
  install -o ai-server -g ai-server -m 644 $S/deploy/boot-autostart/ai-server-$n.container.d/10-boot-autostart.conf "$Q/ai-server-$n.container.d/"
done
install -d -o ai-server -g ai-server -m 755 "$U"
install -o ai-server -g ai-server -m 644 $S/deploy/systemd/ai-server-housekeeping.* "$U"/
AS systemctl --user daemon-reload
AS systemctl --user enable --now ai-server-housekeeping.timer
AS /usr/local/libexec/ai-server/ai-server-housekeeping.sh --dry-run
ls /run/user/999/systemd/generator/default.target.wants/ 2>&1

The generator directory should list both ai-server-*.service units. Then reboot the guest and verify from ai-deploy without touching the services: uptime is small, both containers are healthy with no manual start, GET / returns 200, /docs returns 404, and the dashboard still sends CSP. Also confirm systemctl --user list-timers (as ai-server) shows the housekeeping timer.

Disable and return to manual start:

AS systemctl --user disable --now ai-server-housekeeping.timer
rm -f "$Q"/ai-server-*.container.d/10-boot-autostart.conf
AS systemctl --user daemon-reload

Status: executed on the nested guest on 2026-11-05 (nested-lab evidence only). After install and a guest reboot with no manual start, both containers were healthy within about 2 minutes, ports 8000/8001 were bound to loopback only, / returned 200, /docs and /openapi.json returned 404, the dashboard sent CSP and X-Frame-Options: DENY, both Quadlet units reported generated, and the housekeeping timer was scheduled. The first housekeeping run after boot (about 13 minutes later) was not yet observed; the pre-reboot --dry-run removed nothing, as expected. A single reboot is not host-reboot acceptance: unclean power loss, boot ordering against network/storage, and the physical host remain untested. A forced power-off of the guest (Hyper-V Turn Off, no clean shutdown) was also tested the same day: both containers returned healthy unattended, the docs routes stayed 404, the dashboard kept CSP and X-Frame-Options: DENY, and PRAGMA integrity_check returned ok for all four SQLite files under /var/lib/ai-server. A hypervisor power-off is not physical power loss (no disk write-cache loss). The ai-server journal could not be read by the checking account, so startup logs were not reviewed.

Database location finding (nested lab, 2026-11-05): after the docs-off redeploy the backend used empty databases under /var/lib/ai-server/AI-Server/, while the real data (a local user, a Proxmox mapping, a session, conversations, and diagnostics) remained under /var/lib/ai-server/data/. Cause: since the backend routes moved into the api/ package (commit cb525a4), api/runtime.py loaded .env from ai-backend/ instead of the project root. The mounted /opt/ai-server/.env was therefore never read, so its path and Proxmox settings did not apply and the defaults (HOME-based) were used. The loader is fixed (PROJECT_ROOT, with tests) and was redeployed to the lab as 20261105-dotenv. Verified in the running container: AI_SERVER_AUTH_DB_PATH, AI_SERVER_MEMORY_DB_PATH, and AI_SERVER_LOG_DIR are set from .env and the backend resolves /var/lib/ai-server/data/{auth,memory}.sqlite3 and /var/lib/ai-server/logs; both containers healthy; docs routes 404; anonymous protected route 401; dashboard CSP present. An authenticated login was later verified in the lab, and the admin-only signed-bundle test also required an authenticated session. Live database row counts have not been independently reviewed. The /var/lib/ai-server/AI-Server/ pair is empty and unused; do not delete it until you have reviewed and archived it. A verification that imports only config (as podman exec ... from config import settings does) does not load .env and shows defaults; import api.runtime first.

Before implementing the installer, define and test the following behavior:

• A read-only preflight/plan mode that identifies the supported OS/Proxmox and architecture, required commands and files, disk needs, existing units/images, service account, and intended loopback ports. It must not mutate the host.
• An explicit install/deploy mode that shows its actions and requires confirmation before package, account, file, image, or service changes. Install only pinned, reviewed app prerequisites; do not upgrade the host or alter Proxmox kernel packages.
• Safe secret/config handling: accept protected environment and CA files from an operator-managed path, validate their presence and permissions without printing their contents, and never bake secrets into the image, source bundle, command line, or generated report.
• Versioned image and unit deployment with a retained previous image/config snapshot, health/API checks, and a tested rollback path. Do not remove or overwrite persistent application data as part of rollback.
• Loopback-only service bindings by default and no boot auto-start unless a separately reviewed option is explicitly selected. Report exact versions, paths, unit names, bindings, results, and remaining manual steps.

Treat backend/dashboard deployment as the initial scope. SearXNG should remain an optional, separately reviewed component; NGINX/TLS, Qdrant, llama.cpp, ROCm, ONNX Runtime, and ComfyUI are not ready to be included as automatic install targets. In particular, do not bypass the documented GPU compatibility gate. The nested-lab rollback test restored the same image build it started with; rollback between distinct application versions still needs testing. Until the deployment contract, pinned artifacts, and recovery path are verified in the lab, keep the installer planned and perform deployment steps manually.

LAN access via NGINX with TLS (executed on the nested lab 2026-11-05)

Status: installed on the nested lab guest and verified from a Windows PC and an iPhone on the same LAN, after each imported the local CA. Verified: HTTP redirects to HTTPS, the certificate validates against the imported CA, dashboard sign-in works over HTTPS, 8000 and 8001 are unreachable from the LAN, and /docs stays 404. The /ws/chat WebSocket connects through the proxy (origin and session cookie accepted; the lab has no llama.cpp, so a chat message returns the backend's "Cannot reach llama.cpp" error and token streaming is untested). The initial client-address ambiguity was later mitigated with AI_SERVER_TRUST_PROXY_HEADERS=true; the iPhone lockout test then stopped affecting the desktop administrator. Certificate renewal is not yet verified. Scope is a trusted single-LAN household audience, using a local CA. The backend (8000) and dashboard (8001) stay bound to 127.0.0.1; NGINX is the only listener exposed to the LAN, on 80 (redirect) and 443.

Artifacts:

• deploy/nginx/ai-server.conf: TLS 1.2/1.3 server, same-origin proxy (/api/ and /ws/ to 127.0.0.1:8000, everything else to 127.0.0.1:8001), WebSocket upgrade, per-client login rate limit, short-lived HSTS, 2 MB general body limit, and a dedicated /api/admin/updates location configured for 130 MB uploads (large-upload behavior remains unverified).
• scripts/proxmox/make-lan-cert.sh: creates a local CA (EC P-256, 5 years) constrained by X.509 name constraints to the supplied names, and a server certificate (397 days). Refuses to overwrite, writes keys 0600, prints no key material, and ends with openssl verify.

Procedure (placeholders: LAB_HOST is the guest's LAN IP or DNS name). Take a guest checkpoint first. Run as root on the guest console, starting bash on its own line and cd / before other commands:

apt-get install -y nginx
rm -f /etc/nginx/sites-enabled/default
bash /home/ai-deploy/releases/<release-id>/scripts/proxmox/make-lan-cert.sh --ip LAB_HOST
install -m 0644 /home/ai-deploy/releases/<release-id>/deploy/nginx/ai-server.conf /etc/nginx/conf.d/ai-server.conf
nginx -t && systemctl enable --now nginx

Use --dns NAME instead of, or in addition to, --ip. Then, as the operator (the file holds secrets, so edit it directly), set AI_SERVER_CORS_ORIGINS=https://LAB_HOST in /opt/ai-server/.env and restart the backend unit. Also set AI_SERVER_TRUST_PROXY_HEADERS=true in the same file. Without it the backend sees every LAN device as one address, and five failed logins from any device lock out all sign-ins for 15 minutes (observed on the lab 2026-11-05). Restart the backend afterwards.

Copy only /etc/ai-server/tls/ca.crt to client devices and import it as a trusted root. Never copy ca.key or server.key.

Verification checklist for a new installation:

1. From a separate LAN client: https://LAB_HOST/ loads without a warning, login succeeds, and the chat WebSocket streams.
2. http://LAB_HOST/ redirects to HTTPS.
3. From the LAN client, ports 8000 and 8001 are unreachable.
4. Check which client address the backend sees behind NGINX and rootless Podman port forwarding. The backend login limiter keys on request.client.host; NGINX limits per client address as a mitigation.

Lessons from the lab run: apt install nginx starts NGINX with the Debian default site, so use systemctl restart nginx (not only enable --now) after installing the conf. The CA certificate is /etc/ai-server/tls/ca.crt (only the key is under ca/). On iPhone, installing the profile is not enough: enable full trust under Settings → General → About → Certificate Trust Settings. On Windows, curl may need --ssl-no-revoke, because the local CA has no revocation endpoint.

Rollback: systemctl disable --now nginx, remove /etc/nginx/conf.d/ai-server.conf, and restore AI_SERVER_CORS_ORIGINS. Because the app services never left loopback, no application change is needed.

Caveats: the CA key lives on the server (root-only); move it offline when practical. Server certificates need re-issuing before expiry; clients keep trusting the same CA. Browsers ignore HSTS for IP-literal hosts. Review firewall rules so only 80/443 are reachable.

Signed application updater — staged implementation

Stage 1 (verify-only) is implemented and deployed to the nested lab: administrators can upload a signed .aibundle, verify and stage it, and discard it from Settings → Setup / Deployment. It does not apply or extract the release into the live deployment. Automatic Apply (Stage 2) is not implemented. Its scope is the AI Server backend and static dashboard application only; it must not update Proxmox, Debian packages, kernel, ROCm/HIP, inference runtimes, SearXNG, or operator-managed configuration and secrets.

The Stage 2 target is a signed manifest and a prebuilt, architecture-specific OCI image archive. This differs from the current Stage 1 source-tree payload. The future image archive contains no source build instructions, environment files, CA/private keys, runtime databases, user data, or host package changes. The manifest identifies the release, target architecture, image digest, and data-schema compatibility. A detached Ed25519 signature authenticates the canonical manifest and image digest; the verification public key is provisioned and rotated separately by an operator. The bundle must not be able to add or replace trusted keys.

Treat every uploaded archive as hostile until verified. Enforce a configured upload-size limit; ignore the supplied filename; reject absolute paths, traversal, duplicate entries, links, special files, unexpected members, and decompression-size overflow. Verify the signature, manifest schema, target, and image digest before loading an image or writing deployment files. Stage uploads outside the dashboard web root with restrictive permissions, expire unused uploads, and do not expose archive contents or secrets in logs/errors. Validation and Apply require an authenticated admin, allowed Origin, and CSRF token. The UI shows the exact release, image digest, target, disk requirements, current version, and rollback plan before confirmation.

Applying an update must be handled by a separate updater because the API process will be restarted during deployment. Its fixed operations are limited to validating a staged release, loading/tagging its image, coordinating only the backend/dashboard Quadlet services, checking their health, and restoring the retained prior image on failure. It must not execute bundle-provided scripts or arbitrary commands, change host packages or service bindings, change the boot auto-start policy, or delete persistent application data. Its status channel must not be able to request unrestricted shell execution. The exact privilege model (rootless service-account helper versus a narrowly scoped root-owned broker) remains undecided and must be settled before Stage 2.

Before restart, retain the prior image and deployment metadata and create a consistent backup of application databases. Keep the live .env, CA bundle, and persistent state outside the image and release bundle. Releases must preserve backward compatibility with the previous application/data schema; destructive migrations are excluded until a separately tested data-restore contract exists. Restart the backend and dashboard, verify container health and HTTP endpoints, and only report success after all checks pass. On failure, restore the previous image, restart both services, and verify recovery. Retain a bounded number of verified releases/backups according to an explicit disk policy; never remove the only known-good version or the most recent database backup. If rollback or recovery checks fail, report an actionable failed state and require out-of-band operator recovery.

This is a staged design. Stage 1 (verify-only) is implemented in the repository but only partly lab-verified (see the 2026-11-05 record below): the admin-only upload/verify/discard routes (docs/backend/api.md), the Settings → Setup / Deployment Verify flow, scripts/release/sign-release.py (keygen, build, verify), and a dedicated NGINX location for the large upload. Stage 1 deliberately differs from the design above in two ways: the payload is the existing allow-listed source-tree bundle (payload.type: source-tree) rather than a prebuilt OCI image archive, because no image can be built on the Windows workstation; and the trusted public keys come only from the server .env (AI_SERVER_UPDATE_PUBLIC_KEYS), so a bundle cannot introduce its own key. The bundle is an uncompressed tar of manifest.json, manifest.sig and payload.tar.gz. The operator still applies a staged release at the console using the procedure in this guide. Stage 2 (a separate least-privilege helper that performs backup, build, health gate, switch and automatic rollback) is not started. The helper's privilege model must be resolved. Health and reboot-recovery checks have passed in the nested lab, but should be repeated after the latest release; rollback between two distinct releases remains an outstanding gate. The Apply button stays disabled. The signing private key is created with keygen, encrypted by a passphrase typed at the prompt, and must live outside the repository and off the server. Replay protection, key rotation and an OCI payload remain open. Do not claim atomicity or successful rollback until tested on the nested lab and then the physical host.

The source-tree .aibundle is intentionally not a model delivery format. The allow-listed release builder excludes model files, and the current 128 MiB upload / 256 MiB unpacked limits are unsuitable for large GGUF weights. Models are persistent data under the configured AI_SERVER_MODEL_DIR, outside the application image and release rollback. Planned, not implemented: a separate admin-only signed model-package workflow. It must verify a model-specific manifest and file digest before installation, check free space, install without overwriting or deleting existing models, and never load a model automatically. Keep its verification and recovery behavior independent from application updates; do not raise application-bundle limits to carry model weights.

Stage 1 lab record (nested lab only, release 20261106-updater-verify, 2026-11-05). The release followed the lab procedure. The import gate, extended to also import release_bundle (so the new cryptography dependency is checked before any switch), printed import ok. Both containers reported healthy about 45 s after the switch, GET / on the dashboard returned 200 and /docs returned 404. After the operator added a trusted public key to the server .env, a signed test bundle (release ID 20261106-verify-test, 584 KB, wrapping the 72-file source tree) was uploaded from Settings → Setup / Deployment and reported as verified, with the signer fingerprint, a payload SHA-256 that matched the locally built tarball, 72 files, and a 24-hour expiry, and Apply stayed disabled. The operator then pressed Discard on that release and reported that it appeared to work (observed in the page only; the staging directory and the follow-up GET /api/admin/updates listing were not inspected). Not yet exercised on the lab: rejection of a tampered or wrong-key bundle, expiry purge, an upload near the 128 MiB cap through NGINX (the test bundle was far below the default body limit, so the dedicated upload location is not proven by it), and whether the NGINX configuration was reloaded with the new location. Container health was checked about 45 seconds after the switch and both containers were healthy; reboot recovery after this release was not re-checked.

The backend and Discord bot load project-root .env values with python-dotenv when they start; shell-provided environment values take precedence. The Windows .bat launchers do not source .env for their llama.cpp command-line flags. See configuration documentation for the template and limitations.

Recommended transition sequence

1. Inventory and decision record: document the Proxmox VE 9.x point release/kernel, host deployment choice, RX 6800 XT device, network exposure, storage, recovery method, and desired component versions. Do not begin ROCm installation until the exact target combination is checked against current upstream support information.
2. Prepare the host safely: establish console/SSH recovery, backups, and rollback; update the host using the chosen Proxmox procedure. Review an idempotent preflight/install plan before enabling package repositories, changing kernel/driver packages, or granting GPU device access.
3. Validate the compute stack: install a verified ROCm/HIP stack on the host and confirm the GPU is enumerated. Build llama.cpp with its ROCm backend; run a small-model inference through llama-server, then test Qwen2.5-Coder-7B and Qwen3-8B individually with VRAM headroom and repeatable prompts. Before enabling the proposed operating profiles, also measure Qwen2.5 1.5B Utility residency together with each profile model, including context/KV cache, runtime overhead, system RAM, steady-state load, latency, and model-swap recovery. Do not assume the 16 GB card can support either combination until measured.
4. Port and validate telemetry: implement provider selection by platform, verify the API response schema is unchanged, and test idle/inference load and memory against an independent tool. Test missing permissions and missing telemetry utilities; the dashboard must show unavailable readings explicitly.
5. Deploy supervised services: configure model directory, runtime URL, exact device, backend environment, startup dependencies, logs, service account, and health checks. Validate model switching, startup failure, rollback to the previous model, and reboot recovery.
6. Deploy the web interface safely: choose same-origin proxy or a configured reachable API URL; test chat, model inventory, switching, and telemetry from both the host and a separate client. Keep llama.cpp loopback-only; restrict API reachability until authentication and network policy are in place.
7. Deploy supporting services in stages: install Podman/Quadlet and validate one unit at a time. Bring up NGINX and Qdrant with persistent storage and restricted network bindings. Add ONNX Runtime ROCm EP and ComfyUI in isolated, version-pinned environments/containers after compatibility checks; verify each against a representative workload. Deploy SearXNG and Discord only after their endpoint and secret handling are configured.
8. Record operational handoff: save exact versions, commands/build flags, service names, paths, environment variable names (not secret values), firewall/proxy design, GPU evidence, test results, backup/restore steps, and known limitations in this repository's deployment docs.

Linux acceptance checklist

• [ ] Exact Proxmox VE 9.x point release/kernel and host deployment decision documented.
• [ ] RX 6800 XT driver/runtime combination verified against current upstream support and device enumerated by the chosen llama.cpp build.
• [ ] ROCm/HIP llama.cpp llama-server, ONNX Runtime ROCm EP, and ComfyUI versions/builds recorded; each installed component passes its own GPU-backed smoke test.
• [ ] Podman/Quadlet units survive restart/reboot with persistent data and least-privilege GPU access; NGINX and Qdrant expose only intended interfaces.
• [ ] Small-model inference and both 7B/8B models tested individually; logs confirm GPU offload and adequate VRAM headroom.
• [ ] /api/system returns valid CPU/RAM and live GPU fields; unavailable GPU measurements are null, with an explicit status/error.
• [ ] /api/chat, /api/models, model switching, and recovery tested after service restart and host reboot.
• [ ] Linux backend unit tests pass; backend, llama-server, and dashboard integrate under their documented service lifecycle with persistent model/log paths and correct ownership.
• [ ] Dashboard works from a separate client; API origin, proxy, CORS, WebSocket routing, authentication, firewall, and TLS decisions documented and tested.
• [ ] Observe-only diagnostics API and dashboard operate against live host checks and never perform remediation; verify backend logs and all other protected routes remain accessible only to their authorized dashboard roles through the deployed proxy.
• [ ] llama.cpp remains loopback-only and only the intended services are reachable from approved networks.
• [ ] SearXNG, if deployed, returns JSON results; secrets are stored outside the repository.
• [ ] Discord bot, if deployed, reaches the configured API, reports heartbeat, and handles intended commands without exposing its token.
• [ ] Backup/restore and recovery access verified; versions and known gaps recorded for handoff.

Related documentation

• GPU integration plan
• Windows testing environment
• Architecture
• Backend API