Local Testing Environment
This project uses two pre-physical-server environments. Windows remains the shared-application and GPU prototype environment; the RX 6800 XT is available there for Windows-side tests before it is moved to the server. An additional deployment/integration lab runs Proxmox VE 9.1-1 as a nested Hyper-V guest on the same PC. The nested guest is for Linux/Proxmox deployment preparation and integration testing, not physical-server acceptance.
Windows baseline
- Windows 11
- Python 3.12
- AMD Radeon RX 6800 XT
- ONNX Runtime DirectML for Windows-side ONNX tests
- Visual Studio 2022 Build Tools 17.14.41 with the C++ workload
- Vulkan SDK 1.4.363.0
The project Python environment is created outside the OneDrive workspace at
%LOCALAPPDATA%\AI-Server\venv to avoid synchronizing virtual-environment
files.
Nested Proxmox deployment lab
On 2026-10-16, the guest adapter was moved from Hyper-V NAT to an external
bridge. A reserved static LAN address and the guest's /etc/hosts node mapping
were updated so the configured Proxmox API hostname resolves to the guest.
Keep those values aligned if the lab network changes again.
The guest runs the dashboard and FastAPI backend from /opt/ai-server, with
runtime state under /var/lib/ai-server. The original 2026-10-16 deployment
used transient systemd-run units to start Uvicorn and a Python
standard-library static server as the unprivileged ai-server account, both
bound only to 127.0.0.1; both units were stopped and the ports verified
closed afterward. Dashboard and backend OpenAPI requests returned HTTP 200,
and an unauthenticated inventory request returned HTTP 401.
The guest now also has Podman 5.4.2 and rootless ai-server Quadlet
definitions. The backend and dashboard publish ports 8000/8001 only on
127.0.0.1; Windows cannot reach these loopback-only ports directly. Current
lab status (2026-11-05): boot auto-start and clean/forced guest recovery have
passed; release 20261106-updater-verify was healthy about 45 seconds after
the switch, / returned HTTP 200, and /docs returned 404. A signed
Stage 1 test bundle was verified through the dashboard; the operator reported
that Discard appeared to work. A separate two-release rollback, large upload
through the dedicated NGINX location, and reboot recovery after this latest
release remain unverified. NGINX/TLS was verified earlier from Windows and
iPhone. These are nested-lab results, not physical-host acceptance.
On 2026-10-17, both units were explicitly stopped: systemd reported inactive/dead, no managed containers remained, and ports 8000/8001 had no listeners. Both units were then started again; they reached healthy state and passed dashboard HTTP 200, backend OpenAPI HTTP 200, and anonymous inventory HTTP 401 checks.
A controlled podman kill ai-server-backend tested failure recovery. The
backend Quadlet service restarted once (NRestarts=1); after recovery, the
container was healthy, OpenAPI returned HTTP 200, and live Proxmox inventory
still returned one guest with only name, node, status, type, and
vmid. The dashboard remained healthy throughout, and both host ports
remained bound only to 127.0.0.1.
The configured start-limit boundary was then tested with four backend
container kills within 60 seconds. The first three failures triggered
automatic restarts; the fourth left the unit failed at the limit
(NRestarts=4). systemctl --user reset-failed followed by a manual start
restored the unit and container to active/healthy. OpenAPI returned HTTP 200,
anonymous inventory remained HTTP 401, and listeners remained loopback-only.
This verifies the configured repeated-failure limit and manual recovery path.
At this point in the 2026-10-17 checks, boot auto-start and reboot recovery had
not yet been verified; later nested-lab tests passed both.
Deployment rollback was exercised on 2026-10-17. Before changing the active
unit, the exact backend Quadlet was copied to
/var/lib/ai-server/deploy-backups/rollback-test-20261017/ and the current
image was tagged localhost/ai-server:rollback-known-good-20261017. A
temporary candidate referenced a deliberately nonexistent local image and
used Pull=never, so it could not fetch an image from a registry. The backend
unit failed after the configured start attempts; the dashboard stayed healthy
and returned HTTP 200. Restoring the saved Quadlet, resetting the failed unit,
and starting it manually brought the backend healthy again. OpenAPI returned
HTTP 200, anonymous inventory returned HTTP 401, and the live inventory
client returned one guest with the expected five fields. Both ports remained
loopback-only. The saved baseline is the same image build that was running
before the test; this verifies failure detection and snapshot restoration, not
rollback between two distinct application releases. The backup and known-good
image tag were retained on the lab guest.
The original host-side read-only inventory check ran as ai-server, loaded
the protected guest .env without displaying its contents, and successfully
retrieved one guest using the configured Proxmox token. The response contained
only name, node, status, type, and vmid. The lab-only
missing-CA-keyUsage compatibility setting was enabled with the explicit CA
bundle; certificate chain and hostname verification remained enabled.
The rootless container initially resolved pve.mshome.net to the guest LAN
address but received ConnectionRefusedError connecting to port 8006; the
same address and hostname both connected successfully from the host, where
pveproxy was listening on *:8006. The CA file was present and readable in
the backend container. Inside a throwaway container, host.containers.internal
resolved to 169.254.1.2 and accepted a TCP connection to port 8006.
Additionally, a throwaway container using
--add-host=pve.mshome.net:host-gateway connected through that address. The
backend Quadlet was updated with the corresponding AddHost mapping and the
backend service restarted. The configured inventory client then succeeded
inside the backend container, returning one guest with only name, node,
status, type, and vmid. Both backend and dashboard services were active
and healthy after the restart, and host ports 8000/8001 remained bound only to
127.0.0.1. This retains the configured API hostname and TLS hostname
verification. These are nested-lab results only. Boot auto-start and clean/
forced guest recovery were verified later on 2026-11-05.
On 2026-10-19, live dashboard authentication was tested with the Proxmox
identity ai-dashboard@pve, explicitly mapped to dashboard role viewer.
Login succeeded, remained active after reload, and viewer reads of system
status and VM inventory succeeded. A viewer request to a higher-role route
was denied with HTTP 403, but browser JavaScript could not read the denial
because the deployed backend did not add CORS headers to early auth responses.
The middleware ordering fix and an allowed-origin 403 regression test are
present in the Windows worktree. A subsequent image-build/deployment attempt
did not leave the backend reachable; restoring the retained source and image
snapshot returned the backend API to HTTP 200. The rollback is verified, but
the failed attempt's cause was not isolated, and the CORS fix is not currently
deployed. The live session used HTTP only over a loopback SSH tunnel with
Secure disabled; do not reuse that cookie setting for LAN/production.
Higher-role identities, HTTPS proxy behavior, and the deployed fix were
unverified at that time. Subsequent 2026-11-05 testing exercised HTTPS access
from Windows/iPhone and the proxy-aware login lockout fix. This historical test
does not establish physical-host behavior.
At that point the optional Discord bot and chat/inference were degraded
because their required services/runtime had not been configured. NGINX/TLS,
LAN application exposure, and persistent service supervision were subsequently
set up and tested in the nested lab, but llama.cpp/inference remains
unconfigured there. No nested-lab result validates physical hardware,
GPU/ROCm, network/storage topology, sustained performance, or physical-host
recovery. The non-root ai-deploy SSH account is intended for deployment work
and has no sudo access.
On 2026-11-05, after the guest rebooted (about 11 minutes before the check) and
the operator manually started the backend and dashboard services as root, a
read-only, unauthenticated smoke check ran from ai-deploy against the loopback
ports. Both ports were bound only to 127.0.0.1 and returned HTTP 200. Every
non-parameterized protected GET (16 routes) and every non-parameterized
mutating route (16 routes, empty JSON body) returned 401 without a session;
POST /api/auth/login returned 403 without an Origin header and with a
disallowed Origin. The public routes were / ({"status":"AI backend
online"}) and /api/auth/status ({"authenticated":false}, which also sends
an expiring HttpOnly; SameSite=strict cookie). Findings: the dashboard still
serves through the old SimpleHTTP server with no CSP or frame headers (the lab
image predates the CSP change), and /docs, /redoc, and /openapi.json are
publicly readable. Decision (2026-11-05): disable them by default
(AI_SERVER_ENABLE_API_DOCS) and move the backend health probe to GET /;
at the time of this check, that change was implemented and unit-tested but not
yet deployed. It was subsequently deployed in release
20261106-updater-verify, where /docs returned 404. This check did not cover SQLite
paths and permissions, logs, authenticated reads, or container health status,
because ai-deploy cannot read /var/lib/ai-server or the ai-server
account's journal. Services did not start on their own after this earlier
reboot, before boot auto-start was installed; a later controlled reboot and
forced guest power-off both recovered without manual service start.
Setup
Run setup-windows.bat from the project root. It creates the isolated Python
environment, installs the API dependencies and ONNX Runtime DirectML, then
checks that ONNX Runtime exposes DmlExecutionProvider, then runs a small ONNX
graph through DirectML.
Dashboard browser regressions
The Playwright suite exercises the native browser modules using mocked API responses. It covers module loading, sign-in gating and rejected credentials, administrator/operator/viewer UI controls, panel navigation, and persisted theme, accent, and sidebar preferences; it also checks VM inventory rendering, signed release staging controls, System Health Trends rendering, and Diagnostics report download/native-share behavior. The trends test verifies the current-session sample from mocked telemetry, not long-term storage. The native-share test mocks the platform API and does not verify the iOS Home Screen app on a physical device. These tests do not replace backend authorization tests or validate the nested Proxmox guest. From PowerShell, run:
npm install
npx playwright install chromium
npm run test:frontend
The test server binds only to 127.0.0.1:4173, serves an explicit allowlist of
dashboard assets, and is started and stopped by Playwright. Do not reuse these
mocked results as live API or deployment acceptance.
Portable diagnostics and infrastructure tests
The runbook retrieval and maintenance/remediation simulations are local-only portable behavior. They do not require the nested Proxmox guest, SearXNG, llama.cpp, or the physical server. From PowerShell, run the focused backend tests with the configured project interpreter:
Push-Location .\ai-backend
& "$env:LOCALAPPDATA\AI-Server\venv\Scripts\python.exe" -m unittest discover -s tests -p "test_diagnostic_*.py"
& "$env:LOCALAPPDATA\AI-Server\venv\Scripts\python.exe" -m unittest discover -s tests -p "test_maintenance_simulation.py"
& "$env:LOCALAPPDATA\AI-Server\venv\Scripts\python.exe" -m unittest discover -s tests -p "test_proxmox_contracts.py"
Pop-Location
The tests verify reviewed-runbook filtering and access, deterministic maintenance/remediation outcomes, proposal authorization/confirmation rules, and the read-only Proxmox inventory contract. Passing them does not validate live maintenance, Proxmox mutation, host recovery, or physical-server behavior.
Linux deployment and service lifecycle contract tests
The backend test suite also checks the checked-in Containerfile and Quadlet definitions without invoking Podman or systemd. It verifies the shared image and non-root container identity, loopback-only published ports, persistent backend state, read-only secret/CA mounts, health endpoints, bounded restart settings, and container hardening fields. The existing health-probe tests exercise successful, unavailable, and non-200 responses. Run from PowerShell:
Push-Location .\ai-backend
& "$env:LOCALAPPDATA\AI-Server\venv\Scripts\python.exe" -m unittest tests.test_quadlet_contracts tests.test_container_healthcheck
Pop-Location
These are portable source-contract tests, not Linux integration tests. They do
not validate Quadlet parsing/generation, service-account permissions, Podman
runtime behavior, service enablement, or reboot recovery. The host preparation
script's --check and --report modes were run on the nested guest on
2026-11-05 (see below) but remain unvalidated on the physical server.
On 2026-11-05 both modes were run over SSH as the non-root ai-deploy account
(key authentication through ssh-agent) by piping the script to bash -s. On the
Proxmox VE 9.1.1 guest (Debian 13, kernel 6.17.2-1-pve): --check exited 2 with
the ROCm gate blocked (no RX 6800 XT detected; GPU and Proxmox VE 9 not in
AMD's matrix), which is the expected fail-closed result; --report exited 0
and wrote an owner-only file with no hostnames or addresses. Neither mode
changed packages or configuration, and the temporary report was deleted from the
guest. The memory-label defect (MemTotal::) seen in the first run was fixed
and the corrected output was confirmed on the guest. This is
nested-lab evidence only; lspci is not installed on the guest, so display
details were unavailable, and installer modes were not run.
Start the backend with run-backend.bat. It listens on
http://127.0.0.1:8000.
Start the dashboard in a second terminal with run-dashboard.bat, then open
http://127.0.0.1:8001. Serving the dashboard over HTTP allows it to call the
local API from the browser.
Test the dashboard from a mobile device on the LAN
The normal launchers remain loopback-only. For a full mobile test (including login and API requests), use the opt-in HTTPS LAN launchers below. Do not disable secure cookies or expose the development servers to the public internet. Keep the PC and phone on a trusted private Wi-Fi network and do not configure router port forwarding.
- Find the PC's private Wi-Fi IPv4 address with
ipconfig(for example,192.168.1.50). Reserve that address in the router if possible so it does not change during testing. - Install and initialize mkcert on the PC. In PowerShell, generate a certificate whose subject includes that exact IP address:
powershell
$lanIp = "192.168.1.50"
$certDir = "$env:LOCALAPPDATA\AI-Server\certs"
New-Item -ItemType Directory -Force $certDir
mkcert -install
mkcert -cert-file "$certDir\lan.pem" -key-file "$certDir\lan-key.pem" $lanIp
Find mkcert's CA directory with mkcert -CAROOT. Keep lan-key.pem and
rootCA-key.pem on the PC. Install only the public rootCA.pem on the
phone and explicitly enable trust for it as required by that phone's
OS/browser. Never copy either private key to the phone. If the certificate
is not trusted, stop and fix trust rather than bypassing the browser
warning.
3. In the ignored project-root .env, configure these values using your
address and certificate paths:
dotenv
AI_SERVER_LAN_HOST=192.168.1.50
AI_SERVER_LAN_TLS_CERT=%LOCALAPPDATA%\AI-Server\certs\lan.pem
AI_SERVER_LAN_TLS_KEY=%LOCALAPPDATA%\AI-Server\certs\lan-key.pem
AI_SERVER_CORS_ORIGINS=https://192.168.1.50:8001
AI_SERVER_AUTH_COOKIE_SECURE=true
- Start
run-lan-backend.batandrun-lan-dashboard.batin separate terminals. They bind only to the configured private address and serve HTTPS on ports 8000 (API) and 8001 (dashboard). The dashboard server serves only the HTML, CSS, and explicitly allowlisted JavaScript modules; it does not expose the repository or.env. Openhttps://192.168.1.50:8001on the phone. To use the optional Discord bot while the LAN backend is running, setAI_SERVER_API_BASE=https://192.168.1.50:8000in.envand restart the bot. Its defaulthttp://127.0.0.1:8000points to loopback, where the LAN-only backend is not listening. Restore the loopback URL when returning to the normal backend launcher. - If Windows Firewall blocks the connection, open PowerShell as
Administrator and allow TCP ports 8000 and 8001 only on the Private
profile and only from
LocalSubnet; do not allow these ports on Public networks:
powershell
New-NetFirewallRule -DisplayName "AI Server LAN Test" -Direction Inbound `
-Action Allow -Protocol TCP -LocalPort 8000,8001 -Profile Private `
-RemoteAddress LocalSubnet
Stop both servers with Ctrl+C after testing. Remove the rule when finished:
powershell
Remove-NetFirewallRule -DisplayName "AI Server LAN Test"
The IP address in the certificate, .env, and browser URL must match. If the
address changes, regenerate the certificate and update the CORS origin. The
backend, llama.cpp, and SearXNG normal launchers remain loopback-bound; the
LAN backend can still reach those local services on the PC.
GPU load telemetry
On Windows, the system monitor reads the RX 6800 XT's GPU Engine performance
counters and displays the busiest engine's utilization. The adapter is matched
to its engine counters using its Vulkan device LUID. The dashboard shows
dedicated-memory usage from Windows GPU Adapter Memory counters and the
device-local capacity reported by Vulkan. Counters are sampled in a background
worker because Windows exposes many per-process engine counters; readings
refresh about every six seconds on this machine. If the adapter or counters
are unavailable, the UI shows N/A instead of simulated load or VRAM usage.
The Linux provider is tested on Windows using temporary sysfs fixtures. It
parses the standard AMD amdgpu utilization and VRAM counter files, returns
explicit unavailable results for missing, invalid, or ambiguous data, and
reads optional hwmon values only when the sensor is unambiguous. No live Linux
kernel or GPU is available in this test environment, so driver exposure,
permissions, device identity, and readings under inference remain unverified.
This is telemetry support, not ROCm/HIP runtime validation.
llama.cpp Vulkan chat test
The Windows llama.cpp build uses the upstream source at commit 8bb3cf0,
checked out under %LOCALAPPDATA%\AI-Server\src\llama.cpp. It was configured
and built with Visual Studio 2022 Build Tools 17.14.41, MSVC 19.44, CMake
3.31.6, and Vulkan SDK 1.4.363.0. The llama-cli and llama-server
executables are under
%LOCALAPPDATA%\AI-Server\build\llama.cpp-vulkan\bin\Release.
The dashboard chat now sends requests from FastAPI to the local
llama.cpp-compatible endpoint at http://127.0.0.1:8080. Download the Qwen2.5
1.5B Instruct Q4_K_M model to the model directory and verify its SHA-256:
mkdir "%LOCALAPPDATA%\AI-Server\models" 2>nul
curl.exe -L --fail --output "%LOCALAPPDATA%\AI-Server\models\qwen2.5-1.5b-instruct-q4_k_m.gguf" "https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct-GGUF/resolve/main/qwen2.5-1.5b-instruct-q4_k_m.gguf?download=true"
certutil -hashfile "%LOCALAPPDATA%\AI-Server\models\qwen2.5-1.5b-instruct-q4_k_m.gguf" SHA256
Expected SHA-256:
6a1a2eb6d15622bf3c96857206351ba97e1af16c30d7a74ee38970e434e9407e.
Start the local LLM server and leave its terminal open:
run-llama-server.bat
The launcher loads the model onto Vulkan0 with up to 99 layers offloaded,
listening only on 127.0.0.1:8080. In a separate terminal, start
run-backend.bat and run-dashboard.bat, then submit a prompt in the
dashboard. Each assistant response displays its request latency. If the model
server is stopped, chat returns a visible error rather than a stub reply.
The local Qwen2.5 1.5B model has been verified through the dashboard chat path;
Windows telemetry reports live GPU load and VRAM while it runs.
Explicit web research
The AI Chat panel includes a Research Web checkbox. When enabled, the
next message uses POST /api/research/stream; otherwise chat stays local and
does not search. The complete-response POST /api/research endpoint remains
available. Both expect SearXNG at http://127.0.0.1:8888 by default. Set
SEARXNG_BASE_URL before launching the backend to use another instance.
Research searches up to five sources and attempts to fetch full text from at
most three public HTTP(S) HTML pages. Each fetch is limited to one megabyte
and five seconds, revalidates up to three redirects, and rejects non-public
addresses and nonstandard ports. Unsafe, unavailable, and non-HTML pages fall
back to their SearXNG excerpts; source metadata reports page_status. The
stream returns source metadata before answer tokens, and completed turns are
saved with their citations. Automated unit and API tests cover these
boundaries. On 2026-11-04, a live Windows browser smoke test verified local
chat through the Qwen2.5 1.5B model (1.13 seconds) and Research Web through
the WSL SearXNG service to llama.cpp (6.02 seconds); the research response
included source links. The backend, dashboard, llama.cpp, and SearXNG
endpoints responded successfully during the test. This verifies only the tested
Windows/WSL path; it is not Linux/Proxmox or physical-host acceptance.
Search queries may be forwarded by SearXNG to its configured upstream search
engines, so avoid sending secrets or private data in Research mode.
SearXNG does not support native Windows, so the test environment runs it inside WSL (Ubuntu) with no root access needed. One-time install from the project root:
wsl bash scripts/searxng/install-searxng.sh
The installer downloads uv, a private Python 3.12 (SearXNG needs 3.11 or
newer), and SearXNG into ~/searxng in WSL, then creates ~/searxng/settings.yml
from scripts/searxng/settings.yml with a random secret key. The settings bind
to 127.0.0.1:8888, enable the json format the backend needs, and disable
the bot limiter, which is meant for public instances. Re-running the installer
updates SearXNG and keeps your settings. Start it with run-searxng.bat and
leave that window open; close it or press Ctrl+C to stop. Check it with:
Invoke-RestMethod "http://127.0.0.1:8888/search?q=test&format=json"
The Models page discovers .gguf files in
%LOCALAPPDATA%\AI-Server\models and marks the file reported by llama.cpp's
/props endpoint as loaded. Choose an installed model from AI Chat
dropdown or click its card on the Models page to switch the running server.
The backend verifies that the loopback port belongs to the expected
llama-server, restarts it with the selected file, and only reports success
after /props confirms the new model. A failed load triggers an attempt to
restore the previous model. This interrupts in-flight requests while the
server restarts. Set AI_SERVER_MODEL_DIR before starting the backend to use
a different model directory, and set LLAMA_SERVER_EXECUTABLE if the server
binary is not at its default build path. On Linux, model switching defaults to
the ROCm0 device; set LLAMA_CPP_DEVICE to a device name reported by that
llama.cpp build if it differs.
Additional 4-bit models for the RX 6800 XT are available in the same directory:
| Model | GGUF quantization | File size | SHA-256 |
|---|---|---|---|
| Qwen3-8B | Q4_K_M | 5,027,783,488 bytes (4.68 GiB) | d98cdcbd03e17ce47681435b5150e34c1417f50b5c0019dd560e4882c5745785 |
| Qwen2.5-Coder-7B-Instruct | Q4_K_M | 4,683,074,336 bytes (4.36 GiB) | 1664fccab734674a50763490a8c6931b70e3f2f8ec10031b54806d30e5f956b6 |
Both source model cards specify Apache-2.0 licensing. These are model-weight files, not tools or executable code. Each model is intended to load by itself on the 16 GiB GPU; VRAM is not shared by loading both into the current single-model llama.cpp server at once. Both files have been downloaded and verified against the SHA-256 hashes listed above. Short Vulkan inference tests at a 4096-token context loaded each model and offloaded all 37 Qwen3 or all 29 Qwen2.5-Coder layers to the RX 6800 XT. These smoke tests confirm the GPU execution path, not sustained performance or quality benchmarks.
These are individual-model tests with the current single-model llama.cpp server. The proposed deployment architecture keeps Qwen2.5 1.5B Utility AI resident alongside either Qwen3 8B or Qwen2.5 Coder 7B; that simultaneous residency, routing, and combined VRAM/RAM use have not been tested.
To start one of the installed models instead of the default Qwen2.5 1.5B model without using the dashboard, pass the exact filename to the launcher:
run-llama-server.bat Qwen3-8B-Q4_K_M.gguf
run-llama-server.bat Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf
After it starts, refresh the dashboard; the model inventory identifies the
newly loaded file. The launcher uses Vulkan, up to 99 GPU layers, and a 4096
token context. For Qwen3, it selects llama.cpp's built-in ChatML template
because the current pinned Jinja engine does not support an operation in
Qwen3's embedded template. Verify actual GPU offload and available VRAM in
the llama.cpp startup log and system monitor when testing each larger model.
Both launch paths request --reasoning off --reasoning-format deepseek.
Because --no-jinja bypasses Qwen3's embedded template, generation-time
suppression is not treated as reliable; the backend's standard/streaming
filter is authoritative. Do not use --reasoning-format none, which leaves
thought text in message.content.
To rebuild, open an x64 Visual Studio 2022 Developer Command Prompt and run:
set "VULKAN_SDK=C:\VulkanSDK\1.4.363.0"
set "PATH=%VULKAN_SDK%\Bin;%PATH%"
cmake -S "%LOCALAPPDATA%\AI-Server\src\llama.cpp" -B "%LOCALAPPDATA%\AI-Server\build\llama.cpp-vulkan" -G "Visual Studio 17 2022" -A x64 -DGGML_VULKAN=ON -DGGML_NATIVE=OFF -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_SERVER=ON
cmake --build "%LOCALAPPDATA%\AI-Server\build\llama.cpp-vulkan" --config Release --target llama-cli llama-server -j 8
The earlier tiny smoke-test model is
tinyllamas/stories15M-q4_0.gguf
from the ggml-org/models Hugging Face repository. Its SHA-256 is
66967fbece6dbe97886593fdbb73589584927e29119ec31f08090732d1861739.
With that model, the following command generated text and the verbose runtime
log confirmed offloaded 7/7 layers to GPU on the RX 6800 XT:
"%LOCALAPPDATA%\AI-Server\build\llama.cpp-vulkan\bin\Release\llama-cli.exe" -m "%LOCALAPPDATA%\AI-Server\models\stories15M-q4_0.gguf" -ngl 99 --device Vulkan0 -n 12 -p "Once upon a time" --single-turn --no-display-prompt --verbose
The tiny model verifies the Vulkan build and GPU-offload path; it is not a quality or production-model benchmark. The Qwen2.5 1.5B model is used for coherent chat and latency checks, not as a production model benchmark.
Scope and limitations
- The small DirectML smoke test validates a basic GPU inference path; it does not validate a particular vision or image-generation model.
- No ONNX model is bundled. Add a model and an inference path before measuring vision workloads.
- Image generation still needs a selected DirectML-compatible framework and its own compatible dependency environment.
- GPU backend choices for Linux/Proxmox are separate from this Windows setup.
- Linux AMD GPU telemetry parsing is implemented and fixture-tested; live Linux/Proxmox GPU telemetry and VRAM reporting remain unverified.
- Vision, image generation, and other unconnected workloads still use stubs.