Local Qwen inference server

Objective

Operate a language model on hardware I own, with control over the inference backend, model files, and chat interface. This project is the local foundation for Drabsnarg AI.

Current state

The local system has been tested with an OpenAI-compatible API. The installed software and hardware below are owner-reported facts. This write-up does not claim that external access has been deployed or independently validated.

The project remains active: a locally working stack is a starting point for further measurement, operating procedures, and secure remote access.

Hardware and software

Architecture

Open WebUI provides the browser interface. It sends model requests to the local llama.cpp service, which loads and runs the Qwen model. The browser does not need direct access to the inference endpoint.

Local browser
    ↓
Open WebUI
    ↓  OpenAI-compatible requests
llama.cpp
    ↓
Qwen3-Coder-30B-A3B-Instruct

The public portfolio has a separate hosting lifecycle and does not depend on this machine being online.

Constraints and observations

Both CPU-only and Metal GPU inference have been tested. GPU acceleration on the Vega 64 has not reliably improved performance in the owner’s testing.

That observation does not establish a universal CPU-versus-GPU conclusion. Model quantization, backend version, offload settings, context length, and workload can all change the result. Those conditions need to accompany any published measurements.

No throughput, memory-use, or quality scores are published yet. Exact model quantization and runtime settings are also awaiting documentation.

Validation still to document

• Record exact llama.cpp and Open WebUI versions.
• Record the model file, quantization, and configured context.
• Measure prompt processing and output generation separately.
• Repeat the same workloads under CPU-only and Metal configurations.
• Check memory pressure, response quality, and sustained operation.
• Validate restart and recovery behavior.

Next improvements

Add a repeatable benchmark procedure, document configuration without secrets, and complete the separate secure remote-access project. Multi-user access will require capacity testing before any service expectations are set.