Start with a question
“Is GPU inference faster?” is incomplete without a workload and a configuration. A useful comparison needs a model file, context length, backend version, hardware profile, and repeatable task.
This article describes a proposed procedure for the Drabsnarg local system. It contains no benchmark results.
Measure separate phases
Prompt processing and output generation are different workloads. Report their speeds separately rather than presenting a single number as overall model performance.
Also record time to first token when relevant to the user experience, memory use, and whether the output actually meets the task requirements.
Record enough context
| Field | Why it matters |
|---|---|
| Model and quantization | Establishes the exact model representation |
| Inference build | Backend behavior can change between versions |
| Context configuration | Changes memory requirements and workload |
| CPU/GPU configuration | Identifies what is being compared |
| Prompt and output lengths | Makes the workload interpretable |
| Sampling configuration | Helps explain output variation |
| Warm-up and repetitions | Separates startup effects from repeated operation |
| Memory pressure | Reveals resource constraints beyond throughput |
Proposed procedure
- Capture the configuration and stop unrelated heavy workloads.
- Use a fixed set of prompts with documented expected behavior.
- Perform a warm-up and record startup separately.
- Repeat measurements under equivalent conditions.
- Preserve raw output after removing sensitive material.
- Report variation, failed runs, and limitations alongside any summary.
The llama.cpp project includes benchmarking tools. Follow the options supported by the installed build rather than copying version-dependent commands without checking them.
Current local observation
The owner has tested CPU-only and Metal inference on the Vega 64 system and has not seen reliable improvement from GPU acceleration. That is an owner-reported observation, not a published benchmark comparison.
Exact settings and repeated measurements are needed before drawing broader conclusions.