Make a task-specific shortlist
The linked Llama and Qwen cards describe small instruction-tuned models and their use conditions. Treat them as candidates, not measured winners. Check the exact model version and artifact rather than assuming all derivatives share the base model's behavior or terms.
Budget for the whole device
A local application shares memory and compute with the operating system and other processes. Record the artifact, runtime, context and background workload. Advertised maximum context does not establish what your device can sustain.
Test useful outputs and sustained operation
Create a small permitted evaluation set with explicit acceptance rules. For structured extraction, validate fields and unsupported answers. Repeat after the device has been running, and record failures, completion time and memory. This guide publishes no measured Raspberry Pi or Apple Silicon speed figures.
Keep deployment boundaries explicit
Document where data is processed and whether any fallback sends it to another service. A local first stage does not make a cloud fallback private. See the local versus cloud cost guide and VRAM reduction checklist before scaling the workload.
Frequently asked questions
Can I choose an edge model from parameter count alone?
No. Compare the exact artifact, supported runtime, context, license, available device memory and results on your task. Parameter count does not establish speed or answer quality.
Does a model's advertised context window fit on every edge device?
No. Context support is not a memory guarantee. Start with the context your task needs and measure the complete process, including the operating system and other applications.
Sources & further reading
Primary sources checked Oct 17, 2026. Vendor statements are attributed; editorial advice is our own.
- 1
- 2
Help us keep this useful. Report a factual correction →
