How to Set Up an AI Server for Self-Hosted LLMs
A practical guide to running open-source AI models on your own server — choosing a model, sizing CPU or GPU, serving with Ollama or vLLM, securing the API and keeping costs under control.
Editorial disclosure: Pyalm publishes and maintains the products and free tools discussed on this site. Regulatory statements are linked to primary sources where applicable. No independent professional review is claimed unless a reviewer is explicitly named.
Public AI APIs are the fastest way to start with AI. But when customer, financial or health data can't leave your environment, or API costs get too high, running models on your own server makes sense.
When self-hosting makes sense
- Data can't go to a third party (regulated, confidential or contractual)
- High, steady usage where per-token API costs add up
- You need control over the model version and behaviour
- You need offline or on-premise operation
Step 1: Choose the model
Open-weight model families such as Llama, Mistral, Qwen and Gemma come in different sizes. Smaller models (a few billion parameters) are fast and cheap. Larger ones are more capable but need far more memory. Test several on your tasks before deciding.
Step 2: Size the hardware
| Model size (approx.) | Typical memory need | Hardware |
|---|---|---|
| Small (3–8B), quantised | ~4–8 GB | Strong CPU server or a modest GPU |
| Medium (8–14B) | ~8–30 GB | A single GPU, e.g. 24 GB class |
| Large (30–70B), quantised | ~20–48 GB+ | High-memory GPU(s) |
Quantisation (4-bit or 8-bit) cuts memory needs a lot, with a small quality trade-off. Concurrent users need extra headroom.
Step 3: Serve the model
- Ollama: simple to run, great for internal tools and prototypes
- vLLM: high-throughput serving for many users
- llama.cpp: efficient CPU and edge deployments
Put the model behind an API your apps can call, often OpenAI-compatible, so switching between hosted and self-hosted is easy.
Step 4: Secure and operate
- Authentication and private networking (never expose the model openly)
- Logging and usage limits
- Monitoring of GPU memory, latency and errors
- Backups of configuration and any fine-tuned weights
Cloud GPU or own hardware?
Rent cloud GPUs to start and while usage is uncertain. Consider dedicated hardware when usage is high and steady. See GPU servers for AI.
---
Need servers set up or moved?
Pyalm handles AI server setup, server migration and database hosting for businesses in Dubai, the UAE and India: planned, backed up, monitored and documented.
💬 Talk to us on WhatsApp or call +971 56 938 0690.