Everyone is building on top of AI right now, and almost everyone is doing it the same way: sign up for an API key, send your data to someone else's cloud, and hope the bill stays reasonable. That works for a lot of use cases. But there is a growing category of work where it does not: internal documents, customer records, financial data, anything a legal or HR team would flinch at sending to a third party.
That is the situation I found myself in when scoping a multi-department knowledge platform. The source material included years of internal email archives and operational documents. Sending that to an external API was never going to pass review. So the question became: how far can you get running AI models entirely on your own hardware?
The answer, it turns out, is surprisingly far. This post covers what I learned building on Ollama, from hardware selection through to serving models behind a proper API.
What Ollama actually is
Ollama is an open source tool that runs large language models locally. Think of it as a package manager and runtime for AI models. You install it, pull a model with a single command, and it exposes an HTTP API on your machine that behaves a lot like the commercial APIs you may already know.
ollama pull qwen2.5:32b
ollama run qwen2.5:32bThat is genuinely the whole setup for a basic installation. Under the hood, Ollama handles model quantization formats, GPU and CPU memory management, and request queuing. It supports the major open model families: Qwen, Llama, Mistral, Gemma, DeepSeek, and dozens more.
The important shift in thinking is this: the model is now a file on your disk, and inference is now a process on your server. No usage billing, no rate limits, no data leaving the building.
Hardware: unified memory changes the math
The traditional assumption was that local AI meant an expensive NVIDIA GPU with as much VRAM as you could afford. That is still true for high-throughput workloads, but Apple Silicon has quietly become the most cost-effective entry point for small teams.
I run my setup on a Mac Mini M4 Pro with 48GB of unified memory. The key word is unified: the CPU and GPU share the same memory pool, so a model that needs 20GB to load simply uses 20GB of system memory. There is no separate VRAM ceiling to hit. A comparable NVIDIA setup with that much usable GPU memory costs considerably more and draws far more power.
As a rough guide for sizing:
- 7B parameter models (quantized): around 5 to 6GB of memory, comfortable on 16GB machines
- 14B models: around 10GB, workable on 24GB machines
- 32B models: around 20GB, needs 32GB or more to leave room for the OS and other services
- 70B models: 40GB and up, realistic only on 64GB+ machines or dedicated GPU servers
The rule of thumb: your model plus its context window must fit in memory with headroom to spare. If it does not, performance falls off a cliff.
Choosing a model: bigger is not always the answer
The open model ecosystem moves fast, and the temptation is to always run the largest model your hardware allows. In practice, the right choice depends on the task.
For my knowledge platform work, Qwen2.5 32B hit the sweet spot: strong reasoning, good multilingual handling, and solid instruction following, while still fitting comfortably in memory alongside PostgreSQL and the API layer. For simpler tasks like classification, summarisation, or extracting structured fields from documents, a 7B or 14B model is often indistinguishable in output quality and several times faster.
There is also a second category of model that matters just as much: embedding models. If you are building anything involving search or retrieval, you need a model that converts text into vectors. I use nomic-embed-text, which Ollama serves through the same API. It is small, fast, and good enough that I have never felt the need to swap it out.
Quantization: the trick that makes all of this possible
A 32B parameter model at full precision would need over 60GB of memory. The reason it runs in 20GB is quantization: storing the model weights at reduced precision, typically 4 bits instead of 16.
The quality loss from modern quantization methods is smaller than most people expect. For the majority of business tasks, a well-quantized 32B model outperforms a full-precision 7B model while using similar memory. Ollama pulls quantized versions by default, so you get this benefit without thinking about it, but it is worth knowing the trade-off exists when you compare local model output against cloud APIs running at full precision.
Serving models properly: Ollama behind FastAPI
Ollama's built-in API is fine for experimentation, but I would not expose it directly to users or other services. In my stack, it sits behind a FastAPI layer that handles authentication, request validation, logging, and prompt construction. The pattern looks like this:
import httpx
OLLAMA_URL = "http://localhost:11434/api/generate"
async def ask_model(prompt: str) -> str:
async with httpx.AsyncClient(timeout=120) as client:
response = await client.post(OLLAMA_URL, json={
"model": "qwen2.5:32b",
"prompt": prompt,
"stream": False
})
return response.json()["response"]This separation matters for three reasons. First, your application code never depends on Ollama directly, so swapping models or even swapping Ollama for another runtime is a one-line change. Second, you control what prompts actually reach the model, which is where retrieval context, formatting instructions, and guardrails get injected. Third, you get proper observability: every request logged, every failure traceable.
Where local models fit: retrieval-augmented generation
The most valuable pattern for local models in a business context is RAG: retrieval-augmented generation. Instead of expecting the model to know your business, you store your documents as vectors in a database, retrieve the most relevant chunks for each question, and hand them to the model as context.
My stack for this is PostgreSQL with the pgvector extension. Documents get chunked, embedded via nomic-embed-text, and stored alongside their metadata. A user question gets embedded the same way, pgvector finds the nearest chunks with a similarity search, and those chunks go into the prompt.
The appeal of doing this locally is that the entire pipeline is private. The documents never leave your infrastructure, the embeddings never leave, and the questions never leave. For departments handling sales records, logistics data, or finance documents, that is not a nice-to-have; it is the requirement that makes the project possible at all.
Honest limitations
Local models are not a free replacement for frontier cloud models, and pretending otherwise leads to disappointing projects. A few realities worth stating plainly:
- Raw capability: a quantized 32B model is impressive, but it is not at the level of the largest commercial models for complex reasoning or long, nuanced writing.
- Throughput: one Mac Mini serves one team well. It does not serve a thousand concurrent users. Scaling local inference means real infrastructure planning.
- Context windows: long contexts eat memory fast. A model that fits comfortably with a 4K context may struggle at 32K.
- Maintenance: you own the stack now. Model updates, runtime upgrades, and performance tuning are your job, not a vendor's.
My working rule: use local models where privacy, cost predictability, or data residency are the deciding factors, and use cloud APIs where you need maximum capability on public or low-sensitivity data. Most organisations end up needing both.
Getting started this week
If you want to evaluate this for your own organisation, the path is short:
- Install Ollama on any reasonably modern machine with 16GB or more of memory
- Pull a mid-size model like qwen2.5:14b and test it against your actual use cases, not benchmarks
- Pull nomic-embed-text and prototype a small retrieval flow over a folder of your own documents
- Only after the prototype proves useful, size proper hardware and put a real API layer in front of it
The barrier to entry for private, self-hosted AI has collapsed. Two years ago this required a research team and a server room. Today it requires an afternoon and a machine you may already own. For any organisation sitting on sensitive data it cannot send to the cloud, that changes what is possible.
City Project Solutions builds enterprise systems, ERP integrations, and practical AI solutions for businesses in the UAE and Sri Lanka. If you are exploring private AI for your organisation, get in touch.