What is Ollama?

Introduction

I first heard about Ollama when a colleague mentioned running GPT-like models on a laptop. My first reaction was the same one most engineers have: you can do that?

The short answer is yes. Ollama makes it possible to run large language models locally on consumer hardware. You need only a Mac, a PC, or even a Raspberry Pi and a few minutes to pull a model.

Ollama is an open source tool for running large language models on your own hardware. It wraps llama.cpp, a highly optimized C++ inference engine, and exposes a simple API that mimics the OpenAI chat endpoint. You pull models with a single command, talk to them with curl or any SDK, and the system handles GPU acceleration, memory management, and model loading automatically.

What this article covers: Ollama as a concept, how it works under the hood, the core ideas that shape how you use it, and when it is the right tool.

What this article omits: How to fine-tune models, how to train neural networks, or step-by-step setup guides for specific operating systems.

Prerequisites & Audience

Prerequisites: Basic familiarity with command line tools and what a large language model is.

Primary audience: Developers who have heard about running AI locally but have not dug into the tooling.

Jump to: The problem Ollama solves • How Ollama works • Core concepts • Common use cases • When Ollama is not the right tool • Misconceptions • References

TL;DR: What Ollama is in one pass

• Ollama is a local service that runs AI models on your own hardware.
• It uses llama.cpp for inference, which supports GPU acceleration on Apple Silicon, NVIDIA, and AMD GPUs.
• Models are pulled from a registry and stored as quantized GGUF files that fit on consumer hardware.
• The API is OpenAI compatible so existing tools work without changes.

The Ollama workflow:

This is the basic flow:

Learning Outcomes

By the end of this article, you will be able to:

• Explain what Ollama is and why it exists.
• Describe how Ollama loads and runs models on local hardware.
• Explain the core concepts: models, Modelfiles, templates, and the API.
• Identify when Ollama is a good fit and when it is not.

The problem Ollama solves

Before Ollama, running a local language model meant wrestling with Python dependencies, CUDA toolkit versions, and a patchwork of libraries. You needed PyTorch, transformers, and enough RAM to hold an unquantized model. That stack works for training models. It exceeds what you need for simple questions.

Ollama collapses that stack into a single binary. It installs, starts a background service, and lets you pull models with one command. The model runs on your hardware, your data stays on your machine, and you interact with it through a standard HTTP API.

The pain points Ollama addresses are real:

• API costs add up. Every call to a cloud API costs money. For development, prototyping, or high-volume tasks, those costs stack up fast.
• Data leaves your machine. When you send prompts to a cloud API, you are sending your data to someone else’s server. That is fine for public questions. It is not fine for code snippets, business logic, or anything with sensitive information.
• Setup is hard. llama.cpp existed before Ollama and has been around for years. But using it directly means compiling C++ code, managing model formats, and writing inference code. Ollama removes all of that.
• Cloud APIs go down. Rate limits, outages, and regional restrictions interrupt development workflows. A local model ignores the cloud provider’s status page.

Ollama ignores the problem of model quality. A local 8B parameter model falls short of GPT-4 or Claude Opus. For many tasks, the gap is small. Code completion, summarization, basic Q&A, and formatting data are tasks where a smaller model performs well enough, and the tradeoff of running locally is worth it.

How Ollama works

Ollama is a client-server application written in Go. When you install it, it starts a background daemon that listens on localhost:11434. That daemon is the heart of the system. It manages model loading, memory allocation, and request routing.

The architecture

The architecture is straightforward. A thin CLI sends HTTP requests to the local server, which loads models into memory and runs inference through llama.cpp. The server streams the response as token-by-token JSON.

The CLI is a thin wrapper. Running ollama run llama3.2 translates to an HTTP POST against the local server. The server checks for the model in its blob store, loads it into memory as needed, and passes the prompt to llama.cpp for inference.

Model loading and memory management

One of Ollama’s smartest features is automatic layer offloading. Large language models are split into layers, and each layer consumes a specific amount of memory. Ollama calculates how many layers fit on your GPU and offloads the rest to CPU RAM.

The process works in three phases:

1. Fit. Ollama calculates the memory requirements for the model without allocating anything.
2. Alloc. It reserves GPU VRAM and CPU RAM for the model.
3. Commit. It loads the weights into memory. This is the point of no return.

If a model is too large for your GPU, Ollama adapts. It calculates the optimal split between GPU and CPU layers and runs the model anyway, just slower. That split lets a 70B parameter model run on a MacBook Pro with 32GB of RAM.

Storage

Models live in ~/.ollama/models/ as content-addressed blobs. Each model weight file becomes a SHA256-hashed file, and multiple models sharing the same base weights (like llama3.2 and llama3.2:3b) deduplicate their shared layers. This saves significant disk space when you work with multiple variants of the same model family.

The dual runner system

As of 2025-2026, Ollama maintains two inference runners:

• llamaServer uses CGo bindings to llama.cpp and supports the widest range of models, including multimodal and thinking models.
• ollamaServer is a pure Go inference engine that provides more control and easier maintenance but currently supports only text-only models.

Ollama picks the runner automatically based on the model you request. You can force one with the OLLAMA_NEW_ENGINE environment variable.

Core concepts

Understanding Ollama comes down to a handful of concepts. Get these and you understand how the tool works.

Models

In Ollama, a “model” is a large language model packaged in GGUF format. GGUF (GPT-Generated Unified Format) is a binary format designed for efficient inference. Models come in different sizes, measured by parameter count: 3B, 8B, 70B, and beyond. Smaller models run faster and fit on more hardware. Larger models deliver more capability but require more RAM and GPU power.

You pull models from Ollama’s registry, which hosts thousands of community-contributed models alongside official open source releases from Meta, Google, Microsoft, and others.

ollama pull llama3.2

Tags

Each model has tags that specify variants. The same model base can have multiple tag versions. llama3.2 defaults to the 3B parameter variant. llama3.2:7b loads the 7B variant. llama3.2:latest pulls the most recent version. Tags let you choose between speed and capability.

Modelfiles

A Modelfile is a declarative configuration for a model, inspired by Dockerfiles. It lets you customize how a model behaves without retraining it.

FROM llama3.2

SYSTEM "You are an expert code reviewer. Provide structured, actionable feedback."

PARAMETER temperature 0.3
PARAMETER num_ctx 8192
PARAMETER top_k 50

The key instructions are:

• FROM specifies the base model.
• SYSTEM sets a persistent system message that shapes the model’s behavior.
• PARAMETER tunes inference settings like temperature (randomness), context window size, and top-k sampling.
• TEMPLATE overrides the prompt template, which controls how the model formats conversations.
• ADAPTER applies a Low-Rank Adaptation (LoRA) fine-tuned adapter for specialized behavior.

Modelfiles let you prototype custom model behavior without touching training code. Change the system prompt, adjust temperature, and you have a model tuned for code review, creative writing, or technical support.

Templates

Prompt templates define how conversations are structured. Each model has a default template that formats messages with special tokens. For example, llama3.2 uses:

<|start_header_id|>user<|end_header_id|>

What is Ollama?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

Templates matter because different models expect different formats. When you swap models, the template changes automatically. When you write a custom Modelfile, you can override the template to match a model’s expected format.

The API

Ollama exposes an HTTP API at http://localhost:11434 compatible with the OpenAI chat completions endpoint. Any tool built for OpenAI’s API works with Ollama out of the box.

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "What is Ollama?"}]
  }'

The API also supports streaming responses, tool calling, and Anthropic’s newer message format. This compatibility is one of Ollama’s biggest strengths. Existing applications work with local models without rewriting.

Embeddings

Ollama can generate vector embeddings from text, which are useful for semantic search, Retrieval-Augmented Generation (RAG) pipelines, and clustering. You generate embeddings with a dedicated endpoint:

curl http://localhost:11434/api/embed \
  -d '{
    "model": "nomic-embed-text",
    "input": ["Ollama runs local AI models."]
  }'

Embeddings models run fast on CPU and are smaller than language models. They add semantic search to local applications without spinning up a separate vector database service.

Common use cases

Ollama fits into workflows that need AI capabilities without cloud dependencies. Here are the most common patterns I have seen.

Local development and prototyping

When building applications that use LLMs, you need a model to test against. Running locally means you can iterate without API keys, rate limits, or internet connectivity. You spin up a model, send prompts from your app, and iterate. When you are ready to ship, you point the same code at a cloud provider.

Data-sensitive workflows

Legal documents, medical records, proprietary code. Anything that stays local is a candidate for local inference. Ollama keeps everything on your machine, and the data never leaves.

Automation and scripting

You can call Ollama from shell scripts, CI pipelines, or automation tools. Extract structured data from text, classify content, generate reports, or format output. The API is simple enough to call from bash, Python, or any language with an HTTP client.

Learning and experimentation

Ollama makes it easy to experiment with different models. Pull a model, try it, pull another, compare outputs. You can experiment with Modelfiles, adjust parameters, and see how changes affect behavior. Cloud APIs that charge per token make this kind of freeform experimentation costly.

RAG and semantic search

With embeddings support, Ollama can power local semantic search. Index your documents, generate embeddings, and search by meaning rather than keywords. This is practical for personal knowledge management or small-scale applications that prefer a single local service over a dedicated vector database.

When Ollama is not the right tool

Ollama handles many AI problems well, but falls short for others. Understanding its limits is as important as understanding its strengths.

You need maximum model quality. A local 8B parameter model falls short of 70B+ cloud models or proprietary models like GPT-4. If your task requires the highest possible reasoning, accuracy, or creativity, a cloud API is the better choice.

You need multimodal input. Ollama supports some multimodal models (vision) but only handles text and images. If your application needs to process audio or video, look elsewhere.

You need high throughput at scale. Local hardware has limits. A consumer GPU can handle a few concurrent requests before latency becomes unacceptable. If you are building a production service that needs hundreds of requests per second, Ollama on a single machine will not cut it. You would need a dedicated inference server with GPU clusters.

You need model training or fine-tuning. Ollama runs models. Creating custom models with Modelfiles is prompt engineering, not training. Training a model from scratch or doing full fine-tuning requires a different toolchain.

Your hardware is too weak. Ollama runs on CPU, but performance degrades quickly as models grow. If you have less than 8GB of RAM, even small models will feel sluggish. A machine with 16GB+ is a practical minimum for comfortable use.

Misconceptions

Several myths about Ollama and local AI circulate widely. Clearing them up helps you set realistic expectations.

“Local models are as good as cloud models.” They fall short. Parameter count, training data quality, and post-training alignment all favor cloud models. Local models excel at speed, privacy, and cost, not raw capability. The right model depends on the task.

“Ollama trains models.” Ollama loads and runs models. Modelfiles let you adjust prompts and parameters, which shapes behavior but leaves the model’s weights unchanged. Training and fine-tuning require different tools.

“You need a GPU to run models.” Ollama runs on CPU, and modern CPUs handle small to medium models reasonably well. GPUs speed things up, especially for larger models, but are optional. Apple Silicon’s unified memory architecture is particularly well-suited for running local models.

“Ollama is only for developers.” While developers are the primary audience, anyone who can use a terminal can run models with Ollama. The CLI is designed to be simple: ollama run model-name and start typing. Non-developers can use it for writing assistance, research, and creative tasks.

“Quantized models are useless.” Quantization reduces model precision to save space and memory. A Q4_K_M quantized 7B model uses about 4GB instead of 14GB, with a small but often imperceptible quality tradeoff. For most everyday tasks, quantized models match full-precision versions.

Future trends & evolving standards

The local AI space evolves quickly. Trends worth watching:

Smaller models getting better

Model architecture research (Mixture of Experts, distilled models, improved training data) closes the gap between small and large models. A 3B parameter model today outperforms a 7B model from a year ago. Ollama will run increasingly capable models on increasingly modest hardware.

What this means: Local models become viable for more tasks that currently require cloud APIs. New model releases and engine improvements arrive regularly, so keeping Ollama updated matters.

Multi-agent local workflows

Multiple models collaborating is an emerging pattern. One model handles planning, another handles code generation, a third handles review. Ollama supports running multiple models simultaneously, which makes it well-suited for this kind of setup.

What this means: Local AI will move beyond single-model chat into multi-model orchestration.

Edge AI integration

Ollama is already working on integrating with Edge Tensor Processing Unit (TPU) hardware to bring local inference to embedded devices. This extension brings Ollama beyond laptops and desktops into Internet of Things (IoT) devices and dedicated AI hardware.

What this means: Local AI runs on more than just computers.

Conclusion

Ollama is a local service that runs large language models on consumer hardware. It wraps llama.cpp, exposes an OpenAI-compatible API, and handles the complexity of model loading, quantization, and GPU offloading automatically. The core concepts (models, tags, Modelfiles, templates, and embeddings) cover how the tool organizes and customizes inference.

The tradeoff is clear: local models sacrifice raw capability for privacy, cost control, and development convenience. Understanding that tradeoff helps you choose Ollama when it fits and reach for cloud APIs when a cloud approach serves better.

Next steps

• Ollama documentation: API reference, Modelfile options, and model registry
• llama.cpp repository: The inference engine behind Ollama
• GGUF format documentation: Technical details about the model weight format

References

Official documentation

• Ollama documentation: The official reference for API endpoints, Modelfile options, and model registry.
• Ollama on GitHub: The open source project source code, issues, and release history.
• Ollama download: Installers and installation instructions for all supported platforms.

Core technology

• llama.cpp repository: The C++ inference engine that powers Ollama’s model execution. Includes documentation on supported formats and hardware acceleration.
• GGUF format documentation: Technical details about the model weight format used by Ollama and llama.cpp.

Community resources

• Ollama community models: The model registry hosting thousands of community-contributed models alongside official releases.
• Ollama discussions on Discord: Community discussions and troubleshooting.

Note on verification

Ollama evolves quickly. New models, API changes, and engine improvements arrive regularly. Verify current information against the official documentation and test with the versions you plan to use.