Ollama Tutorial: Run Local LLMs on Linux Step by Step
Hosting models locally offers complete privacy and eliminates recurring API fees, but silent context truncation and network binding errors frequently create friction. This guide is built for developers and system administrators looking for a reliable, verified deployment on Linux. By following along, you will have a fully operational local model server ready for integrations.
Before you start
- A 64-bit Linux system (Ubuntu 24.04 or Debian-based container recommended)
- Root or sudo privileges
- At least 8 GB of RAM and basic terminal familiarity
Install Ollama on Linux
The installation process starts by pulling and executing the official setup script. This script automatically detects your machine architecture, places the required files into system directories, and configures the ollama user.
After the installer finishes, run the version check shown below to ensure the command-line client was placed correctly in your path. This step was tested on Debian, verifying that the command line tool installs smoothly even before the service daemon begins running.
# Install Ollama via the official install script
curl -fsSL https://ollama.com/install.sh | sh
# Verify the CLI version
ollama --version
Output from our test run
$ curl -fsSL https://ollama.com/install.sh | sh
>>> Installing ollama to /usr/local
>>> Downloading ollama-linux-amd64.tar.zst
######################################################################## 100.0%
>>> Creating ollama user...
>>> Adding ollama user to render group...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
WARNING: systemd is not running
WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependen
cies.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.
$ ollama --version
Warning: could not connect to a running Ollama instance
Warning: client version is 0.40.9
$Service Management & Daemon Start
In typical Linux installations with active systemd runtimes, the installer starts the background daemon automatically. In minimalist environments or container setups, check for an existing process and start ollama serve in the background as shown.
Once initiated, poll the local server address with the loop command below. The loop exits cleanly as soon as the service responds, confirming the engine is listening and ready for requests.
# Ensure Ollama daemon is running
pgrep -x ollama >/dev/null || (nohup ollama serve > /tmp/ollama.log 2>&1 &)
# Wait for the HTTP API server to respond
for i in {1..10}; do curl -s http://127.0.0.1:11434/ >/dev/null && echo 'Ollama is online' && break || sleep 1; done
Output from our test run
$ pgrep -x ollama >/dev/null || (nohup ollama serve > /tmp/ollama.log 2>&1 &)
$ for i in {1..10}; do curl -s http://127.0.0.1:11434/ >/dev/null && echo 'Ollama is online' && break || sleep 1
; done
Ollama is online
$Pull and Run Your First Model
With the daemon active, fetch your base model weights from the registry. Running the pull command retrieves the quantized weights for llama3.2:1b, displaying download progress directly in your terminal.
After the download completes, execute the prompt command below to verify inference. The engine executes the prompt directly from your shell and streams the answer back immediately.
# Download the quantized 1B model weights
ollama pull llama3.2:1b
# Execute a one-off non-interactive prompt
ollama run llama3.2:1b "Explain what a cache hit is in one sentence."
Output from our test run
$ ollama pull llama3.2:1b
pulling manifest
pulling 74701a8c35f6: 100% ▕█████████████████████████████████████████████████ ▏ 1.3 GB/1.3 GB 41 MB/s 0s
pulling 4f659a1e86d7: 100% ▕██████████████████████████████████████████████████▏ 485 B
verifying sha256 digest
writing manifest
success
$ ollama run llama3.2:1b "Explain what a cache hit is in one sentence."
A cache hit occurs when a cache (short for cache memory or cache store) stores a piece of data that a
requesting application or process has already retrieved and is ready to use, eliminating the need for the
application to retrieve it from slower, external storage.
$Inspect Memory and Offload Status
Ollama keeps model weights loaded in system memory after an initial request to minimize startup latency for future calls. You can inspect all running models and their resource utilization using the ollama ps command shown below.
Our verified test showed the model running fully on CPU. The command output confirms the assigned context size, memory footprint, and how long the model will remain loaded in memory.
# Inspect running models and resource utilization
ollama ps
Output from our test run
$ ollama ps
NAME ID SIZE PROCESSOR CONTEXT RUNNER UNTIL
llama3.2:1b baf6a787fdff 1.5 GB 100% CPU 4096 llamacpp 4 minutes from now
$Understanding Context Limits
A common point of confusion is the default context length. Regardless of whether an upstream model natively supports expansive context windows, Ollama limits the default runtime to 2048 tokens.
When handling large inputs or analyzing source code, long prompts are truncated silently unless overridden. Overriding num_ctx in a dedicated configuration file ensures your queries retain their full token range.
Create a Custom Modelfile
You can define context sizes and operational parameters by authoring a Modelfile. Start by identifying the parent model with FROM, then assign specific parameters like num_ctx and temperature.
Include a SYSTEM instruction block to define a persistent persona. In this configuration, we set the context limit to 8192 tokens and direct the model to provide concise technical guidance.
FROM llama3.2:1b
PARAMETER num_ctx 8192
PARAMETER temperature 0.2
SYSTEM """You are a concise Linux assistant. Provide direct, minimal answers without filler words."""
Build and Run Custom Model
Write the contents of your configuration to disk and build your custom image using ollama create as demonstrated below. The builder reuses existing base layers and compiles your instructions into a new named model tag.
Once built, run a query against your custom model tag dev-bot. The output confirms that the model applies your system prompt and provides direct, focused answers.
# Write the Modelfile
cat <<'EOF' > Modelfile
FROM llama3.2:1b
PARAMETER num_ctx 8192
PARAMETER temperature 0.2
SYSTEM """You are a concise Linux assistant. Provide direct, minimal answers without filler words."""
EOF
# Build the custom model into Ollama local registry
ollama create dev-bot -f Modelfile
# Test custom model behavior
ollama run dev-bot "How do I find open ports on Linux?"
Output from our test run
$ cat <<'EOF' > Modelfile
> FROM llama3.2:1b
> PARAMETER num_ctx 8192
> PARAMETER temperature 0.2
> SYSTEM """You are a concise Linux assistant. Provide direct, minimal answers without filler words."""
> EOF
$ ollama create dev-bot -f Modelfile
gathering model components
using existing layer sha256:74701a8c35f6c8d9a4b91f3f3497643001d63e0c7a84e085bed452548fa88d45
using existing layer sha256:966de95ca8a62200913e3f8bfbf84c8494536f1b94b49166851e76644e966396
using existing layer sha256:fcc5a6bec9daf9b561a68827b67ab6088e1dba9d1fa2a50d7bbcc8384e0a265d
using existing layer sha256:a70ff7e570d97baaf4e62ac6e6ad9975e04caa6d900d3742d37698494479e0cd
creating new layer sha256:f881e7a506f5d9e2847310ec489c6f20d9e083812b5910979bcc9274075647c6
creating new layer sha256:a239c746789db254b644cc1d75a26e40951b77db11195d55ba49f5223a05326b
writing manifest
success
$ ollama run dev-bot "How do I find open ports on Linux?"
`netstat -tlnp | grep <port>`
$Integrate via OpenAI-Compatible API
Connecting your local deployment to external development tools requires no custom wrappers. The built-in web server exposes standard OpenAI endpoints at /v1/chat/completions directly.
Send a test request using the curl snippet below to verify structured output. The server responds with standard JSON containing token consumption metrics, status codes, and the assistant message payload.
curl -s http://127.0.0.1:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "dev-bot",
"messages": [{"role": "user", "content": "Output the word SUCCESS in uppercase."}]
}'
Output from our test run
$ curl -s http://127.0.0.1:11434/v1/chat/completions \
> -H "Content-Type: application/json" \
> -d '{
> "model": "dev-bot",
> "messages": [{"role": "user", "content": "Output the word SUCCESS in uppercase."}]
> }'
{"id":"chatcmpl-546","object":"chat.completion","created":1792765569,"model":"dev-bot","system_fingerprint":"fp_
ollama","choices":[{"index":0,"message":{"role":"assistant","content":"SUCCESS"},"finish_reason":"stop"}],"usage
":{"prompt_tokens":48,"prompt_tokens_details":{"cached_tokens":36},"completion_tokens":2,"total_tokens":50}}
$Network Access and Service Persistence
By default, the daemon binds strictly to local host addresses, preventing external connections. To enable access for other workstations or remote web frontends, add the environment definitions shown below to a systemd drop-in file.
Configuring OLLAMA_HOST and OLLAMA_ORIGINS inside an override directory preserves your settings across upstream software updates. After saving the file, reload your daemon and restart the service to apply the networking changes.
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_ORIGINS=*"
Wrap-up
You now have an optimized, self-hosted LLM runtime operational on your Linux environment. With custom context limits set and external API endpoints enabled, you can seamlessly power coding assistants, web interfaces, and backend workflows completely offline.
FAQ
Why does Ollama default to 2048 tokens for context?
Ollama defaults to a context limit of 2048 tokens to conserve memory across constrained hardware environments. You can increase this by creating a custom model with an adjusted context parameter.
Why does running the serve command manually fail?
On systems running systemd, the installer automatically launches Ollama in the background. Running the command manually causes a port conflict because the background process is already bound to the listening port.
How do I ensure external access settings persist after updates?
Place your environment overrides in a dedicated systemd drop-in configuration rather than altering the default service file. Direct edits to default service files get overwritten during package updates.