hwdsl2/ollama-server repository overview

⁠InferCrate

Open-source, self-hosted local LLM server.

GitHub: https://github.com/hwdsl2/infercrate⁠

Run local language models on your own hardware with InferCrate. Powered by Ollama⁠, it provides the native Ollama API and an OpenAI-compatible /v1 API subset, with API-key authentication, model management, and CPU or NVIDIA GPU deployment.

Features:

• OpenAI-compatible API: a /v1 API subset for compatible OpenAI SDKs and apps, alongside the native Ollama API.
• Secure by default: Caddy enforces Bearer token authentication for API access; an API key is auto-generated on first start and stored in the persistent volume.
• Model pre-pull: pre-pull models on first start with the OLLAMA_MODELS environment variable.
• Model management: via a helper script (ollama_manage).
• CPU and GPU support: run on CPU or use the :cuda image for NVIDIA GPU acceleration.
• Lightweight CPU image: based on Debian Trixie (slim); approximately 75 MB.
• Automated builds: images are automatically built and published through GitHub Actions⁠.

Also available as part of the Self-Hosted AI Stack⁠, which deploys a complete self-hosted AI stack with a single command.

Also available:

• Related AI services: ScribeCrate⁠, SpeakCrate⁠, EmbedCrate⁠, GatewayCrate⁠, ParseCrate⁠, UplinkCrate⁠

⁠Quick start

Step 1. Start InferCrate:

docker run \
    --name ollama \
    --restart=always \
    -v ollama-data:/var/lib/ollama \
    -p 11434:11434/tcp \
    -d hwdsl2/ollama-server

On first start, an API key is auto-generated and displayed in the container logs. All API requests require this key.

Note: For internet-facing deployments, use a reverse proxy⁠ to add HTTPS. Also replace -p 11434:11434/tcp with -p 127.0.0.1:11434:11434/tcp in the docker run command above, to prevent direct access to the unencrypted port.

Step 2. Get the API key:

# View the key in the container logs
docker logs ollama

# Or retrieve it for use in scripts
infer_api_key="$(docker exec ollama ollama_manage --getkey)"

The API key is displayed in a box labeled InferCrate API key. To display it again at any time:

docker exec ollama ollama_manage --showkey

Step 3. Pull a model:

docker exec ollama ollama_manage --pull llama3.2:3b

Tip: To pull one or more models automatically on first start, set OLLAMA_MODELS before running the container:

docker run \
    --name ollama \
    --restart=always \
    -v ollama-data:/var/lib/ollama \
    -p 11434:11434/tcp \
    -e OLLAMA_MODELS=llama3.2:3b \
    -d hwdsl2/ollama-server

Or add OLLAMA_MODELS=llama3.2:3b to your ollama.env file (see Environment variables⁠).

Step 4. Test with the API:

infer_api_key="$(docker exec ollama ollama_manage --getkey)"

# List models
curl http://localhost:11434/api/tags \
  -H "Authorization: Bearer $infer_api_key"

# Chat completion (streaming)
curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $infer_api_key" \
  -d '{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Hello!"}]}'

Note: The docker exec management commands (ollama_manage) do not require the API key.

To learn more about how to use this image, read the sections below.

⁠Community

• 📬 Get project updates and free deployment guides⁠ (1–2 emails/month)
• 💬 Join the r/selfhostedstack⁠ community for discussions and showcases
• ⭐ Star the repository if you find it useful — it helps others discover it

⁠Requirements

• A Linux server (local or cloud) with Docker installed
• Sufficient disk space for models (3B models ≈ 2GB, 7B models ≈ 4–5GB, 14B+ models ≈ 8–10GB+)
• Sufficient RAM to run models (3B models ≈ 2–4GB, 7B models ≈ 6–8GB, 14B+ models ≈ 12–16GB+)
• TCP port 11434 (or your configured port) accessible

For GPU acceleration (:cuda image):

• NVIDIA GPU with CUDA support
• NVIDIA driver⁠ 575.57.08+ (Linux) or 576.57+ (Windows) installed on the host
• NVIDIA Container Toolkit⁠ installed
• The :cuda image supports linux/amd64 only

⁠Download

Get the trusted build from the Docker Hub registry⁠:

docker pull hwdsl2/ollama-server

For GPU support:

docker pull hwdsl2/ollama-server:cuda

Alternatively, you may download from Quay.io⁠:

docker pull quay.io/hwdsl2/ollama-server
docker image tag quay.io/hwdsl2/ollama-server hwdsl2/ollama-server

Supported platforms: linux/amd64 and linux/arm64. The :cuda tag supports linux/amd64 only.

⁠Environment variables

All variables are optional. If not set, secure defaults are used automatically.

This Docker image uses the following variables, that can be declared in an env file (see example⁠):

Note: In your env file, you may enclose values in single quotes, e.g. VAR='value'. Do not add spaces around =. If you change OLLAMA_PORT, update the -p flag in the docker run command accordingly.

Example using an env file:

cp ollama.env.example ollama.env
# Edit ollama.env and set your values, then:
docker run \
    --name ollama \
    --restart=always \
    -v ollama-data:/var/lib/ollama \
    -v ./ollama.env:/ollama.env:ro \
    -p 11434:11434/tcp \
    -d hwdsl2/ollama-server

⁠Model management

Use docker exec to manage models with the ollama_manage helper script. Models are stored in the Docker volume and persist across container restarts.

List downloaded models:

docker exec ollama ollama_manage --listmodels

Pull a model:

# Small, fast models (recommended for getting started)
docker exec ollama ollama_manage --pull llama3.2:3b
docker exec ollama ollama_manage --pull qwen2.5:7b

# Larger models (require more RAM/VRAM)
docker exec ollama ollama_manage --pull mistral:7b
docker exec ollama ollama_manage --pull phi4:14b
docker exec ollama ollama_manage --pull gemma3:12b

Remove a model:

docker exec ollama ollama_manage --remove llama3.2:3b

Show running models and memory usage:

docker exec ollama ollama_manage --status

Update all models (re-pulls latest versions):

docker exec ollama ollama_manage --update

Show the API key:

docker exec ollama ollama_manage --showkey

Get the API key (machine-readable, for use in scripts):

infer_api_key="$(docker exec ollama ollama_manage --getkey)"

Pull models on first start using the OLLAMA_MODELS variable in your env file:

OLLAMA_MODELS=llama3.2:3b,qwen2.5:7b

⁠Using the API

All API requests require a Bearer token. Retrieve the API key first:

infer_api_key="$(docker exec ollama ollama_manage --getkey)"

Ollama API:

# List models
curl http://localhost:11434/api/tags \
  -H "Authorization: Bearer $infer_api_key"

# Generate (streaming)
curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $infer_api_key" \
  -d '{"model": "llama3.2:3b", "prompt": "Why is the sky blue?"}'

# Chat completion (streaming)
curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $infer_api_key" \
  -d '{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Hello!"}]}'

OpenAI-compatible API (Ollama /v1 subset; works with compatible OpenAI SDK and app workflows):

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $infer_api_key" \
  -d '{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Hello!"}]}'

Python (OpenAI SDK):

from openai import OpenAI

client = OpenAI(
    api_key="<your-api-key>",
    base_url="http://localhost:11434/v1",
)

response = client.chat.completions.create(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

⁠Persistent data

All server data is stored in the Docker volume (/var/lib/ollama inside the container):

/var/lib/ollama/
├── models/           # Downloaded model files
├── .api_key          # API key (auto-generated, or synced from OLLAMA_API_KEY)
├── .initialized      # First-run marker
├── .port             # Saved port (used by ollama_manage)
└── .Caddyfile        # Generated Caddy config (auth proxy)

Back up the Docker volume to preserve your models and API key.

⁠Using docker-compose

cp ollama.env.example ollama.env
# Edit ollama.env and set your values, then:
docker compose up -d
docker logs ollama

Example docker-compose.yml (already included):

services:
  ollama:
    image: hwdsl2/ollama-server
    container_name: ollama
    restart: always
    ports:
      - "11434:11434/tcp"  # For a host-based reverse proxy, change to "127.0.0.1:11434:11434/tcp"
    volumes:
      - ollama-data:/var/lib/ollama
      - ./ollama.env:/ollama.env:ro

volumes:
  ollama-data:
    name: ollama-data

Note: For internet-facing deployments, use a reverse proxy⁠ to add HTTPS. Also change "11434:11434/tcp" to "127.0.0.1:11434:11434/tcp" in docker-compose.yml, to prevent direct access to the unencrypted port.

⁠GPU acceleration (CUDA)

Use docker-compose.cuda.yml to run with NVIDIA GPU support:

docker compose -f docker-compose.cuda.yml up -d

Requirements: NVIDIA GPU, NVIDIA driver⁠ 575.57.08+ (Linux) or 576.57+ (Windows), and the NVIDIA Container Toolkit⁠ installed on the host. The :cuda image is linux/amd64 only.

⁠Using a reverse proxy

For internet-facing deployments, place a reverse proxy in front of Ollama to handle HTTPS termination. The server works without HTTPS on a local or trusted network, but HTTPS is recommended when the API endpoint is exposed to the internet.

Use one of the following addresses to reach the Ollama container from your reverse proxy:

• ollama:11434 — if your reverse proxy runs as a container in the same Docker network as Ollama (e.g. defined in the same docker-compose.yml).
• 127.0.0.1:11434 — if your reverse proxy runs on the host and port 11434 is published (the default docker-compose.yml publishes it).

Note: The Authorization: Bearer header passes through reverse proxies automatically — no special configuration needed.

Example with Caddy⁠ (Docker image⁠) (automatic TLS via Let's Encrypt, reverse proxy in the same Docker network):

Caddyfile:

ollama.example.com {
  reverse_proxy ollama:11434
}

Example with nginx (reverse proxy on the host):

server {
    listen 443 ssl;
    server_name ollama.example.com;

    ssl_certificate     /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass         http://127.0.0.1:11434;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_http_version 1.1;       # required for streaming responses
        proxy_read_timeout 300s;
        proxy_buffering    off;
    }
}

After setting up a reverse proxy, set OLLAMA_HOST=ollama.example.com in your env file so that the correct endpoint URL is shown in the startup logs and ollama_manage --showkey output.

⁠Update Docker image

To update the Docker image and container:

docker pull hwdsl2/ollama-server
docker rm -f ollama
# Then re-run the docker run command from Quick start with the same volume.

Your downloaded models are preserved in the ollama-data volume.

⁠Using with other AI services

Ollama can be used as the local LLM service in a broader self-hosted AI setup.

For full and lightweight Docker Compose stacks, manual docker run examples, and voice/RAG/MCP pipeline examples with SpeakCrate, EmbedCrate, GatewayCrate, InferCrate, ParseCrate, and UplinkCrate, see Self-Hosted AI Stack⁠.

Connect Ollama to LiteLLM:

# In docker-litellm, add Ollama as a model provider:
docker exec litellm litellm_manage \
  --addmodel ollama/llama3.2:3b \
  --base-url http://ollama:11434

⁠Usage counts

See Usage counts⁠.

⁠Technical details

• Base image: debian:trixie-slim for :latest; nvidia/cuda for :cuda
• Image size: ~75MB (CPU) / ~1.7GB (CUDA)
• Ollama: latest release, installed as a static binary
• Auth proxy: Caddy⁠ (always active, enforces Bearer token auth)
• Data directory: /var/lib/ollama (Docker volume)
• Model storage: /var/lib/ollama/models inside the volume
• Ollama API: http://localhost:11434 (or your configured port)
• OpenAI-compatible API: http://localhost:11434/v1

⁠License

Note: The software components inside the pre-built image (such as Ollama, Caddy, and their dependencies) are under the respective licenses chosen by their respective copyright holders. As for any pre-built image usage, it is the image user's responsibility to ensure that any use of this image complies with any relevant licenses for all software contained within.

Copyright (C) 2026 Lin Song
This work is licensed under the MIT License⁠.

Ollama is Copyright (C) 2023 Ollama, and is distributed under the MIT License⁠.

Caddy is Copyright (C) 2015 Matthew Holt and The Caddy Authors, and is distributed under the Apache License 2.0⁠.

This project is an independent Docker setup for Ollama and is not affiliated with, endorsed by, or sponsored by Ollama.