An OpenAI-compatible LLM chat and embedding server: https://github.com/LlamaEdge/LlamaEdge
2.9K
This image starts an OpenAI-compatible API server. It contains
How to use it: https://github.com/LlamaEdge/LlamaEdge/tree/main/docker
On Mac, or Linux / Windows with CPU only:
docker run --name api-server \
-p 8080:8080 \
secondstate/llama-3-8b-nomic-1.5:latest
On Linux / Windows with Nvidia GPU CUDA 11
docker run --name api-server \
-p 8080:8080 --device nvidia.com/gpu=all \
secondstate/llama-3-8b-nomic-1.5:cuda11
On Linux / Windows with Nvidia GPU CUDA 12
docker run --name api-server \
-p 8080:8080 --device nvidia.com/gpu=all \
secondstate/llama-3-8b-nomic-1.5:cuda12
Content type
Image
Digest
sha256:523670f79…
Size
5.7 GB
Last updated
over 2 years ago
docker pull secondstate/llama-3-8b-nomic-1.5