llama-cli

A CLI front-end for llama.cpp.

A decent test command:

time ./llama-cli \
  --no-display-prompt \
  --single-turn \
  --temp 0 \
  -c 16384 \
  -cnv \
  -m Llama-3.1-Tulu-3-8B-Q8_0.gguf \
  -n 1000 \
  -ngl 100 \
  -p 'What is quantum field theory?' \
  -t 10 |
  tee output.txt \
;

but it failed to be deterministic despite --temperature 0. This ran 2x faster at 18 tokens/s for 1000 tokens on P14s on GPU via Vulkan than on CPU which is achievable by removing the -ngl 100.

 Ancestors (16)

llama.cpp
Ollama
Open source LLM
Large language model
Text-to-text model
AI text generation
Generative AI by modality
Generative AI
AI by capability
Artificial intelligence
Machine learning
Computer
Information technology
Area of technology
Technology
 Home

 Incoming links (2)

llama.cpp
Ollama deterministic output