Download llama.cpp – Run Large Language Models Locally on Your PC

llama.cpp is a free, open-source C/C++ inference engine that enables you to run large language models (LLMs) like LLaMA, Mistral, Falcon, and many others directly on your own computer — no internet connection, no API fees, and no GPU required (though GPU acceleration is supported). It is one of the most widely used tools for local AI inference.

Originally developed to run Meta’s LLaMA models on a MacBook, llama.cpp has grown into a comprehensive framework supporting dozens of model architectures in the GGUF quantized format. Quantization dramatically reduces model size (a 7B parameter model can fit in 4-8 GB of RAM), making it feasible to run capable AI models on consumer hardware including laptops and single-board computers.

llama.cpp exposes a command-line interface for interactive chat and text generation, as well as an HTTP server compatible with the OpenAI API format, making it easy to connect to frontends like Open WebUI, LM Studio, and Ollama. It supports CPU, CUDA (NVIDIA), Metal (Apple), ROCm (AMD), and Vulkan backends for hardware-accelerated inference.

Key Features

• CPU and GPU inference – Runs on CPU-only machines and optionally accelerates with CUDA, Metal, ROCm, or Vulkan.
• GGUF model support – Loads quantized GGUF models from Hugging Face for efficient local inference at 2-bit to 8-bit precision.
• OpenAI-compatible API server – Built-in HTTP server with OpenAI-compatible endpoints for drop-in integration with existing tools.
• Multi-model architecture support – Supports LLaMA, Mistral, Falcon, Phi, Gemma, Qwen, DeepSeek, and many other model families.
• Interactive chat mode – Built-in interactive conversation mode with configurable system prompts and sampling parameters.
• Cross-platform – Runs on Windows, macOS, Linux, and ARM devices including Raspberry Pi.
• No data leaves your machine – Fully offline inference for maximum privacy and data security.

How to Install

1. Download the pre-built llama.cpp binary for your OS from the link below (or build from source with cmake).
2. Download a GGUF model file from Hugging Face (e.g., a quantized Mistral or LLaMA model) and place it in a local folder.
3. Run interactive chat: ./llama-cli -m model.gguf -p "You are a helpful assistant." --interactive
4. Or start the API server: ./llama-server -m model.gguf --port 8080 and connect any OpenAI-compatible client to http://localhost:8080.
5. Adjust --n-gpu-layers to offload layers to your GPU for faster inference if a supported GPU is available.