llama.cpp Free code examples in C#, Python and JavaScript

llama-cpp-python builds the engine from source on install, so a C compiler is required. The -hf flag on the CLI downloads GGUF files from Hugging Face and caches them.

# The engine itself, for the server and the CLI:
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp && cmake -B build && cmake --build build -j --config Release

# Try a model without downloading anything by hand:
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# The Python binding builds llama.cpp as part of the install.
pip install llama-cpp-python

THIS IS THE DECISION. The suffix on the GGUF file is the precision, and it is the only knob that changes both memory and quality at once. Q4_K_M is the default for a reason; below Q4 the failures start looking like stupidity rather than compression.

# Same model, four precisions. Sizes are for an 8B model.
#
#   Q8_0     ~8.5GB   near-lossless; use when RAM is not the constraint
#   Q6_K     ~6.6GB   very close to Q8, meaningfully smaller
#   Q5_K_M   ~5.7GB   a small, usually unnoticeable step down
#   Q4_K_M   ~4.9GB   THE DEFAULT. Best quality per gigabyte.
#   Q3_K_M   ~4.0GB   noticeable: more repetition, worse arithmetic
#   Q2_K     ~3.2GB   a demo, not a deployment
#
# Pull a specific one with a tag:
llama serve -hf ggml-org/gemma-4-e4b-it-GGUF:Q4_K_M

# Or quantise a model you already have:
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

n_gpu_layers is the other big lever: -1 offloads everything it can. n_ctx costs memory proportional to its size, so a 32k context on a laptop can be what pushes the model out of RAM rather than the weights.

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="ggml-org/Qwen3.5-0.8B-GGUF",
    filename="*Q4_K_M.gguf",       # the quantisation, by pattern
    n_ctx=4096,                    # context costs memory too
    n_gpu_layers=-1,               # -1 = offload everything possible
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain a database index."}],
)

print(response["choices"][0]["message"]["content"])

In-process means one model per process and no sharing. `llama serve` gives you an OpenAI-compatible endpoint, so the application stops depending on the engine entirely — and that is the shape FreeYantra uses for its own tools.

# Defaults to http://127.0.0.1:8080
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_K_M

# Started with no model, `llama serve` becomes a router: it loads and
# unloads models on demand and forwards each request to the right one.
llama serve

# Then any OpenAI client works against it:
#   OpenAI(base_url="http://127.0.0.1:8080/v1", api_key=PLACEHOLDER)

node-llama-cpp ships prebuilt binaries for the common platforms, so there is usually no compiler step. It is the only first-party-ish Node path to the engine.

npm install node-llama-cpp

# Or skip the binding entirely and run the server:
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_K_M

The suffix is the precision. Q4_K_M is the default recommendation; the steps above it buy little and the steps below it cost real quality.

# For an 8B model:
#   Q8_0     ~8.5GB   near-lossless
#   Q5_K_M   ~5.7GB   a small step down, rarely noticed
#   Q4_K_M   ~4.9GB   THE DEFAULT
#   Q3_K_M   ~4.0GB   repetition and arithmetic start to suffer
#
npx node-llama-cpp pull \
  --dir ./models \
  "https://huggingface.co/ggml-org/Qwen3.5-0.8B-GGUF/resolve/main/Qwen3.5-0.8B-Q4_K_M.gguf"

A model is loaded once and a context is created per conversation. Creating a context per request is the common mistake — the load is the expensive part and it is being repeated.

import { getLlama, LlamaChatSession } from "node-llama-cpp";
import path from "node:path";

const llama = await getLlama();

// Expensive. Once per process.
const model = await llama.loadModel({
  modelPath: path.join("./models", "Qwen3.5-0.8B-Q4_K_M.gguf"),
});

// Cheap-ish. Once per conversation, not once per request.
const context = await model.createContext({ contextSize: 4096 });
const session = new LlamaChatSession({
  contextSequence: context.getSequence(),
});

console.log(await session.prompt("Explain a database index."));

A binding ties the model's lifetime to your process — a deploy unloads gigabytes and reloads them. A server keeps the model warm and turns the engine into an HTTP dependency any client can speak to.

// Terminal: llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF:Q4_K_M
import OpenAI from "openai";

// Required by the client, ignored by the server.
const placeholderKey = "llama";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8080/v1",
  apiKey: placeholderKey,
});

const response = await client.chat.completions.create({
  model: "local",          // llama serve ignores this
  messages: [{ role: "user", content: "Explain a database index." }],
});

console.log(response.choices[0].message.content);