Mithril — Engine — Khazad-dûm

The LazyModelManager loads the model only on first use, keeps it in memory, and provides a streaming bridge between C++ and async Rust.

LazyModelManager — The Core

Manages a single GGUF model in memory. Key behaviors:

Lazy loading: Model is loaded only on first inference request (~2-5 seconds)
Stay resident: Once loaded, stays in memory for fast subsequent calls
Auto-unload: After N seconds of inactivity, frees memory
Metal GPU: Uses Apple Silicon GPU automatically if available

Token Streaming

llama.cpp generates tokens one by one via C callback. The streaming bridge:

llama.cpp generates a token (C/C++ land)
Callback sends it through a tokio::mpsc channel
Async Rust receives it and forwards to the client

The Batch Fix

Critical setting: n_batch = n_ctx. Without this, the model crashes on inputs longer than the default batch size. This ensures the entire context window can be processed in a single batch.

Download as CSV

Model	Size	Use
qwen2:1.5b	~1.2 GB	Request classifier (free routing)
qwen2:7b	~4.5 GB	General coding tasks
llama:8b	~5 GB	Alternative general model
deepseek:6.7b	~4 GB	Code-specialized
phi3:3.8b	~2.3 GB	Compact reasoning model

File	Purpose
`chat_template.rs`
`lazy_model.rs`	Lazy model manager — loads GGUF on first use, auto-unloads after idle timeout.
`mod.rs`
`model_catalog.rs`

Key Concept

LazyModelManager — The Core

Token Streaming

The Batch Fix