Mithril Providers — How to Add & Understand Providers

Providers

Gemini, OpenAI, Anthropic, Groq, Local, CLI

Shared Interface

5 methods

ChatProvider trait

Types

Local GGUF / Cloud API / CLI Tools

Three Types of Providers

Mithril supports three fundamentally different ways to access LLM capabilities:

Type	How It Works	Examples
Local GGUF	Direct inference via llama.cpp FFI. Free, private, fast.	qwen-1.5b, llama-8b
Cloud API	HTTP calls to cloud LLM endpoints. Pay-per-token.	Gemini, OpenAI, Anthropic, Groq
CLI Tools	Subprocess calls to local CLI tools with their own auth.	Kiro, Junie, any chat CLI

Provider Types

flowchart LR
    ORCH(Orchestrator) --> LOCAL(Local GGUF)
    ORCH --> CLOUD(Cloud APIs)
    ORCH --> CLI(CLI Tools)
    CLOUD --> GEM(Gemini)
    CLOUD --> OAI(OpenAI)
    CLOUD --> ANT(Anthropic)
    CLOUD --> GRQ(Groq)
    CLI --> KIRO(kiro-cli)
    CLI --> OTHER(any CLI)

The ChatProvider Trait

Every provider implements this

pub trait ChatProvider: Send + Sync {
    fn name(&self) -> &str;
    fn model(&self) -> &str;
    async fn chat(&self, messages: &[ChatMessage]) -> Result<String>;
    async fn chat_stream(&self, messages: &[ChatMessage], on_chunk: ...) -> Result<String>;
    async fn chat_with_tools(&self, messages: &[ChatMessage], tools: &[ToolDefinition]) -> Result<ToolCallResult>;
    async fn is_available(&self) -> bool;
}

Download as CSV

Provider	Auth	Format	Streaming
Gemini	?key= in URL	contents[{role, parts}]	SSE with streamGenerateContent
OpenAI	Bearer header	messages[{role, content}]	SSE data: lines
Anthropic	x-api-key header	messages[] + system	SSE content_block_delta
Groq	Bearer header	messages[] (OpenAI-compat)	SSE (same as OpenAI)

CLI providers wrap external command-line tools as subprocess calls. They're useful when:

You have access to a CLI with its own authentication (e.g., Kiro with AWS credentials)
The CLI provides models not available via standard API (e.g., proprietary models)
You want to use the CLI's built-in tools and context

How it works

Mithril spawns the CLI process with the prompt as argument
Uses --no-interactive + structured output format
Parses the JSON/text response from stdout
Returns it through the standard ChatProvider interface

Example: Kiro CLI

Under the hood

# What Mithril does internally:
kiro-cli chat "Your prompt here" \
  --model claude-opus-4.6 \
  --no-interactive \
  --output-format stream-json \
  --agent-engine v2

# Output: JSON Lines with runStarted, agent_message_chunk, runFinished

Adding your own CLI provider

Any CLI that accepts a prompt and outputs text can be wrapped as a provider. You just need to adjust the command arguments and output parsing in a new src/providers/your_cli.rs file.

Template

pub struct YourProvider {
    model: String,
}

#[async_trait]
impl ChatProvider for YourProvider {
    fn name(&self) -> &str { "your_provider" }
    fn model(&self) -> &str { &self.model }

    async fn chat(&self, messages: &[ChatMessage]) -> Result<String> {
        // For Cloud: HTTP POST to API
        // For CLI: spawn subprocess, parse output
        todo!()
    }

    async fn chat_stream(&self, messages: &[ChatMessage], on_chunk: ...) -> Result<String> {
        // Stream tokens one by one
        todo!()
    }

    async fn chat_with_tools(&self, messages: &[ChatMessage], tools: &[ToolDefinition]) -> Result<ToolCallResult> {
        // Function calling or fallback to plain chat
        let text = self.chat(messages).await?;
        Ok(ToolCallResult::Text(text))
    }
}

// Register in src/providers/mod.rs create_provider_with_model():
"your_provider" => Ok(Box::new(YourProvider::new(model)))