Building muse — a private multi-model AI tool

I got tired of paying $20/month for a single AI model and wondering where my prompts end up. So I built muse — a private, multi-model AI synthesis tool where you bring your own API keys and pay per token.

The idea

You type a prompt. muse sends it to multiple LLMs concurrently — Claude, GPT, Gemini, and an on-device model. A judge evaluates all responses and returns one synthesized answer with a trust score. You don’t see four competing answers. You get the answer.

You → "How should I structure my portfolio?"

     ┌──────────┐
     │  muse    │
     └────┬─────┘
          │ fan-out (concurrent)
    ┌─────┼──────┬──────┐
    ▼     ▼      ▼      ▼
  Claude  GPT  Gemini  MLX(local)
    │     │      │      │
    └─────┴──────┴──────┘
          │
          ▼
       Judge (Claude)
          │
          ▼
    One answer + trust score: 0.87

Why

Cost. With BYOK (bring your own key), you pay per token. A typical query across 3-4 models costs a few cents. Compare that to $20-40/month for a single provider subscription.

Privacy. No muse server sits in the middle. Prompts go directly from your device to the model API. No telemetry, no data collection, nothing stored externally. On-device mode with MLX is fully airgapped — zero network calls.

Better answers. One model can hallucinate. When multiple models agree, you can trust the answer more. The judge captures this as a trust score.

What’s built

Python backend

The engine is a Python package with a CLI and a local API server:

# CLI — synthesize across all available models
muse "should I use Kafka or Redpanda?"

# Local API server — localhost only
muse serve
curl -X POST http://localhost:8000/muse \
  -d '{"prompt": "your question"}'

The architecture is provider-based. Each model (Anthropic, OpenAI, Gemini) implements a Provider protocol. The orchestrator fans out to all available providers concurrently, collects responses, and hands them to the judge.

iOS app

This is what I’m most excited about. A SwiftUI app with on-device inference via MLX Swift.

The app runs Llama 3.2 3B Instruct (4-bit) directly on Apple Silicon — no server, no API call, no cost. The model is about 1.7 GB in memory and generates responses fast enough for real use.

Cloud providers (Claude, GPT, Gemini) are also available in the app. API keys are stored in the iOS Keychain. Everything stays on-device.

On-device model selection

Getting the right model for mobile was a journey. I tried:

• Llama 3.2 1B — fast and light (~800 MB) but noticeably weak for general chat
• Phi-4 Mini 3.8B — great quality but 2.7 GB RAM and stalled during inference due to GPU memory constraints
• SmolLM 1.7B — hallucinated a math proof when I said “Hi”
• Llama 3.2 3B 4-bit — the sweet spot. ~1.7 GB RAM, solid conversational quality, fast inference

The key technical fixes: the GPU cache limit needs to be large enough for the model’s working set (512 MB, not 20 MB), and generation must be capped at maxTokens to prevent unbounded inference.

What’s next

• OpenRouter migration — replace per-provider API keys with a single OpenRouter key for access to 200+ models
• Persona library — --persona "code reviewer" to shape how models respond
• Chat history — SwiftData persistence in the iOS app

The code is open source: github.com/chanukyapekala/muse