icecream-probe

Choosing and running local AI models is guesswork today. Developers download multi-GB models only to find they don't fit their RAM/VRAM, rely on outdated blog lists, and have no quick way to check if a model (local or cloud) is healthy, fast, or safe to put in front of users.

**icecream-probe** fixes this in one CLI + MCP tool:

- **Right model for your device**: detects your RAM, VRAM, CPU and GPU, pulls **live metadata** from the Hugging Face and Ollama APIs, and uses rule-based memory math to tell you exactly which models and quantizations will run fully on GPU, run partially offloaded, run on CPU only, or not run at all. No stale lists, no AI guessing.
- **Model health & performance testing**: checks health, latency, load, load balancing and failover for both offline (Ollama) and online (cloud) models.
- **Safer chatbots**: built-in guard classifier (Qwen3Guard / Llama Guard) screens user input and model output for toxic or unsafe content.
- **Clear reports**: generates a device + model compatibility report with direct `ollama pull` commands.
- **Works inside AI IDEs**: runs as an MCP server, so agents in tools like Google Antigravity can call it directly.

**Who it's for:** students with limited hardware, developers building local AI apps, and teams needing reliable, safe model deployment.
- **No official Ollama model listing API**: we couldn't list the Ollama library through any documented endpoint. We moved to the official **Hugging Face Hub API** for discovery and metadata, and used Ollama's official `hf.co/{repo}:{quant}` pull support. We use the Ollama registry manifests only as an extra source.
- **Estimating whether a model fits in memory**: file size alone was inaccurate. We added the KV-cache formula (`2 × layers × kv_heads × head_dim × context × 2 bytes`) plus runtime overhead, using values read from each model's `config.json`.
- **Cross-platform hardware detection**: GPU detection differs on NVIDIA, Apple Silicon (unified memory) and CPU-only machines. We built separate detectors (`pynvml`, `sysctl`, `psutil`) with safe fallbacks.
- **Hugging Face API rate limits**: repeated scans got throttled, so we added short-lived timestamped caching so the data stays fresh without hitting limits.
- **Keeping safety checks fast**: running a large guard model slowed chats. We switched to a small guard model (0.6B–1B) to keep latency low.