cli-modelarium launches on PyPI as open‑source LLM comparison tool

The open‑source project cli‑modelarium has been published to the Python Package Index, allowing users to install it with a single pip install cli-modelarium command [4]. The author describes it as a terminal‑based solution for comparing large language models (LLMs) with statistical rigor, positioned between quick chat‑window checks and heavyweight enterprise evaluation platforms.

Key takeaways

• cli‑modelarium is live on PyPI under an Apache 2.0 license [4].
• The tool supports eight cloud providers and local models, handling API keys and rate‑limit details automatically [4].
• It implements bootstrap confidence intervals, paired significance tests, and multiple‑comparison corrections to deliver research‑grade statistics [4].
• Optional hallucination detection flags flag fabricated citations, contradictory claims, and other signs of model “hallucination” [4].
• Cost and latency tracking, including a --max-cost cap, help users stay within budget during comparisons [4].

A terminal‑first approach to LLM benchmarking

The author built cli‑modelarium to fill a gap between informal spot‑checks and complex evaluation dashboards. By installing the package, users can configure provider credentials once—either via a cli-modelarium configure command or environment variables—and then run a single command that sends a prompt to multiple models, records cost per API call, measures time‑to‑first‑token, and returns side‑by‑side outputs [4]. Example usage shows a comparison of Claude and GPT models with a cost ceiling of ten cents, followed by an extended run that adds statistical confidence intervals, hallucination checks, and a separate judge model to score quality [4].

Built‑in statistical and safety features

Beyond basic output comparison, cli‑modelarium incorporates a suite of statistical methods typically reserved for academic research. It uses the bias‑corrected and accelerated (BCa) bootstrap method for confidence intervals, applies paired tests such as McNemar’s test for binary outcomes, and offers correction procedures like Bonferroni and Holm to control false discovery rates [4]. For subjective quality assessments, the tool can invoke a “LLM‑as‑judge” panel, letting multiple judge models vote to reduce single‑model bias [4]. Hallucination detection scans responses for invented citations, contradictory statements, and fabricated names or dates, flagging high‑risk outputs for human review [4].

Why it matters

cli‑modelarium provides developers and researchers with a lightweight, reproducible way to evaluate LLMs without the overhead of cloud dashboards or custom infrastructure. By delivering statistically sound results directly in the terminal, it democratizes rigorous benchmarking and helps users avoid the pitfalls of variance‑driven spot checks. The open‑source nature and Apache 2.0 licensing encourage community contributions and transparency, potentially accelerating the development of best‑practice evaluation tools in the rapidly evolving LLM ecosystem.