99 QA practice sites across 5 categories — tested with Vibium CLI and Vibium MCP. Which is faster, cheaper, and more capable?
| Category | Sites | CLI Total | MCP Total | Speed Ratio | Cost |
|---|---|---|---|---|---|
| General Practice | 28 | 8.0× | ~5× cheaper | ||
| Automation Testing | 41 | 4.2× | ~7× cheaper | ||
| Security Testing | 7 / 11 | 5.2× | ~2.4× cheaper | ||
| API Testing | 16 | 3.3× | ~6× cheaper | ||
| Performance Testing | 3 | 4.1× | ~6× cheaper | ||
| All Sites | 95 / 99 | 450s total | 2,263s total | 5.0× | 5–9× cheaper |
Bar widths relative to longest MCP total (General Practice 864s). 4 Security Testing sites require local Docker or account — excluded from timing.
MCP time is uniformly ~7–50s per site regardless of complexity. CLI time scales with site: from 383ms (ATM Practice) to 87s (Expand Testing timeout). Simplest sites have the widest ratios.
CLI completes in 491ms — the fastest single-site time in the entire dataset. A simple filter-only SPA with no waits exposes MCP's full per-tool-call overhead.
Alert pre-stubbing via eval adds equivalent overhead to both interfaces, nearly equalizing them. Sleep floors and complex setup collapse the gap across the dataset.
CLI receives an Angular SSR Application Error. MCP fully loads the app with 15 products. The most significant cross-interface divergence — same URL, completely different outcomes.
On React/Vue SPAs like TestDino (216 refs), Lambdatest Playground (309 refs), and GreenKart (126 refs), browser_map enumerates all elements. CLI vibium map returns nothing — requiring eval-only workflows.
On http:// origins (Global SQA Demo, Travel Agileway), MCP silently navigates while CLI throws a BiDi "unknown error". MCP is more permissive with insecure origins.
The only site where MCP beat CLI. CLI cold-started at 14.9s on this CDN-heavy docs site; MCP hit a warmer cache at 10.2s. Excluding it, the API batch ratio is 4.8×.
Each MCP browser_navigate + browser_map pair generates its own LLM turn. CLI dispatches via bash and the model barely participates in execution. For pure automation, CLI wins on cost by a wide margin.