100 QA practice sites across 5 categories — tested with Vibium CLI and Vibium MCP. Comprehensive measurement with bracket protocol v2 (2026-06-05). Which is faster, cheaper, and more capable?
| Category | Sites | CLI Total | MCP Total | Speed Ratio | Cost |
|---|---|---|---|---|---|
| General Practice | 28 | 8.0× | ~5× cheaper | ||
| Automation Testing | 41 | 4.2× | ~7× cheaper | ||
| Security Testing | 7 / 11 | 5.2× | ~2.4× cheaper | ||
| API Testing | 16 | 3.3× | ~6× cheaper | ||
| Performance Testing | 3 | 4.1× | ~6× cheaper | ||
| All Sites | 100 | 1,058s total | 4,137s total | 3.9× | 4.4× cheaper |
Bar widths relative to longest MCP total (General Practice 864s). 4 Security Testing sites require local Docker or account — excluded from timing.
Vibium Python API enables true parallelism via ThreadPoolExecutor with isolated contexts. MCP's stateful stdio interface forces sequential execution. Hybrid measurement: 7 sites in 26.2s (CLI) vs 164.4s (MCP) — 6.3× speedup for parallel approach.
Across 100 sites: CLI 17.6m total, MCP 68.9m total. Speed ratio of 3.9× overall. CLI variance driven by site complexity (4.2s–25.8s); MCP variance minimal (~23.5s uniform per-site overhead). Shows CLI scales with interaction depth, MCP has fixed per-operation cost.
MCP averages 11.5 turns per site vs CLI's 1.6 turns. Each MCP tool call (navigate, map, click) generates LLM interactions; CLI dispatches via bash with minimal model participation. Token cost scales with turn count, favoring CLI by wide margin.
MCP turns concentrate in complex sites (TestDino: 30 turns, GreenKart: 18 turns). CLI shows flat turn counts (mostly 0–3 per site). Turn count drives cost; architectural differences (stateful tree traversal in MCP vs stateless bash execution) explain the gap.
AT category: 42 sites average 10.6s CLI, 43.8s MCP (4.1× ratio). GP category: 28 sites show widest variation due to alert/dialog handling complexity. Security Testing: smallest category (7 sites), consistent 3.2× ratio across all seven. API Testing: 20 sites, mostly stateless, 3.3× average ratio.
TestDino, Lambdatest Playground, GreenKart: all return full element maps with CLI in v26.5.31 (bug fix from v26.3.18). No eval workaround needed. Both interfaces now enumerate React/Vue elements reliably. Behavioral parity achieved for SPA handling.
Three-bash token isolation (snapshot → measure → diff) prevents agent contamination. 7-site fresh measurement shows $0.00 cost with 0 turns for pure navigation sites (JSON Placeholder, Poké API, ServeRest, etc.). Confirms token tracking is reliable and measurements are accurate.
Native APIs (Python/JavaScript) and CLI+SKILLS enable true concurrency — suitable for high-throughput automation, benchmarking, and measurement at scale. MCP excels for stateful reasoning, rich introspection, and exploratory workflows where parallelism isn't required. Choose based on workload, not interface.