Open Source QA Research
Vibium
CLI vs MCP
CLI vs MCP
Two browser automation interfaces. 99 practice sites. Which wins?
lana-20.github.io/practice-testing
Scale
99
QA practice
sites tested
sites tested
General Practice
28 sites
Automation Testing
41 sites
Security Testing
11 sites
API Testing
16 sites
Performance Testing
3 sites
Overall Speed
5.0×
CLI is faster
on average
on average
CLI total450s
MCP total2,263s
Speed by Category
CLI faster
everywhere
everywhere
CLI total time vs MCP per category
General Practice
8.0×
Security Testing
5.2×
Automation Testing
4.2×
Performance Testing
4.1×
API Testing
3.3×
Widest Ratio
36×
Automation
Bookstore
Bookstore
CLI: 491ms — fastest in the dataset.
MCP: 17,664ms
A filter-only SPA with no waits exposes MCP's full per-tool-call overhead.
MCP: 17,664ms
A filter-only SPA with no waits exposes MCP's full per-tool-call overhead.
Narrowest Ratio
2.6×
Evil Tester
Alert pre-stubbing via eval adds equal overhead to both interfaces — almost closing the gap entirely. Sleep floors and complex setup collapse the ratio across the dataset.
Behavioral Difference
OWASP
Juice Shop
Juice Shop
Same URL. Completely different outcomes.
CLI
Application
Error
Error
MCP
15 products
loaded
loaded
Angular SSR timing — MCP's browser context handles it, CLI's BiDi layer doesn't.
Cost
5–9×
CLI is cheaper
Each MCP browser_navigate + browser_map pair generates its own LLM turn. CLI dispatches via bash — the model barely participates.
Anomaly
SpaceTraders:
MCP won
MCP won
MCP 10,203ms vs CLI 14,933ms
The only site where MCP beat CLI. CLI cold-started on a CDN-heavy docs site. Excluding it, the API ratio is 4.8×.
The only site where MCP beat CLI. CLI cold-started on a CDN-heavy docs site. Excluding it, the API ratio is 4.8×.
Full Results
99 sites.
Every finding.
Every finding.
Interactive benchmark with per-site CLI vs MCP timing, behavioral diffs, and community-built highlights.
lana-20.github.io/practice-testing
Behavioral Guide → SKILL.md ↗
vibium v26.3.18 · claude-sonnet-4-6