What Happened
Posit published an evaluation of Posit Assistant against Claude Code and Codex. On bluffbench2, which tests whether agents catch subtle data-quality problems during exploratory analysis, Posit Assistant cost 18% less than Claude Code and 25% less than Codex across the tested tasks. Its performance was similar to Claude Code’s and scored 44% higher than Codex’s in that evaluation [1].
Results depended on the model and configuration. With Claude Sonnet 5.5, Posit Assistant scored 42% at $6.76, versus Claude Code’s 38% at $7.27 with a persistent console. A small DeepSWE 1.1 coding evaluation showed lower costs for Posit Assistant, but too few tasks to establish a reliable performance difference [1]. The research notes do not establish new releases for Streamlit, Quarto, Jupyter, Shiny, Gradio, Chainlit, Modal or Replicate.
Why It Matters to Businesses
For data applications, an agent that produces plausible code but misses a data-quality issue can be more costly than one that uses extra tokens. Evaluate tools on the errors your analysts actually encounter, not coding scores alone. Posit attributes its lower token use partly to direct plot access, truncated file output, parallel actions and caching—design choices that affect both cost and what evidence an agent can inspect [1].
Kimbodo Engineering Perspective
These results are a useful purchasing signal, not a universal ranking. Agent performance changes with the underlying model, tool access and task mix; the small coding evaluation does not justify a broad performance claim [1]. We would choose the application framework separately from the agent: Streamlit, Shiny or Gradio may serve an interface; Quarto or Jupyter may support analysis and publishing; Chainlit may suit a conversational interface; Modal or Replicate may supply compute or model execution. The right combination depends on users, deployment controls and operational requirements, not this benchmark.
How We Would Implement It
- Build a representative test set of analysis tasks with known data-quality defects, expected outputs and review criteria.
- Run candidate agents with the same models, datasets, permissions and spending limits; record defect detection, task success, latency and total cost.
- Keep the user interface separate from analysis execution. Run data access and model calls behind authenticated services with explicit tool permissions.
- Capture plots, queries and validation results as reviewable evidence. Require human approval before an agent publishes findings or changes production data.
Risks, Costs and Security
Benchmark savings may not transfer to a different workload, model price or deployment setup [1]. Caching can reduce repeat costs but needs access-aware keys and retention limits; truncated output can save tokens but hide relevant records. Treat notebooks, uploaded files and retrieved text as untrusted input, restrict credentials and outbound access, and log agent actions without retaining sensitive data unnecessarily.
Where Kimbodo Comes In
Kimbodo builds and operates this in production for businesses — see our AI Application Development practice, or Estimate My AI Application.
