#019 • 6 May 2025

Contents of this Newsletter

📊

Benchmarking

⏰ : 8 min

📓

Recent Blogs

⏰ : 8 min

🗞️

AI News

⏰ : 5 min

MAIN STORY

The Benchmarks Are Not Good Enough For Us

The Human Brain Is The Benchmark

Sooner or later, every team building with AI hits the same realization: benchmarks don’t cut it. They measure accuracy, precision, latency—but not how people actually use AI. Tools like LLM Arena (an open-source evaluation platform for comparing large language models across tasks like reasoning, summarization, and question answering) help, but still miss the mark when it comes to real, messy, human workflows.

Today, AI isn’t just a tool—it’s a thinking partner. It helps us brainstorm, organize thoughts, recall context, and challenge assumptions—like a smart assistant inside our head. It simulates thinking, not just output.

This shift is rooted in a concept called cognitive offloading—when we rely on external tools to lighten our mental load. Humans have done this for centuries with notebooks, calendars, calculators—and now, with AI. But AI goes further: it helps us explore ideas, test hypotheses, reason through decisions, and refine solutions.

Its real power lies in enhancing how we think. And that’s exactly what traditional benchmarks miss. Which brings us to the obvious question: if that’s how people use AI in the real world, why are we still measuring it like a student taking a timed exam?

How Does The Benchmark Fall Short?

Most benchmarks were built for static tasks: translate a sentence, answer a trivia question, solve a math problem. But that’s not how AI shows up in real work. In real-world use, AI is collaborative, contextual, and iterative. It helps navigate ambiguity, weigh trade-offs, and adapt as the task evolves. It's less “ask and answer,” more “trial, reflect, refine.”

Yet today’s benchmarks are:

Single-shot — Measure isolated answers, ignoring the back-and-forth collaboration real users rely on.
Static — Use rigid prompts that don’t evolve with the workflow or changing goals.
Human-detached — Miss the dynamic between user and AI, where the real power lies.
Purely performance-based — Focus only on outputs, not how AI expands human thinking or creativity.

If AI is meant to augment how we think, testing it without the human in the loop is like evaluating a pen by how well it writes—with no ink.

How Humans Actually Use AI?

Let’s go back to cognitive offloading. It helps us focus, avoid overload, and think more clearly and AI supports this like no tool before.

Memory Management: AI remembers context so you don’t have to.
Brainstorming: AI generates options, you curate or combine.
Search Heuristics: You guide the direction, AI fills in the blanks.
Iterative Reasoning: You explore together, not in isolation.

This is a thinking partner, one that remembers, challenges, and evolves with you. Over time, it becomes a loop that sharpens how you think.

That’s why the human brain is the benchmark. Not because AI should mimic our minds, but because its value lies in extending them. If a benchmark misses that dynamic, it misses the point.

If you’re curious to go deeper into the psychology behind this shift, we highly recommend this video by David Shapiro that explores how AI helps humans navigate complex problems by supporting intuition, feedback loops, and heuristic thinking.

AI as a thinking partner illustration — AI as a thinking partner

What We're Doing at Quash

We realized early on: traditional benchmarks don’t capture how AI fits into our product. That’s because our workflows aren’t static, AI is embedded across the QA lifecycle.

From generating test cases and scripts to analyzing branch diffs and surfacing bugs, AI helps users stay in flow and make smarter decisions. So instead of judging models on one-shot outputs, we evaluate them across real QA workflows—like test generation, execution, and change analysis to see how well they actually support the user’s journey.

Recently, we ran a focused evaluation of different LLMs across our test case generation workflow. The differences were clear: some models missed edge cases, others repeated steps, while a few stood out with thoughtful coverage and depth. But we’re not treating these results as the final word. Instead, they’ve helped us identify where to build stronger systems around the models, so we can reduce hallucinations, catch invalid logic, and fine-tune outputs. We’re continuously evolving our AI pipelines to align better with the strengths of newer models and push the overall quality higher, not just by switching models but by designing smarter systems around them.

AI Model	PRD Quality	Figma Quality	Code Quality	Output Score	Remarks
OpenAI GPT 4.1	High	High	Good	8.43	Best overall performance. Good coverage, details, and accuracy.
	Low	High	Good	8.38
	High	Low	Good	8.08
	Low	Low	Good	7.94
OpenAI o4-mini	High	High	Good	8.05	Good, but lacks as much detail. Also gave a few invalid test cases.
	Low	High	Good	7.93
	High	Low	Good	7.65
	Low	Low	Good	7.10
Anthropic Claude 3.7 Sonnet	High	High	Good	8.39	Best in coverage, good details, but giving some duplicates (can be improved)
	Low	High	Good	8.33
	High	Low	Good	7.96
	Low	Low	Good	7.54
Gemini 2.5 Pro	High	High	Good	7.98	Great detailing, accuracy, but missed out on coverage.
	Low	High	Good	7.89
	High	Low	Good	7.45
	Low	Low	Good	7.37

Because for us, a good AI tool isn’t just accurate. It should remember context, run actions, look things up, and spot patterns that move the work forward.

That’s the bar we care about.

Check out our Recent Blogs

BDD Testing vs. TDD: Which Approach Suits Your Team Better?

Choosing between BDD and TDD impacts how your team collab...

Read article

Test-Driven Development (TDD) Testing: A 2025 Field Guide

In this 2,100-word field guide, we break down Test-Driven Development for...

Read article

news

This Week in AI

OpenAI to Acquire AI Coding Startup Windsurf for $3B

OpenAI is set to acquire Windsurf (formerly Codeium), an AI-powered coding assistant startup, for around $3 billion — its largest deal yet. The move strengthens OpenAI’s position in the competitive AI coding space, as tools like Copilot and Gemini continue to evolve.

OpenAI Rolls Back GPT-4o Update After User Backlash

OpenAI is reverting its latest GPT-4o update after complaints that the model became overly agreeable and sycophantic. CEO Sam Altman confirmed the rollback and bid farewell to GPT-4 in a post, saying, “You kicked off a revolution,” and noting the model's legacy will be preserved for history.

Meta Launches AI App for Personalized Assistant Experience

Meta has released the first version of its standalone Meta AI app, powered by Llama 4. The app offers a personalized assistant that remembers context, supports voice interactions, and connects seamlessly with Meta’s platforms and smart glasses. It also includes a “Discover” feed to explore how others use AI.

Claude Adds Integrations and Smarter Research Mode

Anthropic has introduced Integrations for Claude, letting users connect apps and tools directly to the AI. This gives Claude deeper context about your work—like project history or task updates—and enables it to take informed actions. Advanced Research Mode has also expanded to search the web, Google Workspace, and connected integrations, generating detailed, citation-backed reports after 45 minutes of research. Web search is now available to all paid Claude users globally.

Meta Launches Llama API in Preview for Easier Dev Access

Meta has announced the Llama API in limited free preview—offering the ease of closed model APIs with the freedom of open source. Developers can explore Llama 4 models like Scout and Maverick through an interactive playground, spin up API keys instantly, and build apps using lightweight Python and TypeScript SDKs. It’s also OpenAI SDK-compatible, so existing apps can migrate easily.

Benchmarks Are Dead—Your Brain Is the New Standard