I put every tool in a category through the same battery, score it out of 100, and publish a dated PASS, BORDERLINE or FAIL. The inputs are public, so you can run the test yourself and tell me where I'm wrong. A new one lands every couple of weeks.
A new test every couple of weeks. No spam, unsubscribe anytime. By subscribing you agree to receive the newsletter and to our Privacy Policy.
Same published inputs, dated and re-runnable. No sponsorships; the verdict isn't for sale.
Each test adds a row. A re-test gets its own dated row, so nothing quietly gets overwritten. The back catalogue is the proof I'm not just chasing whatever launched this week.
| Tool | Category | As of | Score | Verdict | Cost / result | Buy or skip |
|---|
Rows marked pending are queued for the next run. Bands: PASS 75 and up, BORDERLINE 55 to 74, FAIL under 55.
Every tool in a category meets the same battery and the same scoring. The number I care about most is the cost of a result you can actually use.
One task set, one set of inputs, run identically. No friendly demos and no improvising to flatter a particular product.
Quality, reliability, speed, setup friction, cost per result, workflow fit, and the limits nobody advertises. It all collapses into one number out of 100.
Sticker price lies. I track what one output you can trust actually costs, once the retries and the re-dos are in.
Every verdict is pinned to a version and a date, with the inputs published so you can run it yourself and check me.
First run: AI meeting notetakers — tl;dv, Otter, Fireflies, Granola and Fathom (free plans, as of 2026-07-13). This is the same meeting audio every tool received. Download it, feed it to any notetaker, and score it against the fixed set below — written before testing.
Download the audio (MP3) · three speakers, ~2:16 · voices generated with ElevenLabs
20 terms scored: Northwind, Postgres, MySQL, RDS, Kubernetes, HPA, EC2, CloudWatch, TikTok, UTM, Google Analytics, Mailchimp, Datadog, HubSpot, Salesforce, Notion, LinkedIn, SOC 2, Jira, AWS.
10 action items scored: book the Saturday maintenance window (post in Slack by Thursday); open an AWS support ticket with the CloudWatch logs; grant Carla Google Analytics edit access; send three Mailchimp subject-line options for a vote; tune the Datadog latency alert to the 95th percentile; write a dedup script and test it in the Salesforce sandbox; put the $12k budget proposal in Notion; email the SOC 2 auditor to confirm the August date; everyone update Jira tickets before standup; (GA access restated by the PM).
Result: tl;dv 89 · Otter 85 · Fireflies 83 · Granola 66. Action items: 9/10 for everyone except tl;dv, which caught 8 — it dropped the one task addressed to “everyone” rather than to a person. Terms: tl;dv 19/20, Otter 17, Granola 15, Fireflies 14 (Fireflies hears well but writes brand names in lowercase). Speaker labels: tl;dv and Fireflies 93%, Otter 73% (it found only two of the three people), Granola 0% (it doesn’t separate speakers, by design).
Nobody transcribed “RDS.” All five invented a different word instead: Alderdays (Otter), Deus (Fireflies), older days (Granola, Fathom), DS (tl;dv). Three of five heard “EC2” as “easy to.” Every hard error in the test landed on the same speaker — the quietest voice, with the strongest accent. The voices are AI-generated, so treat that as a reproducible probe, not proof of bias in the wild: download the file and try your own recording.
Fathom is not scored. It only records live calls and identifies speakers by who is in the meeting, not by voice — so it cannot be handed a published file and checked. That is a limitation, not a verdict. This is a mini-run (two core tasks plus speaker labels); the full battery adds the rest.
A litmus strip doesn't care about the marketing, and neither does the score. Drag it: the same paper gives every tool the same reading.
You get the scorecard, the raw inputs, and a straight buy-or-skip call, every couple of weeks.
No spam, unsubscribe anytime. By subscribing you agree to receive the newsletter and to our Privacy Policy.