How we test AI note takers
Every tool hears exactly the same audio under the same conditions. A script, not a reviewer's impression, counts the mistakes.
The recordings
- A four-person meeting: meeting ES2008a from the AMI Meeting Corpus (AMI Consortium, University of Edinburgh), licensed CC BY 4.0, about 17 minutes. We use the headset mix of the audio and the corpus's human-made transcript as the reference.
- A non-native speaker: a scripted weekly meeting of about 2.5 minutes read by a Lithuanian speaker of English. The script is the reference transcript.
Planned for a later round, not yet tested: a noisy version of the first recording.
How each tool is run
Each recording is played into one Google Meet call, as a shared browser tab with its audio. Tools that join as a bot are added by pasting the meeting link; tools that record on the computer (Granola and Fathom, on a Mac) run on the same machine. After the call, we copy the transcript and the summary exactly as the tool produces them, with no edits.
Two things we removed before scoring: lines that a tool wrote from our own microphone and that were not in the audio under test, and the speaker names and timestamps (see below). We keep those lines in the test notes.
What we measure
| Measure | How | In round 1 |
|---|---|---|
| Word error rate (WER), meeting | Wrong, missed and added words divided by the words in the reference transcript. Capitalisation, punctuation, filler words ("um", "uh"), speaker names and timestamps are ignored. | Scored |
| Word error rate, non-native speaker | The same WER on the non-native speaker recording. | Scored |
| Summary completeness | Before the test we list each recording's decisions and action items. We count how many the summary contains, and check borderline cases by hand. | Scored |
| Cost | The cheapest paid plan and the free plan's limits, from the tools' own pricing pages on the test date. | Listed. Cost per meeting hour is not calculated yet. |
| Speaker labels | A sample of lines checked by hand: is each sentence attributed to the right person? | Not scored: every voice arrived on one audio channel (see limits) |
| Speed | Minutes from the end of the call until the notes are ready. | Not measured yet |
Limits of this method
- Every tool hears the audio through Meet's compression, not the original file.
- One audio source means every voice arrives on one channel, so speaker labels are not a fair test; we do not score them.
- The non-native result is one speaker and one recording. It says how the tools handled this voice, not Lithuanian accents in general.
- AMI is a public dataset and widely used, so a tool may have seen it before.
- All tools were tested on free plans, on one day. Fathom offered a Premium trial at sign-up. No trial confirmation email arrived, so we believe the test ran on the free plan, but we did not check the plan screen on the test day. Free-plan limits changed the result once: Fireflies hid most of its transcript of the second recording, so it is scored on its summary only.
- Bots keep recording until stopped, so recording lengths shown in the apps are longer than the audio.
- Where we changed the scoring after seeing the outputs (for example, accepting a reworded checklist item), we say so in the article and apply the same rule to every tool.
Fairness rules
- Every result shows the test date and the plan used.
- We retest every 3–6 months, because these tools change fast.
- Affiliate commissions never change a number or a ranking. If the best tool pays us nothing, it still wins.
- Vendors can ask us to rerun a test, and we publish corrections.
The scoring script
A script (tools/measure.py) aligns each transcript with the reference word by word and counts the differences. It treats "forty two" and "42", "OK" and "okay", and "€12.50" and "12,50 euro" as the same. The script is not published yet; the rules it applies are the ones on this page. Questions about a result: contact.