Meeting Tools Lab

How we test AI note takers

Last updated: 15 November 2026

Every tool hears exactly the same audio under the same conditions. A script, not a reviewer's impression, counts the mistakes.

The recordings

  1. A four-person meeting: meeting ES2008a from the AMI Meeting Corpus (AMI Consortium, University of Edinburgh), licensed CC BY 4.0, about 17 minutes. We use the headset mix of the audio and the corpus's human-made transcript as the reference.
  2. A non-native speaker: a scripted weekly meeting of about 2.5 minutes read by a Lithuanian speaker of English. The script is the reference transcript.

Planned for a later round, not yet tested: a noisy version of the first recording.

How each tool is run

Each recording is played into one Google Meet call, as a shared browser tab with its audio. Tools that join as a bot are added by pasting the meeting link; tools that record on the computer (Granola and Fathom, on a Mac) run on the same machine. After the call, we copy the transcript and the summary exactly as the tool produces them, with no edits.

Two things we removed before scoring: lines that a tool wrote from our own microphone and that were not in the audio under test, and the speaker names and timestamps (see below). We keep those lines in the test notes.

What we measure

MeasureHowIn round 1
Word error rate (WER), meetingWrong, missed and added words divided by the words in the reference transcript. Capitalisation, punctuation, filler words ("um", "uh"), speaker names and timestamps are ignored.Scored
Word error rate, non-native speakerThe same WER on the non-native speaker recording.Scored
Summary completenessBefore the test we list each recording's decisions and action items. We count how many the summary contains, and check borderline cases by hand.Scored
CostThe cheapest paid plan and the free plan's limits, from the tools' own pricing pages on the test date.Listed. Cost per meeting hour is not calculated yet.
Speaker labelsA sample of lines checked by hand: is each sentence attributed to the right person?Not scored: every voice arrived on one audio channel (see limits)
SpeedMinutes from the end of the call until the notes are ready.Not measured yet

Limits of this method

Fairness rules

The scoring script

A script (tools/measure.py) aligns each transcript with the reference word by word and counts the differences. It treats "forty two" and "42", "OK" and "okay", and "€12.50" and "12,50 euro" as the same. The script is not published yet; the rules it applies are the ones on this page. Questions about a result: contact.