How AI meeting notes work, from recording to summary

An AI note taker does four jobs in a row. It captures the sound, turns speech into written words, works out who said what, and then shortens the words into a summary. Each job can go wrong in its own way, and knowing the order helps you find where a mistake came from.

Step one: the audio

Everything starts with a recording of the call or the room. Some tools join a video call as a participant. Others run inside the video call software, or record from a phone or laptop microphone.

The quality of this audio sets the ceiling for everything after it. Speech gets distorted by background noise, echoes and the way it was recorded¹. A tidy transcript can't be made from a muddy recording.

Step two: speech becomes text

Speech recognition turns the audio into written words. The result is the transcript. In Microsoft Teams, for example, transcription creates a real-time written record of what was said during a meeting².

Accuracy is usually rated with a measure called word error rate, or WER¹. If a maker quotes a figure for it, ask what audio it was tested on. Your office, with its own voices and noise, is a different test.

Accents, pronunciation, pitch, volume and speed all change how well the software hears¹. That's why one person's words come out clean and another's don't.

Step three: who said what

A transcript with no names is hard to use. Speaker diarisation (also spelt diarization) fixes that. It splits an audio stream into segments according to the identity of each speaker³. It's meant to answer the question "who spoke when?"³.

Note what it does and doesn't do. Diarisation separates voices into turns. Giving each voice a true identity takes speaker recognition on top³. In Teams, participants can view the transcript afterwards with timestamps and speaker attribution².

If two people talk at once, or someone joins on a shared speakerphone, the labels can slip. Check them.

Step four: the summary

Automatic summarisation is the process of shortening a set of data computationally⁴. For meetings, the tool feeds the transcript to a language model and asks for a shorter version: topics, decisions and next steps. Wikipedia notes that tools in Microsoft 365 and Google Workspace use large language model-based summarisation to generate meeting recaps⁴.

A summary only knows what's in the transcript. If the transcript missed a word, the summary builds on the gap. If something was agreed by a nod, or by a conversation after the call ended, it isn't there at all.

Why a person still reads it

Each stage passes its errors along. A wrong word in the transcript can become a wrong fact in the summary, and the summary will read smoothly either way. The page What to check in AI meeting notes before you share them lists what to look for.

The differences between the transcript and the summary, and when you want each, are covered in Transcript or summary: what each one is for. Whether and how to tell people about the recording is on Telling people you are recording a meeting.