It’s a nice engineering approach, but I’m interested in the motivation. Um and ah is distracting in a transcript, where you can naturally pause to take in information; in speech however it can serve as a focusing point to indicate the next part is important. See https://medium.com/better-humans/dont-worry-about-saying-um-... for example. The weirdly obsessive zeal that orgs like Toastmasters have about eliminating them is weird.
Disfluencies aren’t necessarily bad even if the word starts with “dis”!
Having heard radio interviews with and without 'internal editing' to remove ums and ahs, most of the time I'd rather the edited version. It's more concise and focused, and I find it easier to comprehend. Too many ums and ahs and my mind wanders, and if it's radio, I can't go easily go back to try again. When I've listened to podcasts or audiobooks, I could never easily go back a little to try again either, and I gave up on them (even though I have some content I really want to listen to, it's too frustrating, so it's not happening). But I'm sure other people have different preferences.
I also don't care for writing that could have been made a lot more concise. It's a lot of work to make things shorter, but I think it's worthwhile.
It just goes to show that people have very different views. I think when I hear people thinking out loud (ums and ahs) it's a marker that they are actually engaging with the question, thinking through an answer and not bullshitting without thinking.
The younger generation seems to love listening at 1.2x or faster. I think it’s a preference for a fast information dopamine hit. I may argue it’s even a shallow approach that prefers against pausing and time for careful reflection. Meanwhile, book reading is at an all time low seemingly because no one has a preference or patience for careful study and reflection.
The most popular academic theory (IIRC) is that "um" and "uh" are conversational placeholders that say, "don't talk, I'm not finished speaking yet". Which obviously serves no purpose in a monologue.
To me they just indicate lack of confidence on the part of the speaker.
There's a correlation between speaking with confidence and bullshitting / corner cutting. Hard, nuanced questions require more thinking time to produce a nuanced answer. But a bullshitter will just confidently answer subtly wrong stuff. But they won't say "uh"! Is that really better?
> in speech however it can serve as a focusing point to indicate the next part is important
it's... exact opposite?
the main (attempted) use for ummms is to keep continuation of speech despite the pause. And the main complaint is exactly that it ruins the focus and doesn't give respite
>The weirdly obsessive zeal that orgs like Toastmasters have about eliminating them is weird.
If you speak with disfluencies, you probably didn't sufficiently rehearse your speech. If you didn't rehearse enough, you probably didn't put much effort into writing it either, so why should I put much effort into listening? It's the same principle as AI slop.
Not necessarily true, more rehearsal isn't the key to fluent oratory.
Many people can speak off the cuff fluently and confidently, avoiding "like", "um", and other filler words. And even if you're not speaking fluently, leaving silences as punctuation is more effective, IMO.
Many impressive speakers I've met actually cite Toastmasters! So their obsessive zeal actually does work.
More rehearsal does work too sometimes, but it does sometimes lead to speeches "sounding too rehearsed".
> Many people can speak off the cuff fluently and confidently, avoiding "like", "um", and other filler words.
I don't think that's true, we usually just don't notice filler words in the same way we are surprised that people usually don't even talk in whole sentences, in contrast to written text or movies (which also use written text).
Not to promote something, but Wispr Flow does that for me automatically if I trigger a setting for it..
While it's a commercial product with a subscription, I spent a long time on the free tier not even hitting their limits until I started using it so extensively that I wanted to pay for it.
And I've used Whisper in the past, mostly for tinkering. I tried it for a couple of use cases but haven't touched the base project in a while. But I do regularly use Faster-Whisper-XXL, an open source project based on Whisper, for subtitle generation.
Though, for subtitle generation, I decided to support the project and mainly use the non-public build of Faster-Whisper-XXL Pro built for donators to the open source project.
The extra features smooth out the subtitle editing process very substantially. Toss in "--roformer_overlap 0.125 --roformer_vram 16 --best_of 15 --ff_vocal_extract mb-roformer --vad_method pyannote_v3" to the cli parameters (and sometimes --realign) and you have much less work to do in SubtitleEdit or Tero Subtitler afterwards to clean it up.
Surprisingly, it's the whisper model itself that does that. I find that it's also good with false starts, often correcting something like: "uhm, we could...we can go there" to just "we can go there", if spoken rapidly enough.
Is love to hear more about subtitle generation. Specifically, can you label different speakers? I'd be using this for meeting transcription. Thank you.
When I was doing podcasts regularly, it made me acutely aware of various people's speech mannerisms. (Somewhat similarly, recording a lot of videos during COVID made me very aware of a variety of my own mannerisms--especially overactive hand motions.)
This approach seems kind of backwards to me. Why try to detect everything except the thing you're trying to remove instead of either sampling a few uhs and ums and treating them as noise to be silenced (with a sharp crossfade to the noise floor that doesn't interrupt speech flow) or finetuning a model to detect them specifically for full automation?
> instead of either sampling a few uhs and ums and treating them as noise to be silenced
If you're not paying ttention, ctting out specific sounds can easily cause more trouble. I for one would be quite pset if I couldn't hear the pire's reasoning for calling a foul.
I think the “What it won’t touch” section shows why the entire concept is unsound. Here it is with a different first sentence, and (other than the third sentence no longer matching erm’s reality) it’s perfectly coherent:
> It leaves um, uh, er and elongated versions (ummmm, uhhhhh) alone. Those sound like fillers but they’re doing real work in the sentence, and cutting them automatically would change what someone said. The rule erm follows: only remove things that are sound, not language.
> It also doesn’t touch repeated words, false starts, or long thinking pauses. Those aren’t noise on top of the speech; they are the speech, just messier than the speaker would like. Cleaning them up is an editorial decision about which take to keep, and erm doesn’t have an opinion about that.
Think about it. Cleaning these things-that-can-be-just-sounds-but-can-also-very-much-be-load-bearing up is an editorial decision. At the very least, you need to judge based on the surrounding content whether the removal of an um would change the meaning at all; and I don’t think text alone is adequate for that.
>> It leaves um, uh, er and elongated versions (ummmm, uhhhhh) alone.
Something's already gone wrong here. Uh and er refer to the same sound. Uh is the American spelling. Er is British; to them a following "r" like that is just a kind of vowel.
They're quite different vowel sounds in the same sense that "back" and "back" use "quite different vowel sounds" when pronounced by American vs British speakers.
But not in any other sense.
> in case it wasn’t clear: I was quoting from the start of the article in that sentence.
You don't seem to be quoting from the article at all, actually. You've combined two different sentences in a way that grossly misrepresents what the article says. But that's not really relevant to the point here.
I wonder if with enough input data and transcription you could “fingerprint” where a speaker personality has habits of interjecting “ums” leading to more hardy analysis. Novel approach, but gets me thinking
No, you run an entire second pass LLM over the output of Whisper. "no uhhh three no four." should just output four the numeral not even f.o.u.r.
Hi, my name is fragmede. Judging by the date on my computer it's been four months since it's since I've t touched the transcription directory on computer and tried to improve on the state of wisprflow. Mines pretty good but it just doesn't... ah you can't drag me back in.
I would love to see support for videos and removal of custom filler words (I say 'basically' and 'like' a lot and have so far failed to improve myself on this).
This is great, I've tried out automated podcast editing tools before and they cut too aggressively in my experience. What are you thinking about doing next with this now that you've gotten the alignment snapping working cleanly for 'um' and 'ah', are you thinking of expanding the tool?
What an awesome tool and idea. I’d be keen to see if it can integrate with video editing tools.
Ideally it would slice the video in the timeline without actually removing anything, so you can scrub through your video and try with and without each disfluency (thank you - awesome word) & decide case by case which to keep!
I think it is harder to remove those from your own speech. I have been doing that for few months now and I still get back at it when I am in hurry or stressed.
In my experience native English speakers are particularly bad, generally when speaking a second language people are less likely to add random filler words.
Also the type of filler word for some reason is often different between UK and US: British people tend to be "umm"-ers and Americans are more likely to add "you know" (although "umm" is also common).
Once you notice it it's impossible to ignore and many, many native English speakers are actually terrible at speaking and add filler words to the point where it's very distracting
Really cool stuff and definitely going to try it; I’m also finding it wild that Google put effort into adding ums and erms into their text to speech model a while back. AI puts it in, AI helps take it out.
This post is mostly about how surprisingly hard it is to cut filler words out of speech cleanly. Apparently, stripping ums isn't a find and replace type thing, because Whisper's timestamps are off by up to a few hundred ms and cutting on them chops syllables or leaves stutters. So, I built a tool, erm, that starts from Whisper's guess, finds where each word actually starts and stops in the audio, and snaps the cuts to silence so there's no click, with ffmpeg doing the splicing.
It’s a nice engineering approach, but I’m interested in the motivation. Um and ah is distracting in a transcript, where you can naturally pause to take in information; in speech however it can serve as a focusing point to indicate the next part is important. See https://medium.com/better-humans/dont-worry-about-saying-um-... for example. The weirdly obsessive zeal that orgs like Toastmasters have about eliminating them is weird.
Disfluencies aren’t necessarily bad even if the word starts with “dis”!
Having heard radio interviews with and without 'internal editing' to remove ums and ahs, most of the time I'd rather the edited version. It's more concise and focused, and I find it easier to comprehend. Too many ums and ahs and my mind wanders, and if it's radio, I can't go easily go back to try again. When I've listened to podcasts or audiobooks, I could never easily go back a little to try again either, and I gave up on them (even though I have some content I really want to listen to, it's too frustrating, so it's not happening). But I'm sure other people have different preferences.
I also don't care for writing that could have been made a lot more concise. It's a lot of work to make things shorter, but I think it's worthwhile.
It just goes to show that people have very different views. I think when I hear people thinking out loud (ums and ahs) it's a marker that they are actually engaging with the question, thinking through an answer and not bullshitting without thinking.
I agree to you, when it's in person. I think what your describing is mostly the beginning of an answer.
Just randoms "um" inbetween because your struggling to build sentences can get annoying both in person and online
The younger generation seems to love listening at 1.2x or faster. I think it’s a preference for a fast information dopamine hit. I may argue it’s even a shallow approach that prefers against pausing and time for careful reflection. Meanwhile, book reading is at an all time low seemingly because no one has a preference or patience for careful study and reflection.
The most popular academic theory (IIRC) is that "um" and "uh" are conversational placeholders that say, "don't talk, I'm not finished speaking yet". Which obviously serves no purpose in a monologue.
To me they just indicate lack of confidence on the part of the speaker.
There's a correlation between speaking with confidence and bullshitting / corner cutting. Hard, nuanced questions require more thinking time to produce a nuanced answer. But a bullshitter will just confidently answer subtly wrong stuff. But they won't say "uh"! Is that really better?
Sure, that figures. Much of this is surely subjective.
> in speech however it can serve as a focusing point to indicate the next part is important
it's... exact opposite?
the main (attempted) use for ummms is to keep continuation of speech despite the pause. And the main complaint is exactly that it ruins the focus and doesn't give respite
It can be a focusing point when someone wants to highlight the deliberate use of euphemism, removing those would be, um, unwise.
Although that is probably the less common use.
I think you’re both right. But you’re right regarding writing and your parent comment is right regarding speech.
Occasional ums and ahs are fine but when every other phrase starts with a long aaaaah it can be pretty unpleasant to listen to.
So, if this project's source Audio were Beavis and Butthead, you would be enthused?
>The weirdly obsessive zeal that orgs like Toastmasters have about eliminating them is weird.
If you speak with disfluencies, you probably didn't sufficiently rehearse your speech. If you didn't rehearse enough, you probably didn't put much effort into writing it either, so why should I put much effort into listening? It's the same principle as AI slop.
Not necessarily true, more rehearsal isn't the key to fluent oratory.
Many people can speak off the cuff fluently and confidently, avoiding "like", "um", and other filler words. And even if you're not speaking fluently, leaving silences as punctuation is more effective, IMO.
Many impressive speakers I've met actually cite Toastmasters! So their obsessive zeal actually does work.
More rehearsal does work too sometimes, but it does sometimes lead to speeches "sounding too rehearsed".
> Many people can speak off the cuff fluently and confidently, avoiding "like", "um", and other filler words.
I don't think that's true, we usually just don't notice filler words in the same way we are surprised that people usually don't even talk in whole sentences, in contrast to written text or movies (which also use written text).
As with all things ... Don't be opinionated and make it an option for the user.
Looks interesting, would be a nicer article though if there was a demo with before/after to show the results, and why the previous ideas didn't work
for something dealing with audio you do need to play the audio really
Not to promote something, but Wispr Flow does that for me automatically if I trigger a setting for it..
While it's a commercial product with a subscription, I spent a long time on the free tier not even hitting their limits until I started using it so extensively that I wanted to pay for it.
And I've used Whisper in the past, mostly for tinkering. I tried it for a couple of use cases but haven't touched the base project in a while. But I do regularly use Faster-Whisper-XXL, an open source project based on Whisper, for subtitle generation.
Though, for subtitle generation, I decided to support the project and mainly use the non-public build of Faster-Whisper-XXL Pro built for donators to the open source project.
The extra features smooth out the subtitle editing process very substantially. Toss in "--roformer_overlap 0.125 --roformer_vram 16 --best_of 15 --ff_vocal_extract mb-roformer --vad_method pyannote_v3" to the cli parameters (and sometimes --realign) and you have much less work to do in SubtitleEdit or Tero Subtitler afterwards to clean it up.
Surprisingly, it's the whisper model itself that does that. I find that it's also good with false starts, often correcting something like: "uhm, we could...we can go there" to just "we can go there", if spoken rapidly enough.
Is love to hear more about subtitle generation. Specifically, can you label different speakers? I'd be using this for meeting transcription. Thank you.
When I was doing podcasts regularly, it made me acutely aware of various people's speech mannerisms. (Somewhat similarly, recording a lot of videos during COVID made me very aware of a variety of my own mannerisms--especially overactive hand motions.)
This approach seems kind of backwards to me. Why try to detect everything except the thing you're trying to remove instead of either sampling a few uhs and ums and treating them as noise to be silenced (with a sharp crossfade to the noise floor that doesn't interrupt speech flow) or finetuning a model to detect them specifically for full automation?
> instead of either sampling a few uhs and ums and treating them as noise to be silenced
If you're not paying ttention, ctting out specific sounds can easily cause more trouble. I for one would be quite pset if I couldn't hear the pire's reasoning for calling a foul.
I think the “What it won’t touch” section shows why the entire concept is unsound. Here it is with a different first sentence, and (other than the third sentence no longer matching erm’s reality) it’s perfectly coherent:
> It leaves um, uh, er and elongated versions (ummmm, uhhhhh) alone. Those sound like fillers but they’re doing real work in the sentence, and cutting them automatically would change what someone said. The rule erm follows: only remove things that are sound, not language.
> It also doesn’t touch repeated words, false starts, or long thinking pauses. Those aren’t noise on top of the speech; they are the speech, just messier than the speaker would like. Cleaning them up is an editorial decision about which take to keep, and erm doesn’t have an opinion about that.
Think about it. Cleaning these things-that-can-be-just-sounds-but-can-also-very-much-be-load-bearing up is an editorial decision. At the very least, you need to judge based on the surrounding content whether the removal of an um would change the meaning at all; and I don’t think text alone is adequate for that.
>> It leaves um, uh, er and elongated versions (ummmm, uhhhhh) alone.
Something's already gone wrong here. Uh and er refer to the same sound. Uh is the American spelling. Er is British; to them a following "r" like that is just a kind of vowel.
Um… no. Quite different vowel sounds.
(Also, in case it wasn’t clear: I was quoting from the start of the article in that sentence.)
They're quite different vowel sounds in the same sense that "back" and "back" use "quite different vowel sounds" when pronounced by American vs British speakers.
But not in any other sense.
> in case it wasn’t clear: I was quoting from the start of the article in that sentence.
You don't seem to be quoting from the article at all, actually. You've combined two different sentences in a way that grossly misrepresents what the article says. But that's not really relevant to the point here.
This resonates with our crusade to eradicate Ums once and for all.
- Ums Considered Harmful: https://hamanlp.org/research/ums/
- Related paper: https://hamanlp.org/SIGBOVIK_2026.pdf
I wonder if with enough input data and transcription you could “fingerprint” where a speaker personality has habits of interjecting “ums” leading to more hardy analysis. Novel approach, but gets me thinking
This is fascinating! I'm going to try this on a certain clip from Jurassic Park.
...
No, you run an entire second pass LLM over the output of Whisper. "no uhhh three no four." should just output four the numeral not even f.o.u.r.
Hi, my name is fragmede. Judging by the date on my computer it's been four months since it's since I've t touched the transcription directory on computer and tried to improve on the state of wisprflow. Mines pretty good but it just doesn't... ah you can't drag me back in.
I would love to see support for videos and removal of custom filler words (I say 'basically' and 'like' a lot and have so far failed to improve myself on this).
It takes about 30 seconds in Audacity and will give an infinitely better result. Also works on any other sound
Doesn't sound true. Unless audacity already has a tool for this exactly... How would you do it on 30 seconds or less?
It doesn't and ums aren't the only consistent tic you often want to clean up--"you know," long pauses, etc.
This is great, I've tried out automated podcast editing tools before and they cut too aggressively in my experience. What are you thinking about doing next with this now that you've gotten the alignment snapping working cleanly for 'um' and 'ah', are you thinking of expanding the tool?
What an awesome tool and idea. I’d be keen to see if it can integrate with video editing tools.
Ideally it would slice the video in the timeline without actually removing anything, so you can scrub through your video and try with and without each disfluency (thank you - awesome word) & decide case by case which to keep!
What a very cool utility.
I think it is harder to remove those from your own speech. I have been doing that for few months now and I still get back at it when I am in hurry or stressed.
In my experience native English speakers are particularly bad, generally when speaking a second language people are less likely to add random filler words.
Also the type of filler word for some reason is often different between UK and US: British people tend to be "umm"-ers and Americans are more likely to add "you know" (although "umm" is also common).
Once you notice it it's impossible to ignore and many, many native English speakers are actually terrible at speaking and add filler words to the point where it's very distracting
BTW, any recommendations for AI tools that remove the laugh track? I don't even mind the awkward acting without the missing laughter.
there is a aah counter in toast master !! this is the software that helps !!
Really cool stuff and definitely going to try it; I’m also finding it wild that Google put effort into adding ums and erms into their text to speech model a while back. AI puts it in, AI helps take it out.
Disfluencies are not necessarily "filler". They can convey mood or hesitation. Cutting them can change the meaning.
A trivial example is "umm... well... (sigh) okay" versus just "okay". Not okay!
This post is mostly about how surprisingly hard it is to cut filler words out of speech cleanly. Apparently, stripping ums isn't a find and replace type thing, because Whisper's timestamps are off by up to a few hundred ms and cutting on them chops syllables or leaves stutters. So, I built a tool, erm, that starts from Whisper's guess, finds where each word actually starts and stops in the audio, and snaps the cuts to silence so there's no click, with ffmpeg doing the splicing.
https://github.com/dougcalobrisi/erm