How an AI picks “the best moment” in a long video (and what it still gets wrong)

An AI does not watch your video. It reads it.

The first thing to know about an AI clip generator is what it receives. Not the pictures: the words. The video is transcribed first, and the transcription engine returns every word with its start and end time in milliseconds (flysub uses AssemblyAI for this step). The words are then grouped into sentences, and each sentence becomes one line of a script that looks like this:

[412|t=1534] The first thing I did was fire my best client.

The number is the line id, and t is the second where the sentence starts. The model reads a few hundred such lines and answers with ids: the line that hooks, the line that carries the payoff, the last line. It never answers with a time.

That constraint is deliberate. A language model has no idea where second 1,534 falls in the audio; it reads text. By asking it only for sentence numbers, you keep it on the one thing it is good at, judging what a passage says, and you leave the clock to code that can actually read it.

The five rules of a good clip, as written in the prompt

The instructions flysub sends with the transcript cast the model as a senior editor of vertical shorts, a specialist of podcasts and interviews, looking for passages that can be published as they are: cut only at the start and at the end, never in the middle. A good clip, the prompt says, obeys five rules.

1. It stands alone. Someone who lands on it while scrolling, without having seen the episode, understands from the first sentence what it is about. So the first line never starts with a connective that continues what came before (“and”, “but”, “so”, “because”), a pronoun that points back to something already said (“it”, “those”, “this reduction”), or a reaction (“that’s a good point”, “as I was saying”). If the answer only makes sense with the question, the clip starts on the question, provided it fits in one short sentence.
2. It hooks in the first 3 seconds. The first line is short and strong: a question that intrigues, a blunt or counter-intuitive claim, a striking number, a story that begins, a confession, an emotion.
3. It develops one idea. No digression.
4. It ends on its payoff. The answer, the conclusion, the lesson, the punchline or the twist. Never on a question without its answer, never on a sentence that announces a follow-up (“two things: the first…” without the second), never in the middle of a story.
5. It runs 20 to 90 seconds, ideally 30 to 60. A complete 75-second idea beats a 45-second idea cut short. Under 20 seconds is reserved for a punchline that needs nothing else.

The prompt also lists what to leave out: the episode’s introduction and conclusion, the guest’s presentation, thanks, sponsors and calls to subscribe; logistics and technical incidents (“let me plug in my mic”); a long question followed by the mere start of an answer; a company’s or a product’s sales pitch; and two passages that carry the same idea, where only the better one survives.

Why those lengths, and why the first 3 seconds? The platforms set the outer boundary and the audience sets the inner one. A YouTube Short can run up to 3 minutes, and a Reel longer than 3 minutes is no longer recommended to new audiences. Nobody is stopped by the ceiling. Clips fail at the floor, when they start in the middle of an idea. The head matters because the platforms measure it: YouTube’s Shorts analytics report the share of viewers who “stayed to watch past the initial seconds”, Instagram tells creators to start with a hook in the first 3 seconds and watch the view rate after 3 seconds, and TikTok’s Creator Academy asks for an attractive element ideally within the first 5 seconds. The 3 seconds in flysub’s prompt are the strictest of the three.

Four marks and one score

For each passage, the model writes a 20-word analysis before it grades anything, then gives four marks from 1 to 10, severe and compared across passages (“a 9 or a 10 remains rare”):

• Hook: do the first 3 seconds stop the scroll?
• Self-contained: does a stranger understand everything with nothing else?
• Value: do you learn or feel something, do you want to share it?
• Payoff: does the ending reward the watch?

The code turns the four marks into one score out of 100. The weights are not equal:

The model also writes a title of 70 characters at most, in the language of the transcript, with one hard rule: it promises nothing that is not said between the first and the last line. No number, name or conclusion borrowed from elsewhere in the episode. A title that oversold the clip was the most common defect of our first version.

How many clips? About one per 5 minutes of video, between 5 and 10, and that number is a ceiling, not a target. The prompt is explicit: never pad with average passages; if only three are good, return three.

Then the code checks everything

A model can be confidently wrong, so every passage it proposes goes through deterministic checks before it becomes a clip. These are not AI. They are rules written in code, and they run the same way every time.

• Start on a sentence start. If the proposed first line begins mid-sentence, the code backs up to the beginning of that sentence, up to 3 lines; beyond that, the passage is rejected.
• End on a real sentence end, never before the payoff. The end moves forward up to 2 lines to reach a sentence that actually finishes; a passage that ends on an unfinished sentence is rejected.
• A connective at the start pulls in the previous sentence. If the first line still begins with “and”, “because” or “so”, the code includes the sentence before it, the one that sets the context, as long as the clip stays under 90 seconds. Once only.
• Too long? Shorten from both ends. The end is brought back to the payoff line. If the clip is still over 90 seconds, the start moves closer to the payoff, onto the latest sentence start that keeps the clip under 60 seconds (or, failing that, under 85) and begins neither with a connective nor with a pronoun. The model often picks too long a set-up; its payoff is usually right.
• No dead air in the head. A silence of more than 2 seconds inside the first 3 seconds, and the clip starts at the next sentence instead. Nobody waits through silence in a feed.
• 15 seconds minimum. Shorter than that after all corrections, the passage is dropped.
• Cut between words, with room to breathe. The final cut snaps to word boundaries: 180 milliseconds before the first word, 320 milliseconds after the last one, and never within 60 milliseconds of a neighbouring word. A clip never opens or closes in the middle of a syllable.

Each rejection has a name in the logs: wrong ids, start mid-sentence, unfinished end, too long, too short, dead air, overlap with a clip already taken. That is how the rules get tuned: not from a feeling, from counts.

A second AI reads every candidate again

Since 25 September 2026, every candidate that survives the guardrails is read a second time by the model, in a different role: the editor-in-chief who approves shorts before publication. It sees the proposed passage with the 3 lines before and the 3 lines after, the exact duration and the silences, and it is told to be demanding: a short that starts without context or cuts its payoff is a failure; when in doubt, reject.

In that pass the model can move the start and the end within the lines it sees, grades the four criteria again, decides whether to keep the clip, and names its reason when it does not: start without context, ending without payoff, wrong duration, intro or outro, duplicate of a better candidate, no interest. It also rewrites the title to match the final passage, because the first pass often titled a longer passage than the one that was kept.

To give the second pass something to reject, the first pass is asked for twice as many candidates as the target (three times as many when you ask for “5 more clips” on a video already clipped, since the lines already used are masked). The extra cost is small: about $0.035 per import of an hour and forty minutes.

What the AI still gets wrong

When we audited the first version of this pipeline on 24 September 2026, 4 clips out of 27 were publishable. The rest started in the middle of an idea, cut their payoff, or carried a title that promised what came after the clip. The rules above come from that audit. They fixed most of it, not all of it. Here is what still goes wrong, and why.

• Moments that live in the picture. A demo on screen, a face that reacts, a gesture, a chart: none of it is in the transcript. The model cannot pick what it cannot read. If your best moment is visual, it will not be in the first batch.
• Humour carried by the voice. Irony and deadpan read flat on paper. A line that kills in the room can score 5 on value in text.
• Long-range references. The pronoun rule catches “it” and “this”, not a name introduced at minute 2 and reused at minute 40 as if everyone knew it. The second pass sees 3 lines of context, not 40 minutes.
• Transcription mistakes. A misheard proper noun becomes a wrong title, and word-by-word captions repeat the error on screen until you fix it.
• Two voices at once. Cross-talk produces sentences that start and end nowhere. The code rejects them, and the moment is lost.
• The audience it does not know. The model grades for “a stranger”. Your subscribers are not strangers: an inside joke that fails the stand-alone test may be your best clip for them.

And when the model returns nothing usable, flysub does not show an error. A fallback without AI ranks 20-to-60-second blocks by speech density, keeps those that end on a full sentence, and completes the list up to three proposals with a flat score of 35. You will see them with that score: a proposal, not a recommendation.

Check a clip in 60 seconds

1. Watch the first 3 seconds without sound.If you do not know what the clip is about by then, it starts too late or too early.
2. Read the title against the clip.Every word of the title must be said in the clip. Numbers and names first.
3. Play the last 5 seconds.The clip must end on the answer, not on the question or on the sentence that announces it.
4. Scrub to the sentence before the in-point.If the clip only makes sense with it, move the start back to it.
5. Check the captions word for word.Proper nouns and numbers first: a word-by-word caption repeats a transcription error on screen.

For the publishing side (related video, title, first comment), see our step-by-step guide to making a Short from a long video; for the craft itself, what video clipping is.