How Tennis Rally Detection Works
August 27, 2026
Record an hour of tennis and you have maybe eighteen minutes of tennis. The rest is walking to the baseline, picking up balls, checking the score, and standing around.
Rally detection is the job of finding those eighteen minutes for you. Here is what it involves, and why the answer changes depending on which tool you use.
The part that is harder than it sounds
You would think a point is easy to find. It is not, and the difficulty is not where people expect.
A rally does not start at the serve. If a clip begins at ball contact you have lost the toss, and a serve without its toss looks wrong to anyone who plays. The useful start is a beat earlier — the player settled at the line, about to begin.
A rally does not end on the last shot either. The point is over when the ball is dead, but the watchable version runs on a little: the ball rolling to the fence, the reaction, whatever gets said. Cut on the final bounce and every clip feels amputated.
So detection is not really “find the ball”. It is “decide where a person would have cut” — twice a point, a hundred times a session, consistently enough that the clips feel like they were edited rather than sliced.
That consistency is most of the value. Doing it by hand for one match is an evening’s work, and the reason people stop doing it is not that it is difficult. It is that it is boring, and they have already watched the match.
Why some tools care where you put your camera
This is the practical difference between tools, and it is worth understanding before you record rather than after.
Some rally detection works by following the ball. Knowing where the ball is, frame by frame, on a court the software has mapped out, is what makes line calls and shot speed possible. If you want to know whether that ball was in, or how fast your serve is, this is what produces those numbers. SwingVision is the best known of these and it is genuinely good at it.
A tennis ball is small and fast, though, and following one takes a cooperative view. That is why the setup advice for these tools is so specific:
- camera behind the baseline, not off to the side
- completely still — a tripod, not propped on a bag
- doubles is harder than singles, because four players and quick net exchanges give the tracker more chances to lose the ball
That is not a flaw. It is what following a ball requires. But it does mean the footage has to be planned, and video you already shot from the wrong end of the court cannot be rescued afterwards.
Other tools work from the scene instead. Rather than locating the ball, they read what is happening: two players in position at opposite baselines, one about to serve, is a point starting. Someone jogging to the fence after a ball is not. It is the same judgement you make watching from the sideline, and you do not track the ball to make it either.
The practical consequence is that the camera angle matters far less. Side-on, up in the stands, slightly off-square, phone leaning against a bag — if a person watching could tell a point was being played, so can the software.
For footage that already exists, that is often the whole question. A season of matches shot from wherever there was room is a difficult library for a ball tracker and an ordinary one for a scene reader.
What you end up with
Worth being blunt about this, because it is where the two really part company.
Scene-based detection will not tell you your first-serve percentage, your average rally length in shots, or whether that ball caught the line. Those numbers come from knowing exactly where the ball was, and this kind of tool never knew.
What it gives you instead is the tennis: every point as its own clip, the dead time gone, ready to watch, trim, keep or throw away.
If the numbers are what you are after, use something built to produce numbers, and film from the baseline. If you mostly want to watch the tennis without the standing around — or share three good points with someone — you never needed the ball’s coordinates for that.
| What you want | What to look for |
|---|---|
| Line calls, shot speed, court positioning | Ball tracking |
| Just the points, from footage you already have | Scene-based detection |
| Both, and you film from behind the baseline anyway | Ball tracking gives you both |
Filming so detection works well
Whichever kind you use, a few things change the result more than anything in the software:
Keep the whole court in frame. Every method degrades when players leave the picture, for the same reason — what is not in shot cannot be judged.
Don’t zoom or pan during points. A moving camera makes a hard problem harder and buys nothing.
Record straight through. Detection works from the contrast between play and everything else. Starting and stopping to “help” removes the contrast, and it is more work for you.
Don’t worry about length. An hour is fine. Length is much more of a problem for you than for the software.
Where Rallytics fits
Rallytics is the second kind. It reads the session rather than following the ball, which is why it does not mind where you filmed from, and why it has nothing to say about whether a ball was in.
You get each rally as its own clip, with the beginning and end placed where a person would have put them. Keep the ones worth keeping, drop the rest, and build a highlight reel from what is left. It runs as a desktop app on Mac and Windows, and your original recording never leaves your machine.
Download Rallytics — free to start, and a full session takes a few minutes to come back.