Building a Real-Time Human Pose Estimation Platform for Sports Performance Analysis
Athletes who train without a coach in the room get no feedback on form. We built a human pose estimation platform that reads an athlete's body from a phone camera, compares every movement to the trainer's, and scores the technique before the next rep begins.
- → 34% higher subscription revenue after launch
- → 91% agreement with certified trainers on form scores
- → <80 ms from camera frame to on-screen feedback
- → 140+ exercises across 12 sports at launch
Project Details
The client is a US sports technology startup with a paid training app and about 40,000 subscribers when we started. Users followed video workouts from certified trainers, then trained alone with no way to know whether a squat looked anything like the trainer’s. The founders wanted the app to watch the athlete’s back: estimate the person’s pose from one phone camera, compare it to the trainer’s pose, and say what to fix in real time.
Business Challenge: Creating Accurate Sports Pose Analysis
The startup had tried an off-the-shelf pose detection SDK a year before we met. It ran fine on a demo phone in a bright studio. On a mid-range Android in a garage gym, it dropped to 9 frames per second, lost the human skeleton whenever a dumbbell crossed the torso, and had no notion of what a correct rep looked like anyway. “Wrong feedback” tickets outnumbered billing tickets three to one, and annual-plan churn sat near 6.5% a month.
Three problems had to be solved together, and each one alone would have sunk the old SDK again in real-world scenarios.
Real-Time Performance Requirements
A chest press rep lasts about two seconds. Feedback that arrives after the rep is history, so the loop from camera frame to key points to trainer comparison to on-screen cue had to fit inside 100 ms on a phone, at 30 frames a second.
Multi-Person Detection in Sports Scenarios
Group classes, partner drills, and a kid wandering through the living room all put multiple people in the input image. The old SDK merged two bodies into one skeleton. The new system had to keep each person's pose separate and know which one is being coached.
Deployment Across Diverse Hardware
About 58% of subscribers were on Android across 210 device models, and 30% of those phones were three or more years old. One pose estimation model tuned for a flagship iPhone would have shut most of the paying audience out. Depth sensors were off the table for the same reason.
Technologies Behind the Pose Estimation Platform
Our Human Pose Estimation Solution
One decision shaped everything: the phone does the fast work, the cloud does the precise work. On the device, a pose detection model built on MediaPipe and OpenCV tracks 17 body joints at 30 fps and drives the live cues, with three model sizes so older phones still hit 24 fps. In the cloud, a heavier PyTorch model trained on high-resolution representations of the same frames handles 3D pose estimation after the session, the same deep learning techniques that top the public human pose estimation benchmarks, retrained on gym footage. Real-time performance on the phone, precision on the server.
Multi-person pose estimation runs on a top-down approach: an object detection stage boxes every human body in the frame, then a single-person pose estimator runs inside each box. A bottom-up approach that detects keypoints across the whole image was about 20% faster in crowded scenes, but it swapped limbs between two athletes standing close together roughly one frame in eight. Top-down methods won.
The comparison with the trainer’s pose is where most of the custom work went. Trainers recorded each exercise from three angles, and their 3D pose sequences became the references: 140 exercises across 12 sports at launch. During a rep, the athlete’s body pose is normalized for limb length and camera angle, time-aligned to the reference, and scored joint by joint. A Scikit-Learn classifier turns the deviation into a named cue, “elbows flaring” instead of “left elbow 14 degrees off”. Models trained on 48,000 annotated frames from 60 trainers reached 91% agreement with the trainers’ own scores.
Live Form Scoring Against the Trainer
During each rep, the athlete's key points match the trainer's reference, and an accuracy score appears on screen with a plain-language cue. Feedback lands in under 80 ms, so the next rep can already be better than the last one.
Multi-Person Tracking in Group Scenes
Person detection isolates every human body in the input image, and each one gets its own skeleton. The coached athlete stays locked when a partner crosses the frame, which used to reset the old SDK about twice a minute.
3D Pose Reconstruction After the Session
Uploaded sessions run through the cloud model, which lifts the 2D pose to 3D and catches errors one camera angle hides, such as hip shift in a squat. The corrected accuracy history appears within about 15 minutes of finishing.
Exercise Accuracy History
Every exercise keeps a per-session accuracy chart, so an athlete can watch a dumbbell chest press climb from 65% to 98% over a month.
Coach Dashboard With Remote Sessions
Coaches review any athlete's skeleton overlay next to their own reference and open a WebRTC session when a cue alone falls short. One coach now covers about 3x the athletes they could review by video.
Agile Methodology
Project Journey
The project ran eight months in two-week sprints, and the first two went to filming: 60 trainers, 140 exercises, three camera angles each, plus 2,000 clips shot by subscribers in their own gyms. Discovery settled the split between on-device estimation for cues and cloud 3D pose for history, and fixed the 100 ms feedback budget every later decision was measured against. Live form scoring reached a closed group of 300 subscribers in sprint 6, three months before public launch.
How the Pose Estimation Platform Works
- The app shows the trainer's demo and confirms the entire body is in frame before the set starts.
- A detector boxes every person in the frame. The athlete is locked by position and size; others are tracked separately.
- The on-device model estimates 17 body joints per frame, smoothed over time so a blurred hand cannot jump the score.
- Each rep is normalized, time-aligned to the reference, and scored per joint. A classifier names the error.
- The athlete sees the accuracy percentage and one cue per rep, such as "knees past toes", within 80 ms of the frame.
- After the session, Apache Beam pipelines run the heavier model, recover the 3D pose, and update the chart and the coach dashboard.
Development Process Flow
Machine learning development for a computer vision task like this one rarely survives a single big release, because a model that scores well on studio footage meets a subscriber’s garage on day one. The two-week cadence put a working build on real phones 16 times before launch, and the sprint 6 closed group changed the cue wording, the camera setup screen, and the fallback model size. Athletes asked for two of those three changes.
How We Delivered the Human Pose Estimation Application
- We define the project goal together, agree on priority features, and set a realistic delivery date and budget.
- We build a ranked list of everything the product needs, starting with what matters most to the business.
- Work is broken into 2-week cycles. At the start of each, we select the next set of features to deliver.
- The team builds, tests, and integrates features throughout the sprint.
- At the end of every sprint, you see working software and give feedback that shapes the next cycle.
- Each sprint produces a shippable piece of the product. We review what worked, adjust, and move forward.
UI/UX Design: Feedback an Athlete Can Read Mid-Rep
An athlete mid-set has about half a second of attention to spare and is looking at the phone from six feet away. That rules out most of what a computer vision team would love to show, so the skeleton overlay and per-joint angles live in the coach dashboard. The athlete sees a large accuracy percentage, one cue of three words or fewer, and a color that shifts from red to green as the rep improves.
The exercise screen keeps the setup honest. It opens with the trainer’s demo photo, the duration (a 25-minute dumbbell chest press session, say), and two actions: Start exercise and Review accuracy history. Before the set, a camera check confirms the entire human body is visible, and the phone is roughly level, because about 40% of bad scores in the closed group came from framing while the form was fine.
Accuracy history is the screen subscribers open most. Each exercise gets a month view with a line chart of form accuracy per session, so progress from 65% to 98% over four weeks is visible at a glance, plus a share button, because athletes send these charts to their coaches and, judging by the analytics, to each other.
Results
Before
- ✕Off-the-shelf pose detection at ~9 fps on mid-range Android, skeleton lost whenever equipment crossed the body
- ✕No comparison with a trainer's pose, so subscribers got a demo video and guessed
- ✕Two people in frame merged into one skeleton about twice a minute
- ✕"Wrong feedback" tickets outnumbered billing tickets 3:1, annual-plan churn near 6.5% a month
- ✕Coaches reviewed sent-in videos by hand, roughly 12 athletes per coach per week
After
- ✔30 fps on-device tracking of 17 key points, <80 ms from frame to feedback
- ✔91% agreement with certified trainers on form scores across 140+ exercises in 12 sports
- ✔Multi-person tracking keeps the athlete locked in crowded scenes, with about one ID swap per hour of footage
- ✔Form-related tickets down 72%, monthly churn on the annual plan down to 3.1%
- ✔One coach dashboard with skeleton overlays and live sessions, about 35 athletes per coach per week
Impact of the Pose Estimation Platform After Launch
Verified Reviews
Our Reputation on Top Platforms
LITSLINK holds a 4.8 rating on top platforms. Clients who review our artificial intelligence and software development work most often mention engineers who test on the customer’s real hardware, communication that survives an eight-month engagement, and models that keep improving after launch.
Have a Computer Vision Project in Mind?
Need human pose estimation for a product that has to work on the phones your users already own? Tell us what the camera should track, and a LITSLINK specialist gets back to you within 48 hours.
Thank you for your message. It has been sent.
