Building watchable digital twins of 64 World Cup games

Summary

A lot of us saw the cool World Cup data visualizations going around social media, like this one from Alexander Bogachev: Source: Bogachev's tweet here I wanted to make my own but go a lot deeper, so I went looking to see where the data comes from.It turns out a bunch of companies sell data to sports teams and gamblers that they build using computer vision on live broadcasts. It works something like this:Source: SoccerNetObject detection and OCR for players / ball trackingField geometry identification for coordinate transform (using the penalty box corners, center circle, goal lines)Mix of inference and human tagging to log events (like passes, interceptions, shots, penalties)There's a lot more complexity around event tagging and player tracking, which requires visual embeddings for the players to maintain a best guess of their position when their jersey number isn't visible, but this is the high level.The completeness of this tracking data varies by provider. It generally falls into the following categories: Each row May include Matches one match competition, date, home & away teams, stadium, score / outcome, # periods Roster one player player name, team, position, jersey number, started? Events one event event type, timestamp, duration, actor, detail (varies by type) Tracking one frame each player's XY position, ball XYZ position I found 5 providers that offer free data samples for developers. Here's a quick comparison of their level of completeness: Source # Games What Frames/game Link PFF 64 Videos + events + tracking ~160k Data StatsBomb 426 Events + tracking ~3-4k Data Metrica 3 Events + tracking ~160k Data SoccerNet 550 Videos + raw CV data ~160k Data WhoScored Many Avg team X position (what Bogachev used?) ~100 Example game Since I'm building high-resolution digital twins of the games, I need a minimum of ~2 data frames per second (~11k frames total). PFF data is the most complete and polished with XY coordinates for all players at 30Hz. This resolution allow...