How we made our LeRobot video reader up to 15× faster

Summary

Robot data is converging on LeRobot LeRobot has emerged as the dominant open format for robot learning data. But running data operations on it isn't easy: decoding frames is expensive and memory-intensive, and every step before the GPU - decoding, transforming, annotating frames - is where pipelines bog down while your GPU sits idle, whether you're running inference or training. We've been working to make that part as easy as possible. As part of that, we recently introduced a native LeRobot reader in Daft (Daft #7090). daft.datasets.lerobot reads a dataset straight from Hugging Face into a dataframe with one row per frame. With load_video_frames, it decodes each camera into an image column. However, the initial version was painfully slow. The problem: one remote open per frame A LeRobot v3 dataset stores each camera's video as MP4 shards - files that pack the frames of many episodes back to back. To decode a frame, the reader opens the shard and seeks to the frame's timestamp. Opening an MP4 also means reading its index - the metadata that maps timestamps to byte positions in the file - before any seeking can happen. The original reader did all of this per row: every frame re-opened its shard, and each open re-read the index over the network. On remote datasets that came to roughly 3 seconds per frame, and total cost grew linearly with frame count - even when consecutive rows wanted neighboring frames from the same file. The fix: batch decode by shard The decode is now a batch UDF (Daft #7184): instead of being called once per row, the function receives 16 consecutive rows at a time, so it can plan the decode across them. For each batch it does three things. 1. Group the rows by shard. One pass over the batch groups the rows by the shard they point to. Each row's target time is its episode's start offset within the shard plus the frame's timestamp within the episode. The result is one list per shard, holding the row indices and target times that shard needs to serv...