LoRA Speedrun – a public wall-clock leaderboard for fine-tuning techniques

Summary

How fast can you LoRA-fine-tune Qwen2.5-1.5B to ≥ 57% on GSM8K — on a single L40S? This is modded-nanogpt for fine-tuning: a frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is independently re-run 3× with fresh seeds on identical hardware before it counts. Attempting and verifying are free: official timing runs on a Modal L40S sandbox, and Modal's free monthly compute credits cover full runs — so anyone can compete, and anyone can re-verify any record with one command. Current record: 6m 05s by @Saivineeth147 — Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. # Date Author Train time GSM8K Δ Technique 0 2026-07-18 @Saivineeth147 11m 57s 59.4% — Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR. No tricks. (report) 1 2026-07-18 @Saivineeth147 6m 05s 61.1% −49% Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. (report) Full history with verification reports: records/RECORDS.md Base model Qwen/Qwen2.5-1.5B (base, not instruct) — frozen Training data GSM8K train split (7,473 examples) — the only allowed data Goal ≥ 57.0% exact-match on GSM8K test (0-shot, greedy, strict #### N extraction) Hardware 1× NVIDIA L40S (48 GB) — Modal sandbox, network-blocked Metric Wall-clock time of your training run. Lower is better. Constraint Base weights frozen; PEFT adapter only, ≤ 30M trainable params Machine-readable spec: spec.yaml. Full rules: TASK.md. You control everything else: LoRA rank and placement, quantization, learning-rate schedules, sequence packing, data subset selection and ordering, custom kernels, when to stop. Train on 1,000 well-chosen examples for 90 seconds if you can make it clear the bar. LoRA/QLoRA is how most real-world fine-tuning actually happens, and the technique space exploded — DoRA, rsLoRA, PiSSA, LoRA+, NEFTune, Unsloth kernels, rank-adaptive methods — but there's n...