Twelve Ways to Be Wrong About AI-Assisted Coding

Summary

Twelve Ways to Be Wrong About AI-Assisted Coding Suppose your manager asks you next week to demonstrate that the AI coding tools your company signed up for are worth the subscription cost. Would you measure lines of code generated, or tickets closed? Or would you send out a survey asking whether developers feel more productive? Each of those approaches is flawed in a different way; the sections below explain why. Note: this post is about how people are assessing AI, not at LLM-assisted coding itself; with a little rewording, these criticisms could be applied to a lot of the claims that have been made about agile development, test-driven development, and other practices. If I’ve learned anything in the last twenty years, it’s that software engineering would be a lot further ahead today if we had been willing to let our peers in the human sciences teach us how to study these kinds of things properly. Also, if you’d a one-day introduction to the research methods you should use to avoid making these errors, please reach out. I’m not qualified to teach it, but I know people who are, and I could probably talk them into doing it… Counting Lines of Code Generated Proxy metrics stand in for concepts that are hard to measure directly, and lines of code is one of the oldest. LLMs generate more code, but not necessarily better outcomes: a team that sees a 40% increase in lines of code per developer after adopting LLM tools has measured verbosity, not productivity. Deleting 2000 lines of tangled logic and replacing it with 200 clean ones is an improvement that looks like a loss on this metric [Sadowski2019]. More code also means more to read, maintain, and debug, and AI’s contribution to that future burden does not appear in the line count. Timing Artificial Tasks A widely cited study found that developers who used GitHub Copilot completed a task 55% faster than those who did not [Peng2023]. The task was implementing an HTTP server in JavaScript from scratch, in ninety minutes; ...