Building an internal agent: Code-driven vs. LLM-driven workflows

Summary

When I started this project, I knew deep in my heart that we could get an LLM plus tool-usage to solve arbitrarily complex workflows. I still believe this is possible, but I’m no longer convinced this is actually a good solution. Some problems are just vastly simpler, cheaper, and faster to solve with software. This post talks about our approach to supporting both code and LLM-driven workflows, and why we decided it was necessary.This is part of the Building an internal agent series.Why determinism mattersWhen I joined Imprint, we already had a channel where folks would share pull requests for review. It wasn’t required to add pull requests to that channel, but it was often the fastest way to get someone to review it, particularly for cross-team pull requests.I often start my day by skimming for pull requests that need a review in that channel, and quickly realized that often a pull request would get reviewed and merged without someone adding the :merged: reacji onto the chat. This felt inefficient, but also extraordinarily minor, and not the kind of thing I want to complain about. Instead, I pondered how I could solve it without requiring additional human labor.So, I added an LLM-powered workflow to solve this. The prompt was straightforward:Get the last 10 messages in the Slack channelFor each one, if there was exactly one Github pull request URL, extract that URLUse the Github MCP to check the status of each of those URLsAdd the :merged: reacji to messages where the associated pull request was merged or closedThis worked so well! So, so well. Except, ahh, except that it sometimes decided to add :merged: to pull requests that weren’t merged. Then no one would look at those pull requests. So, it worked in concept–so much smart tool usage!–but in practice it actually didn’t solve the problem I was trying to solve: erroneous additions of the reacji meant folks couldn’t evaluate whether to look at a given pull request in the channel based on the reacji’s presence.(As ...