Solving Semantle with the Wrong Embeddings

Summary

In my last post, I wrote about a solver for Semantle (the word game where guesses are scored based on semantic similarity to a hidden target word) that Ethan Jantz and I built while at the Recurse Center. If your goal is (as ours was) to solve Semantle in as few guesses as possible, that solver works great. But it requires knowing exactly what embedding model the game itself is using, and the exact cosine similarities between every word in the vocabulary. After writing the post, a former colleague from grad school, Daniel Vitek, reached out with an idea for a solver that doesn’t rely on that knowledge, and instead only uses information about which guesses are better than others. It takes more guesses to win, but it’s more robust, in that it works even with a different underlying embedding model than the game uses. To me, it also feels more aligned with what it’s actually like to play Semantle. Daniel shared a notebook showing how it works, which I adapted into this solver. Which guess is closer? Let’s say I’m playing Semantle and guess 1 is “biology” (which scored 26.03) and guess 2 is “philosophy” (which scored 3.09). The most important piece of information I receive from the scores the game returns is the relative ordering: “biology” (guess 1) is closer to the target word than guess 2 (“philosophy”). That would make sense if the hidden target word were “medical.” For this solver, that relative rank is the core of what we need. Here, guess 1 corresponds to embedding g1g_1g1​, guess 2 corresponds to embedding g2g_2g2​, and the target corresponds to embedding ttt. We can imagine a boundary in embedding space containing all points that are equally similar to both guesses (i.e. words that make the same angle with g1g_1g1​ as with g2g_2g2​). On one side of that boundary, g1g_1g1​ would be more aligned with ttt; on the other, g2g_2g2​ would be. Since Semantle told us g1g_1g1​ was closer, the target must live on the g1g_1g1​ side of that boundary. Here’s what that would l...