LLM poetry and the "greatness" question: Experiments by Gwern and Mercor

Summary

I have been looking at the ability of large language models (LLMs) to write poetry for more than five years. With every new model, the technical abilities improve: pro models can produce lines and stanzas, handle rhyme and meter, and create surprising and witty turns of phrase, produce impressive metaphors. Technique is not the problem.Does any LLM output rise to the level of good poetry, or even great poetry? It’s a different question than studies about favorable ratings ask. I’m interested in greatness, what it is, and can LLMs get there. My definition of “great” poetry, which I’ve used over thirty years of reading, teaching, and writing poetry, is simple: a great poem is both particular and universal. A great poem is “about” a specific person or moment embedded in a particular culture, composed in such a way that it reaches across time and distance to resonate with readers outside that culture.Poets work inside an historical network of existing poems. A new poem resonates when it activates prior reading in the mind of the reader. Lines and images from older work resonate with each other and with new work. That resonance is the mechanism by which the particular becomes universal.LLMs are trained on existing texts. LLMs have in their training data most of the digitized poetry in existence; their output tends to reflect patterns in that data. A model asked to write a poem about grief will draw upon images and phrasings about grief in earlier poems. (Primer on vector embeddings.) But LLMs do not yet, at this writing, have culture. Without culture, I’m not sure there can be great poetry.GwernIf anyone can crack this nut, it is my favorite thinker on how exactly LLMs and poetry work: Gwern. His recent experiments using LLMs to write formal verse in the Pindaric mode “on the occasion of visiting a biomedical lab and watching the release of unnecessary mice,” to “complete” a poem by William Empson, and to write a romantic poem are, in my view, required reading for all wo...