How Gnutella Scaled to Handle Query Traffic

Summary

(This is part two of my Gnutella exploration series. Please read the Gnutella Explanation article first if you are not already familiar with Gnutella.) At its peak, Limewire was installed on a third of desktop computers and estimates from the year prior showed a concurrent node count in the millions. It hit this level of adoption while staying decentralized and avoiding the need for a global file index or coordination servers. More importantly, the network still survives today despite minimal upkeep. Early versions of Gnutella searched the network using a technique called flood routing. Your computer sent a query to its neighbors. Those neighbors sent it to their neighbors. The process continued until the query ran out of hops or reached a machine with a matching file. Flood routing was beautifully simple, but it created a snowball effect for bandwidth usage. Most of the computers receiving a query had nothing relevant to offer. They still had to accept the message, inspect it, remember it, and forward it onward. In the previous article, I described Gnutella queries as though they simply flooded across the network. This was true, but I was kinda lying to keep the article short. Flood routing was OK when Gnutella was an experimental network with a few thousand users. Blindly bothering every reachable computer was no longer viable once the network hit mainstream adoption. Based on my Internet Archive archaeology digs, this practice likely came to an end around the year 2003. Flood routing is a convenient half-truth we tell for the sake of moving an explanation along, like the time I told my son that semicolons in JavaScript are optional. Later Gnutella clients (and the ones surviving today) still pass queries through a mesh of peers, but they are much more selective about which peers they decide to bother. They do this without central file indexes and without requiring any participant to reveal an exact list of shared files. The newer search system was called the Quer...