Speeding Up the Back End with Graph Theory (2019)

Summary

Here at Sensor Tower we handle large volumes of data, so to keep things snappy for our customers we need to think carefully about how we process and serve that data.Understanding the data we're handling is a fundamental part of improving the way we serve it, and by analyzing how an important back end service worked, we were able to speed it up by a factor of four.BackgroundWe have many user-facing endpoints in Sensor Tower. So many, in fact, that we have numerous dashboards to keep tabs on how the system behaves.A few months ago, we noticed that a particular and very important endpoint was very sluggish: While all the other endpoints of the same type had <50ms latencies, this particular service took a leisurely 300 to 500ms to respond. Here's a diagram of how that looked:The customer doesn't look very happy up there! So, we decided to take some time and do an in-depth analysis of that endpoint.Okay, now to a more serious diagram. Here's what that endpoint looked like:The numbered steps in the diagram above perform the following operations:Decode a Protobuf string and build a Ruby object from it;Modify the object;Encode the Ruby object back to Protobuf.In essence, the endpoint receives Protobuf-encoded strings, does some work on them, and returns a processed version of the Protobuf-encoded string to the client. If you don't know what Protobuf is, that's okay; I didn't either. You can think of it as something similar to JSON: A serialized tree structure encoded using a binary format rather than text.Once we pinpointed that the endpoint slowness was due to Protobuf parsing, the next step was to try and find bottlenecks in the algorithm. The proper way to do this (which is not to just to read the code thoroughly) is by using a profiler.With the help of ruby-prof (generate profile data) and KCacheGrind (view profile data), we were able to identify two methods, #find_all and #encode, that took a large portion of the CPU time:While a profiler is a useful tool to identify p...