Gorilla: A fast, scalable, in-memory time series database (2016)

Summary

Gorilla: A fast, scalable, in-memory time series database – Pelkonen et al. 2015 Error rates across one of Facebook’s sites were spiking. The problem had first shown up through an automated alert triggered by an in-memory time-series database called Gorilla a few minutes after the problem started. One set of engineers mitigated the immediate issue. A second group set out to find the root cause. They fired up Facebook’s time series correlation engine built on top of Gorilla, and searched for metrics showing a correlation with the errors. This showed that copying a release binary to Facebook’s web servers (a routine event) caused an anomalous drop in memory used across the site… In the 18 months prior to publication, Gorilla helped Facebook engineers identify and debug several such production issues. An important requirement to operating [these] large scale services is to accurately monitor the health and performance of the underlying system ad quickly identify and diagnose problems as they arise. Facebook uses a time series database to store system measuring data points and provides quick query functionalities on top. As of Spring 2015, Facebook’s monitoring systems generated more than 2 billion unique time series of counters, with about 12 million data points added per second – over 1 trillion data points per day. Here then are the design goals for Gorilla: Store 2 billion unique time series, identifiable via a string key Insertion rate of 700 million data points (time stamp and floating point value) per minute Data to be retained for fast querying over the last 26 hours Up to 40,000 queries per second at peak Reads to succeed in under one millisecond Support time series with up to 15 second granularity (4 points/minute/time series) Two in-memory replicas (not co-located) for DR Continue to serve reads in the face of server crashes Support fast scans over all in-memory data Handle continued growth in time series data of 2x per year! To meet the performance requireme...