The secret life of data in Valkey

Summary

The secret life of data in Valkey Technical Deep Dive 2026-07-21 · Kyle Davis If you’ve used Valkey for very long you’re probably aware of the different primary data types in the core: strings, hashes, lists, sets, sorted sets, and streams. Each data type has an accompanying set of commands that manipulate that data type. For example, you use HSET to set a field and value pair to a hash, but you don’t use, for example, HSET to add an element to a list. What if I told you that there is an entire hidden layer that you never see unless you use a specific OBJECT command and can’t control unless you start tinkering with the configuration? And that this could have a profound effect on your total usable storage: enough that careful configuring could mean significant savings in your infrastructure. Read on to find out how you can understand and tweak the real underlying structures. Engineering tradeoffs Compact memory representation is highly important in Valkey: most users aren’t limited by performance but they are limited by the amount of memory available. When you’re representing data in RAM there often isn’t a clear winner for a technique that maximizes both performance and compactness at all data sizes. Like every engineering project, Valkey attempts to balance tradeoffs of performance and compactness. In this case, the balance is achieved by an encoding. All data in Valkey has an encoding, and this abstracts the actual in-memory representation from how it’s accessed (i.e. the commands used). Take the example of these three HSETs (assume an empty database): > HSET hash0 field "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > HSET hash1 field "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" > HSET hash2 field "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" At first glance you might observe that each value is one character longer than the previous. However, what you probably don’t see is that despite the single char...