Sparkey: Light up your Hashes
Sparkey
Sparkey is an extremely simple persistent key-value store. You could think of it as a read-only hashtable on disk and you wouldn't be far off. It is designed and optimized for some server side usecases at Spotify but it is written to be completely generic and makes no assumptions about what kind of data is stored.
Some key characteristics:
Supports data sizes up to 2^63 - 1 bytes.
Supports iteration, get, put, delete
Optimized for bulk writes.
Immutable hash table.
Any amount of concurrent independent readers.
Only allows one writer at a time per storage unit.
Cross platform storage file.
Low overhead per entry.
Constant read startup cost
Low number of disk seeks per read
Support for block level compression.
Data agnostic, it just maps byte arrays to byte arrays.
What it's not:
It's not a distributed key value store - it's just a hash table on disk.
It's not a compacted data store, but that can be implemented on top of it, if needed.
It's not robust against data corruption.
The usecase we have for it at Spotify is serving data that rarely gets updated to users or other services. The fast and efficient bulk writes makes it feasible to periodically rebuild the data, and the fast random access reads makes it suitable for high throughput low latency services. For some services we have been able to saturate network interfaces while keeping cpu usage really low.