Should I use a zstd dictionary?
Small messages compress badly on their own because each one starts from nothing. A trained dictionary gives every message the same head start. Paste a few hundred real messages (log lines, JSON events, API responses) and this page trains dictionaries of several sizes on most of them, then compresses the rest one at a time, with and without. It's libzstd 1.5.7 compiled to WebAssembly, running here in your browser.
| Dictionary | Ratio | Bytes/msg | Gap closed | Pays off after |
|---|
How to read it
- Ratio: original bytes over compressed bytes for the held-out messages, each compressed as its own frame (standard frame header included).
- Whole set as one stream: all held-out messages concatenated and compressed together. That's roughly the ceiling: what you'd get if every message could see all the others. Gap closed is how much of the distance from no dictionary to that ceiling a dictionary covers.
- Pays off after: messages until the bytes saved cover shipping the dictionary once. Matters when the dictionary has to travel to each reader.
- Pick: the smallest dictionary within 2% of the best one. Bigger dictionaries cost memory on both ends and take longer to load.
- A dictionary is only as good as its samples. In my runs on GitHub events, a dictionary larger than about a quarter of the sample bytes did worse than a smaller one, and sizes like that are flagged here. Ten samples already helped a lot; the last few percent took thousands.
- Data drifts. A dictionary trained on yesterday's events lost about 4% of its ratio, a month-old one about 9%. Retrain when the shape of your messages changes, and keep the dictionary ID in the frame so old messages still decode.
The split into training and test messages is random but fixed, so the same input gives the same answer. With very few messages the test set is small and the numbers are noisy. The measurements behind the notes: a zstd dictionary needs fewer samples than you think.