After this saga, I wanted to close the series with some numbers about the impact of this algorithm.
If you’ll recall, I started this whole series discussing variable-sized integers. I was using this list of numbers to compare the values. There are 44,956 items in the list.
| Algorithm | Size |
| Raw | 359.648 |
| Varint | 224,780 |
| Delta + Varint | 45,970 |
| Delta + Varint + Gzip | 5,273 |
| FastPFor | 24,717 |
You can see some interesting details about the various options. Delta + Varint + Gzip is a really good setup, if you can assume that the output pattern has a high degree of repetition. In some cases, that is possible, but that isn’t a generic property. Aside from that, FastPFor is winning hands down in terms of the output size of the data.
There is also another important aspect. The size of the output here is too big to fit in a single 8KB buffer, so I’ll need to use multiple for that. This is not something that you can really do with GZip. Here is the cost across multiple buffers:
This particular list would fit into 4 pages:
- 14,080 entries at 8,048 bytes
- 14,080 entries at 8,063 bytes
- 15,616 entries at 8,030 bytes
- 1,180 entries at 644 bytes
But the compression ratio is only part of that. Let’s talk about the performance aspect. On my machine, you can run the encoding process in 1,115 ticks and decoding in just 462 ticks.
To give some context, that means that you can do the encoding at a rate of ~400,000,000 / second and decoding at a rate of about 1 billion per second.
My perf team also asked me to mention that they haven’t gotten the chance to look at the code yet, so things are likely to improve.
The entire premise of FastPFor inside of RavenDB relies on these fast decoding & encoding times. It makes it super cheap to iterate over a vast amount of numbers, and the B+Tree layout we have means that updating a posting list’s page is trivial. Read it, merge, and encode it back. It’s fast enough that there is really no other place for meaningful optimizations / complexity.