Lucene uses what is effectively the FastPFOR algorithm inspired by the JavaFastPFOR library. As for using the C++ library, I do not know if it is practical since Lucene is written in Java.
powturbosays:
In my tests SIMD bitpacking offer no speed advantage over optimized scalar bitpacking when used with large buffers (see simplebenchmark in FastPFor). This is valable for most applications (ex. inverted index). A realistic benchmark should compare SIMD/Scalar bitpacking only on large buffers.
If you are going to take the data from RAM, bring it all the way to L1 cache, load it in registers, then push it out all the back to RAM… you are IO bound… your CPU runs empty and so, saving CPU cycles becomes irrelevant. To make things worse, you can pretty much forget about using more than one core because your L3 cache is going to be overwhelmed by one core.
So? So you avoid decompressing whole arrays to RAM.
This particular library does not handle disk storage at all (by design). However, there is no particular problem with updates and this library. In fact, it compresses super fast so recompressing updating blocks should be quite fast.
Excellent. Very easy to use! Thanks!
Has FastPFOR been used/evaluated in a real contex such as lucene text search?
It’s used in https://github.com/manticoresoftware/columnar which can be used with Manticore Search – https://github.com/manticoresoftware/manticoresearch/
@anonymous
Lucene uses what is effectively the FastPFOR algorithm inspired by the JavaFastPFOR library. As for using the C++ library, I do not know if it is practical since Lucene is written in Java.
In my tests SIMD bitpacking offer no speed advantage over optimized scalar bitpacking when used with large buffers (see simplebenchmark in FastPFor). This is valable for most applications (ex. inverted index). A realistic benchmark should compare SIMD/Scalar bitpacking only on large buffers.
@powturbo
If you are going to take the data from RAM, bring it all the way to L1 cache, load it in registers, then push it out all the back to RAM… you are IO bound… your CPU runs empty and so, saving CPU cycles becomes irrelevant. To make things worse, you can pretty much forget about using more than one core because your L3 cache is going to be overwhelmed by one core.
So? So you avoid decompressing whole arrays to RAM.
We have demonstrated directly the benefit of SIMD bit packing in our latest paper (see http://arxiv.org/abs/1401.6399).
If you’re out of disk-space, is there a way to handle updates in a way that won’t require additional scratch space?
@Garen
This particular library does not handle disk storage at all (by design). However, there is no particular problem with updates and this library. In fact, it compresses super fast so recompressing updating blocks should be quite fast.