Daniel Lemire's blog

A new release of simdjson: runtime dispatching, 64-bit ARM support and more

, 2 min read

JSON is a ubiquitous data exchange format. It is found everywhere on the Internet. To consume JSON, software uses tools called JSON parsers. Earlier this year, we released the first version of our JSON parsing library, simdjson. It is arguably the fastest standard-compliant parser in the world. It…

Science and Technology links (July 27th 2019)

, 1 min read

There are thick ice deposits on the Moon. Water in space is important as it can be used to create fuel and to sustain life. Most animals do not suffer heart attacks and strokes, the way human beings do. It appears that the difference may be due to a specific genetic mutation. Heart attacks are…

How fast can a BufferedReader read lines in Java?

, 2 min read

In an earlier post, I asked how fast the getline function in C++ could run through the lines in a text file. The answer was about 2 GB/s, certainly over 1 GB/s. That is slower than some of the best disk drives and network connections. If you take into account that software rarely only need to…

Arbitrary byte-to-byte maps using ARM NEON?

, 3 min read

Modern processors have fast instructions that can operate on wide registers (e.g., 128-bit). ARM processors, the kind of processors found in your phone, have such instructions called “NEON”. Sebastian Pop pointed me to some of his work doing fast string transformations using NEON instructions.…

Programming competition with $1000 in prizes: make my code readable!

, 1 min read

Colm MacCárthaigh is organizing a programming competition with three 3 prizes: $500, $300, $200. The objective? Produce the most readable, easy to follow, and well tested implementation of the nearly divisionless random integer algorithm. The scientific reference is Fast Random Integer Generation…

Science and Technology links (July 20th 2019)

, 2 min read

Researchers solve the Rubik’s cube puzzle using machine learning (deep learning). There has been a rise in the popularity of “deep learning” following some major breakthroughs in tasks like image recognition. Yet, at least as far as recommender systems are concerned, there are reasons to be…

Science and Technology links (July 13th 2019)

, 2 min read

Drinking juice increases your risk of having cancer. College completion rates are increasing and this may have to do with lower standards.1. Australia is going to try a flu vaccine that was partially designed with help from an artificial intelligence called Sam. Sam acquires knowledge and has new…

Parsing JSON quickly on tiny chips (ARM Cortex-A72 edition)

, 2 min read

I own an inexpensive card-size ROCKPro64 computer ($60). It has a ARM Cortex-A72 processors, the same processors you find in the recently released Raspberry Pi 4. These are processors similar to those you find in your smartphone, although they are far less powerful than the best Qualcomm and Apple…

Parsing JSON using SIMD instructions on the Apple A12 processor

, 3 min read

Most modern processors have “SIMD instructions“. These instructions operate over wide registers, doing many operations at once. For example, you can easy subtract 16 values from 16 other values in a single SIMD instruction. It is a form of parallelism that can drastically improve the…

Science and Technology links (July 6th, 2019)

, 2 min read

Jim Keller, the vice president of silicon engineering at Intel, is optimistic regarding the continued exponential progress in computing: “It’s going to keep going, Moore’s law is relentless.” England did away with free college. The result? England’s shift has resulted in increased…

A fast 16-bit random number generator?

, 2 min read

In software, we often need to generate random numbers. Commonly, we use pseudo-random number generators. A simple generator is wyhash. It is a multiplication followed by an XOR: uint64_t wyhash64_x; uint64_t wyhash64() { wyhash64_x += 0x60bee2bee120fc15; __uint128_t tmp; tmp =…

Science and Technology links (June 29th 2019)

, 1 min read

Consuming pornography has no effect on sexual desire for one’s partner. Exercise may increase the likelihood that a woman will become pregnant. Consuming more protein make it less likely that you will become frail as you age. There is no correlation between how often university professors fly…

Bounding the cost of the intersection between a small array and a large array

, 5 min read

Consider the scenario where you are given a small sorted array of integers (e.g., [1,10,100]) and a large sorted array ([1,2,13,51,…]). You want to compute the intersection between these two arrays. A simple approach would be to take each value in the small array and do a binary search in the…

Science and Technology links (June 22nd 2019)

, 1 min read

An implemented chip can improve long-term memory. It is currently impractical, but a start-up company will try to bring this technology to market. Never before so many simultaneous and distinct new approaches have been in development in the history of drug research. Once more, we have evidence…

How fast is getline in C++?

, 1 min read

A standard way to read a text file in C++ is to call the getline function. To iterate over all lines in file and sum up their length, you might do as follows: while(getline(is, line)) { x += line.size(); } How fast is this? On a Skylake processor with a recent GNU GCC compiler without any disk…

What should we do with “legacy” Java 8 applications?

, 2 min read

Java is a mature programming language. It was improved over many successive versions. Mostly, new Java versions did not break your code. Thus Java was a great, reliable platform. For some reason, the Oracle engineers decided to break things after Java 8. You cannot “just” upgrade from Java 8 to…

Science and Technology links (June 15th 2019)

, 2 min read

We know that naked mole rats are ageless mammals, in the sense that their mortality rate appear constant (not increasing with age). We believe that many sea creatures are similarly ageless. However, new research shows that sea urchins actually age in reverse: their mortality rate diminishes with…