Daniel Lemire's blog

Deep learning: the silver bullet?

, 4 min read

In 2016, we saw a wide range of breakthroughs having to do with artificial intelligence and deep learning in particular. Google, Facebook, and Baidu announced several breakthroughs using deep learning. Google also defeated Go. Deep learning is one specific class of machine learning algorithms. It…

How expensive are the union and intersection of two unordered_set in C++?

, 2 min read

If you are using a modern C++ (C++11 or better), you have access to set data structures (unordered_set) which have the characteristics of a hash set. The standard does not provide us with built-in functions to compute the union and the intersection of such sets, but we can make our own. For…

Resizing arrays can be slow in Swift

, 2 min read

Swift a recent high-performance programming language. It is still primarily used develop iOS applications, but it has the potential to be a general-purpose language. The lack of maturity of the language is still apparent. For example, it is almost trivially easy to create source code that the Swift…

How quickly can you remove spaces from a string?

, 4 min read

Sometimes programmers want to prune out characters from a string of characters. For example, maybe you want to remove all line-ending characters from a piece of text. Let me consider the problem where I want to remove all spaces (‘ ‘) and linefeed characters (‘\n’ and ‘\r’). How would…

Best programming language for high performance (January 2017)?

, 5 min read

I keep hoping that the field of programming language will evolve. I am a bit tired to program in Java and C… I’d like better languages. I am particularly interested in what I generally call “high-performance programming”. I want to pick languages where I can get the most out of my hardware.…

Predicting the future job market: the librarians

, 2 min read

People spend a lot of time worrying that robots and computers are going to wipe out all jobs. My belief is that the job market is a lot more complex and simplistic reasonings (“better robots means fewer jobs”) are likely misleading. I recently observed that despite the widespread introduction…

Betting against techno-unemployment

, 6 min read

There are millions of truck drivers in the US today. In particular, there are about 1.7 million tractor-trailer (human) drivers. There are many more professional truck drivers in the US, but tractor-trailer drivers are the “archetypal truck drivers”. In 2016, we had several demonstrations of…

Can your C compiler vectorize a scalar product?

, 3 min read

If you have spent any time at all on college-level mathematics, you have probably heard of the scalar product: float scalarproduct(float * array1, float * array2, size_t length) { float sum = 0.0f; for (size_t i = 0; i < length; ++i) { sum += array1[i] * array2[i]; } return…

The threat of technological unemployment

, 6 min read

There is a widely reported threat to our economy: robots are going to replace human workers. It is nothing new… In 1930, Keynes, the famous economist introduced the term “technological unemployment”… We are being afflicted with a new disease of which some readers may not yet have heard the…

Don´t let the experts define science!

, 8 min read

We know more than we can tell. We all know what a democracy is… We know that France, Canada, the USA, Japan… are democracies… Russia and China are not democracies. However, I have never seen a definition of “democracy” that fits the actual use of the term. One formal definition is…

Performance overhead when calling assembly from Go

, 2 min read

The Go language allows you to call C functions and to rewrite entire functions in assembly. As I have previously documented, calling C functions from Go comes with a significant overhead. It still makes sense, but only for sizeable functions, or when performance is irrelevant. What about functions…

What is a useful theory?

, 5 min read

I was an adept, as a teenager and a young adult, of thinkism. Thinkism is the idea that intelligence alone can solve problems. I thought I was smart so that I could just sit down and solve important problems. One after the other. Whatever contributions I ended up making had little to do with…

Science and technology: what happened in 2016

, 8 min read

This year, you are able to buy CRISPR-based gene editing toolkits for $150 on the Internet as well as autonomous drones, and you can ask your Amazon Echo to play your favorite music or give you a traffic report. You can buy a fully functional Android tablet for $40 on Amazon. If you have made it to…

How to build robust systems

, 2 min read

Millions of little things go wrong in your body every minute. Your brain processes the data in a noisy manner. Even trained mathematicians can’t think logically most of the time. In our economy, most companies fail within a decade. Most products are flawed. Over a million people die every year in…

Don´t assume that safety comes for free: a Swift case study

, 5 min read

Most modern languages try to be “safer” by checking runtime values in the hope of producing more secure and less buggy software. Sadly, it makes it harder to reason about the performance of the code. And, sometimes, the cost can be far from negligible. Let me illustrate it through a story using…

Getting a job in the software industry

, 3 min read

I am routinely asked about how to break into the software industry as a programmer. It is true that there is high demand for qualified programmers, but not all jobs are equal. There are good and bad tracks. And most jobs aren’t that good. How do you get a good job as a programmer? Here is my…

Software evolves by natural selection

, 2 min read

Software evolves by natural selection, not by intelligent design. It is a massive trial-and-error process. There are many thousands of programmers working every day to build new things in the hope of replacing the old… From time to time, you will hear about a new fantastic piece of computer…

On metadata

, 1 min read

On metadata