Daniel Lemire's blog

, 4 min read

Good ideas are overrated

As a college student, I was convinced that the most important part of science and engineering was to have good and original ideas. If you contemplate Einstein, it is not hard to come to this conclusion… The man had a never ending stream of great ideas. Notice however that we never read Einstein himself, or examine his workbooks… instead, we just hear about his ideas.

The same might be said about Google. Brin and Page came up with the idea of using hyperlinks to rank web pages. And they went on to become billionaires, just like that. We rarely talk about how the then dominating player (AltaVista) failed to keep its index updated for months and tried hard to milk every penny out of visitors by plastering its web site with ads… while Google just offered a working search engine kept up-to-date. Matt Might, a computer science professor, recently wrote that “the limiting reagent for scientific progress is good ideas”.

Matt presents the scientific model in a way that is familiar to many, but contrary to my own experience. One imagines scientists idle, waiting for good ideas to come to them. Maybe we can imagine Einstein who sits by his window receiving all these great ideas, as if they were telegraphed from an Oracle directly to his brain. And then, once the idea hits, it is all over in an instant. Einstein grabs his pen and writes the paper in a minute. Brin and Page imagine PageRank and then build their search engine in a week-end, taking the world by storm.

When I supervise graduate students, I often see them struggling with this model. They have ideas, but they do not know which ones are good. Some of them choose to wait… maybe a great idea will come eventually? It never does.

My model is different. Instead of imagining Einstein as someone receiving these deep insights magically, I imagine an obsessive thinker who keeps asking… “yes, but what does it mean?”… or “yes, but how does it work?”. Whereas others are satisfied with recopying the accepted answers, or accepting the unknowns, he redoes the work, more carefully. I imagine Brin and Page not as people who surfed on a brilliant idea… but rather as people who took a problem that might have been considered solved, or too technical, that is “how do you build a search engine”… and they revisited it, putting a lot more care into it.

When I look around, I see a deeply flawed world with thousands of superior ideas waiting to be tried out. I am literally never running out of good ideas and I doubt others are. I am not exceptionally smart, but give me any interesting problem, in science or engineering, and I am sure I can come up with five approachable questions that have not been answered yet. In fact, I think that almost anybody can do this… Give me a few years to become really knowledgeable, and I can probably prototype a solution in a few weeks.

I have two young boys. Almost every day they come up with a science question that I cannot answer. Once a week, they come up with a question that I cannot answer with Wikipedia. Many of these questions are genuinely interesting and could make a research project of its own.

What is genuinely scarce is interest and motivation. Let me put it another way. Imagine that I can go back in time to 1996. I know that the next critical piece of infrastructure is going to be a search engine. I know this, but there are already well entrenched leaders. I am in Montreal, and nobody is going to fund me, or join me to work on this problem… So I have to move to Silicon Valley where I will live in my car for a time. Then I have to find the ressources to build the new Google. It is really hard work. It is also very frustrating work because I know that Brin and Page are around… and maybe they will still come up on top.

Or maybe I could go back to 1900. I basically have a major in Physics from the 1990s. I know a lot of Physics that even Einstein cannot yet imagine. I could find a job to feed myself while I hack away at brilliant research papers. Or maybe I could try to rewrite one of Stephen King’s famous novels and get it published before he could… by going back in time.

Would I succeed? Maybe I would, maybe I would get discouraged. What is clear is that, even with perfect hindsight, success would be far from easy. Even with a perfect knowledge of all the great ideas… success would be really hard work.

Experienced software programmers know that, often, it is just as hard to understand someone’s code than to rewrite it from scratch. There are many ways to re-express this same idea. Some might say that execution is everything. But I think that the truth is even harsher: the concept of good idea is ill-defined. In hindsight, we try to explain success as “having a good idea”, but even if you had received this “good idea” on a silver platter, you might not have done anything with it in the end.

If you imagine the world a being two-dimensional, then there are many dead-ends, and few correct paths. The truth is that there are far more than two dimensions. You are not stuck in a dead-end. You are not out of good ideas. You are just bored.