Daniel Lemire's blog

, 3 min read

Combine smart people with crazily hard projects

Back in college, professors assigned crazily hard problems… and I was forced to talk with my peers to figure out how they fared… and eventually teaming up with some of them. The same pattern has repeated itself again and again in my life: smart people and really hard problems. That’s how I am able to learn new things…

It is a powerful mix. Smart people challenge you while simultaneously allowing you to build up your confidence as you eventually match their expertise. Hard problems force you to fail, which is how you learn.

Neither of these things is particularly scarce today. There are billions of people connected to the Internet… many of them much smarter than you… yet all of them a few bits away from you… And we have more hard problems at our disposal than ever before in the history of humanity… I think that smart people meeting hard problems are more than just a learning opportunity… I think that’s how civilization moves forward.

Sadly, however, I fear that this potent mix is too often absent:

  • Most office work is organized so as to push hard problems, and failure, at the margin. The only kind of failures that are routinely present as “zero-sum failures”. Maybe the budget this year is fixed, and so only some projects will be funded… maybe your project will fail to secure funding. You will have failed, but not as a result of working on a difficult problem… it is “artificial failure”: you did not even try.- Smart people are often also pushed at the margins. Smart people are those who can make a dent on hard problems… but if the hard problems are pushed at the margin, then who needs them? When employers come to see me, a college professor, they never ask for “the smartest student I know”, they ask for “a good programmer”. They never say that they have hard and interesting problems to offer them… they talk about being able to offer them good salaries…

You might think that colleges would act as a safe haven… but I fear that they often do not. To succeed in academia, it helps to appear smart… but tackling hard problems is entirely optional. In fact, the secret for getting a research grant is… propose to do something you already know how to do… That’s not just a theoretical concern of mine. Just last week, I was in a meeting with a smart young professor. She explained to us, quite openly, that she intentionally did not include an important but difficult idea from her last research grant application… The implication was clearly that since it was a hard problem she did not know how to tackle, it would stack the odds against her to include it as a target. The net result is that grant applications are often conservative, if not openly boring. They must appear new and exciting… but the underlying problems must be easy if not already solved. So where do you find smart people working on hard problems? I don’t think you find them in managed environment… I think you find them in the cracks… And I think that it contributes to making the future really hard to predict. The future is often literally being built by friends in a dark garage… it does not get planned by Wall Street or the government.