, 4 min read
Why I still program
People expect that, as you grow older, you give up practical jobs such as programming for more noble tasks such as managing a team and acquiring funding. This especially true in academia where “real professors” delegate the details, keeping only the “big picture stuff”. In other words, organizations are geared toward vertical collaboration: a hierarchical structure where people on top supervise other (cheaper) employees. In research, this means that the senior scientists have the ideas which junior scientists implement. Over time, the senior scientists may become unable to do what the junior scientists do, but they will become experts at acquiring funding. This model can scale up: the senior scientist can direct middle-level scientists who then supervise the younger scientists, and so on. Jorge Cham referred to this model as the Profzi scheme because it works best when funding is abundant and ever increasing.
The counterpart is horizontal collaboration. In this model, the senior scientists do everything, from having the big idea to executing it. They prefer to automate or avoid busy work when possible. Collaboration is mostly used to get a different point of view and complementary expertise. This model still works when funding is scarce, but it fails to scale up with respect to the number of people involved: horizontal collaboration is necessarily intimate.
The type of work that each model supports best differs. I conjecture that vertical collaboration favors long-term plans and predictable results. I believe horizontal collaboration favors serendipity and “wild” ideas.
As a sign that I favor horizontal collaboration, I still program even though I am old. This is unusual. It is so unusual as to raise eyebrows. Some programming takes time, a lot of time. I can spend two or three months a year programming. Presumably, my time is too valuable to be spent on a lowly task like programming that can be best done by people earning a fraction of my income. So why do I still program?
Maybe my best advocate would be the master himself, Donald Knuth:
People who discover the power and beauty of high-level, abstract ideas often make the mistake of believing that concrete ideas at lower levels are relatively worthless and might as well be forgotten. (…) on the contrary, the best computer scientists are thoroughly grounded in basic concepts of how computers actually work, and indeed that the essence of computer science is an ability to understand many levels of abstraction simultaneously.
But I also have my own arguments:
- I want my work to be significant, to have impact. Yet even the widely cited research papers are rarely read. Very few research papers have significant impact. However, it is comparatively easier to do work that matters with software. For example, recently a team from Facebook integrated one of my compressed bitmap index library in Apache Hive: the Hadoop-based framework for data warehousing. I am willing to bet good money that nobody at Facebook read the original paper for which I wrote this software.
- Time and time again, implementing my ideas has forced me to understand them better. A common scenario is that something that sounded reasonable on paper suddenly feels unwieldy when you must implement it. I also often discover bugs in my mathematical arguments through implementation. Could I outsource this work to others? Maybe. But the process would not be as fruitful.
- You do get better at programming over time. I have been building up my expertise for decades. It is enjoyable to start from scratch and solve a difficult problem in days when you know that others would take weeks or months to do the same.
If my arguments are reasonable, and if even Donald Knuth is on my side, why does it still surprise people when I admit to be a programmer-scientist? I believe that the rejection of programming as a lower activity can be explained by the Theory of the leisure class. In effect, we do not seek utility but prestige. There is no prestige in tool-making, cooking or farming. To maximize your prestige, you must rise up to the leisure class: your work must not be immediately useful. There is more prestige in being a CEO or a politician, than in being a nurse or a cook. Scientists who supervise things from afar have more prestige. Programming is akin to tool-making, thus people from the leisure class won’t touch it. People will call themselves engineer or analyst or developer, but rarely “programmer” because it is too utilitarian.
Warning: Not everyone should be programming. It is a time consuming activity. Because I program so much, there are many other exciting things I cannot do.
Further reading: Brooks, The Design of Design, 2010.
Subscribe to this blog
in a reader
or by Email.