, 8 min read
Personal reflections on 2017
Year 2017 is coming to an end. As a kid, I would have surely imagined that by 2017, I would be living in space. But then I can chat with Amazon’s servers through the “Alexa” persona. It is not space, but it is cool.
It is the first year I used a tablet (an iPad) for actual work. It is not a kind of laptop/tablet hybrid, but rather a genuine post-PC tool. It worked though it was also infuriating at times. I plan to write more about in 2018. We are not yet in the post-PC era, sadly.
Tech companies like Facebook and Google are believed to have reached very high levels of political powers. The web is a power law, with most of the traffic being captured by a handful of large corporations. The dream of the web as a democratic place where all of us could be both builders and consumers, with no middleperson, seems to wane. Facebook and Google decide what you watch and though they make good decisions on the whole, it does seem like we are moving back to the TV era in some sense… with a few big players holding most of the power. It sometimes feels like independent bloggers like myself are more irrelevant than ever before… and, yet, every number I look at suggests that, on average, my posts get more readers than ever before. Is my blog still relevant? I think it is important to ask such questions. As an experiment, I added a PayPal link where people can offer donations to help support the blog. I don’t want to talk too much about money, but there are more people than I thought that gave more money than I would have ever guessed. These people told me in a very convincing manner that this blog is valuable to them. Thus I have mixed feelings about the state of the web. There has been no apocalypse wiping me out.
Lots of people have told me about how bad they feel about politics, and about how we seem to have lost some of our freedom of speech… but I get the impression that much of the loss is self-inflicted. Let us rejoice first on how all these powerful men fell from grace because of their sexually abusive ways. Surely, something right happened if this could occur.
Videoconferencing is free and everywhere but it is bad. In fact, in 2017, I have often insisted on in-person meetings. We need to be able to meeting in person through our computers in a way that makes sense. Watching a video rendition of the other person is not great. As a testimony to my engagement toward live meetings, I have personally organized lunch meetings in Montreal (open to all, in French). One of the best move I made all year. I have also decided to mostly abandon videoconferencing with my grad students… until the technology gets better.
I have become department chair. Mostly it has to do with making people work together better. It helps being a stoic (philosophically speaking) when doing such jobs. That is, you set aside how you feel about the problems, you ignore what you can’t change, and you focus what little you can do to find solutions.
This year, our Roaring bitmap indexes kept become more and more widely used. It was very rewarding. It was adopted by Microsoft (for VSTS). Pilosa is a new and already very popular distributed index system that’s based on Roaring (written in Go). At this point, the various libraries are built and developed by a web of super clever engineers. My role is mostly limited in complaining about how difficult it is to build and release Java code using tools like Maven. I am not sure what lesson to draw of the success that Roaring bitmaps received. Obviously, something was done right. I hope to be able to draw the right lessons. A few things come to mind:
- Software is not primarily about code and bits, it is primarily about people. People matter. A great deal. For example, in the initial stages of working on Roaring, we got in touch with the Druid folks (Metamarkets) and it led to a paper. They provided real-world feedback and validation that made a huge difference. The optimized C implementation was the result of an array of collaborations that lead us to a nice experience-report paper. It would have been impossible to pull something like this in a closed campus laboratory. No matter how smart you are, there are more clever ideas out there than you know.
- Simple designs coupled with lots of strong engineering can make for a potent if underestimated mix. People always seek the “big idea”… but big ideas are overrated. Great solutions are often based on simple ideas that are executed with care.
In 2017, I acquired my first smartphone (an iPhone 7). Yes. It is crazy that I did not own one before, but there is a purpose to my madness. I am deliberately picking with care what enters my life. For example, I did away with television fifteen years ago, something people often don’t take seriously. Not having a smartphone allowed me to a spectator to the smartphone phenomenon. Yet I knew that I would have to cave it one day, and I decided that this year, I needed that phone. It came about when I set up a meeting Montreal with Stephen Canon. I went to meet Stephen, but the restaurant I suggested was closed. Reasonably enough, Stephen messaged me, but I had relied on my wife’s smartphone… and though I was with my wife, my wife wasn’t carrying on smartphone… Long story short, I never got to meet Stephen. So now I own a smartphone. Socially speaking, not owning a smartphone has become a serious handicap. (I am aware that this is not a new insight.)
In technology, this year was the year of deep learning. We know it works because it is being used by companies with lots of data (Google, Apple, etc.) to solve hard problems. I have been very impressed by the machine translations offered by DeepL.
Too many people asked me why I am not jumping into deep learning. The answer is much the same as to why I am not buying any bitcoins. My best estimate is that at this stage in the game, you are as likely to overestimate as to underestimate the value of your investments. You should not expect to profit massively from these technologies at this time, unless you invested earlier.
I am placing my own bets on problems that are much less glamorous, and mostly untouched. Here are a few things that I have in mind:
- New non-game applications of virtual reality (VR). VR is getting cheap and really accessible. There are some modest applications of VR outside games… such as medical rehabilitation and e-Learning… but I am betting that there is a lot more than we know to discover.
- For all the talk of AI and big data, data engineering is a mess in 2017. We figured out that throwing everything into an Oracle database was great for Oracle, but not so good building modern applications. So we created data lakes and they have turned into data swamps. The clever AI kernel that makes the press release… is maybe 2% of your codebase… the rest is made of the pipes laid out in your data swamp. But it is hard to recruit data-swamp engineers. I am betting that lots of highly valuable future technology will live in the data swamp.
- Software performance is a messy place. Dan Luu has a great post about how modern computers fall short of the Apple 2 when it comes to latency. It is great that I can run a 3D shooter like Doom in pure JavaScript within my browser… but it is less great that when editing a LaTeX document online using a tool like Overleaf, my browser will effectively freeze for what feels like a second from time to time. My wife is going mad with her brand-new Android phone because, too often, when she clicks on a button or a link, there is a noticeable delay while the phone seems to become unconscious for a second or two… I’m helping a colleague right now do some standard statistical analysis… the thing is slow (running for weeks) to the point of making it unusable. Given how standard the problem is, I assumed I could just find ready-made solutions… and I could, but they are immature and undocumented. Not to mention that they are not so fast. I guess it explains how people like John D Cook can make a good living by consulting at the intersection of mathematics, statistics and computing. Maybe the best-kept secret in computing today is that software performance matters more than it did, not less.
- I remember giving talks about ten years ago, and summarizing the state of processors: “you are running an Intel processor, so are your friends, so are the servers you connect to”. There were non-Intel processors all over the place, but they were pretty much limited to embedded computing… an important field to be sure, but one you could leave to specialists. That’s not even close to being true today. You are more likely to be reading my words on a computer running an ARM processor than an Intel processor. And these Intel processors have become more complicated beasts.Some people are willing to bet that ARM processors may soon enter the server market and displace (partially) Intel. I find this credible.If you care about performance and you have to deal with ever more exotic hardware, your life is not going to get easier.