Daniel Lemire's blog

, 3 min read

Infinite storage: we are almost there…

Seven years ago, I wrote a blog post called What is infinite storage? The blog post was a response to a Physics professor and colleague of mine who objected to my claim that we are soon reaching “infinite storage”. To be precise, my claim is that as far as computers are concerned, we will soon be able to assume that there is no limit on what we can store. Formally speaking, this can never be true. But, on the face of it, nothing is ever infinite. What scientists mean by infinite is “so large that it might as well be infinite”. For example, as far as my family in concerned, there is an infinite supply of bread and milk at the local convenience store. In theory, we could drink so much milk that the convenience store could not supply us… but it can’t realistically happen. In any case, there are 3 convenience stores and a supermarket within walking distance of my house. Also, I am rich enough that I can afford to pay for all the milk and bread that my family can consume. So, at some point in the XXth century, we reached the “infinite bread and milk” threshold. So what about storage? I have defined “infinite storage” as “the ability to buy a cheap 10TB disk at my local electronics store”. Why 10TB? Because that’s how much storage you need to record everything you see in a year. You know these Google glasses? They won’t be able to record everything you see in a year… but they could if you could find a cheap 10TB memory chip. They could also record everything you see in a week if you had cheap 256GB memory cards.

So where are we? I think that, in the last two years, we have made substantial progress. Seven years ago, I wrote:

Currently a portable 1TB drive can be had for $569.

What about today?

We still don’t have cheap and portable 10TB drives. They are apparently coming in 2014 according to some reports. Still, you can record everything you see in a year for $600 using two 5TB portable drives. That’s pretty good!

Effectively, this means that you can install a camera in your house, and keep a record everything that ever happens… for a mere $600 a year.

However, these portable drives are a bit large. You can’t imagine hooking them up permanently to a Google glass device. The SD cards are closer to what I want, but to record everything you see in a year on memory cards, you would need to spend 20K$. Basically you would need to buy a new SD card each week at $500 a piece. Some of us can afford it, but I cannot.

What would happen if many of us started wearing glasses that record everything? I bet that the prices would fall and we would soon all be able to afford it.

Why would you want to record everything? That’s a separate topic, but I can imagine that people who are losing their memory might greatly benefit from such records, especially if you have the computer power to process the data and replay the significant bits at will. Maybe this is what Google will be offering in 10 years.

As prices fall further, it will soon become cost-effective to record everything that happens around your car while you are driving it. I think it is only a matter of time before insurance companies entice us to record everything in this manner.

As storage becomes infinite, software will continue to become more important. I would bet that the winners will be companies able to recruit the best minds of the software industry. We will soon be limited by how fast software engineers can solve data problems. If these ideas sound crazy, consider that Google is keeping a record of all emails I have sent and received since 2004. All 15GB of it. Today, storing 15GB of data in the cloud is trivial. Ten years ago, it would have sounded futuristic.

Non-disclosure: I do not own any Google stock. In fact, I do not own any technology stock. I probably should.