Daniel Lemire's blog

, 2 min read

When a terabyte is small

3 thoughts on “When a terabyte is small”

  1. I think this is one big obstacle for current research in IR. The time spent dealing with “infrastructure” is getting bigger. This leaves less time for real research. I think that, in the broad field of IR, “industry research” is going to produce much more results in the next years than “academia research”.

    Google’s Peter Norvig is quoted saying – Google does not have the best minds, they have a great infrastructure that allows them to experiment much faster.

    How can academia deal with this?

  2. Kevembuangga says:

    LOL!!!
    You are probably not old enough to know that rule:
    No matter the size of the drive it is ALWAYS 95/98% full so for the “next run” (whatever this is) you have first to upgrade.
    This is probably even more “solid” than Moore’s law.
    In the very early 70s a 5 megabytes drive was “large”…

  3. Kevembuangga says:

    BTW, why not using outsourced storage and computation power?
    The NYT did it:
    http://open.blogs.nytimes.com/tag/hadoop/

    (via Lukas Biewald http://www.lukasbiewald.com/?p=134 )