Daniel Lemire's blog

, 2 min read

Downloading files faster by tweaking headers

2 thoughts on “Downloading files faster by tweaking headers”

  1. Hi @lemire Thanks for following up on this issue.
    using R curl package you can specify headers to accept gzip content and curl will automatically download and unzip for you

    "Accept-Encoding": "deflate, gzip"

    I had to use it in my own package to support fast downloads in regions with low internet speed (developing countries). More importantly, using curl open the door to more speed enhancements including asynchronous downloads which cuts the download time even further especially if the dwonload server allows for concurrent downloads. I am not a network guru, but I think we can combine the goodies of curl and Rppsimdjson in a new package.

  2. Dr. Shishir S. Urdhwareshe says:

    And thanks also @Mohammed O.E Abdallah for suggesting simdjason library