Daniel Lemire's blog

, 3 min read

Speed up Python without effort using generator expressions

3 thoughts on “Speed up Python without effort using generator expressions”

  1. Carlos says:

    Not only that, but expression 2 doesn’t leak a variable!

    $ python
    Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53)
    [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin
    Type “help”, “copyright”, “credits” or “license” for more information.
    >>> i
    Traceback (most recent call last):
    File “”, line 1, in
    NameError: name ‘i’ is not defined
    >>> sum([i for i in xrange(100000)])
    4999950000L
    >>> i
    99999
    >>> sum(j for j in xrange(100000))
    4999950000L
    >>> j
    Traceback (most recent call last):
    File “”, line 1, in
    NameError: name ‘j’ is not defined
    >>>

  2. Running code through cProfile is not fair since it adds an overhead for each function call.

    Run these commands instead:

    echo “sum([x for x in xrange(1000000)])” > t1.py

    echo “sum((x for x in xrange(1000000)))” > t2.py

    python t1.py

    python t2.py

  3. Andres says:

    Hi Daniel!
    Reading this post I try myself to do it. The results were not the same as you, I post it here. I work on python and I think that is correct that this happens, because generators in some operations are more expensive that simple list. when you have to read a big file generators are the solution, but in other cases lists are the solution!

    cProfile.run(‘sum((x for x in xrange(1000000)))’)
    1000004 function calls in 1.305 CPU seconds

    Ordered by: standard name

    ncalls tottime percall cumtime percall filename:lineno(function)
    1000001 0.565 0.000 0.565 0.000 :1()
    1 0.000 0.000 1.305 1.305 :1()
    1 0.000 0.000 0.000 0.000 {method ‘disable’ of ‘_lsprof.Profiler’ objects}
    1 0.740 0.740 1.305 1.305 {sum}

    cProfile.run(‘sum([x for x in xrange(1000000)])’)
    3 function calls in 0.540 CPU seconds

    Ordered by: standard name

    ncalls tottime percall cumtime percall filename:lineno(function)
    1 0.357 0.357 0.540 0.540 :1()
    1 0.000 0.000 0.000 0.000 {method ‘disable’ of ‘_lsprof.Profiler’ objects}
    1 0.182 0.182 0.182 0.182 {sum}