waterwaterwaterwaterwaterwaterwaterwaterwaterwaterPollenation web design
07/07/2004

Pyrex for Python modules

cats:

    I came across Pyrex some time ago and finally had a quick play with it tonight. Pyrex is really designed to make writing extension modules easy but it seemed to have the potential to improve the speed of normal Python modules too. Here's a trivial Python module, mod.py, I used as a start:

    def oddsquares(n):
        total = 0
        for i in xrange(n):
            if i%2 == 1:
                total = total+(i*i)
        return total
    

    Copying that to cmod.pyx and turning the module into a .so is relatively painless and immediately gives a performance boost:

    $ pyrexc cmod.pyx; gcc -c -fPIC -I/usr/include/python2.3/ cmod.c; gcc -shared cmod.o -o cmod.so
    $ python /usr/lib/python2.3/timeit.py "from mod import oddsquares;
    oddsquares(1000)"
    1000 loops, best of 3: 967 usec per loop
    python /usr/lib/python2.3/timeit.py "from cmod import oddsquares;
    oddsquares(1000)"
    1000 loops, best of 3: 571 usec per loop
    

    Now, the fun really starts when you tweak the module to be more Pyrex-friendly:

    def oddsquares(int n):
        cdef int i, total
        total = 0
        for i from 0 < = i < n:
            if i%2 == 1:
                total = total+(i*i)
        return total
    
    $ python /usr/lib/python2.3/timeit.py "from cmod import oddsquares;
    oddsquares(1000)"
    10000 loops, best of 3: 35.5 usec per loop
    

    Almost 30 times as fast for such a small amount of effort, impressive stuff! And, as you increase the value of n Pyrex has an even greater effect:

    $ python /usr/lib/python2.3/timeit.py "from mod import oddsquares; oddsquares(100000)"
    10 loops, best of 3: 1.38e+06 usec per loop
    $ python /usr/lib/python2.3/timeit.py "from cmod import oddsquares; oddsquares(100000)"
    100 loops, best of 3: 2.29e+03 usec per loop
    

    I have no idea whether this sort of performance increase is typical but Pyrex certainly looks like a very useful tool.