Python
Python has not a reputation as an efficient language, as shown by Pereira et al 2017, or Georgiou et al 2017 and Pereira et al 2021.
TL;DR
Don’t use pandas loc or iloc. Scipy seems slightly faster than numpy.
Datastructures:
Let’s run some own experiments. We start with access to datastructures especially numpy and pandas.
np_row | np_fill | np_col | pandasloc | pandasloccolfirst | pandasiloc | pdtonp | pdtonp_expl | |
---|---|---|---|---|---|---|---|---|
1 | 4.12656e-05 | 1.0104e-06 | 7.2745e-05 | 0.0111469 | 0.0101949 | 0.00923493 | 9.07452e-05 | 7.61744e-05 |
2 | 8.2077e-05 | 8.96602e-07 | 0.000114476 | 0.0206702 | 0.0205129 | 0.0181202 | 0.000122865 | 0.000125629 |
4 | 0.000161479 | 9.996e-07 | 0.000201557 | 0.041093 | 0.0412492 | 0.0389662 | 0.000205946 | 0.000270107 |
8 | 0.000356612 | 1.114e-06 | 0.0003609 | 0.0813604 | 0.0813056 | 0.0739316 | 0.000389416 | 0.000391635 |
16 | 0.000690099 | 1.425e-06 | 0.000704012 | 0.163096 | 0.163492 | 0.145289 | 0.000712648 | 0.000698786 |
32 | 0.00131142 | 1.9228e-06 | 0.00135474 | 0.323704 | 0.324672 | 0.289665 | 0.00143766 | 0.00135203 |
64 | 0.00263158 | 2.9556e-06 | 0.00260746 | 0.648613 | 0.654005 | 0.575187 | 0.00270987 | 0.00276178 |
128 | 0.00538865 | 5.1514e-06 | 0.00525566 | 1.36646 | 1.34012 | 1.18164 | 0.00547093 | 0.00544242 |
Linear Algebra
Matrix Multiplikation
numpy | |
---|---|
1 | 9.9739e-05 |
2 | 0.000139488 |
4 | 0.000114884 |
8 | 0.000120791 |
16 | 0.000183942 |
32 | 0.000405003 |
64 | 0.00162623 |
128 | 0.00782108 |
256 | 0.0386351 |
512 | 0.194442 |
1024 | 1.2193 |
2048 | 11.0974 |
Real FFT
scipy | numpy | pyfftw | pyfftw-aligned | scipy.rfft | numpy.rfft | |
---|---|---|---|---|---|---|
1 | 0.00381239 | 0.00151092 | 0.0585787 | 0.0574758 | 0.00302329 | 0.00135853 |
2 | 0.00322401 | 0.00155525 | 0.0534272 | 0.0543276 | 0.00314442 | 0.00135439 |
4 | 0.00360894 | 0.00151672 | 0.0547443 | 0.0471189 | 0.00323722 | 0.00135196 |
8 | 0.0034119 | 0.00151119 | 0.047058 | 0.0466947 | 0.00312754 | 0.00137967 |
16 | 0.00342473 | 0.00159911 | 0.0491221 | 0.0466457 | 0.00373785 | 0.00145432 |
32 | 0.003446 | 0.00173954 | 0.0475046 | 0.0478198 | 0.00324612 | 0.00162357 |
64 | 0.00388431 | 0.00197752 | 0.0536956 | 0.0535495 | 0.0037881 | 0.00170258 |
128 | 0.00433995 | 0.00292303 | 0.0629871 | 0.0616636 | 0.00391455 | 0.00212671 |
256 | 0.00535025 | 0.00368953 | 0.0565794 | 0.0594629 | 0.00439391 | 0.00279817 |
512 | 0.0071481 | 0.00603329 | 0.0615802 | 0.0629647 | 0.00569091 | 0.00413892 |
1024 | 0.0101502 | 0.0104215 | 0.0698446 | 0.0650553 | 0.00781038 | 0.00629772 |
2048 | 0.0168405 | 0.0207966 | 0.0686745 | 0.0789321 | 0.0115867 | 0.011525 |
4096 | 0.0317679 | 0.0476814 | 0.0949736 | 0.0986773 | 0.0221407 | 0.0228395 |
8192 | 0.0651446 | 0.112784 | 0.146115 | 0.13402 | 0.0482237 | 0.052777 |
16384 | 0.136167 | 0.226666 | 0.217261 | 0.217812 | 0.0976548 | 0.106355 |
32768 | 0.286817 | 0.469508 | 0.39762 | 0.398721 | 0.218348 | 0.234994 |
65536 | 0.597191 | 1.20375 | 0.947739 | 0.967633 | 0.444679 | 0.520851 |
131072 | 1.38737 | 3.09616 | 2.10764 | 2.10954 | 1.11791 | 1.29787 |
Complex FFT
scipy | numpy | pyfftw | pyfftw-aligned | gpyfft | gpyfft-nocopy | scipy.rfft | numpy.rfft | |
---|---|---|---|---|---|---|---|---|
1 | 0.00336861 | 0.00161249 | 0.0548753 | 0.0482731 | 0.395371 | 0.205374 | 0.00338208 | 0.00150055 |
2 | 0.00325483 | 0.001634 | 0.0448886 | 0.0444545 | 0.277287 | 0.231672 | 0.00328468 | 0.00152817 |
4 | 0.00322713 | 0.00163326 | 0.0445435 | 0.0455871 | 0.356396 | 0.313082 | 0.00335697 | 0.00147627 |
8 | 0.00334265 | 0.00171216 | 0.0520503 | 0.0454285 | 0.435177 | 0.390907 | 0.00338687 | 0.00156424 |
16 | 0.00351015 | 0.00177305 | 0.0523957 | 0.0461696 | 0.45355 | 0.408727 | 0.00335801 | 0.00159903 |
32 | 0.00383642 | 0.00194109 | 0.0484208 | 0.0459848 | 0.61425 | 0.57502 | 0.00349584 | 0.00175208 |
64 | 0.00369906 | 0.0022227 | 0.0606762 | 0.0535553 | 0.624321 | 0.584427 | 0.0035127 | 0.00188932 |
128 | 0.0042883 | 0.00301635 | 0.0554015 | 0.0548098 | 0.951812 | 0.915097 | 0.00410201 | 0.0023974 |
256 | 0.0053246 | 0.00406811 | 0.0556712 | 0.0553767 | 0.944038 | 0.899359 | 0.00462181 | 0.00289644 |
512 | 0.00678661 | 0.00645472 | 0.0590778 | 0.0584005 | 1.36361 | 1.31239 | 0.00575479 | 0.00446222 |
1024 | 0.0102666 | 0.011006 | 0.0597801 | 0.0595775 | 2.10603 | 2.06098 | 0.00766799 | 0.00680423 |
2048 | 0.0172736 | 0.0218563 | 0.0641989 | 0.0640106 | 3.18386 | 3.1457 | 0.0122486 | 0.0119234 |
4096 | 0.0362455 | 0.0456738 | 0.0765759 | 0.0742328 | 6.1314 | 6.11184 | 0.0231177 | 0.0234235 |
8192 | 0.0779121 | 0.0981208 | 0.106211 | 0.106677 | 1.81492 | 1.75577 | 0.0488487 | 0.052597 |
16384 | 0.162319 | 0.203807 | 0.151713 | 0.150241 | 1.86237 | 1.82282 | 0.102897 | 0.106645 |
32768 | 0.371336 | 0.4336 | 0.245122 | 0.2475 | 2.20302 | 2.09415 | 0.220809 | 0.235322 |
65536 | 0.78366 | 0.928682 | 0.46866 | 0.453954 | 2.28698 | 2.18018 | 0.462715 | 0.495351 |
131072 | 1.64225 | 2.21782 | 1.05342 | 1.05408 | 5.05316 | 4.90513 | 1.07257 | 1.1733 |