Python performance profiling in Pycharm

John D K -

Testing performance of Python programs can be done in many different ways, environments and modules. In previous article we saw how to do basics tests and measure the execution time: Python test performance and measure time elapsed in seconds with modules time, datetime, timeit, cProfile. In this post I want to present new way of measuring python performance in a Linux top command way. It is supported text and graphical information representing information about the running Python program from a process or a single Python file. The tool has fancy name as py-spy and you can use it in PyCharm or as a console command.

Here you can find more information about the program: Py-Spy: A sampling profiler for Python programs

The advantages of using this sampling profiler are:

  • you can test the program while it's running based on the process
  • you have top like output
  • good visual output save to a image
  • working on Linux, OSX and Windows and all python versions: 2.7, 3.3, 3.7

This is the link for the video tutorial: Powerful Python Performance Profiler

Installation of the tool can be done by:

pip install py-spy

If you want to measure performance in PyCharm then you can:

  • open PyCharm
  • Show terminal - ALT + F12
  • Be sure that your virtual environment is used by the name in the brackets before the command (if you use one):
(myenv) user@user-machine:~/PycharmProjects/python/tests$
  • install it by:
pip install py-spy

You have two options of usage(it's the same if you test performance in PyCharm or in the console):

py-spy --pid 12345
# OR
py-spy -- python myprogram.py

the result of this is:

Collecting samples from 'python ProfilingIsolation.py' (python v3.6.6)
Total Samples 300
GIL: 0.00%, Active: 100.00%, Threads: 1

  %Own   %Total  OwnTime  TotalTime  Function (filename:line)                                                                                                                                                                                                        
 63.00%  63.00%   0.630s    0.630s   test (ProfilingIsolation.py:21)
 15.50%  15.50%   0.220s    0.220s   after (ProfilingIsolation.py:15)
  6.00%   6.00%   0.115s    0.115s   after (ProfilingIsolation.py:13)
  5.50%   5.50%   0.070s    0.070s   after (ProfilingIsolation.py:14)
  3.50%   3.50%   0.035s    0.035s   test (ProfilingIsolation.py:20)
  3.00%   3.00%   0.030s    0.030s   test (ProfilingIsolation.py:19)
  2.50%   2.50%   0.025s    0.025s   test (ProfilingIsolation.py:18)
  1.00%   1.00%   0.020s    0.020s   after (ProfilingIsolation.py:12)
  0.00% 100.00%   0.000s     1.50s   run (cProfile.py:16)
  0.00% 100.00%   0.000s     1.50s   runctx (cProfile.py:100)
  0.00%   0.00%   0.010s    0.010s   before (ProfilingIsolation.py:5)
  0.00%   0.00%   0.100s    0.100s   before (ProfilingIsolation.py:7)
  0.00%   0.00%   0.090s    0.090s   before (ProfilingIsolation.py:6)
  0.00% 100.00%   0.000s     1.50s   run (cProfile.py:95)
  0.00%   0.00%   0.155s    0.155s   before (ProfilingIsolation.py:8)
  0.00% 100.00%   0.000s     1.50s   <module> (<string>:1)
  0.00%  72.00%   0.000s    0.720s   <module> (ProfilingIsolation.py:25)
  0.00%   0.00%   0.000s    0.355s   <module> (ProfilingIsolation.py:23)

Press Control-C to quit, or ? for help.

The other way of visualization is by using:

py-spy --flame profile.svg --pid 12345
# OR
py-spy --flame profile.svg -- python myprogram.py

The result of this execution is visible below:

The tested code of myprogram.py is below:

import itertools
import cProfile

def before():
    for i in range(1, 1000000):
        dictA = {'Java': 1, 'Python': 2, 'C++': 3}
        dictB = {'Python': 3, 'C++': 4, 'Fortran': 5}
        dictC = {**dictA, **dictB}Linux, OSX and Windows


def after():
    for i in range (1, 1000000):
        dictA = {'Java': 1, 'Python': 2, 'C++': 3}
        dictB = {'Python': 3, 'C++': 4, 'Fortran': 5}
        dictA.update(dictB)

def test():
    for i in range (1, 1000000):
        dictA = {'Java': 1, 'Python': 2, 'C++': 3}
        dictB = {'Python': 3, 'C++': 4, 'Fortran': 5}
        dictC = dict(itertools.chain(dictA.items(), dictB.items()))

cProfile.run('before()')
cProfile.run('after()')
cProfile.run('test()')

which is another way of performance tests in Python. You can create this method and test different executions inside it.