Memory Profiling in Python¶

It is quite important to know what is going wrong in your code, to perform better. One tool to help you do what you need to do is to use a memory profiler.

• PyPI - Memory Profiler

for execution speed use:

• PyPI - Line Profiler
• CProfile

Install memory profile via pip:

In [1]:

!pip install --user --quiet memory_profiler

!pip install --user --quiet memory_profiler

After installing you might need to restart your kernel!

Write an example with a function and add the @profile decorator to indicate which function shall be profiled.

In [2]:

%%writefile example.py

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

%%writefile example.py @profile def my_func(): a = [1] * (10 ** 6) b = [2] * (2 * 10 ** 7) del b return a if __name__ == '__main__': my_func()

Overwriting example.py

In [3]:

# Execute the module with the memory_profiler being loaded:
!python3 -m memory_profiler example.py

# Execute the module with the memory_profiler being loaded: !python3 -m memory_profiler example.py

Filename: example.py

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
     2   39.406 MiB   39.406 MiB           1   @profile
     3                                         def my_func():
     4   47.141 MiB    7.734 MiB           1       a = [1] * (10 ** 6)
     5  199.766 MiB  152.625 MiB           1       b = [2] * (2 * 10 ** 7)
     6   47.254 MiB -152.512 MiB           1       del b
     7   47.254 MiB    0.000 MiB           1       return a

Of course you can also add this directly into the file:

In [4]:

%%writefile example.py
from memory_profiler import profile

@profile
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

%%writefile example.py from memory_profiler import profile @profile def my_func(): a = [1] * (10 ** 6) b = [2] * (2 * 10 ** 7) del b return a if __name__ == '__main__': my_func()

Overwriting example.py

In [5]:

!python3 example.py

!python3 example.py

Filename: /home/spack/computer-resources/Python/example.py

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
     3     39.4 MiB     39.4 MiB           1   @profile
     4                                         def my_func():
     5     47.1 MiB      7.7 MiB           1       a = [1] * (10 ** 6)
     6    199.7 MiB    152.6 MiB           1       b = [2] * (2 * 10 ** 7)
     7     47.2 MiB   -152.5 MiB           1       del b
     8     47.2 MiB      0.0 MiB           1       return a

It can be loaded into an ipython session with %load_ext memory_profiler

In [6]:

%load_ext memory_profiler

%load_ext memory_profiler

In [7]:

%%writefile example.py
def my_func():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_func()

%%writefile example.py def my_func(): a = [1] * (10 ** 6) b = [2] * (2 * 10 ** 7) del b return a if __name__ == '__main__': my_func()

Overwriting example.py

In [8]:

from example import my_func

from example import my_func

In [9]:

%mprun -f my_func my_func()

%mprun -f my_func my_func()

Filename: /home/spack/computer-resources/Python/example.py

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
     1     52.2 MiB     52.2 MiB           1   def my_func():
     2     59.8 MiB      7.6 MiB           1       a = [1] * (10 ** 6)
     3    212.4 MiB    152.6 MiB           1       b = [2] * (2 * 10 ** 7)
     4     59.9 MiB   -152.6 MiB           1       del b
     5     59.9 MiB      0.0 MiB           1       return a

Line profiler¶

Line profiler works similar.

In [10]:

!pip install --user --quiet line_profiler

!pip install --user --quiet line_profiler

In [11]:

%load_ext line_profiler

%load_ext line_profiler

In [12]:

%lprun -f my_func my_func()

%lprun -f my_func my_func()

Timer unit: 1e-06 s

Total time: 0.127 s
File: /home/spack/computer-resources/Python/example.py
Function: my_func at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           def my_func():
     2         1       4842.0   4842.0      3.8      a = [1] * (10 ** 6)
     3         1      78616.0  78616.0     61.9      b = [2] * (2 * 10 ** 7)
     4         1      43539.0  43539.0     34.3      del b
     5         1          3.0      3.0      0.0      return a

In [ ]:

---

Last update: January 28, 2022
Created: November 29, 2021