Time profiling
Profile in IPython shell
Install line_profiler
with conda/pip install line_profiler.
Profile time usage of function unit_deviation line-by-line:
$ ipython
> %load_ext line_profiler
> %lprun?
> from deviation_pandas import main, unit_deviation, inputFilePath, outputFilePath, isTrain
> %lprun -f unit_deviation main(inputFilePath, outputFilePath, isTrain)
Here main(...) is the caller of the profiled function unit_deviation.
You can have multiple -f <func> arguments for profiling multiple functions
at the same time.
Profile with decorator
Firstly add decorator @profile above the target function (such as unit_deviation).
Then profile the function with kernprof -l deviation_pandas.py.
Here -v means print result in console instead of save to a file (named deviation_pandas.py.lprof).
-l mean line-by-line.
No matter if the -v is used, kernprof always save the profiling results in
a file named deviation_pandas.py.lprof.
You can view it with python -m line_profiler deviation_pandas.py.lprof
Memory profiling
Install memory_profiler
with conda/pip install memory_profiler.
Profile in IPython shell
Profile memory useage of function unit_deviation line-by-line:
$ ipython
> %load_ext memory_profiler
> %memit?
> from deviation_pandas import main, unit_deviation, inputFilePath, outputFilePath, isTrain
> %memit main(inputFilePath, outputFilePath, isTrain)
peak memory: 370.30 MiB, increment: 183.06 MiB
> %mprun?
> %mprun -f unit_deviation main(inputFilePath, outputFilePath, isTrain)
Here peak memory means the max amount of memory used.
increment means the new memory used by function unit_deviation only.
Profile with decorator
Add from memory_profiler import profile, and decorator @profile above the target function
(for example, unit_deviation or main), and run python deviation_pandas.py.
Time-based memory usage
To plot time versus whole memory useage, run:
$ mprof run deviation_pandas.py
$ mprof plot
py-spy
Install py-spy with cargo install py-spy (you need
install rust with asdf beforehand).
An error /usr/bin/ld: cannot find -lunwind occured on Ubuntu 18.04.
Fix it with apt install libunwind-dev.
Profile script deviation_pandas.py with
/home/leo/.asdf/installs/rust/stable/bin/py-spy -- python deviation_pandas.py.
Ref: