Configurations

To make IPython loading autoreload extension automatically, and quit without confirmation, run ipython profile create and edit the template file (whose name is in the output), uncomment the corresponding lines.

Or run the following lines to generate profile directly:

$ cat << EOF > ~/.ipython/profile_default/startup/50-autoreload.ipy
%load_ext autoreload
%autoreload 2
%config TerminalInteractiveShell.confirm_exit = False
EOF

If not config it globally, you can run commands above in IPython shell.

Autoreload make the change to function and class method reloaded automatically after the file saved.

Workflow

Basic workflow

The basic workflow is: IPython console + editor (vim), and optionally tmux (or i3wm, etc).

Define one-line statements and expressions in IPython console; Define and update function or class in IPython console with %edit magic; Visualize data with:

%matplotlib
import matplotlib.pyplot as plt
plt.plot(...)

With tmux, you can create an editor pane and an IPython console pane in a tmux window. In IPython shell, import all functions in the developed file (name it myfile.py for example) with from myfile import *, and you can use all functions of myfile in IPython shell.

Ref: https://ipython.org/ipython-doc/3/config/extensions/autoreload.html

Live development

"Live development" means writing codes at the "developing point" (DP) in the runtime, with all variables, environments and data loaded.

In PyCharm

No need to add stub codes into the source file. But you need a PyCharm IDE, which is not very keyboard friendly, and remote develop (via SSH) is complicated.

1. Write a unit test (with unittest or pytest) to cover the DP;

2. Add a breakpoint at the DP;

3. "Debug" the test, then the process stops at the DP;

4. Write new codes, add them to your source file;

5. Change the DP, start the next iteration of live development.

In IPython console

You need only IPython console and an editor, minimal, fast and keyboard friendly. Both OK for local and remote development. You can exploit full functionality of IPython console, such as tab-completion, running any valid statements and evaluating any expressions. You can define and edit a function with %edit my_func magic of IPython. Display the function body with ??my_func. After the function is fulfilled, copy it to your source code.

But you need insert some stub codes into your source codes, and comment out or remove them when the functions is fulfilled.

1. Write a unit test (with unittest or pytest) to cover the DP;

2. Add embed() at the DP (and from IPython import embed);

3. Run the test with python -m unittest your_module.YourClassTest.test_your_func, then the process stops at the DP and an IPython console is fired;

4. Test your expressions in the console, if OK add them to your source file;

5. Change the DP, start the next iteration of live development.

With ipdb

ipdb in IPython console is more "live" than IPython console decribed above, for you can run the program step by step. And the tab-completion still works well in the debugger.

But you can't use magic here. So you can't edit function with %edit, and can't run system command with !.

See ipdb section in dsnote Debug Python Script.

Debug

Install ipdb with sudo pip3 install ipdb.

Add from ipdb import set_trace and set_trace() into myfile, the line before the the breakpoint.

In IPython shell, run %pdb and the function. It will stopped at the set_trace(). Use h for available commands.

You can add multiple set_trace() for multiple breakpoints. Use c to jump to the next breakpoint.

Other tips

• %autocall: with %autocall 2, you can run len mylist instead of len(mylist), or df.info instead of df.info(), to save typing;

• %load: Copy text from source file into IPython console. You can load specific line ranges: %load -r start:end myscript.py. Or load specific functions or classes: %load -s myfunc,MyClass myscript.py. See %load? for details;

• %quickref shows a quick reference sheet;

• ll to list files in current directory;

• %cd changes working directory;

• %who and %whos list all interactive variables;

• _ is the last command output;

• Use _oh to see all output;

• %edit edit a string and run it, for example: edit _;

• %time is the shell command time equivalent;

• Run shell command: ! only executes a shell command. With !!, you can store the command output for later use. Reference the output with _<no>, for example _38.

Rerun some codes multiple times with rerun

The following codes show the command history and rerun the first to 3rd line:

%hist -n
%rerun 1:3

Rerun some codes multiple times with macro

The following codes defines a macro named "my_macro" which contains the first to third line, the 5th and the 8th lines in history. And run it:

%hist -n
macro my_macro 1-3 5 8
my_macro

List functions with wildcard

>>> import numpy as np
>>> np.*load*?
np.__loader__
np.load
np.loads
np.loadtxt
np.pkgload

Run scripts in Current environment

The variables aa, bb defined outside the script can be loaded into the script with run -i magic:

$ cat << EOF > demo.py
def func2(x, y):
    return bb * x + y
print(func2(aa, 4))
EOF
$ ipython
>>> aa, bb = 3, 4
>>> run -i demo.py
16

Ref: https://ipython.org/ipython-doc/3/interactive/magics.html