DarkMatter in Cyberspace

Python Data Analysis Environment in RStudio-Server Style

Demo Code

Wavelet Denoise

Run the folling codes as test case:

import numpy as np
import matplotlib.pyplot as plt
import pywt

original = pywt.data.camera()

noiseSigma = 16.0
image = original + np.random.normal(0, noiseSigma, size=original.shape)

wavelet = pywt.Wavelet('haar')
levels  = int(np.floor(np.log2(image.shape[0])))

waveletCoeffs = pywt.wavedec2(image, wavelet, level=levels)

threshold = noiseSigma * np.sqrt(2 * np.log2(image.size)) 
newWaveletCoeffs = list(map(lambda x: pywt.threshold(x,threshold), waveletCoeffs))
newImage = pywt.waverec2(newWaveletCoeffs, wavelet)

plt.imshow(original)
plt.imshow(image)
plt.imshow(newImage)

Visual Studio Code

See dsnote Visual Studio Code Notes for initial setup.

Install Python Extension Pack and VS Code Jupyter Notebook Previewer in the Extensions bar. After the Python language server pack is downloaded and installed (see the progression in the status bar), restart VSC.

Now you have 3 options:

  • shift+enter: run selected codes (or current line) in Python console provided by Python Extension Pack. You need specify the conda environment with Python: Select interpreter in command palette. Then it starts conda environment automatically. But no way to use IPython console;

  • ctrl+enter: run selected codes (or current line) in active terminal. This is implemented by add keyboard shortcut in user's keybindings.json file, see dsnote "Visual Studio Code Notes" for details. You need specify conda environment and start Python shell IPython console manually. For a feature (bug?) of IPython, you can't send multiple lines to IPython shell. You can use bpython or ptpython shell instead (install with pip).

  • ctrl+alt+enter: run in Jupyter notebook provided by VS Code Jupyter Notebook Previewer. Output to Jupyter notebook instead of console let you see the plot right after the plot command (for example, plt.imshow(newImage) in above codes), without plt.show(), which is necessary for image show in console.

Spyder

This is the most light-weight solution except bare IPython console, It's included in Anaconda package, so no need to install.

The keyboard shortcuts and image display are both good. You can jump between editor and embedded IPython console easily. And run current line or selected block easily with Keyboard shortcuts.

The main problem is the lost of funding of this project(2018.9). The developers work mainly for maintaining 3.x version, and the lack of vim mode plugin. (There's one on github but not published to conda).

See dsnote Spyder Notes for details.

Rodeo

Verified on Ubuntu 16.04.

Jupyter notebook has no console with notebook. Rodeo has console with notebook in RStudio style. But Rodeo has no server edition. And by far Rodeo only support .py format. You can't write Jupyter notebook in Rodeo.

Another problem is the Keyboard shortcuts. Rodeo use Alt-Enter to run current line, which is conflict with i3wm. And there's NO way to customize this shortcut.

So Rodeo is a good exploring and analyse environment, while Jupyter notebook is good for presenting the results.

I installed Rodeo's Linux editon on the server. SSH to server from my laptop in MobaXterm and run:

. activate anaconda
cd <project_root>
/opt/Rodeo/rodeo

Or set python path in Preferences, so no need to start Rodeo in virtual environment.

Highlights:

  • vim-mode in editor: [Preferences > Editor > Key Bindings] to vim;

  • Better code hints and completion of module names, function names, function parameters in both editor and console, than Jupyter notebook;

To set working directory in session level, run cd <target-foder> in console window. To set working directory in system level, set the path in [Preferences > Python > Default Working Directory].

Changing the font size [Preferences > Global > Font Size].

Useful key shortcuts:

  • Ctrl-1: jump to editor;

  • Ctrl-2: jump to console;

  • Ctrl-,: open preference window;

  • Ctrl-Enter: run the current line, or run selected multiple lines, which is very convenient in vim-mode;

See http://rodeo.yhat.com/docs/#keyboard-shortcuts for all shortcuts.

Jupyter Notebook

See dsnote "Jupyter Notebook Notes".

Enthought Canopy

Verified on Ubuntu 16.04.

Likes Rodeo, this IDE puts text editor and console side by side, so you can run the current script (Ctrl-R) or selected text (Ctrl-Shift-R) in console easily.

Jump between editor and console with Ctrl-Tab.

Canopy supports image display inline.

To install Canopy, download the installer (a .sh file more than 600MB), make it executable and run. Then start Canopy with https_proxy=https://localhost:1080 http_proxy=https://localhost:1080 ./canopy, because it's package manager will be stuck without the proxy. Canopy uses its own package manager. Start it in welcome page, or [Window -> Package Manager] in editor.

Config the proxy settings in [Edit -> Preferences -> Network]. Choose System settings and click button Test connection setup.

It has no vim bindings.

It auto-completion looks better than Rodeo. For example, when in a function, it can auto-complete the name of the parameters with TAB key, while Rodeo can't.

It creates some folders in the home directory, which is annoying.

Wing IDE

To make Wing use Anaconda environment, set [Project > Project Properties > Python Executable] to the python executable in the target environment, for example: /home/leo/apps/miniconda3/envs/anaconda/bin/python. Then check it with [Source > Show Python Environment > Effective Python Path].

To use vim mode in editor, set [Edit > Preferences > User Interface > Keyboard > Personality] to "VI/VIM".

To run current line or selected texts in Python console with keyboard shortcut Shift-Enter, build a script (such as "myfunc.py") in the folder "~/.wingpersonal6/scripts", which is defined at [Edit > Preferences > IDE Extension Scripting > Search Path].

In this file add the following functions:

import wingapi

def runCurLine():
    app = wingapi.gApplication
    app.ExecuteCommand('evaluate-sel-in-shell', restart_shell=False,
                       whole_lines=True)
    app.ExecuteCommand('focus-current-editor')
    app.ExecuteCommand('exit-visual-mode')

Then in [Edit > Preferences > User Interface > Keyboard > Custom Key Bindings], click "Insert" button. Press Shift-Enter when "Key" is focused. Input runCurLine in "Command" text box, which is the function name defined in your script file.

To focus Pythn shell with keyboard, add the following function into your script file:

def focusShell():
    app = wingapi.gApplication
    app.ExecuteCommand('show-panel-python-shell', flash=True, grab_focus=True)

In [Edit > Preferences > User Interface > Keyboard > Custom Key Bindings], click "Insert" button. Press Alt-u when "Key" is focused. Input focusShell in "Command" text box.

Or input show-panel-python-shell in "Command" text box, so no need to define a custom function.

Now you can focus Python shell with Alt-u. Focus editor with ESC key.

Note:

Test a command Wing provided (such as show-panel-python-shell ), or user-defined (such as focusShell) by [Edit > Command by Name].

After modified a function, save your script file and reload it in Wing with [Edit > Reload All Scripts].

Problems

Wing can't embed image into the IDE environment. The image window is a new window seperating from the IDE. The process of drawing the image persists until you close the image window.

PyCharm

PyCharm's project-oriented workflow makes it suitable for project development, instead of casual data exploration. The latter is fit for file-oriented workflow.

Send current cell (a cell is a code block seperated by #%%, which is the same with that in Spyder) to console with Shift-Enter.

Pycharm's Scientific Mode (toggle with [View > Scientific Mode]) is exellent for data exploration, for smart code auto-completion, embeded image and documentation window.

Pycharm need relatively large memory resources, or it will very slow.

Jupyter notebook support in Pycharm is still poor. Keyboard shortcuts doesn't work. Connecting to Jupyterhub failed, so you can't run codes in a notebook.

Published

Last Updated

Category

Tech

Tags

Contact