DarkMatter in Cyberspace
  • Home
  • Categories
  • Tags
  • Archives

Rebuild Python/R Environment on a Offline Host


Environments Manged by Pip

Download all packages with pip download -r requirements.txt and copy them to the offline host. Install them on the offline host with pip install --no-index --find-links /path/to/some/dir/ -r requirements.txt.

If you manage environment with pipenv , create the requirements.txt file from Pipfile with pipenv lock -r.

Environments Managed by Conda

Conda environment is portable. Make sure develop and production host have the same conda installation folder and OS specification. In the following demo the OS of both hosts are CentOS 6.8, and the installation folder the default one: /root/miniconda3 or /root/anaconda3.

Install Anaconda and R packages in the conda env on the develop machine (with Internet access). Tar the $MINICONDA_HOME/envs/<target-env> folder and copy it to the offline machine. Extract to the $MINICONDA_HOME/envs of the offline host. Activate the new environment and run app codes. The following codes demonstrates packaging the entire Anaconda and R environment with conda and restore it on an offline host:

ssh GW-191
bash Miniconda3-latest-Linux-x86_64.sh
. ~/.bashrc
conda create -n operator anaconda          # takes 2 hours
conda install -n operator r-base r-psych   # takes 30 minutes
. activate operator
R
> library(psych)
> describeBy(CO2$uptake, list(CO2$Plant))
> ^d
. deactivate
cd ~/miniconda3/envs/
tar zcf operator.tgz operator    # 831MB for tgz file, 2.7GB for *operator* folder
scp operator.tgz GW-192:~/

ssh GW-192
cd ~/miniconda3/envs/
tar xf ~/operator.tgz
. activate operator
ipython
>>> import pandas
>>> data = pandas.DataFrame([(3, 33), (4, 44)])
>>> ^d
R
> library(psych)
> describeBy(CO2$uptake, list(CO2$Plant))
> ^d

So this is a fully functional Anaconda and R environment.

Or you can use Anaconda installer instead of Miniconda when the Internet access is slow:

ssh GW-193
bash Anaconda3-5.1.0-Linux-x86_64.sh
. ~/.bashrc
conda create -n renv r-base r-psych
. activate renv
R
> library(psych)
> describeBy(CO2$uptake, list(CO2$Plant))
> ^d
. deactivate
cd anaconda3/envs/
tar zcf renv.tgz renv/   # 263MB for tgz file, 771MB for *renv* folder
scp renv.tgz GW-194:~/

ssh GW-194  # this is the offline host
cd /root/anaconda3/envs
tar xf ~/renv.tgz
. activate renv
> library(psych)
> describeBy(CO2$uptake, list(CO2$Plant))
> ^d

Other options:

  • Copy all $MINICONDA_HOME/pkg/*.bz2 files to offline host, install with conda install -n myenv *.bz2.

Ref:

  • Python Packages Offline Installation

  • Generating a requirements.txt

  • export environment to an offline machine



Published

Mar 19, 2018

Last Updated

Mar 20, 2018

Category

Tech

Tags

  • backup 8
  • conda 5
  • package 28
  • pip 6
  • python 136
  • R 5
  • restore 3

Contact

  • Powered by Pelican. Theme: Elegant by Talha Mansoor