Restore / Save Conda Environments¶
Sometimes it just works and you do not want to change anything.
This is when you can create an environment file and safe your current python installed packges and versions.
What you can do:
- Backup an existing environment
- Restore an environment
- Design an environment
Backup¶
It is a good choice to create a backup of a working environment. Regardless if you plan on setting it up somewhere else or not.
In [1]:
Copied!
!module list
!module list
Currently Loaded Modulefiles: 1) anaconda3/2020.07-gcc-8.4.1 3) proj/8.1.0-gcc-8.4.1 2) geos/3.9.1-gcc-8.4.1 >
In [3]:
Copied!
# Check What environment are present
# Typically there are some root installed environments
!conda env list
# Check What environment are present
# Typically there are some root installed environments
!conda env list
# conda environments:
#
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/anaconda3-2020.07-l2bohj4adsd6r2oweeytdzrgqmjl64lt
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-5qezliv3bkrc63cv2zk2lyh3xioozd45
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/miniconda3-4.9.2-htox4jjktvvm7b4d3rfwpwpwa4i2kkhk
/home/spack/envs/test
/home/swd/anaconda3
/home/swd/manual/intelpython/intelpython/latest
/home/swd/manual/intelpython/intelpython/latest/envs/2021.4.0
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda2-2019.10-eaw5h5kknfwa3bsjnjhqfhytzfxgd3nf
base * /home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2020.07-4onhdcbnvyvwfdxxbelrplxjihjsknex
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2020.11-v46okvjlotnor7yrmpqnbn63xt3a2qyq
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2021.05-hrg2ocjseksc4af3osaiipzu433xctef
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/miniconda2-4.7.12.1-3r6cvxekrhf32f6tdeout3aq7xxsgjit
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/miniconda3-4.10.3-l23y36yispi4ue5w7mhyc4mllio4ylfm
But there is one called test in /home/spack/envs/test which we are going to backup.
In [7]:
Copied!
# Check What is installed inside your environment
!conda list -p /home/spack/envs/test
# Check What is installed inside your environment
!conda list -p /home/spack/envs/test
# packages in environment at /home/spack/envs/test: # # Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 4.5 1_gnu ca-certificates 2021.9.30 h06a4308_1 certifi 2021.10.8 py39h06a4308_0 ld_impl_linux-64 2.35.1 h7274673_9 libffi 3.3 he6710b0_2 libgcc-ng 9.3.0 h5101ec6_17 libgomp 9.3.0 h5101ec6_17 libstdcxx-ng 9.3.0 hd4cf53a_17 ncurses 6.2 he6710b0_1 openssl 1.1.1l h7f8727e_0 pip 21.2.4 py39h06a4308_0 python 3.9.7 h12debd9_1 readline 8.1 h27cfd23_0 setuptools 58.0.4 py39h06a4308_0 sqlite 3.36.0 hc218d9a_0 tk 8.6.11 h1ccaba5_0 tzdata 2021a h5d7bf9c_0 wheel 0.37.0 pyhd3eb1b0_1 xz 5.2.5 h7b6447c_0 zlib 1.2.11 h7b6447c_3
In [8]:
Copied!
# Export the definition (versions, packages) to a YAML file
!conda env export -p /home/spack/envs/test > environment.yaml
# Export the definition (versions, packages) to a YAML file
!conda env export -p /home/spack/envs/test > environment.yaml
In [9]:
Copied!
!cat environment.yaml
!cat environment.yaml
name: null channels: - defaults dependencies: - _libgcc_mutex=0.1=main - _openmp_mutex=4.5=1_gnu - ca-certificates=2021.9.30=h06a4308_1 - certifi=2021.10.8=py39h06a4308_0 - ld_impl_linux-64=2.35.1=h7274673_9 - libffi=3.3=he6710b0_2 - libgcc-ng=9.3.0=h5101ec6_17 - libgomp=9.3.0=h5101ec6_17 - libstdcxx-ng=9.3.0=hd4cf53a_17 - ncurses=6.2=he6710b0_1 - openssl=1.1.1l=h7f8727e_0 - pip=21.2.4=py39h06a4308_0 - python=3.9.7=h12debd9_1 - readline=8.1=h27cfd23_0 - setuptools=58.0.4=py39h06a4308_0 - sqlite=3.36.0=hc218d9a_0 - tk=8.6.11=h1ccaba5_0 - tzdata=2021a=h5d7bf9c_0 - wheel=0.37.0=pyhd3eb1b0_1 - xz=5.2.5=h7b6447c_0 - zlib=1.2.11=h7b6447c_3 prefix: /home/spack/envs/test
Note: Versions and packages are listed. As well as the path where it was installed
With that file you will be able to rebuild this environment at anytime.
If you want to use pip later to restore your environment, you can use
./envs/test/bin/pip list format=freeze > requirements.txt
However, this is quite different from conda. You need to have the correct version of python available otherwise you can not restore the environment.
Restore¶
Maybe you get a environment file from a colleague or it's your backup, there are ways to restore your environment.
Please note that conda is slow and resolving dependencies can be slow.
In [13]:
Copied!
# Create the environment in a special path
!conda env create -p /home/spack/envs/restore -f environment.yaml
# Create the environment in a special path
!conda env create -p /home/spack/envs/restore -f environment.yaml
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.3
latest version: 4.10.3
Please update conda by running
$ conda update -n base -c defaults conda
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate /home/spack/envs/restore
#
# To deactivate an active environment, use
#
# $ conda deactivate
In [14]:
Copied!
!conda env list
!conda env list
# conda environments:
#
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/anaconda3-2020.07-l2bohj4adsd6r2oweeytdzrgqmjl64lt
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-5qezliv3bkrc63cv2zk2lyh3xioozd45
/home/spack-root/opt/spack/linux-centos8-skylake_avx512/gcc-8.3.1/miniconda3-4.9.2-htox4jjktvvm7b4d3rfwpwpwa4i2kkhk
/home/spack/envs/restore
/home/spack/envs/test
/home/swd/anaconda3
/home/swd/manual/intelpython/intelpython/latest
/home/swd/manual/intelpython/intelpython/latest/envs/2021.4.0
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda2-2019.10-eaw5h5kknfwa3bsjnjhqfhytzfxgd3nf
base * /home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2020.07-4onhdcbnvyvwfdxxbelrplxjihjsknex
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2020.11-v46okvjlotnor7yrmpqnbn63xt3a2qyq
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/anaconda3-2021.05-hrg2ocjseksc4af3osaiipzu433xctef
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/miniconda2-4.7.12.1-3r6cvxekrhf32f6tdeout3aq7xxsgjit
/home/swd/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.4.1/miniconda3-4.10.3-l23y36yispi4ue5w7mhyc4mllio4ylfm
Please note that we have another environment like this, called restore:
/home/spack/envs/restore
In [15]:
Copied!
!conda list -p /home/spack/envs/restore
!conda list -p /home/spack/envs/restore
# packages in environment at /home/spack/envs/restore: # # Name Version Build Channel _libgcc_mutex 0.1 main _openmp_mutex 4.5 1_gnu ca-certificates 2021.9.30 h06a4308_1 certifi 2021.10.8 py39h06a4308_0 ld_impl_linux-64 2.35.1 h7274673_9 libffi 3.3 he6710b0_2 libgcc-ng 9.3.0 h5101ec6_17 libgomp 9.3.0 h5101ec6_17 libstdcxx-ng 9.3.0 hd4cf53a_17 ncurses 6.2 he6710b0_1 openssl 1.1.1l h7f8727e_0 pip 21.2.4 py39h06a4308_0 python 3.9.7 h12debd9_1 readline 8.1 h27cfd23_0 setuptools 58.0.4 py39h06a4308_0 sqlite 3.36.0 hc218d9a_0 tk 8.6.11 h1ccaba5_0 tzdata 2021a h5d7bf9c_0 wheel 0.37.0 pyhd3eb1b0_1 xz 5.2.5 h7b6447c_0 zlib 1.2.11 h7b6447c_3
When you want to use pip to restore the environment:
# create a virtual environment
python3 -m venv env
# use that to install the packages
./env/bin/pip install -r requirements.txt
Here it will use the python3 that you give, meaning that it will not be Python 3.9 as we specified above. Download the correct python interpreter first. here or any other way.
Design¶
It is possible to write your own environment file and use it for build a special environment.
There are three things you need:
- name
- channels (mostly conda-forge and defaults)
- dependencies (packages)
An example yaml file
name: selfdesigned
channels:
- conda-forge
- defaults
dependencies:
- numpy
- pandas
- xarray
- pip:
- metpy
you can specify versions using numpy=1.0 or python=3.6. If no version is given the newest compatible version will be used.
In [16]:
Copied!
%%writefile design.yaml
name: selfdesigned
channels:
- conda-forge
- defaults
dependencies:
- numpy
- pandas
- xarray
- pip:
- metpy
%%writefile design.yaml
name: selfdesigned
channels:
- conda-forge
- defaults
dependencies:
- numpy
- pandas
- xarray
- pip:
- metpy
Writing design.yaml
In [17]:
Copied!
!conda env create -f design.yaml -p /home/spack/envs/design
!conda env create -f design.yaml -p /home/spack/envs/design
Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies. Conda may not use the correct pip to install your packages, and they may end up in the wrong place. Please add an explicit pip dependency. I'm adding one for you, but still nagging you.
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.3
latest version: 4.10.3
Please update conda by running
$ conda update -n base -c defaults conda
Downloading and Extracting Packages
pandas-1.3.3 | 13.0 MB | ##################################### | 100%
numpy-1.21.2 | 6.2 MB | ##################################### | 100%
ncurses-6.2 | 985 KB | ##################################### | 100%
libcblas-3.9.0 | 11 KB | ##################################### | 100%
pytz-2021.3 | 242 KB | ##################################### | 100%
ld_impl_linux-64-2.3 | 667 KB | ##################################### | 100%
_libgcc_mutex-0.1 | 3 KB | ##################################### | 100%
sqlite-3.36.0 | 1.4 MB | ##################################### | 100%
tk-8.6.11 | 3.3 MB | ##################################### | 100%
pip-21.3 | 1.2 MB | ##################################### | 100%
libblas-3.9.0 | 12 KB | ##################################### | 100%
python_abi-3.9 | 4 KB | ##################################### | 100%
python-3.9.7 | 27.5 MB | ##################################### | 100%
tzdata-2021c | 120 KB | ##################################### | 100%
wheel-0.37.0 | 31 KB | ##################################### | 100%
libgfortran5-11.2.0 | 1.7 MB | ##################################### | 100%
zlib-1.2.11 | 86 KB | ##################################### | 100%
libopenblas-0.3.17 | 9.2 MB | ##################################### | 100%
libstdcxx-ng-11.2.0 | 4.2 MB | ##################################### | 100%
libgfortran-ng-11.2. | 19 KB | ##################################### | 100%
_openmp_mutex-4.5 | 22 KB | ##################################### | 100%
six-1.16.0 | 14 KB | ##################################### | 100%
libffi-3.4.2 | 57 KB | ##################################### | 100%
python-dateutil-2.8. | 240 KB | ##################################### | 100%
ca-certificates-2021 | 139 KB | ##################################### | 100%
libgcc-ng-11.2.0 | 887 KB | ##################################### | 100%
xz-5.2.5 | 343 KB | ##################################### | 100%
libzlib-1.2.11 | 59 KB | ##################################### | 100%
xarray-0.19.0 | 614 KB | ##################################### | 100%
setuptools-58.2.0 | 1011 KB | ##################################### | 100%
openssl-3.0.0 | 2.9 MB | ##################################### | 100%
readline-8.1 | 295 KB | ##################################### | 100%
libgomp-11.2.0 | 427 KB | ##################################### | 100%
liblapack-3.9.0 | 11 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Ran pip subprocess with arguments:
['/home/spack/envs/design/bin/python', '-m', 'pip', 'install', '-U', '-r', '/home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt']
Pip subprocess output:
Collecting metpy
Using cached MetPy-1.1.0-py3-none-any.whl (361 kB)
Collecting pooch>=0.1
Using cached pooch-1.5.2-py3-none-any.whl (57 kB)
Collecting scipy>=1.2.0
Downloading scipy-1.7.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (28.5 MB)
Collecting pint>=0.10.1
Using cached Pint-0.17-py2.py3-none-any.whl (204 kB)
Requirement already satisfied: pandas>=0.24.0 in /home/spack/envs/design/lib/python3.9/site-packages (from metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (1.3.3)
Requirement already satisfied: numpy>=1.17.0 in /home/spack/envs/design/lib/python3.9/site-packages (from metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (1.21.2)
Collecting matplotlib>=3.0.1
Downloading matplotlib-3.4.3-cp39-cp39-manylinux1_x86_64.whl (10.3 MB)
Requirement already satisfied: xarray>=0.14.1 in /home/spack/envs/design/lib/python3.9/site-packages (from metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (0.19.0)
Collecting traitlets>=4.3.0
Downloading traitlets-5.1.0-py3-none-any.whl (101 kB)
Collecting pyproj>=2.3.0
Downloading pyproj-3.2.1-cp39-cp39-manylinux2010_x86_64.whl (6.3 MB)
Collecting pyparsing>=2.2.1
Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting kiwisolver>=1.0.1
Downloading kiwisolver-1.3.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
Collecting pillow>=6.2.0
Downloading Pillow-8.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
Requirement already satisfied: python-dateutil>=2.7 in /home/spack/envs/design/lib/python3.9/site-packages (from matplotlib>=3.0.1->metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (2.8.2)
Collecting cycler>=0.10
Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Requirement already satisfied: pytz>=2017.3 in /home/spack/envs/design/lib/python3.9/site-packages (from pandas>=0.24.0->metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (2021.3)
Collecting packaging
Using cached packaging-21.0-py3-none-any.whl (40 kB)
Collecting appdirs
Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting requests
Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting certifi
Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Requirement already satisfied: setuptools>=40.4 in /home/spack/envs/design/lib/python3.9/site-packages (from xarray>=0.14.1->metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (58.2.0)
Requirement already satisfied: six in /home/spack/envs/design/lib/python3.9/site-packages (from cycler>=0.10->matplotlib>=3.0.1->metpy->-r /home/spack/computer-resources/Python/condaenv.wjxlpjet.requirements.txt (line 1)) (1.16.0)
Collecting idna<4,>=2.5
Downloading idna-3.3-py3-none-any.whl (61 kB)
Collecting charset-normalizer~=2.0.0
Downloading charset_normalizer-2.0.7-py3-none-any.whl (38 kB)
Collecting urllib3<1.27,>=1.21.1
Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Installing collected packages: urllib3, pyparsing, idna, charset-normalizer, certifi, requests, pillow, packaging, kiwisolver, cycler, appdirs, traitlets, scipy, pyproj, pooch, pint, matplotlib, metpy
Successfully installed appdirs-1.4.4 certifi-2021.10.8 charset-normalizer-2.0.7 cycler-0.10.0 idna-3.3 kiwisolver-1.3.2 matplotlib-3.4.3 metpy-1.1.0 packaging-21.0 pillow-8.3.2 pint-0.17 pooch-1.5.2 pyparsing-2.4.7 pyproj-3.2.1 requests-2.26.0 scipy-1.7.1 traitlets-5.1.0 urllib3-1.26.7
#
# To activate this environment, use
#
# $ conda activate /home/spack/envs/design
#
# To deactivate an active environment, use
#
# $ conda deactivate
In [18]:
Copied!
!/home/spack/envs/design/bin/pip list
!/home/spack/envs/design/bin/pip list
Package Version ------------------ --------- appdirs 1.4.4 certifi 2021.10.8 charset-normalizer 2.0.7 cycler 0.10.0 idna 3.3 kiwisolver 1.3.2 matplotlib 3.4.3 MetPy 1.1.0 numpy 1.21.2 packaging 21.0 pandas 1.3.3 Pillow 8.3.2 Pint 0.17 pip 21.3 pooch 1.5.2 pyparsing 2.4.7 pyproj 3.2.1 python-dateutil 2.8.2 pytz 2021.3 requests 2.26.0 scipy 1.7.1 setuptools 58.2.0 six 1.16.0 traitlets 5.1.0 urllib3 1.26.7 wheel 0.37.0 xarray 0.19.0
FIN