Python Environments

Directly from [1]: Anaconda is a distribution of the Python and R programming languages for scientific computing (data science, machine learning applications, large-scale data processing, predictive analytics, etc.), that aims to simplify package management and deployment.

Loading the Module

module load Anaconda3/2020.07
The first time you use conda, you’ll need to run conda init bash or something similar in order for conda to interact with your BASH environment appropriately.

Getting Started

Create your Environment

Use the conda create command to build a virtual environment. The environment files are saved to '/home/<username>/.conda/envs'  directory, this directory is linked to your ada research storage volume. You can specify the other location with the help of --prefix the option in the command. You should set meaningful environment names for easy readability and reference.

With conda

The user can specify the name, python versions, packages, and package versions at the environment creation time. Once a conda environment is created, a user can install packages using either conda install <package> or pip install <package>.

conda create --name=env_name <package1> <package2> ... <packageN>

or

conda create --name=env_name

followed by
conda activate env_name
pip install <package1> <package2> <package3> ... <packageN>

Using an environment

In order to “activate” a conda environment, you’ll need to run something like the above conda activate env_name.

If you’re in an interactive session (on the login node or in an srun session), that’s all you’ll need.

If you’re in a batch session, you must first run the following command within your SLURM batch script in order for your BASH environment to interact with conda correctly.
eval "$(conda shell.bash hook)"

To deactivate the environment :
conda deactivate

Installing Packages

You can install required packages using conda or pip. Attempt to install packages as much as possible with pip, then use conda if the packages are unavailable through pip. The general process is to activate an environment and then run pip/conda commands to install new packages into this environment. It is imperative that the environment you’re attempting to install the new package into is activated when you run the below commands.

With pip

Pip organizes packages alongside python interpreters better than conda (at the time this was written; 20210901), so it’s advisable to at least start with pip. To use pip for installation into an existing environment, run the following command with your conda environment activated:

pip install <package-name>

Pip works by downloading necessary files into /tmp and then using those files to install into your ~/.conda/env/env_name. After the installation finishes, the downloaded files are removed from the temporary directory. For large packages, /tmp sometimes gets filled. In these cases, you can specify an alternate temporary directory for this part of the installation.

TMPDIR="/nfs/ada/<group>/users/<username>" pip install <package-name>

With conda

To install a specific package into an existing current environment “env_name”, use the following:

conda install <package-name>

If you want to install the package in a specific  environment, mention the required environment name using the below command :

conda install --name env_name <package-name>

Additional Conda Commands

List all of your environments :
conda info --envs orconda env list
Delete a Conda Environment :
conda remove --name <environment-name> --all orconda env remove --name <environment-name>
View a list of packages installed in an environment:

If the environment is already activated, run

conda list orconda list -n <environment-name>

Understanding Storage of conda packages

The environment files are saved by default in the directory ‘/home/username>/.conda/envs‘ The --prefix option in the command allows you to specify other locations.

 


[1]: https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)