[FIXED] How do I tell conda/mamba to use windows/linux/osx channels that are different from the system OS?

Issue

I would like to help someone solve their problem with installing a particular mamba environment: New mamba environment force torch CPU and I don't know why

However, they use Windows, and I am on macOS.

How can I tell mamba to use pytorch/win-64 and conda-forge/win-64 channels instead of osx-arm subchannels?

I know I can specify channels using -c but how do I specify the system subdirectory?

Solution

The CONDA_SUBDIR variable works well for this. For example,

CONDA_SUBDIR=win-64 mamba create -dn foo -c pytorch -c conda-forge pytorch

Answered By - merv

Answer Checked By - Clifford M. (PHPFixing Volunteer)

Read More

[FIXED] How to share conda environments across platforms

Issue

The conda docs at http://conda.pydata.org/docs/using/envs.html explain how to share environments with other people.

However, the docs tell us this is not cross platform:

NOTE: These explicit spec files are not usually cross platform, and      
therefore have a comment at the top such as # platform: osx-64 showing the  
platform where they were created. This platform is the one where this spec
file is known to work. On other platforms, the packages specified might not
be available or dependencies might be missing for some of the key packages
already in the spec.

NOTE: Conda does not check architecture or dependencies when installing 
from an explicit specification file. To ensure the packages work correctly,
be sure that the file was created from a working environment and that it is 
used on the same architecture, operating system and platform, such as linux-
64 or osx-64.

Is there a good method to share and recreate a conda environment in one platform (e.g. CentOS) in another platform (e.g. Windows)?

Solution

This answer is given with the assumption that you would like to make sure that the same versions of the packages that you generally care about are on different platforms and that you don't care about the exact same versions of all packages in the entire dependency tree. If you are trying to install the exact same version of all packages in your entire dependency tree that has a high likelihood of failure since some conda packages have different dependencies for osx/win/linux. For example, the recipe for otrobopt will install different packages on Win vs. osx/linux, so the environment list would be different.

Recommendation: manually create an environment.yaml file and specify or pin only the dependencies that you care about. Let the conda solver do the rest. Probably worth noting is that conda-env (the tool that you use to manage conda environments) explicitly recommends that you "Always create your environment.yml file by hand."

Then you would just do conda env create --file environment.yml

Have a look at the readme for conda-env.

They can be quite simple:

name: basic_analysis
dependencies:
  - numpy
  - pandas

Or more complex where you pin dependencies and specify anaconda.org channels to install from:

name: stats-web
channels:
  - javascript
dependencies:
  - python=3.4   # or 2.7 if you are feeling nostalgic
  - bokeh=0.9.2
  - numpy=1.9
  - nodejs=0.10
  - flask
  - pip:
    - Flask-Testing

In either case, you can create an environment with conda env create --file environment.yaml.

NOTE: You may need to use .* as a version suffix if you're using an older version of conda.

Answered By - Eric Dill

Answer Checked By - Clifford M. (PHPFixing Volunteer)

Read More

[FIXED] How to make conda or mamba verbose using environment variable?

Issue

I am looking for an environment variable I can set to make mamba verbose while it is resolving the dependency to see if it is stuck at a certain stage.

Solution

This can be done either through Conda configuration (i.e., .condarc) or an environment variable. The environment variable has the highest priority, so I recommend setting default behavior via the configuration such that the environment variable is still available for overriding.

There is an (older) article detailing the Conda configuration system: "The Conda Configuration Engine for Power Users".

Conda Configuration

The configuration variable is verbosity. Here is the description:

$ conda config --describe verbosity
# # verbosity (int)
# #   aliases: verbose
# #   Sets output log level. 0 is warn. 1 is info. 2 is debug. 3 is trace.
# # 
# verbosity: 0

which shows 0 is the default. It can be set with

conda config --set verbosity 2  # debug by default

Environment Variable

All Conda configuration variables can be set via CONDA_* environment variables. To override the verbosity configuration, one can use:

## temporary setting
CONDA_VERBOSITY=2 conda info

## permanent setting, e.g., in a .bashrc or .bash_profile
export CONDA_VERBOSITY=2

Answered By - merv

Answer Checked By - Marilyn (PHPFixing Volunteer)

Read More

[FIXED] How to get base conda installation directory (not CONDA_PREFIX)?

Issue

I'm trying to figure out the best way to get the base conda installation directory.

I wrote this but I know this isn't the best way to do it:

(base) -bash-4.2$ which conda
/usr/local/devel/ANNOTATION/jespinoz/anaconda3/bin/conda

(base) -bash-4.2$ which conda | python -c "import sys; print('/'.join(sys.stdin.read().split('/')[:-2]))"
/usr/local/devel/ANNOTATION/jespinoz/anaconda3

Is there some environment variable I'm missing?

Solution

One trick is to echo CONDA_PREFIX from activated base:

conda run -n base bash -c "echo \${CONDA_PREFIX}"

There is also CONDA_ROOT, but I don't think that is defined when no environment is activated.

Answered By - merv

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More

[FIXED] How do I run a python file in Atom? Conda env?

Issue

I don't quite know what to do. I use VSCode and Jupyter Notebook and conda env. I just downloaded Atom and it keeps saying no kernal for grammar python. I have a similar problem if I try using the conda command in Terminal where it doesn't recognize the conda command until I:

export PATH=/Users/edgar/anaconda3/bin:$PATH

How do I make my atom run my python code? Thank you very much.

Solution

to set up atom to become a python ide you need packages like:

Community Packages (14) /home/simone/.atom/packages
├── Hydrogen@2.14.1
├── atom-ide-ui@0.13.0
├── autocomplete-python@1.16.0
├── hydrogen-python@0.0.8
├── ide-python@1.5.0
├── intentions@1.1.5
├── linter@2.3.1 (disabled)
├── linter-flake8@2.4.0
├── linter-ui-default@1.8.1
└── python-autopep8@0.1.3

and to run atom on a conda / pyenv environment you just need to:

$ cd [path to project]
$ conda activate [env]
$ atom .

so that atom will use that python env to run the scripts.

Answered By - simone viozzi

Answer Checked By - Pedro (PHPFixing Volunteer)

Read More

[FIXED] How to set up conda-installed R for use with RStudio?

Issue

I've been trying to set up my R using conda (eventually to use with Beaker Notebook) and I want to be able to use RStudio with my conda-installed version of R.

My method of installing R:

conda install -c r r
conda install -c r r-essentials
conda install -c r r-rserve
conda install -c r r-devtools
conda install -c r r-rcurl
conda install -c r r-RJSONIO
conda install -c r r-jpeg
conda install -c r r-png
conda install -c r r-roxygen2
conda install --channel https://conda.anaconda.org/bioconda bioconductor-edger

I ran that version of R (I only installed this version)

> version
               _                           
platform       x86_64-apple-darwin11.0.0   
arch           x86_64                      
os             darwin11.0.0                
system         x86_64, darwin11.0.0        
status                                     
major          3                           
minor          3.1                         
year           2016                        
month          06                          
day            21                          
svn rev        70800                       
language       R                           
version.string R version 3.3.1 (2016-06-21)
nickname       Bug in Your Hair   

Running R in Jupyter is kind of buggy. For example, when it outputs errors, it outputs to stdout and splits every character in the string with a linebreak. I want to use RStudio but I don't want to install another version of R.

How can I route my conda version of R into RStudio?

Here's my .bash_profile not sure if this will be useful:

$ cat ~/.bash_profile
# added by Anaconda3 4.0.0 installer
export PATH="/Users/jespinoz/anaconda/bin:$PATH"

export RSTUDIO_WHICH_R=/Users/jespinoz/anaconda/bin/R 

I've been trying to follow these tutorials but I am lost. I'm really not too familiar with environment variables and such things.

(1) https://support.rstudio.com/hc/en-us/community/posts/207830688-Using-RStudio-with-conda

(2) Launch mac eclipse with environment variables set

when I looked for my R it directed me to:

$ which R
/Users/jespinoz/anaconda/bin/R

but the directions from (1) is using this path which is very confusing:

/Users/jespinoz/anaconda/lib/R/bin/R

I tried doing what this guy did and added this to my .bash_profile but it didn't work. I even made a .bashrc but it still didn't work (I sourced both after I added the lines)

export RSTUDIO_WHICH_R=/Users/jespinoz/anaconda/bin/R

How to tell RStudio to use R version from Anaconda

Unfortunately, anaconda has no tutorial for this in https://docs.continuum.io/anaconda/ide_integration

Solution

So long as which R shows up a working R interpreter (which it should do if you have installed the r package from conda and activated your environment) then launching rstudio from that same environment should pick it up just fine.

For a test, on ArchLinux, I built and installed: https://aur.archlinux.org/packages/rstudio-desktop-git/

.. then force removed the R interpreter (pacman -Rdd r), then installed r from conda (conda install -c r r) and it worked fine. I then closed my terminal and opened a new one (so that the correct conda environment was not activated and successfully launched RStudio with the following command: RSTUDIO_WHICH_R=/home/ray/r_3_3_1-x64-3.5/bin/R rstudio

I think the crux is to launch RStudio from the right environment? Your ~/.bash_profile and ~/.bashrc are only sourced when you run bash. For environment variables to be set so that the your desktop environment knows about them, on Linux, you should put them in ~/.profile or else in /etc/pam.d (you may need to logout or shutdown after making those changes) and on OS X, you should check out https://apple.stackexchange.com/q/57385

Answered By - Ray Donnelly

Answer Checked By - David Marino (PHPFixing Volunteer)

Read More

[FIXED] How to use Jupyter notebooks in a conda environment?

Issue

Typically one runs jupyter notebook or jupyter-notebook or ipython notebook in a terminal to start a Jupyter notebook webserver locally (and open the URL in the browser). When using conda and conda environments, what is the best way to run a Jupyter notebook which allows to import Python modules installed in the conda environment?

As it seems, this is not quite straight forward and many users have similar troubles.

Most common error message seems to be: after installing a package XYZ in a conda environment my-env one can run import XYZ in a python console started in my-env, but running the same code in the Jupyter notebook will lead to an ImportError.

This question has been asked many times, but there is no good place to answer it, most Q&A's and Github tickets are quite messy so let's start a new Q&A here.

Solution

Disclaimer: ATM tested only in Ubuntu and Windows (see comments to this answer).

---

Jupyter runs the user's code in a separate process called kernel. The kernel can be a different Python installation (in a different conda environment or virtualenv or Python 2 instead of Python 3) or even an interpreter for a different language (e.g. Julia or R). Kernels are configured by specifying the interpreter and a name and some other parameters (see Jupyter documentation) and configuration can be stored system-wide, for the active environment (or virtualenv) or per user. If nb_conda_kernels is used, additional to statically configured kernels, a separate kernel for each conda environment with ipykernel installed will be available in Jupyter notebooks.

In short, there are three options how to use a conda environment and Jupyter:

Option 1: Run Jupyter server and kernel inside the conda environment

Do something like:

conda create -n my-conda-env         # creates new virtual env
conda activate my-conda-env          # activate environment in terminal
conda install jupyter                # install jupyter + notebook
jupyter notebook                     # start server + kernel inside my-conda-env

Jupyter will be completely installed in the conda environment. Different versions of Jupyter can be used for different conda environments, but this option might be a bit of overkill. It is enough to include the kernel in the environment, which is the component wrapping Python which runs the code. The rest of Jupyter notebook can be considered as editor or viewer and it is not necessary to install this separately for every environment and include it in every env.yml file. Therefore one of the next two options might be preferable, but this one is the simplest one and definitely fine.

Option 2: Create special kernel for the conda environment

Do something like:

conda create -n my-conda-env                               # creates new virtual env
conda activate my-conda-env                                # activate environment in terminal
conda install ipykernel                                    # install Python kernel in new conda env
ipython kernel install --user --name=my-conda-env-kernel   # configure Jupyter to use Python kernel

Then run jupyter from the system installation or a different conda environment:

conda deactivate          # this step can be omitted by using a different terminal window than before
conda install jupyter     # optional, might be installed already in system e.g. by 'apt install jupyter' on debian-based systems
jupyter notebook          # run jupyter from system

Name of the kernel and the conda environment are independent from each other, but it might make sense to use a similar name.

Only the Python kernel will be run inside the conda environment, Jupyter from system or a different conda environment will be used - it is not installed in the conda environment. By calling ipython kernel install the jupyter is configured to use the conda environment as kernel, see Jupyter documentation and IPython documentation for more information. In most Linux installations this configuration is a *.json file in ~/.local/share/jupyter/kernels/my-conda-env-kernel/kernel.json:

{
 "argv": [
  "/opt/miniconda3/envs/my-conda-env/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "my-conda-env-kernel",
 "language": "python"
}

Option 3: Use nb_conda_kernels to use a kernel in the conda environment

When the package nb_conda_kernels is installed, a separate kernel is available automatically for each conda environment containing the conda package ipykernel or a different kernel (R, Julia, ...).

conda activate my-conda-env    # this is the environment for your project and code
conda install ipykernel
conda deactivate

conda activate base            # could be also some other environment
conda install nb_conda_kernels
jupyter notebook

You should be able to choose the Kernel Python [conda env:my-conda-env]. Note that nb_conda_kernels seems to be available only via conda and not via pip or other package managers like apt.

Troubleshooting

Using Linux/Mac the command which on the command line will tell you which jupyter is used, if you are using option 1 (running Jupyter from inside the conda environment), it should be an executable from your conda environment:

$ which jupyter
/opt/miniconda3/envs/my-conda-env/bin/jupyter
$ which jupyter-notebook   # this might be different than 'which jupyter'! (see below)
/opt/miniconda3/envs/my-conda-env/bin/jupyter-notebook

Inside the notebook you should see that Python uses Python paths from the conda environment:

[1] !which python
/opt/miniconda3/envs/my-conda-env/bin/python
[2] import sys; sys.executable
'/opt/miniconda3/envs/my-conda-env/bin/python'
['/home/my_user',
 '/opt/miniconda3/envs/my-conda-env/lib/python37.zip',
 '/opt/miniconda3/envs/my-conda-env/lib/python3.7',
 '/opt/miniconda3/envs/my-conda-env/lib/python3.7/lib-dynload',
 '',
 '/opt/miniconda3/envs/my-conda-env/lib/python3.7/site-packages',
 '/opt/miniconda3/envs/my-conda-env/lib/python3.7/site-packages/IPython/extensions',
 '/home/my_user/.ipython']

Jupyter provides the command jupyter-troubleshoot or in a Jupyter notebook:

!jupyter-troubleshoot

This will print a lot of helpful information about including the outputs mentioned above as well as installed libraries and others. When asking for help regarding Jupyter installations questions, it might be good idea to provide this information in bug reports or questions.

To list all configured Jupyter kernels run:

jupyter kernelspec list

Common errors and traps

Jupyter notebook not installed in conda environment

Note: symptoms are not unique to the issue described here.

Symptoms: ImportError in Jupyter notebooks for modules installed in the conda environment (but not installed system wide), but no error when importing in a Python terminal

Explaination: You tried to run jupyter notebook from inside your conda environment (option 1, see above), there is no configuration for a kernel for this conda environment (this would be option 2) and nb_conda_kernels is not installed (option 3), but jupyter notebook is not (fully) installed in the conda environment, even if which jupyter might make you believe it was.

In GNU/Linux you can type which jupyter to check which executable of Jupyter is run.

This means that system's Jupyter is used, probably because Jupyter is not installed:

(my-conda-env) $ which jupyter-notebook
/usr/bin/jupyter

If the path points to a file in your conda environment, Jupyter is run from inside Jupyter:

(my-conda-env) $ which jupyter-notebook
/opt/miniconda3/envs/my-conda-env/bin/jupyter-notebook

Note that when the conda package ipykernel is installed, an executable jupyter is shipped, but no executable jupyter-notebook. This means that which jupyter will return a path to the conda environment but jupyter notebook will start system's jupyter-nootebook (see also here):

 $ conda create -n my-conda-env
 $ conda activate my-conda-env
 $ conda install ipykernel
 $ which jupyter            # this looks good, but is misleading!
 /opt/miniconda3/envs/my-conda-env/bin/jupyter
 $ which jupyter-notebook   # jupyter simply runs jupyter-notebook from system...
 /usr/bin/jupyter-notebook

This happens because jupyter notebook searches for jupyter-notebook, finds /usr/bin/jupyter-notebook and calls it starting a new Python process. The shebang in /usr/bin/jupyter-notebook is #!/usr/bin/python3 and not a dynamic #!/usr/bin/env python. Therefore Python manages to break out of the conda environment. I guess jupyter could call python /usr/bin/jupyter-notebook instead to overrule the shebang, but mixing system's bin files and the environment's python path can't work well anyway.

Solution: Install jupyter notebook inside the conda environment:

 conda activate my-conda-env
 conda install jupyter
 jupyter notebook

Wrong kernel configuration: Kernel is configured to use system Python

Note: symptoms are not unique to the issue described here.

Symptoms: ImportError in Jupyter notebooks for modules installed in the conda environment (but not installed system wide), but no error when importing in a Python terminal

Explanation: Typically the system provides a kernel called python3 (display name "Python 3") configured to use /usr/bin/python3, see e.g. /usr/share/jupyter/kernels/python3/kernel.json. This is usually overridden by a kernel in the conda environment, which points to the environments python binary /opt/miniconda3/envs/my-conda-env/bin/python. Both are generated by the package ipykernel (see here and here).

A user kernel specification in ~/.local/share/jupyter/kernels/python3/kernel.json might override the system-wide and environment kernel. If the environment kernel is missing or the user kernel points to a python installation outside the environment option 1 (installation of jupyter in the environment) will fail.

For occurrences and discussions of this problem and variants see here, here, here and also here, here and here.

Solution: Use jupyter kernelspec list to list the location active kernel locations.

$ conda activate my-conda-env
$ jupyter kernelspec list
Available kernels:
  python3 /opt/miniconda3/envs/my-conda-env/share/jupyter/kernels/python3

If the kernel in the environment is missing, you can try creating it manually using ipython kernel install --sys-prefix in the activated environment, but it is probably better to check your installation, because conda install ipykernel should have created the environment (maybe try re-crate the environment and re-install all packages?).

If a user kernel specification is blocking the environment kernel specification, you can either remove it or use a relative python path which will use $PATH to figure out which python to use. So something like this, should be totally fine:

$ cat ~/.local/share/jupyter/kernels/python3/kernel.json
{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

Correct conda environment not activated

Symptoms: ImportError for modules installed in the conda environment (but not installed system wide) in Jupyter notebooks and Python terminals

Explanation: Each terminal has a set of environment variables, which are lost when the terminal is closed. In order to use a conda environment certain environment variables need to be set, which is done by activating it using conda activate my-conda-env. If you attempted to run Jupyter notebook from inside the conda environment (option 1), but did not activate the conda environment before running it, it might run the system's jupyter.

Solution: Activate conda environment before running Jupyter.

 conda activate my-conda-env
 jupyter notebook

Broken kernel configuration

Symptoms: Strange things happening. Maybe similar symptoms as above, e.g. ImportError

Explanation: If you attempted to use option 2, i.e. running Jupyter from system and the Jupyter kernel inside the conda environment by using an explicit configuration for the kernel, but it does not behave as you expect, the configuration might be corrupted in some way.

Solution: Check configuration in ~/.local/share/jupyter/kernels/my-kernel-name/kernel.json and fix mistakes manually or remove the entire directory and re-create it using the command provided above for option 2. If you can't find the kernel configuration there run jupyter kernelspec list.

Python 2 vs 3

Symptoms: ImportError due to wrong Python version of the Jupyter kernel or other problems with Python 2/3

Explanation: The kernel configuration can have all sorts of confusing and misleading effects. For example the default Python 3 kernel configuration will allow me to launch a Jupyter notebook running on Python 2:

conda create -n my-conda-env
conda activate my-conda-env
conda install python=2
conda install jupyter
jupyter notebook

The default Python 3 kernel:

$ cat ~/.local/share/jupyter/kernels/python3/kernel.json
{
 "argv": [
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

After creating a new Jupyter Notebook with the Python 3 kernel, Python 2 from the conda environment will be used even if "Python 3" is displayed by Jupyter.

Solution: Don't use Python 2 ;-)

Answered By - lumbric

Answer Checked By - David Marino (PHPFixing Volunteer)

Read More

[FIXED] How do I find the name of the conda environment in which my code is running?

Issue

I'm looking for a good way to figure out the name of the conda environment I'm in from within running code or an interactive python instance.

The use-case is that I am running Jupyter notebooks with both Python 2 and Python 3 kernels from a miniconda install. The default environment is Py3. There is a separate environment for Py2. Inside the a notebook file, I want it to attempt to conda install foo. I'm using subcommand to do this for now, since I can't find a programmatic conda equivalent of pip.main(['install','foo']).

The problem is that the command needs to know the name of the Py2 environment to install foo there if the notebook is running using the Py2 kernel. Without that info it installs in the default Py3 env. I'd like for the code to figure out which environment it is in and the right name for it on its own.

The best solution I've got so far is:

import sys

def get_env():
    sp = sys.path[1].split("/")
    if "envs" in sp:
        return sp[sp.index("envs") + 1]
    else:
        return ""

Is there a more direct/appropriate way to accomplish this?

Solution

You want $CONDA_DEFAULT_ENV or $CONDA_PREFIX:

$ source activate my_env
(my_env) $ echo $CONDA_DEFAULT_ENV
my_env

(my_env) $ echo $CONDA_PREFIX
/Users/nhdaly/miniconda3/envs/my_env

$ source deactivate
$ echo $CONDA_DEFAULT_ENV  # (not-defined)

$ echo $CONDA_PREFIX  # (not-defined)

In python:

import os
print(os.environ['CONDA_DEFAULT_ENV'])

for the absolute entire path which is usually more useful:

Python 3.9.0 | packaged by conda-forge | (default, Oct 14 2020, 22:56:29) 
[Clang 10.0.1 ] on darwin
import os; print(os.environ["CONDA_PREFIX"])
/Users/miranda9/.conda/envs/synthesis

---

The environment variables are not well documented. You can find CONDA_DEFAULT_ENV mentioned here: https://www.continuum.io/blog/developer/advanced-features-conda-part-1

The only info on CONDA_PREFIX I could find is this Issue: https://github.com/conda/conda/issues/2764

Answered By - NHDaly

Answer Checked By - Mildred Charles (PHPFixing Admin)

Read More

[FIXED] What do bioconda's perl-net-ftp and perl-lwp-protocol-https do?

Issue

I was given some code of a previous intern and I'm supposed to find out what it does.

It is a snakemake pipeline that has a my_pipeline.yaml file which is supposed to contain all the 'general packages needed for the pipeline' The file looks like this:

name: my_pipeline

channels:
    - conda-forge
    - bioconda

dependencies:
    - snakemake
    - perl-net-ftp
    - perl-lwp-protocol-https

I googled both perl-net-ftp and perl-lwp-protocol-https but I was unable to find docs on them.

All I can gather from my limited understanding as a newbie: ftp stands for "file transfer protocol" and I definitely heard that sometime before, perl is a language (that I don't know). HTTPS is another protocol, sure...

Regarding the perl-net-ftp I even found the bioconda link but the description is unfortunately empty.

Same goes for the perl-lwp-protocol-https - the description on conda is blank.

In I see that the pipeline is using wget in the snakefile. Is it safe to assume that the 2 packages are needed inorder to run wget?

I'd be glad for an explanation for a beginner and of course documentation if your queries get better results.

https://anaconda.org/bioconda/perl-net-ftp

Solution

For Bioconda packages, it is usually better to look at the Bioconda Package Index rather than Anaconda Cloud (though admittedly the metadata here could be improved). Visiting the Bioconda pages for the perl-lwp-protocol-https and perl-net-ftp packages will lead to links to their respective upstream description pages.

Bioconda Page	Meta::CPAN Page
perl-lwp-protocol-https	LWP::Protocol::https
perl-net-ftp	Net::FTP

Broadly, these are Perl modules that implement functions to communicate with remote servers, which usually means downloading or uploading files in the context of a Perl script.

wget does not depend on Perl or those modules. So, it is still unclear the particular purpose of those. Maybe don't rule out the possibility that they aren't actually needed - sometimes people start writing things one way and then forget to remove components that they no longer require.

Answered By - merv

Answer Checked By - Dawn Plyler (PHPFixing Volunteer)

Read More