Installing a Python working environment with UH software

These instructions are specific to installing on a reasonably recent Mac, but most of the steps are similar on other machines. Some notes about Linux are included. The instructions are based on using the conda package manager with Miniconda and the conda-forge package collection, which provides platform-independent package management for Python and other software in self-contained user-specific environments. This is highly recommended, and will be required for this course. (Historical note: conda was developed as open source by a company known at the time as Continuum.io, and the original package collection was called Anaconda. It was so successful that the company changed its name to Anaconda. They still provide their collection, but a larger community-led effort developed the conda-forge system and collection which includes a broader range of packages and which updates them more promptly than Anaconda.

The instructions also assume that you are installing most things from scratch; if you have remnants of earlier installations or attempts, conflicts could occur, and these can be confusing and difficult to track down. It can be more efficient to put an old installation aside and re-install from scratch than to troubleshoot such problems.

Install Miniconda base

To begin, install the current 64-bit Python 3 Miniconda base that matches your operating system. We will use the shell script version (*.sh) of the installer, not a GUI version (*.pkg on the Mac). We don’t even need a browser to download the installer. Open a terminal window so you can run commands. Then you can download the Mac installer by executing:

curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh

if your machine has an Intel processor, or:

curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh

if you have a machine with the M1 or M2 chip (“Apple Silicon”, which uses the ARM architecture).

For Linux:

curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

After downloading, use the bash interpreter to execute the shell script, like this:

bash Miniconda3-latest-MacOSX-x86_64.sh

but substituting the corresponding file name if you are on Apple Silicon or Linux. Do not use “sudo”; execute the script as a normal user, and let the installation occur in the default location in your home directory. Use the spacebar to page down through the user agreement, hit return, and “yes” to accept it. I suggest answering “no” to:

Do you wish the installer to initialize Miniconda3
by running conda init? [yes|no]

We will take care of this manually in a minute, when the installer has finished.

Are you running zsh, or bash? If you are on Linux, it is almost certainly bash. If you are on an older Mac, it might be bash. If you are on a newer Mac, with Catalina or later, it is probably zsh. The banner at the top of the Mac terminal window will show which one you are using. You can also get a list of processes you are running in the terminal, including either bash or zsh, by executing:

ps

If it says “-bash”, ignore the hyphen; it is just bash. If you are running zsh, then execute, on the command line:

~/miniconda3/bin/conda init zsh

or if you are running bash:

~/miniconda3/bin/conda init bash

Either way, this command will tell you that it is modifying a file, “.zshrc” or “.zprofile” if you are running zsh, and otherwise either “.bash_profile” or “.bashrc”. Note the name of the file being modified; you might need it later.

Next, put some configuration entries in your .condarc file by cutting and pasting the following lines into your terminal. (The backslashes are line continuations.) You might need to hit “return” after pasting, so the last command will be executed:

~/miniconda3/bin/conda config --add channels conda-forge
~/miniconda3/bin/conda config --set channel_priority strict
~/miniconda3/bin/conda config --set auto_activate_base false
~/miniconda3/bin/conda config --append create_default_packages ipython \
         --append create_default_packages pip \
         --append create_default_packages "blas=*=openblas"

(I am recommending a subset of the configuration options described in https://gist.github.com/ocefpaf/863fc5df6ed8444378fbb1211ad8feb1.)

Now quit the terminal application completely (this is necessary with OSX; on Linux you only need to close the terminal window and open a new one), restart it, and check that the conda executable is found. Execute:

which conda

Depending on the shell you are running, it should return either a path that starts with your home directory followed by miniconda3/condabin/conda (if running bash), or multiple lines of shell code defining a conda function (if running zsh).

Make a conda working environment

In the configuration step above we included auto_activate_base false so that when a terminal application starts it will have access to conda but it will not yet have access to conda-installed python and libraries. Although we could make the base environment also the primary working environment, there are some advantages to working instead in one or more named environments, and always explicitly activating the desired environment. Let’s make such an environment, calling it “py312”, for example. (At the time of this writing this matches the python version that will be installed by default.) We can start with a subset of desired packages:

conda create -n py312  matplotlib scipy jupyter spyder

Activate it:

conda activate py312

Notice that your prompt now starts with the name of your activated environment, in parentheses. If you are going to want this to be your working environment whenever you start up, use your text editor to append this same line to the shell configuration file that was modified when you ran the “conda init …” command.

Now you are ready to install some additional packages that we will need, using the following command:

conda install netcdf4 xarray statsmodels\
            basemap basemap-data-hires\
            cython meson-python

OS X only: install conda-forge compilers:

conda install c-compiler fortran-compiler netcdf-fortran

Install Mercurial

Mercurial has recently become available as a conda package, so:

conda install mercurial

Alternatively, it is also fine to install Mercurial downloaded from the web site (OS X) or via your package manager (Linux). As of 2024-09-03 there seems to be a problem with the conda-forge package.

When you install Mercurial, it is a good idea to use a text editor (e.g., nano) to put a minimal configuration file in your home directory. It must be called .hgrc, and should look something like this:

[ui]
username= John Doe <jdoe@hawaii.edu>

[extensions]
purge=
pager=

[diff]
git = True

[pager]
pager = less -FRX

OS X only, optional: Install the system C compiler

Although you are installing a conda-forge packge with a C compiler, it is a good idea to also install the C compiler that goes with OS X. Instead of doing this by installing the huge Xcode, just get the command-line tools via:

xcode-select --install

OS X only, optional: install Homebrew

In the long run, you will probably find it useful to install Homebrew: it provides a way to install on a Mac just about anything you would find on a Linux system, including packages that are not available from conda-forge. To install, go to the Homebrew web site and follow the instructions. Note that you will need to have installed the C compiler (command-line tools) as described above.

Linux only: Install gfortran and the fortran netcdf4 library

For pygamma and pytide, we need gfortran. Use your package manager to install gfortran, e.g., sudo apt install gfortran. The netcdf4 libraries will also be available via the package manager. On Ubuntu: sudo apt install libnetcdff-dev.

(The essential C compiler, gcc, is probably already installed by default on your Linux system. If not, install it.)

Install UH software

Select a location for the source code, all of which will be downloaded as Mercurial repositories. I recommend the following sequence of commands, which will create the directory structure that I use:

cd
mkdir currents
mkdir currents/programs
mkdir currents/tide
mkdir currents/topog
mkdir currents/topog/etopo

In the following I will assume that this is what you have done. Next, download all the repos:

cd ~/currents/programs
hg clone https://currents.soest.hawaii.edu/hgstage/codas3
hg clone https://currents.soest.hawaii.edu/hgstage/pycurrents
hg clone https://currents.soest.hawaii.edu/hgstage/pytide
hg clone https://currents.soest.hawaii.edu/hgstage/pygamma

Note

Downloading and installing codas3 is optional. It should be needed only if you work with shipboard ADCP data processing; but it is fairly quick and easy, and can’t hurt.

Remember, you should be in your working conda environment at this point. Assuming you followed the earlier instructions and created a “py312” environment, your command line prompt should have a “(py312)” prefix to show that. If it doesn’t, activate the environment now:

conda activate py312

Compile and install codas3 first:

cd ~/currents/programs/codas3
./waf configure --python_env
./waf build
./waf install

The --python_env option above will make the installation be in the Miniconda environment instead of the default /usr/local. Either will work, but keeping everything together makes it easier to keep track of what you have installed, and to delete it all if you want to make a fresh start some time.

Next, install the pycurrents Python package:

cd ~/currents/programs/pycurrents
./runsetup.py

The next two Python packages are optional. They include Fortran extension code, so they require a compatible Fortran compiler and environment. Pygamma additionally requires a compatible netcdf fortran library and development environment.

Now pytide:

cd ~/currents/programs/pytide
pip install .

And pygamma:

cd ~/currents/programs/pygamma
pip install .

For pycurrents we need topography data; we will use Etopo1:

cd ~/currents/topog/etopo
curl -O ftp://currents.soest.hawaii.edu/pub/outgoing/etopo1_for_pycurrents.zip
unzip etopo1_for_pycurrents.zip
ls -l

That last line should show something like this:

total 1738224
-rw-rw-r--  1 efiring  efiring  365714041 Jan  9  2015 etopo1_for_pycurrents.zip
-rw-rw-r--  1 efiring  efiring  466624802 May  5  2011 etopo1_ice_g_i2.bin
-rw-rw-r--  1 efiring  efiring        264 Dec  4  2009 etopo1_ice_g_i2.hdr
-rw-rw-r--  1 efiring  efiring   51854400 Aug  6  2011 etopo1_ice_g_i2_s3.bin
-rw-rw-r--  1 efiring  efiring    5764800 Aug  6  2011 etopo1_ice_g_i2_s9.bin

You may delete the zip file.

For pytide we need tide model data from OSU. We will start with the TPXO7.2 global model and the Hawaii model, reformatted for more efficient access:

cd ~/currents/tide
curl -O ftp://currents.soest.hawaii.edu/pub/outgoing/tidemodels_for_pytide.zip
unzip tidemodels_for_pytide.zip
ls -l

That should show you the zip file and the 4 files extracted from it. Again, you may delete the zip file.

Last, we need to symbolically link the data directories into the Miniconda directory so that the python modules using the data can find it. The following assumes your Miniconda installation was made in the “miniconda3” subdirectory of your home directory, which is the default location:

cd ~/miniconda3
ln -s ~/currents/topog .
ln -s ~/currents/tide .
ls
cd

Using symbolic links like this means that if something ever goes horribly wrong with your Miniconda installation, you can delete it and install a new one without having to download the data files again–you only need to redo this linking step. Putting these links into the miniconda3 directory is probably the only time you should work directly with that directory tree, as opposed to managing it with conda and using additional package installation tools such as waf and pip.

Finished!

Quick test

To check some basic functionality, change to a directory that is not inside any of the source code directories and run ipython. For example:

cd
ipython

At the IPython prompt, execute:

%matplotlib qt5
from pycurrents.plot.maptools import mapper
m = mapper([-163, -153], [18, 25])
m.topo()
m.grid()

You should see a nice map of the main Hawaiian Islands.

If you installed gfortran and pytide, then you can also check pytide in the same IPython session:

import numpy as np
import matplotlib.pyplot as plt
from pytide import model
tidemod = model('haw')
t = np.linspace(0, 30, 1000)
vel = tidemod.velocity(2015, t, -158, 22.75)
h = tidemod.height(2015, t, -158, 22.75)
fig, (axv, axh) = plt.subplots(nrows=2, sharex=True)
axv.plot(t, vel.u, 'r', label='U')
axv.plot(t, vel.v, 'b', label='V')
axh.plot(t, h.h, 'g', label='H')
axh.set_xlabel('days in 2015')
axv.set_ylabel('m/s')
axh.set_ylabel('m')
axv.set_title('Barotropic velocity')
axh.set_title('Tide height')
axv.legend(loc='upper right')

You just plotted the tide model hindcast of elevation and velocity at Station ALOHA for the first month of 2015.