Installing a Python working environment with UH software

These instructions are specific to installing on a reasonably recent Mac, but most of the steps are similar on other machines. Some notes about Linux are included. The instructions are based on using the conda package manager with Miniconda and the conda-forge package collection, which provides platform-independent package management for Python and other software in self-contained user-specific environments. This is highly recommended, and will be required for this course. (Historical note: conda was developed as open source by a company known at the time as Continuum.io, and the original package collection was called Anaconda. It was so successful that the company changed its name to Anaconda. They still provide their collection, but a larger community-led effort developed the conda-forge system and collection which includes a broader range of packages and which updates them more promptly than Anaconda. Ideally, one would be able to use both sources together, and in fact, most of the time this works fine. There are some conflicts, however, so Anaconda and we recommend that for now one should stick exclusively to one or the other. We need the coverage and timeliness of conda-forge, so that is what we are describing here.)

The instructions also assume that you are installing most things from scratch; if you have remnants of earlier installations or attempts, conflicts could occur, and these can be very confusing and difficult to track down.

Install Miniconda base

To begin, install the current Miniconda base that matches your operating system. Choose Python 3. The installers are based on the command-line interpreter, bash–they are shell scripts. After downloading, use a terminal command line to execute, e.g.:

bash Miniconda3-latest-MacOSX-x86_64.sh

if on a Mac, or the corresponding version if on Linux. Do not use “sudo”; execute the script as a normal user, and let the installation occur in your home directory. Answer “N” when asked whether you want an entry added to your .bash_profile.

When this installation has finished, execute:

miniconda3/bin/conda

and follow the instructions to add one or two lines to your .bash_profile file in your home directory. For example, since my home directory on my Mac is /Users/efiring, the added lines are:

PATH=~/miniconda3/bin:"${PATH}"
. /Users/efiring/miniconda3/etc/profile.d/conda.sh

Now quit the terminal application completely (this is necessary with OSX; on Linux you only need to close the terminal window and open a new one), restart it, and check that the correct python is being found. Execute:

which python
python --version

These commands should show something like:

/Users/efiring/miniconda3/envs/py37/bin/python
Python 3.7.0

Install additional conda packages

One of the new commands that will be found in miniconda3/bin is conda, which is the package management tool. Packages can be installed from the default location or from alternatives. We will use conda-forge. Add a file called .condarc to your home directory. It should contain the following two lines:

channels:
    - conda-forge

Now we need to create a conda environment in which we will install the packages we need. We will start with a subset:

conda create -n py37 future matplotlib scipy ipython jupyter

Activate it:

conda activate py37

Now you are ready to install some additional packages that we will need, using the following commands (or do it in one step by concatenating the packages to a single argument list):

conda install requests netcdf4 pandas xarray statsmodels
conda install cartopy basemap basemap-data-hires

Very likely we will find additional package that we will want to install later.

Install Mercurial

Install the Mercurial package directly from its web site; it will work with the Python 2 version that is included with OS X. (In general, it is best not to install packages into the OS X Python; this is an exception.)

If you are on Linux, Mercurial might already be installed; if not, then install it using your Linux distribution’s package manager. On Ubuntu, for example, it would be:

sudo apt-get install mercurial

On either OS, when you install Mercurial, it is a good idea to use a text editor to put a minimal configuration file in your home directory. It must be called .hgrc, and should look something like this:

[ui]
username= John Doe <jdoe@hawaii.edu>

[extensions]
purge=
pager=

[diff]
git = True

[pager]
pager = less -FRX

Install a C compiler (OS X only)

You will need the C compiler that goes with OS X. You can get it by installing all of Xcode (it’s huge), and then the command-line tools, but I recommend that you just install the latter directly via the following command:

xcode-select --install

The C compiler (gcc) is usually installed by default and is always available from the native package manager on Linux.

Install gfortran and the fortran netcdf4 library

This section is optional.

For pygamma and pytide, we need gfortran. There are probably other ways of getting a working version, but the way I have done it on my machine is using Homebrew:

brew install gcc

In earlier Homebrew versions gfortran was available as an independent installation, but now it is available only as part of the gcc suite.

Now install the fortran libraries for netcdf:

brew install netcdf --with-fortran

On Linux, use your package manager to install gfortran, e.g., “sudo apt-get install gfortran”. The netcdf4 libraries will also be available via the package manager. On Ubuntu: “sudo apt install libnetcdf-dev”.

Install UH software

Select a location for the source code, all of which will be downloaded as Mercurial repositories. I recommend the following sequence of commands, which will create the directory structure that I use:

cd
mkdir currents
mkdir currents/programs
mkdir currents/tide
mkdir currents/topog
mkdir currents/topog/etopo

In the following I will assume that this is what you have done. Next, download all the repos:

cd currents/programs
hg clone http://currents.soest.hawaii.edu/hgstage/codas3
hg clone http://currents.soest.hawaii.edu/hgstage/pycurrents
hg clone http://currents.soest.hawaii.edu/hgstage/pytide
hg clone http://currents.soest.hawaii.edu/hgstage/pygamma

Compile and install codas3 first:

cd codas3
./waf configure --python_env
./waf build
./waf install

Note

If the build stage gives an error on OSX, try re-doing the configure stage with an additional specification:

./waf configure AR=/usr/bin/ar --python_env
./waf build
./waf install

The --python_env option above will make the installation be in the Miniconda environment instead of the default /usr/local. Either will work, but keeping everything together makes it easier to keep track of what you have installed, and to delete it all if you want to make a fresh start some time.

Next, install the pycurrents Python package, which depends on successful installation of codas3 as described above:

cd ../pycurrents
./runsetup.py

The next two Python packages are optional. They include Fortran extension code, so they require a compatible Fortran compiler and environment. If you run into an error, check to see whether your environment includes Fortran-related variables, and/or whether you have a Fortran compiler other than gfortran installed.

Now pytide:

cd ../pytide
pip install .

And pygamma:

cd ../pygamma
pip install .

For pycurrents we need topography data; we will use Etopo1:

cd ../../topog/etopo

Now download ftp://currents.soest.hawaii.edu/pub/outgoing/etopo1_for_pycurrents.zip and unzip it into the location to which you cd’ed above. In trying to be helpful, gui tools often do the wrong thing, and this is a typical example. First, if you download by clicking with some browsers like Safari, the file name might get an extension added or removed. Second, either the downloader or the file browser might “helpfully” expand the archive, put it in the wrong place, add extensions to the file names, etc. Therefore you need to make sure that one way or another you end up with something like this:

(test) efiring@manini2:~/currents/topog/etopo$ ls -l
total 1738224
-rw-rw-r--  1 efiring  efiring  365714041 Jan  9  2015 etopo1_for_pycurrents.zip
-rw-rw-r--  1 efiring  efiring  466624802 May  5  2011 etopo1_ice_g_i2.bin
-rw-rw-r--  1 efiring  efiring        264 Dec  4  2009 etopo1_ice_g_i2.hdr
-rw-rw-r--  1 efiring  efiring   51854400 Aug  6  2011 etopo1_ice_g_i2_s3.bin
-rw-rw-r--  1 efiring  efiring    5764800 Aug  6  2011 etopo1_ice_g_i2_s9.bin

You don’t need to keep the zip file, but you need the four others.

For pytide we need tide model data from OSU. We will start with the TPXO7.2 global model and the Hawaii model, reformatted for more efficient access. Change directory to its destination:

cd ../../tide

Download ftp://currents.soest.hawaii.edu/pub/outgoing/tidemodels_for_pytide.zip and unzip it into your current location.

Again, make sure you end up with 4 files in that directory. Here they are, as listed in the zip archive using the unzip tool, which is also the best tool to use for unzipping:

(test) efiring@manini2:~/currents/tide$ unzip -l tidemodels_for_pytide.zip
Archive:  tidemodels_for_pytide.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
       57  03-09-10 10:51   Model_tpxo7.2
       45  07-23-10 09:19   Model_haw
329350721  12-02-10 12:48   uh_le_tpxo7.2
106296262  10-08-11 16:12   uh_le_haw
 --------                   -------
435647085                   4 files

Last, we need to symbolically link the data directories into the Miniconda environment so that the python modules using the data can find it. The following assumes your Miniconda installation was made in the “miniconda3” subdirectory of your home directory, which is the default location:

cd ~/miniconda3
ln -s ~/currents/topog .
ln -s ~/currents/tide .

Using symbolic links like this means that if something ever goes horribly wrong with your Miniconda installation, you can delete it and install a new one without having to download the data files again–you only need to redo this linking step.

Finished!

Quick test

To check some basic functionality, change to a directory that is not inside any of the source code directories and run ipython. For example:

cd
ipython

At the IPython prompt, execute:

%matplotlib qt5
from pycurrents.plot.maptools import mapper
m = mapper([-163, -153], [18, 25])
m.topo()
m.grid()

You should see a nice map of the main Hawaiian Islands.

If you installed gfortran and pytide, then you can also check pytide in the same IPython session:

import numpy as np
import matplotlib.pyplot as plt
from pytide import model
tidemod = model('haw')
t = np.linspace(0, 30, 1000)
vel = tidemod.velocity(2015, t, -158, 22.75)
h = tidemod.height(2015, t, -158, 22.75)
fig, (axv, axh) = plt.subplots(nrows=2, sharex=True)
axv.plot(t, vel.u, 'r', label='U')
axv.plot(t, vel.v, 'b', label='V')
axh.plot(t, h.h, 'g', label='H')
axh.set_xlabel('days in 2015')
axv.set_ylabel('m/s')
axh.set_ylabel('m')
axv.set_title('Barotropic velocity')
axh.set_title('Tide height')
axv.legend(loc='upper right')

You just plotted the tide model hindcast of elevation and velocity at Station ALOHA for the first month of 2015.