.. _Installing-with-UH: Installing a Python working environment with UH software --------------------------------------------------------- These instructions are specific to installing on a reasonably recent Mac, but most of the steps are similar on other machines. Some notes about Linux are included. The instructions are based on using the conda_ package manager with Miniconda_ and the conda-forge_ package collection, which provides platform-independent package management for Python and other software in self-contained user-specific environments. This is highly recommended, and will be required for this course. (Historical note: conda was developed as open source by a company known at the time as Continuum.io, and the original package collection was called Anaconda_. It was so successful that the company changed its name to Anaconda. They still provide their collection, but a larger community-led effort developed the conda-forge_ system and collection which includes a broader range of packages and which updates them more promptly than Anaconda. The instructions also assume that you are installing most things from scratch; if you have remnants of earlier installations or attempts, conflicts could occur, and these can be confusing and difficult to track down. It can be more efficient to put an old installation aside and re-install from scratch than to troubleshoot such problems. .. _conda: https://conda.io .. _Miniconda: https://conda.io/miniconda.html .. _Anaconda: https://www.anaconda.com/download/ .. _Mercurial: https://www.mercurial-scm.org .. _Homebrew: https://brew.sh .. _Etopo1: https://www.ngdc.noaa.gov/mgg/global/global.html .. _OSU: http://volkov.oce.orst.edu/tides/ .. _conda-forge: https://conda-forge.org/ Install Miniconda base ~~~~~~~~~~~~~~~~~~~~~~ To begin, install the current 64-bit Python 3 Miniconda_ base that matches your operating system. We will use the shell script version (``*.sh``) of the installer, not a GUI version (``*.pkg`` on the Mac). We don't even need a browser to download the installer. Open a terminal window so you can run commands. Then you can download the Mac installer by executing:: curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh if your machine has an Intel processor, or:: curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh if you have a machine with the M1 or M2 chip ("Apple Silicon", which uses the ARM architecture). For Linux:: curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh After downloading, use the ``bash`` interpreter to execute the shell script, like this:: bash Miniconda3-latest-MacOSX-x86_64.sh but substituting the corresponding file name if you are on Apple Silicon or Linux. Do not use "sudo"; execute the script as a normal user, and let the installation occur in the default location in your home directory. Use the spacebar to page down through the user agreement, hit return, and "yes" to accept it. I suggest answering "no" to:: Do you wish the installer to initialize Miniconda3 by running conda init? [yes|no] We will take care of this manually in a minute, when the installer has finished. Are you running zsh, or bash? If you are on Linux, it is almost certainly bash. If you are on an older Mac, it might be bash. If you are on a newer Mac, with Catalina or later, it is probably zsh. The banner at the top of the Mac terminal window will show which one you are using. You can also get a list of processes you are running in the terminal, including either bash or zsh, by executing:: ps If it says "-bash", ignore the hyphen; it is just bash. If you are running zsh, then execute, on the command line:: ~/miniconda3/bin/conda init zsh or if you are running bash:: ~/miniconda3/bin/conda init bash Either way, this command will tell you that it is modifying a file, ".zshrc" or ".zprofile" if you are running zsh, and otherwise either ".bash_profile" or ".bashrc". Note the name of the file being modified; you might need it later. Next, put some configuration entries in your .condarc file by cutting and pasting the following lines into your terminal. (The backslashes are line continuations.) You might need to hit "return" after pasting, so the last command will be executed:: ~/miniconda3/bin/conda config --add channels conda-forge ~/miniconda3/bin/conda config --set channel_priority strict ~/miniconda3/bin/conda config --set auto_activate_base false ~/miniconda3/bin/conda config --append create_default_packages ipython \ --append create_default_packages pip \ --append create_default_packages "blas=*=openblas" (I am recommending a subset of the configuration options described in https://gist.github.com/ocefpaf/863fc5df6ed8444378fbb1211ad8feb1.) Now **quit the terminal application completely** (this is necessary with OSX; on Linux you only need to close the terminal window and open a new one), restart it, and check that the conda executable is found. Execute:: which conda Depending on the shell you are running, it should return either a path that starts with your home directory followed by ``miniconda3/condabin/conda`` (if running bash), or multiple lines of shell code defining a conda function (if running zsh). Make a conda working environment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In the configuration step above we included ``auto_activate_base false`` so that when a terminal application starts it will have access to conda but it will *not* yet have access to conda-installed python and libraries. Although we could make the base environment also the primary working environment, there are some advantages to working instead in one or more named environments, and always *explicitly* activating the desired environment. Let's make such an environment, calling it "py310", for example. (At the time of this writing this matches the python version that will be installed by default.) We can start with a subset of desired packages:: conda create -n py310 matplotlib scipy jupyter spyder Activate it:: conda activate py310 Notice that your prompt now starts with the name of your activated environment, in parentheses. If you are going to want this to be your working environment whenever you start up, use your text editor to append this same line to the shell configuration file that was modified when you ran the "conda init ..." command. Now you are ready to install some additional packages that we will need, using the following command:: conda install requests netcdf4 pandas xarray statsmodels\ basemap basemap-data-hires Install Mercurial ~~~~~~~~~~~~~~~~~ Mercurial has recently become available as a conda package, so:: conda install mercurial When you install Mercurial, it is a good idea to use a text editor (e.g., ``nano``) to put a minimal configuration file in your home directory. It must be called ``.hgrc``, and should look something like this:: [ui] username= John Doe [extensions] purge= pager= [diff] git = True [pager] pager = less -FRX Install a C compiler (OS X only) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You will need the C compiler that goes with OS X. You can get it by installing all of Xcode (it's huge), and then the command-line tools, but I recommend that you just install the latter directly via the following command:: xcode-select --install On Linux, the C compiler (gcc) is usually installed by default and is always available from the native package manager. Recommended: install Homebrew (OS X only) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are two reasons for installing Homebrew_: first, because it provides a way to install on a Mac just about anything you would find on a Linux system; and second, because it provides a reliable way to install gfortran and the fortran netcdf libraries, which are needed for my ``pygamma`` and ``pytide`` packages, as well as for many other things. To install, go to the Homebrew_ web site and follow the instructions. Note that you will need to have installed the C compiler (command-line tools) as described above. Install gfortran and the fortran netcdf4 library ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This section is optional. For ``pygamma`` and ``pytide``, we need ``gfortran``. Although gfortran and the netcdf-fortran are now available via conda for the Mac, I don't have enough experience with them to provide instructions for their use. Therefore, for now, I am keeping the instructions I have used for getting them from Homebrew:: brew install gcc This brings in gfortran along with the underlying gcc C compiler. Now install the fortran libraries for ``netcdf``:: brew install netcdf On Linux, use your package manager to install ``gfortran``, e.g., ``sudo apt install gfortran``. The netcdf4 libraries will also be available via the package manager. On Ubuntu: ``sudo apt install libnetcdff-dev``. Install UH software ~~~~~~~~~~~~~~~~~~~ Select a location for the source code, all of which will be downloaded as Mercurial repositories. I recommend the following sequence of commands, which will create the directory structure that I use:: cd mkdir currents mkdir currents/programs mkdir currents/tide mkdir currents/topog mkdir currents/topog/etopo In the following I will assume that this is what you have done. Next, download all the repos:: cd ~/currents/programs hg clone https://currents.soest.hawaii.edu/hgstage/codas3 hg clone https://currents.soest.hawaii.edu/hgstage/pycurrents hg clone https://currents.soest.hawaii.edu/hgstage/pytide hg clone https://currents.soest.hawaii.edu/hgstage/pygamma .. note:: Downloading and installing codas3 is optional. It should be needed only if you work with shipboard ADCP data processing; but it is fairly quick and easy, and can't hurt. Remember, you should be in your working conda environment at this point. Assuming you followed the earlier instructions and created a "py310" environment, your command line prompt should have a "(py310)" prefix to show that. If it doesn't, activate the environment now:: conda activate py310 Compile and install ``codas3`` first:: cd ~/currents/programs/codas3 ./waf configure --python_env ./waf build ./waf install The ``--python_env`` option above will make the installation be in the Miniconda environment instead of the default ``/usr/local``. Either will work, but keeping everything together makes it easier to keep track of what you have installed, and to delete it all if you want to make a fresh start some time. Next, install the ``pycurrents`` Python package:: cd ~/currents/programs/pycurrents ./runsetup.py The next two Python packages are optional. They include Fortran extension code, so they require a compatible Fortran compiler and environment. Pygamma additionally requires a compatible netcdf fortran library and development environment. Now ``pytide``:: cd ~/currents/programs/pytide pip install . And ``pygamma``:: cd ~/currents/programs/pygamma pip install . For ``pycurrents`` we need topography data; we will use Etopo1_:: cd ~/currents/topog/etopo curl -O ftp://currents.soest.hawaii.edu/pub/outgoing/etopo1_for_pycurrents.zip unzip etopo1_for_pycurrents.zip ls -l That last line should show something like this:: total 1738224 -rw-rw-r-- 1 efiring efiring 365714041 Jan 9 2015 etopo1_for_pycurrents.zip -rw-rw-r-- 1 efiring efiring 466624802 May 5 2011 etopo1_ice_g_i2.bin -rw-rw-r-- 1 efiring efiring 264 Dec 4 2009 etopo1_ice_g_i2.hdr -rw-rw-r-- 1 efiring efiring 51854400 Aug 6 2011 etopo1_ice_g_i2_s3.bin -rw-rw-r-- 1 efiring efiring 5764800 Aug 6 2011 etopo1_ice_g_i2_s9.bin You may delete the zip file. For ``pytide`` we need tide model data from OSU_. We will start with the TPXO7.2 global model and the Hawaii model, reformatted for more efficient access:: cd ~/currents/tide curl -O ftp://currents.soest.hawaii.edu/pub/outgoing/tidemodels_for_pytide.zip unzip tidemodels_for_pytide.zip ls -l That should show you the zip file and the 4 files extracted from it. Again, you may delete the zip file. Last, we need to symbolically link the data directories into the Miniconda directory so that the python modules using the data can find it. The following assumes your Miniconda installation was made in the "miniconda3" subdirectory of your home directory, which is the default location:: cd ~/miniconda3 ln -s ~/currents/topog . ln -s ~/currents/tide . ls cd Using symbolic links like this means that if something ever goes horribly wrong with your Miniconda installation, you can delete it and install a new one without having to download the data files again--you only need to redo this linking step. Putting these links into the miniconda3 directory is probably the *only* time you should work directly with that directory tree, as opposed to managing it with conda and using additional package installation tools such as waf and pip. Finished! Quick test ~~~~~~~~~~ To check some basic functionality, change to a directory that is not inside any of the source code directories and run `ipython`. For example:: cd ipython At the IPython prompt, execute:: %matplotlib qt5 from pycurrents.plot.maptools import mapper m = mapper([-163, -153], [18, 25]) m.topo() m.grid() You should see a nice map of the main Hawaiian Islands. If you installed gfortran and pytide, then you can also check pytide in the same IPython session:: import numpy as np import matplotlib.pyplot as plt from pytide import model tidemod = model('haw') t = np.linspace(0, 30, 1000) vel = tidemod.velocity(2015, t, -158, 22.75) h = tidemod.height(2015, t, -158, 22.75) fig, (axv, axh) = plt.subplots(nrows=2, sharex=True) axv.plot(t, vel.u, 'r', label='U') axv.plot(t, vel.v, 'b', label='V') axh.plot(t, h.h, 'g', label='H') axh.set_xlabel('days in 2015') axv.set_ylabel('m/s') axh.set_ylabel('m') axv.set_title('Barotropic velocity') axh.set_title('Tide height') axv.legend(loc='upper right') You just plotted the tide model hindcast of elevation and velocity at Station ALOHA for the first month of 2015.