Refs #2 Example notebook with files too

docs/installs/fermi/.ipynb_checkpoints/fermi-checkpoint.ipynb

@@ -0,0 +1,382 @@
+{
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 2",
+   "language": "python",
+   "name": "python2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.5"
+  },
+  "name": "",
+  "signature": "sha256:1bae707cd89c1671128b0810cb43168425b2b5be834a6f71b3cf3c734858a1ac"
+ },
+ "nbformat": 3,
+ "nbformat_minor": 0,
+ "worksheets": [
+  {
+   "cells": [
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "<h1>Installation of Sassena 1.4.1 on cluster fermi.ornl.gov</h1>\n",
+      "\n",
+      "<a id='Table of Contents'></a><h3>Table of Contents</h3>  \n",
+      "\n",
+      "<a href='#packages'>Download packages</a>    \n",
+      "<a href='#boost'>Install boost library</a>  \n",
+      "<a href='#fftw'>Install FFTW3 library</a>  \n",
+      "<a href='#sassena'>Install Sassena</a>  \n"
+     ]
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "(<a href='#Table of Contents'>Top</a>)<a id='packages'></a><h3>Download packages</h3>  \n",
+      "\n",
+      "* [sassena-library-package-v1.4.1.zip](files/sassena-library-package-v1.4.1.zip)\n",
+      "* [boost_1_47_0.tar.gz](files/boost_1_47_0.tar.gz)\n",
+      "* [fftw-3.3.tar.gz](files/fftw-3.3.tar.gz)\n",
+      "\n",
+      "Let's assume sassena will be installed under <code>/sw/sassena/</code>, where we have downloaded all these packages."
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "unzip sassena-library-package-v1.4.1.zip && mv sassena-library-package/ 1.4.1/\n",
+      "mv boost_1_47_0.tar.gz fftw-3.3.tar.gz 1.4.1/ && cd 1.4.1"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "(<a href='#Table of Contents'>Top</a>)<a id='boost'></a><h3>Install boost library</h3>\n",
+      "Sassena uses an older version of boost"
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "#!/bin/bash\n",
+      "module load openmpi-x86_64\n",
+      "PREFIX=/sw/sassena/1.4.1\n",
+      "cd $PREFIX\n",
+      "tar -xzf boost_1_47_0.tar.gz\n",
+      "cd $PREFIX/boost_1_47_0"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "There are required changes to a few of the source files in  boost 1.47 prior to compiling, <i>if you are compiling with gcc > 4.6</i>:\n",
+      "\n",
+      "Edit file ./boost/config/stdlib/libstdcpp3.hpp and substitute line 36:  \n",
+      "<code>|| defined(_GLIBCXX__PTHREADS)</code>  \n",
+      "with these two lines  \n",
+      "<code>|| defined(_GLIBCXX__PTHREADS) \\\n",
+      "|| defined(_GLIBCXX_HAS_GTHREADS)</code>\n",
+      "\n",
+      "Substitute <code>TIME\\_UTC</code> with <code>TIME\\_UTC\\_</code> in all files with the following command:  \n",
+      "<code>for file in `grep -R TIME_UTC * |cut -d ':' -f 1|sort -u|tr '\\n' ' '`;do\n",
+      "   perl -pi -e 's/TIME\\_UTC/TIME\\_UTC\\_/g' $file\n",
+      "done</code>\n",
+      "\n",
+      "After these changes we can compile safely."
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "module load openmpi-x86_64  # Load the environment for mpi compilation\n",
+      "PREFIX=/sw/sassena/1.4.1\n",
+      "TARGETDIR=/sw/sassena/1.4.1\n",
+      "mkdir -p $TARGETDIR/include/boost_1_47_0\n",
+      "mkdir -p $TARGETDIR/lib64/boost_1_47_0\n",
+      "cd $PREFIX/boost_1_47_0\n",
+      "./bootstrap.sh --includedir=$TARGETDIR/include/boost_1_47_0 --libdir=$TARGETDIR/lib64/boost_1_47_0\n",
+      "echo \"using mpi ; \" >> project-config.jam\n",
+      "./b2 install"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "The previous commands will compile boost and save artifacts to the include and lib64 directories \n",
+      "For completeness, here are the contects of the previously loaded openmpi-x86_64 module:  \n",
+      "<pre>#%Module 1.0\n",
+      "#\n",
+      "#  OpenMPI module for use with 'environment-modules' package:\n",
+      "#\n",
+      "conflict                mpi\n",
+      "prepend-path            PATH            /usr/lib64/openmpi/bin\n",
+      "prepend-path            LD_LIBRARY_PATH /usr/lib64/openmpi/lib\n",
+      "prepend-path            PYTHONPATH      /usr/lib64/python2.6/site-packages/openmpi\n",
+      "prepend-path            MANPATH         /usr/share/man/openmpi-x86_64\n",
+      "setenv                  MPI_BIN         /usr/lib64/openmpi/bin\n",
+      "setenv                  MPI_SYSCONFIG   /etc/openmpi-x86_64\n",
+      "setenv                  MPI_FORTRAN_MOD_DIR     /usr/lib64/gfortran/modules/openmpi-x86_64\n",
+      "setenv                  MPI_INCLUDE     /usr/include/openmpi-x86_64\n",
+      "setenv                  MPI_LIB         /usr/lib64/openmpi/lib\n",
+      "setenv                  MPI_MAN         /usr/share/man/openmpi-x86_64\n",
+      "setenv                  MPI_PYTHON_SITEARCH     /usr/lib64/python2.6/site-packages/openmpi\n",
+      "setenv                  MPI_COMPILER    openmpi-x86_64\n",
+      "setenv                  MPI_SUFFIX      _openmpi\n",
+      "setenv                  MPI_HOME        /usr/lib64/openmpi\n",
+      "</pre>"
+     ]
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "(<a href='#Table of Contents'>Top</a>)<a id='fftw'></a><h3>Install FFTW3 library</h3>  \n",
+      "Library for fast fourier transform"
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "module load openmpi-x86_64\n",
+      "PREFIX=/sw/sassena/1.4.1\n",
+      "cd $PREFIX/\n",
+      "tar -xzf fftw-3.3.tar.gz\n",
+      "cd $PREFIX/fftw-3.3\n",
+      "./configure --prefix=$PREFIX\n",
+      "make && make install\n",
+      "./configure --prefix=$PREFIX --enable-shared\n",
+      "make clean && make && make install"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "<h2>Fireworks</h2>"
+     ]
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "(<a href='#Table of Contents'>Top</a>)<a id='sassena'></a><h3>Install Sassena</h3>  \n",
+      "In addition to <code>boost</code> and <code>fftw</code>, sassena requires knowing the location of other libraries:  \n",
+      "\n",
+      "* BLAS\n",
+      "* LAPACK\n",
+      "* HDF5\n",
+      "\n",
+      "Here we provide in painful detail of how to export all neccessary environment variables to compile in cluster fermi.ornl.gov. Many of these variables are commented out, but you may need to uncomment some if your environment is unable to find them:"
+     ]
+    },
+    {
+     "cell_type": "code",
+     "collapsed": false,
+     "input": [
+      "module load openmpi-x86_64\n",
+      "PREFIX=/sw/fermi/sassena/1.4.1\n",
+      "cd $PREFIX\n",
+      "tar -zxf sassena-v1.4.1.tar.gz\n",
+      "\n",
+      "#######################\n",
+      "#  Boost environment  #\n",
+      "#######################\n",
+      "BOOST_ROOT=$PREFIX/boost_1_47_0\n",
+      "Boost_INCLUDE_DIR=${BOOST_ROOT}\n",
+      "BOOST_LIBRARYDIR=$PREFIX/lib64/boost_1_47_0\n",
+      "Boost_LIBRARY_DIRS=$BOOST_LIBRARYDIR\n",
+      "Boost_FILESYSTEM_LIBRARY=$BOOST_LIBRARYDIR/libboost_filesystem.so\n",
+      "Boost_FILESYSTEM_LIBRARY_DEBUG=$BOOST_LIBRARYDIR/libboost_filesystem.so\n",
+      "DBoost_FILESYSTEM_LIBRARY_RELEASE=$BOOST_LIBRARYDIR/libboost_filesystem.so\n",
+      "Boost_SYSTEM_LIBRARY=$BOOST_LIBRARYDIR/libboost_system.so\n",
+      "Boost_SYSTEM_LIBRARY_DEBUG=$BOOST_LIBRARYDIR/libboost_system.so\n",
+      "Boost_SYSTEM_LIBRARY_RELEASE=$BOOST_LIBRARYDIR/libboost_system.so\n",
+      "Boost_REGEX_LIBRARY=$BOOST_LIBRARYDIR/libboost_regex.so\n",
+      "Boost_REGEX_LIBRARY_DEBUG=$BOOST_LIBRARYDIR/libboost_regex.so\n",
+      "Boost_REGEX_LIBRARY_RELEASE=$BOOST_LIBRARYDIR/libboost_regex.so\n",
+      "Boost_THREAD_LIBRARY=$BOOST_LIBRARYDIR/libboost_thread.so\n",
+      "Boost_THREAD_LIBRARY_DEBUG=$BOOST_LIBRARYDIR/libboost_thread.so\n",
+      "Boost_THREAD_LIBRARY_RELEASE=$BOOST_LIBRARYDIR/libboost_thread.so\n",
+      "Boost_PROGRAM_OPTIONS_LIBRARY=$BOOST_LIBRARYDIR/libboost_program_options.so\n",
+      "Boost_PROGRAM_OPTIONS_LIBRARY_DEBUG=$BOOST_LIBRARYDIR/libboost_program_options.so\n",
+      "Boost_PROGRAM_OPTIONS_LIBRARY_RELEASE=$BOOST_LIBRARYDIR/libboost_program_options.so\n",
+      "Boost_SERIALIZATION_LIBRARY=$BOOST_LIBRARYDIR/libboost_serialization.so\n",
+      "Boost_SERIALIZATION_LIBRARY_DEBUG=$BOOST_LIBRARYDIR/libboost_serialization.so\n",
+      "Boost_SERIALIZATION_LIBRARY_RELEASE=$BOOST_LIBRARYDIR/libboost_serialization.so\n",
+      "\n",
+      "#####################\n",
+      "# BLAS environment  #\n",
+      "#####################\n",
+      "#BLASLIB=acml\n",
+      "#BLAS_ROOT=/cm/shared/apps/acml/5.3.1/gfortran64\n",
+      "#BLAS_DIR=$BLAS_ROOT/lib\n",
+      "#BLASDIR=$BLAS_ROOT/lib\n",
+      "#BLAS_LIBRARIES=$BLASDIR/libacml.so\n",
+      "#LD_LIBRARY_PATH=$BLASDIR:LD_LIBRARY_PATH\n",
+      "#INCLUDE=$BLAS_ROOT/include\n",
+      "#module load acml/gcc/64/5.3.1\n",
+      "#BLAS_LIBRARIES=/cm/shared/apps/acml/5.3.1/gfortran64/lib/libacml.so\n",
+      "BLAS_LIBRARIES=/usr/lib64/libblas.so\n",
+      "\n",
+      "#######################\n",
+      "# LAPACK environment  #\n",
+      "#######################\n",
+      "#module load lapack\n",
+      "#LAPACK_LIBRARIES=/usr/lib64/liblapack.so\n",
+      "LAPACK_LIBRARIES=/usr/lib64/atlas/liblapack.so.3\n",
+      "\n",
+      "#####################\n",
+      "# FFTW environment  #\n",
+      "#####################\n",
+      "export FFTW3_HOME=$PREFIX\n",
+      "\n",
+      "#####################\n",
+      "# HDF5 environment  #\n",
+      "#####################\n",
+      "HDF5_INCLUDE_DIR=/usr/include\n",
+      "HDF5_LIBRARY_DIR=/usr/lib64\n",
+      "HDF5HL_LIBRARY_DIR=/usr/lib64\n",
+      "HDF5_LIBRARIES=${HDF5_LIBRARY_DIR}/libhdf5.so\n",
+      "HDF5_LIBRARY=${HDF5_LIBRARY_DIR}/libhdf5.so\n",
+      "HDF5HL_LIBRARY=${HDF5_LIBRARY_DIR}/libhdf5_hl.so\n",
+      "LD_LIBRARY_PATH=hdf5:$LD_LIBRARY_PATH\n",
+      "\n",
+      "########################################\n",
+      "# Sassena environment and compilation  #\n",
+      "########################################\n",
+      "\n",
+      "SASSENA_HOME=$PREFIX/sassena-v1.4.1\n",
+      "BUILDIR=$PREFIX/build\n",
+      "mkdir -p $BUILDIR && cd $BUILDIR\n",
+      "\n",
+      "SASSENA_HOME=$PREFIX/sassena-v1.4.1\n",
+      "BUILDIR=$PREFIX/build\n",
+      "mkdir -p $BUILDIR && cd $BUILDIR\n",
+      "\n",
+      "cmake -D CMAKE_EXE_LINKER_FLAGS=\"-Wl,-rpath,$LD_LIBRARY_PATH\" -DCMAKE_INSTALL_PREFIX=$PREFIX \\\n",
+      "$SASSENA_HOME -DBoost_INCLUDE_DIR=${Boost_INCLUDE_DIR} -DBOOST_LIBRARYDIR=${BOOST_LIBRARYDIR} \\\n",
+      "-DBoost_LIBRARY_DIRS=${Boost_LIBRARY_DIRS} -DBoost_FILESYSTEM_LIBRARY=${Boost_FILESYSTEM_LIBRARY} \\\n",
+      "-DBoost_FILESYSTEM_LIBRARY_DEBUG=${Boost_FILESYSTEM_LIBRARY_DEBUG=} \\\n",
+      "-DBoost_FILESYSTEM_LIBRARY_RELEASE=${Boost_FILESYSTEM_LIBRARY_RELEASE} \\\n",
+      "-DBoost_SYSTEM_LIBRARY=${Boost_SYSTEM_LIBRARY} \\\n",
+      "-DBoost_SYSTEM_LIBRARY_DEBUG=${Boost_SYSTEM_LIBRARY_DEBUG} \\\n",
+      "-DBoost_SYSTEM_LIBRARY_RELEASE=${Boost_SYSTEM_LIBRARY_RELEASE} \\\n",
+      "-DBoost_REGEX_LIBRARY=${Boost_REGEX_LIBRARY} \\\n",
+      "-DBoost_REGEX_LIBRARY_DEBUG=${Boost_REGEX_LIBRARY_DEBUG} \\\n",
+      "-DBoost_REGEX_LIBRARY_RELEASE=${Boost_REGEX_LIBRARY_RELEASE} \\\n",
+      "-DBoost_THREAD_LIBRARY=${Boost_THREAD_LIBRARY} \\\n",
+      "-DBoost_THREAD_LIBRARY_DEBUG=${Boost_THREAD_LIBRARY_DEBUG} \\\n",
+      "-DBoost_THREAD_LIBRARY_RELEASE=${Boost_THREAD_LIBRARY_RELEASE} \\\n",
+      "-DBoost_PROGRAM_OPTIONS_LIBRARY=${Boost_PROGRAM_OPTIONS_LIBRARY} \\\n",
+      "-DBoost_PROGRAM_OPTIONS_LIBRARY_DEBUG=${Boost_PROGRAM_OPTIONS_LIBRARY_DEBUG} \\\n",
+      "-DBoost_PROGRAM_OPTIONS_LIBRARY_RELEASE=${Boost_PROGRAM_OPTIONS_LIBRARY_RELEASE} \\\n",
+      "-DBoost_SERIALIZATION_LIBRARY=${Boost_SERIALIZATION_LIBRARY} \\\n",
+      "-DBoost_SERIALIZATION_LIBRARY_DEBUG=${Boost_SERIALIZATION_LIBRARY_DEBUG} \\\n",
+      "-DBoost_SERIALIZATION_LIBRARY_RELEASE=${Boost_SERIALIZATION_LIBRARY_RELEASE} \\\n",
+      "-DHDF5_INCLUDE_DIR=${HDF5_INCLUDE_DIR} -DHDF5_LIBRARY_DIR=${HDF5_LIBRARY_DIR} \\\n",
+      "-DHDF5HL_LIBRARY_DIR=${HDF5HL_LIBRARY_DIR} -DHDF5_LIBRARIES=${HDF5_LIBRARIES} \\\n",
+      "-DHDF5_LIBRARY=${HDF5_LIBRARY} -DHDF5HL_LIBRARY=${HDF5HL_LIBRARY} -DCMAKE_BUILD_TYPE=Debug \\\n",
+      "-DBLAS_LIBRARIES=${BLAS_LIBRARIES} -DLAPACK_LIBRARIES=${LAPACK_LIBRARIES} $SASSENA_HOME\n",
+      "\n",
+      "make && make install"
+     ],
+     "language": "python",
+     "metadata": {},
+     "outputs": []
+    },
+    {
+     "cell_type": "markdown",
+     "metadata": {},
+     "source": [
+      "artifacts (shared libraries and executables) will be stored in <i>/sw/fermi/sassena/1.4.1/build</i>. At this point it is convenient to create a modules file to load the appropriate environment. In the case of fermi.ornl.gov, we save module file <code>1.4.1</code> under directory <code>/sw/fermi/modulefiles/sassena/</code>. The contents of the module file:  \n",
+      "<pre>#%Module\n",
+      "proc ModulesHelp { } {\n",
+      "  puts stderr \"SASSENA\"\n",
+      "  puts stderr \"\"\n",
+      "}\n",
+      "\n",
+      "# One line description\n",
+      "module-whatis \"Sassena 1.4.1, MPI\"\n",
+      "\n",
+      "module load openmpi-x86_64\n",
+      "set SASSENAHOME /sw/fermi/sassena/1.4.1\n",
+      "set SASSENADATABASE $SASSENAHOME/database\n",
+      "\n",
+      "prepend-path LD_LIBRARY_PATH $SASSENAHOME/build\n",
+      "prepend-path PATH $SASSENAHOME/build:$SASSENAHOME/bin\n",
+      "\n",
+      "setenv       SASSENA_DATABASE $SASSENADATABASE\n",
+      "\n",
+      "puts stderr \"Sassena 1.4.1 MPI. Executable is 'sassena'\\nSASSENA_DATABASE=$SASSENADATABASE\"\n",
+      "</pre>\n",
+      "\n",
+      "And of course, we make sure that directory <code>/sw/fermi/modulefiles</code> is part of environment variable\n",
+      "<code>MODULEPATH</code>:  \n",
+      "\n",
+      "<pre>fermi-login1:build> echo $MODULEPATH\n",
+      "/SNS/users/jbq/.local/modulefiles:/sw/fermi/modulefiles:/etc/modulefiles/:/usr/share/Modules/modulefiles:\n",
+      "</pre>\n",
+      "\n",
+      "Also, we include file <code>.version</code> in <code>/sw/fermi/modulefiles/sassena/</code> with the following contents:\n",
+      "\n",
+      "<pre>#%Module1.0\n",
+      "##\n",
+      "##  The desired version of EPW\n",
+      "##\n",
+      "set ModulesVersion \"1.4.1\"\n",
+      "</pre>\n",
+      "\n",
+      "This file will set file <code>1.4.1</code> as the default module. With all this in place, we call the sassena module to load the right environment in our queue scripts. for instance:\n",
+      "\n",
+      "<pre>\n",
+      "#!/bin/bash\n",
+      "\n",
+      "#PBS -N sassenaInc\n",
+      "#PBS -j oe\n",
+      "#PBS -l walltime=12:00:00\n",
+      "#PBS -l nodes=4:ppn=16\n",
+      "\n",
+      "#Request four nodes and using 16 cores per node \n",
+      "source $MODULESHOME/init/bash\n",
+      "module load sassena\n",
+      "\n",
+      "cd $PBS_O_WORKDIR\n",
+      "mpirun -np 64 sassena --config=sassenaInc.xml &> sassenaInc.log\n",
+      "</pre>"
+     ]
+    }
+   ],
+   "metadata": {}
+  }
+ ]
+}