Overview
=========

chirp_test is a package to translate CrIS L1b and AIRS L1c files 
to a common, more generic L1c format.

Source Repos
-------------

https://github.com/motteler/chirp_test.git
https://github.com/strow/airs_deconv.git
https://github.com/strow/ccast.git

Compilation
------------

The Matlab compiler bundles related files to make a "deployable
archive," and writes a short shell script to set some paths and call
the compiled program.  In addition to the CHIRP repo, CHIRP depends
on the ccast and airs_decon git repos.  airs_decon is the AIRS to
CrIS translation, and is the heart of the AIRS-parent CHIRP product.
CCAST is the UMBC CrIS L0 to L1b package, and is used here mainly
for support routines.

The deployable archive also includes a number of support files.
For example for CrIS-parent this includes a bias vector, the NIST
table of leap-seconds, and a file ALGVERS used to pass the git
version to the product attributes.  These dependencies and support
files are set in the Makefile, for compiled files, and as Matlab
"addpath" statements for interpreted files.

To compile CHIRP, check the paths and files in the Makefile, and
type "make cris_all" and "make airs_all".  This produces executables
chirp_cris_main, chirp_airs_main, and wrappers run_chirp_cris_main.sh
and run_chirp_airs_main.sh

Running CHIRP
--------------

cris_chirp_main and airs_chirp_main translate one AIRS or CrIS
granule to one CHIRP granule, per call.  They can be run as either
interpreted or compiled functions.  Runtime parameters and some
attributes are set in YAML spec files,

  chirp_AQ_demo.yaml  - AIRS-parent YAML spec
  chirp_J1_demo.yaml  - CrIS J1-parent YAML spec
  chirp_SN_demo.yaml  - CrIS SNPP-parent YAML spec

These can be edited for single-granule tests; for that case the
processing options airs_l1c, airs_dir, and chirp_dir should be set
to match the desired test granule.  The Makefile includes examples
of running compiled files on a single granule.

For more general testing, scripts jpl_airs_loop and jpl_cris_loop
read these YAML specs, use them as templates, loop on days and
granules, update temporary versions of a single-granule YAML spec,
and call cris_chirp_main or airs_chirp_main.  This is intended as a
rough simulation of the JPL production environment.

To run compiled files, Matlab needs space to unpack the deployable
archive.  The default path is a dotfile subdirectory in the user's
home directory, a bad choice for a production environment.  This can
be changed with the environment variable MCR_CACHE_ROOT, which
should be set before running the compiled file.  For example, to
unpack the archive in /home/motteler/runlib and run in the current
working directory

  MCR_CACHE_ROOT=/home/motteler/runlib
  export MCR_CACHE_ROOT
  run_chirp_airs_main.sh /opt/matlab/R2020a chirp_AQ_gran.yaml

We set MCR_CACHE_ROOT in a separate step rather than in the run
script because the latter is overwritten with each compilation.
The run script parameters include a path to a local matlab install
for the runtime, bin, and sys directories.  But the compiled file
should not need a valid Matlab licence to run.


CHIRP Files
============

production loop
----------------
  airs2chirp      - translate AIRS to CHIRP granules
  cris2chirp      - interpolate CrIS to CHIRP granules
  chirp_airs_main - top-level wrapper for airs2chirp
  chirp_cris_main - top-level wrapper for cris2chirp

  umbc_cris_loop  - set cris2chirp options and loop on days
  umbc_airs_loop  - set airs2chirp options and loop on days
  jpl_cris_loop   - set cris2chirp options and loop on days
  jpl_airs_loop   - set airs2chirp options and loop on days

  chirp_batch     - run chirp translation as a batch job

global attributes
------------------
  airs_src_attr   - airs-parent per-granule attributes
  cris_src_attr   - cris-parent per-granule attributes
  copy_airs_attr  - copy attributes from AIRS data
  copy_cris_attr  - copy attributes from CRIS data
  airs_geo_attr   - geo-dependent AIRS attributes  
  write_prod_attr - write prod_attr global attributes

support and test
-----------------
  utc_string      - datenum to UTC char array
  nasa_fname      - build a nasa product filename
  read_airs_h4    - read an AIRS HDF4 granule
  read_netcdf_h5  - general purpose netcdf H5 reader
  read_yaml_cfg   - read yaml proc opts and prod attr
  write_yaml_cfg  - write yaml proc opts and prod attr
  airs_obs_id     - generate obs id strings for AIRS
  nedn_est        - NEdN estimate for the AIRS-to-CrIS translation

  airs_fov_gen    - FOV polygons and range from AIRS L1b data
  test_fov_gen    - quick test of matlab version of airs_fov_gen
  bias_correct    - apply a simple bias correction
  mk_airs_bias    - bias file for AIRS-parent CHIRP
  bias_check      - compare radiance and BT bias corrections

  a2cris_lin      - airs2cris linearization test
  a2cris_srfs     - airs2cris linearization with SRF plots
  airs_num_synth  - multi-granule count of L1cNumSynth values
  check_synth     - counts for synthetic and fixed channels
  nedn_interp     - NEdN interpolation demo
  pen_lift        - add NaNs for pen lift at frequency gaps

  quick_look      - quick look at chirp granules
  quick_attrs     - quick look at chirp attributes
  cmp_parent      - basic 1-granule AIRS and CHIRP comparison
  cmp_gran_jpl    - compare last 2 runs of the same jpl granule
  cmp_gran_umbc   - compare last 2 runs of the same umbc granule

YAML config files
-----------------
  chirp_AQ_init.yaml  - AIRS parent yaml job spec file
  chirp_J1_init.yaml  - CrIS j01 parent yaml job spec file
  chirp_SN_init.yaml  - CrIS npp parent yaml job spec file

  chirp_AQ_gran.yaml  - machine-written AIRS yaml granule file
  chirp_J1_gran.yaml  - machine-written CrIS j01 yaml granule file
  chirp_SN_gran.yaml  - machine-written CrIS npp yaml granule file

netCDF format
--------------
  chirp_1330.cdl  - CHIRP 1330 netCDF CDL template
  chirp_1330.nc   - CHIRP 1330 netCDF initial (aka template) file
  chirp_1000.nc   - (temporary, a) symlink to chirp_1330.nc


Notes
======

netCDF notes
-------------
Starting from an existing netCDF file, you can dump the CDL format
spec, edit it extensively, add or delete variables, write an empty
netCDF file from the CDL spec, and write data to the netCDF file.
This is convenient for managing relatively complex formats.  Some
examples:

 - chirp initial netCDF file from a CDL spec
     ncgen -o chirp_master.nc chirp_master.cdl

 - CDL spec extrated from a netCDF file
      ncdump -h uw_sdr_test.nc > uw_sdr_test.cdl

 - initial netcdf file from a CDL spec
      ncgen -o ncgen_test.nc uw_sdr_test.cdl

compilation notes
------------------
The environment variable MCR_CACHE_ROOT will set an alternate
path to the .mcrCache9.8 (or whatever) directory.  The directory
is set at runtime, not at compile time, and you can move it around
by changing MCR_CACHE_ROOT without recompiling.  ctfroot called at
runtime returns this path.  Beware "ls" may miss .mcrCache9.8 since
it's a "dotfile".  We don't want the "-C" option, that creates the
cache directory in the run directory.