Issue in running the tracking algorithm for CM1 output

[ealucy]

My goal is to track vorticity features within tropical cyclone model output in CM1. Currently, when running the code, I'm getting an error that the algorithm doesn't seem to be seeing the files within the directory that I have housed them:

(aug23_env) el381212@turing:/nfs/tcdynasty/lucy$ python run_generic_tracking.py config.yml
2023-12-05 20:01:29,557 - pyflextrkr.idfeature_driver - INFO - Identifying features from raw data
2023-12-05 20:01:30,181 - pyflextrkr.idfeature_driver - INFO - Total number of files to process: 0
2023-12-05 20:01:30,184 - pyflextrkr.idfeature_driver - INFO - Done with features from raw data.
2023-12-05 20:01:30,184 - pyflextrkr.tracksingle_driver - INFO - Tracking sequential pairs of idfeature files
2023-12-05 20:01:30,185 - pyflextrkr.tracksingle_driver - INFO - Total number of files to process: 0
2023-12-05 20:01:30,185 - pyflextrkr.tracksingle_driver - INFO - Done with tracking sequential pairs of idfeature files
2023-12-05 20:01:30,185 - pyflextrkr.gettracks - INFO - Tracking features sequentially from single track files
2023-12-05 20:01:30,186 - pyflextrkr.gettracks - INFO - Total number of files to process: 0
Traceback (most recent call last):
File "/nfs/tcdynasty/lucy/run_generic_tracking.py", line 61, in
tracknumbers_filename = gettracknumbers(config)
^^^^^^^^^^^^^^^^^^^^^^^
File "/nfs/knight/mamba_aug23/envs/aug23_env/lib/python3.11/site-packages/pyflextrkr/gettracks.py", line 74, in gettracknumbers
logger.debug(f"files[0]: {files[0]}")
~~~~~^^^
IndexError: list index out of range

I'll also attach my config file:

---

ERA5 vorticity anomaly tracking configuration file

Identify features to track

run_idfeature: True

Track single consecutive feature files

run_tracksingle: True

Run tracking for all files

run_gettracks: True

Calculate feature statistics

run_trackstats: True

Link merge/split tracks

run_mergesplit: True

Map tracking to pixel files

run_mapfeature: True

Start/end date and time

startdate: '20000101_000004'
enddate: '20000101_000008'

Parallel processing set up

run_parallel: 1 (local cluster), 2 (Dask MPI)

run_parallel: 1
nprocesses: 32 # Number of processors to use if run_parallel=1

databasename: 'cm1out_'
#databasename: ERA5_SFvortPV_

Specify date/time string format in the file name

E.g., radar_20181101.011503.nc --> yyyymodd.hhmmss

E.g., wrfout_2018-11-01_01:15:00 --> yyyy-mo-dd_hh:mm:ss

time_format: 'yyyymodd_hhmmss'

Input files directory

clouddata_path: '/nfs/tcdynasty/lucy/cm1/'

Working directory for the tracking data

root_path: '/nfs/tcdynasty/lucy/cm1_tracking/'

root_path: '/pscratch/sd/j/jmarquis/ERA5_waccem/Bandpassed/'

Working sub-directory names

tracking_path_name: 'vtracking'
stats_path_name: 'vortstats'
pixel_path_name: 'vortracking'

Specify types of feature being tracked

This adds additional feature-specific statistics to be computed

feature_type: 'generic'

Specify data structure

datatimeresolution: 1/3600 # hours
pixel_radius: .015625 # km
x_dimname: 'ni'
y_dimname: 'nj'
time_dimname: 'time'
time_coordname: 'time'
x_coordname: 'x'
y_coordname: 'y'
field_varname: 'rel_vort'

Feature detection parameters

label_method: 'skimage.watershed'

peak_local_max params:

plm_min_distance: 15 # min_distance - distance buffer between maxima; num grid points
plm_exclude_border: 5 # exclude_border - distance buffer between maxima and the domain sides; num grid points
plm_threshold_abs: 0 # threshold_abs - minimum magnitude of PSI' required to define a maxima

watershed params:

cont_thresh: 0.00002 # PSI' contour defining outermost of flood-filled object area
compa: 0 #"compactness factor" - (how much you'll let a flood fill spread into a neighbor's domain. Zero or < 100 seemed ok.)

field_thresh: [1.6, 1000] # variable thresholds

min_size: .1 # Min area to define a feature (km^2)
R_earth: 6378.0 # Earth radius (km)

Tracking parameters

timegap: 1/3600 # hour
othresh: 0.3 # overlap percentage threshold
maxnclouds: 100 # Maximum number of features in one snapshot
nmaxlinks: 10 # Maximum number of overlaps that any single feature can be linked to
duration_range: [6, 800] # A vector [minlength,maxlength] to specify the duration range for the tracks

Flag to remove short-lived tracks [< min(duration_range)] that are not mergers/splits with other tracks

0:keep all tracks; 1:remove short tracks

remove_shorttracks: 1

Set this flag to 1 to write a dense (2D) trackstats netCDF file

Note that for datasets with lots of tracks, the memory consumption could be very large

trackstats_dense_netcdf: 1

Minimum time difference threshold to match track stats with cloudid pixel files

match_pixel_dt_thresh: 60.0 # seconds

Link merge/split parameters to main tracks

maintrack_area_thresh: .1 # [km^2] Main track area threshold
maintrack_lifetime_thresh: 60/3600 # [hour] Main track duration threshold
split_duration: 30/3600 # [hour] Split tracks <= this length is linked to the main tracks
merge_duration: 30/3600 # [hour] Merge tracks <= this length is linked to the main tracks

Define tracked feature variable names

feature_varname: 'feature_number'
nfeature_varname: 'nfeatures'
featuresize_varname: 'npix_feature'

Track statistics output file dimension names

tracks_dimname: 'tracks'
times_dimname: 'times'
fillval: -9999

Output file base names

finalstats_filebase: 'trackstats_final_'
pixeltracking_filebase: 'vort_tracks_'

List of variable names to pass from input to tracking output data

pass_varname:

• 'rel_vort'

All the files are housed in the /nfs/tcdynasty/lucy/cm1/ directory, but it seems to me that they're not being found by the code. Any assistance is much appreciated!

[feng045]

Based on your config, the code would be searching for input files like this:
/nfs/tcdynasty/lucy/cm1/cm1out_yyyymodd_hhmmss.nc

And the files date/time must be within this range:
startdate: '20000101_000004'
enddate: '20000101_000008'

You should check to make sure that matches your input files.

[ealucy]

Yes, they match. The files are titled like this: 'cm1out_20000101_000004.nc'. Curious!

[feng045]

I just realized that your startdate and enddate only differ by 4 seconds. The code calculates the date/times from your filenames (hence the specified datetime format 'yyyymodd_hhmmss' in the config), and then only keeps those that fall within your specified startdate and enddate for processing.

What does your file names look like? Can you put the list of your full file names here?

[ealucy]

Yes, that is correct. This is the file list:
cm1out_20000101_000004.nc
cm1out_20000101_000005.nc
cm1out_20000101_000006.nc
cm1out_20000101_000007.nc
cm1out_20000101_000008.nc
There are only these five files, as the entire dataset is not housed locally. I was hoping to test the tracker on these few to get an idea of how it works before attempting to do so on the entire dataset.

[feng045]

I think I may know why. The function in PyFLEXTRKR converting input file datetimes did not use the digits down to seconds precision. See the code at this line.

You can try making a larger datetime window that include all the files you have, e.g.,
startdate: '20000101_000000'
enddate: '20000101_001000'