Index out of bound error

[VISION0310]

Hello author's and all contributors,

I am running flextrkr for mcs tracking with merged Tb + precipitation (GPM) data as input data.
and getting following errror -

2024-07-23 09:59:07,312 - pyflextrkr.trackstats_driver - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//stats/trackstats_20231214.0000_20231215.2300.nc
2024-07-23 09:59:07,394 - pyflextrkr.trackstats_driver - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//stats/trackstats_sparse_20231214.0000_20231215.2300.nc
2024-07-23 09:59:07,397 - pyflextrkr.identifymcs - INFO - Identifying MCS based on Tb statistics
2024-07-23 09:59:07,431 - pyflextrkr.identifymcs - INFO - Number of tracks to process: 1342
2024-07-23 09:59:07,786 - pyflextrkr.identifymcs - INFO - Number of Tb defined MCS: 36
2024-07-23 09:59:08,057 - pyflextrkr.identifymcs - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//stats/mcs_tracks_20231214.0000_20231215.2300.nc
2024-07-23 09:59:08,058 - pyflextrkr.matchtbpf_driver - INFO - Matching Tb tracked MCS with precipitation to calculate PF statistics
2024-07-23 09:59:08,200 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231214_163000.nc
2024-07-23 09:59:08,200 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231214_033000.nc
2024-07-23 09:59:08,201 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231214_123000.nc
2024-07-23 09:59:08,201 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231215_203000.nc
2024-07-23 09:59:08,201 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231214_213000.nc
2024-07-23 09:59:08,202 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231214_043000.nc
2024-07-23 09:59:08,202 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231215_103000.nc
2024-07-23 09:59:08,202 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231214_173000.nc
2024-07-23 09:59:08,340 - pyflextrkr.matchtbpf_func - INFO - /home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231215_013000.nc
2024-07-23 09:59:08,343 - distributed.worker - WARNING - Compute Failed
Key: matchtbpf_singlefile-f3726840-e511-45fe-970c-0fbf33dfac45
State: executing
Function: execute_task
args: ((<function matchtbpf_singlefile at 0x7f55be8d3b00>, '/home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data//tracking/cloudid_20231214_213000.nc', array([ 1, 24, 3, 40, 9, 5, 2, 12, 7, 13, 8, 4, 18, 6],
dtype=int32), array([[-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999],
[ 62, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,
-9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999, -9999,

kwargs: {}
Exception: "IndexError('index 37 is out of bounds for axis 1 with size 37')"

Traceback (most recent call last):
File "/home/PANL/piyush/PyFLEXTRKR/config/../runscripts/run_mcs_tbpf.py", line 75, in
pfstats_filename = match_tbpf_tracks(config)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/PANL/piyush/PyFLEXTRKR/pyflextrkr/matchtbpf_driver.py", line 144, in match_tbpf_tracks
final_result = dask.compute(*results)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/PANL/piyush/anaconda3/envs/flextrkr/lib/python3.11/site-packages/dask/base.py", line 662, in compute
results = schedule(dsk, keys, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/PANL/piyush/PyFLEXTRKR/pyflextrkr/matchtbpf_func.py", line 218, in matchtbpf_singlefile
(sublandmask[ipfy, ipfx] >= np.min(landfrac_thresh)) & \

IndexError: index 37 is out of bounds for axis 1 with size 37

#############################################
Configuration file as follows
##############################################

Global GPM based MCS tracking configuration file

Tracking uses collocated Tb + Precipitation

Processing steps:

Maybe worth nesting these in sets of dictionaries

run_idfeature : True
run_tracksingle : True
run_gettracks : True
run_trackstats : True
run_identifymcs : True
run_matchpf : True
run_robustmcs : True
run_mapfeature : True
run_speed: True

Parallel processing set up

run_parallel: 1 (local cluster), 2 (Dask MPI)

run_parallel: 1
nprocesses : 8 # Number of processors to use if run_parallel=1
dask_tmp_dir: '/tmp' # Dask temporary directory if run_parallel=1
timeout: 360 # [seconds] Dask timeout limit

Start/end date and time

startdate: '20231214.0000'
enddate: '20231215.2300'

Specify tracking input data date/time string format

This is the preprocessed file that contains Tb & rainrate

E.g., databasename20181101.011503.nc --> yyyymodd.hhmmss

E.g., databasename2018-11-01_01:15:00 --> yyyy-mo-dd_hh:mm:ss

time_format: 'yyyymoddhh'
databasename: 'merg_' # Note: include all strings before the time (including "_", ".")

Input files directory

clouddata_path: '/home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data/input/'

Working directory for the tracking data

root_path: '/home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data/'

Working sub-directory names

tracking_path_name: 'tracking'
stats_path_name: 'stats'
pixel_path_name: 'mcstracking'

Land mask file (optional, leave it an empty string if not available)

landmask_filename: '/home/PANL/piyush/PHD/DATA/MCS_Tracking_Input_Data/tracking_data/input/IMERG_landmask_saag.nc'
landmask_varname: 'landseamask'
landmask_x_dimname: 'lon'
landmask_y_dimname: 'lat'
landmask_x_coordname: 'lon'
landmask_y_coordname: 'lat'
landfrac_thresh: [0,90] # Define the range of fraction for land (depends on what value is land the landmask file)

Specific to GPM Tb+IMERG combined dataset

pixel_radius: 10.0 # [km] Spatial resolution of the input data
datatimeresolution: 1.0 # [hour] Temporal resolution of the input data

Variable names in the input data

tb_varname: 'Tb'
pcp_varname: 'precipitation'
clouddatasource: 'gpmirimerg'
time_dimname: 'time'
x_dimname: 'lon'
y_dimname: 'lat'
time_coordname: 'time'
x_coordname: 'lon'
y_coordname: 'lat'
idclouds_hourly: 1 # 0: No (use all available time frames in input files), 1: Yes (only use hourly time frame)
idclouds_minute: 30 # 0 or 30 min, which minute mark to use the Tb data
idclouds_dt_thresh: 5 # [minute], time difference allowed between actual data and idclouds_minute

Specify types of feature being tracked

This adds additional feature-specific statistics to be computed

feature_type: 'tb_pf'

Cloud identification parameters

mincoldcorepix: 4 # Minimum number of pixels for the cold core
smoothwindowdimensions: 10 # Dimension of the Box2DKernel filter on Tb.
medfiltsize: 5 # Window size to perform medfilt2d to fill missing Tb pixels, must be an odd number
geolimits: [-60, -360, 60, 360] # 4-element array to subset domain boundaries [lat_min, lon_min, lat_max, lon_max]
area_thresh: 800 # [km^2] Minimum area to define a cloud
miss_thresh: 0.4 # Missing data fraction threshold. If missing data exceeds this, the time frame will be omitted.
cloudtb_core: 225.0 # [K]
cloudtb_cold: 241.0 # [K]
cloudtb_warm: 261.0 # [K]
cloudtb_cloud: 261.0 # [K]
absolutetb_threshs: [160, 330] # K [min, max] absolute Tb range allowed.
warmanvilexpansion: 0 # Not working yet, set this to 0 for now
cloudidmethod: 'label_grow'

Specific parameters to link cloud objects using PF

linkpf: 1 # Set to 1 to turn on linkpf option; default: 0
pf_smooth_window: 5 # Smoothing window for identifying PF
pf_dbz_thresh: 3 # [dBZ] for reflectivity, or [mm/h] for rainrate
pf_link_area_thresh: 648.0 # [km^2]

Tracking parameters

othresh: 0.5 # overlap fraction threshold. Clouds that overlap more than this between times are tracked.
timegap: 3.1 # [hour] If missing data duration longer than this, tracking restarts
nmaxlinks: 50 # Maximum number of clouds that any single cloud can be linked to
maxnclouds: 3000 # Maximum number of clouds in one snapshot
duration_range: [2, 400] # A vector [minlength,maxlength] to specify the duration range for the tracks

Flag to remove short-lived tracks [< min(duration_range)] that are not mergers/splits with other tracks

0:keep all tracks; 1:remove short tracks

remove_shorttracks: 1

Set this flag to 1 to write a dense (2D) trackstats netCDF file

Note that for datasets with lots of tracks, the memory consumption could be large

trackstats_dense_netcdf: 1

Minimum time difference threshold to match track stats with cloudid files

match_pixel_dt_thresh: 60.0 # seconds

MCS Tb parameters

mcs_tb_area_thresh: 40000 # [km^2] Tb area threshold
mcs_tb_duration_thresh: 4 # [hour] Tb minimum length of a mcs
mcs_tb_split_duration: 12 # [hour] Tb tracks smaller or equal to this length will be included with the MCS splits from
mcs_tb_merge_duration: 12 # [hour] Tb tracks smaller or equal to this length will be included with the MCS merges into
mcs_tb_gap: 1 # [unitless] Allowable temporal gap in Tb data for MCS area threshold

MCS PF parameters

mcs_pf_majoraxis_thresh: 100 # [km] MCS PF major axis length lower limit
max_pf_majoraxis_thresh: 1800 # [km] MCS PF major axis length upper limit
mcs_pf_durationthresh: 4 # [hour] PF minimum length of mcs
mcs_pf_majoraxis_for_lifetime: 20 # [km] Minimum PF size to count PF lifetime
mcs_pf_gap: 1 # [unitless] Allowable temporal gap in PF data for MCS characteristics

Specify rain rate parameters

pf_rr_thres: 2.0 # Rain rate threshold [mm/hr]
nmaxpf: 3 # Maximum number of precipitation features that can be within a cloud feature
nmaxcore: 20 # Maximum number of convective cores that can be within a cloud feature
pcp_thresh: 1.0 # Pixels with hourly precipitation larger than this will be labeled with track number
heavy_rainrate_thresh: 10.0 # Heavy rain rate threshold [mm/hr]

MCS PF parameter coefficients [intercept, slope]

These parameters are derived with pf_rr_thres: 2 mm/h

coefs_pf_area: [1962, 0] # 1% [changed slope to 0: independent of lifetime]

coefs_pf_area: [2119.02, 61.143] # 3%

coefs_pf_area: [2874.05, 89.825] # 5% [recommended]

coefs_pf_area: [4160.82, 93.077] # 7%

coefs_pf_area: [4988.15, 138.172] # 10%

coefs_pf_rr: [2.72873, 0.0008317] # 1%

coefs_pf_rr: [2.81982, 0.0135463] # 3%

coefs_pf_rr: [3.01657, 0.0144461] # 5% [recommended]

coefs_pf_rr: [3.14895, 0.0150174] # 7%

coefs_pf_rr: [3.34859, 0.0172043] # 10%

coefs_pf_skew: [0.036384, 0.0022199] # 1%

coefs_pf_skew: [0.072809, 0.0104444] # 3%

coefs_pf_skew: [0.194462, 0.0100072] # 5% [recommended]

coefs_pf_skew: [0.256639, 0.0106527] # 7%

coefs_pf_skew: [0.376142, 0.0095545] # 10%

coefs_pf_heavyratio: [0.750260, 0.4133300] # 5%

coefs_pf_heavyratio: [3.419024, 0.4387090] # 10% [recommended]

coefs_pf_heavyratio: [4.753215, 0.4886454] # 15%

coefs_pf_heavyratio: [4.592209, 0.6107371] # 20%

coefs_pf_heavyratio: [8.389616, 0.5079337] # 25%

Define tracked feature variable names

feature_varname: 'feature_number'
nfeature_varname: 'nfeatures'
featuresize_varname: 'npix_feature'

Track statistics output file dimension names

tracks_dimname: 'tracks'
times_dimname: 'times'
pf_dimname: 'nmaxpf'
fillval: -9999

MCS track stats file base names

mcstbstats_filebase: 'mcs_tracks_'
mcspfstats_filebase: 'mcs_tracks_pf_'
mcsrobust_filebase: 'mcs_tracks_robust_'
pixeltracking_filebase: 'mcstrack_'
mcsfinal_filebase: 'mcs_tracks_final_'

Feature movement speed parameters

lag_for_speed: 1 # [unitless] lag intervals between tracked features to calculate movement
track_number_for_speed: "pcptracknumber"
track_field_for_speed: 'precipitation'
min_size_thresh_for_speed: 20 # [km] Min PF major axis length to calculate movement
max_speed_thresh: 50 # [m/s] Speeds larger than this will be replaced by temporal filter
~

Kindly guide me to resolve this issue .
Regards Piyush