[21pt] Test adjustment of SRCs using the ML-estimated missing flow volumes

[CarsonPruitt-NOAA]

The data schema according to Arash is as follows:

It contains all the below channel properties computed for 2 flow conditions 1- bankfull 2- in-channel. Our experiments with branch 0 showed the in-channel estimates were more apt to account for missing volume and is stored in inchan_ChannelVolume (m3). Also, the ML derived roughness can be used and we will work to refine that with ML derived bathymetric information.

---

hf_id: reference fabric id (can be interchanged 99% of the time with NHDPlusV2 COMID)
hf_source: the source geospatial fabric used
owp_y_bf: Machine Learned depth at bankfull
owp_y_inchan: Machine Learned depth at in channel
owp_tw_bf: Machine Learned topwidth at bankfull
owp_tw_inchan: Machine Learned topwidth at in channel
owp_ahg_c: Machine Learned AHG c coefficient
owp_ahg_f: Machine Learned AHG f coefficient
owp_ahg_a: Machine Learned AHG a coefficient
owp_ahg_b: Machine Learned AHG b coefficient
owp_ahg_k: Machine Learned AHG k coefficient
owp_ahg_m: Machine Learned AHG m coefficient
owp_dingman_r: Machine Learned channel shape r Dingman parameter
LENGTHKM: the reference fabric length
roughness: an approximation of roughness learned from CFIM SRCs, the NHDPlus Network and USGS rating curves.
inchan_ChannelArea (m2): The cross sectional area at in channel (using AHGestimation)
inchan_ChannelPerimeter (m): The cross sectional perimeter at in channel
bf_ChannelArea (m2): The cross sectional area at bankfull (using AHGestimation)
bf_ChannelPerimeter (m): The cross sectional perimeter at bankfull
inchan_ChannelVolume (m3): The channel volume at in channel
inchan_ChannelBedArea (m2): The channel bed area at in channel
bf_ChannelVolume (m3): The channel volume at bankfull
bf_ChannelBedArea (m2): The channel bed area at bankfull

More info:

inchan_ChannelVolume (m3) = LENGTHKM * 1000 * inchan_ChannelArea (m2)
In case FIM hydrofabric is going to be used instead of reference fabric LENGTHKM need to come from FIM hydrofabric to recompute inchan_ChannelVolume (m3).

[CarsonPruitt-NOAA]

As a validation measure, I saved a flow file for an event on 4/4/2024 in Wheeling, WV. The flow file can be used to create a candidate FIM to be validated against the drone aerial photo. The files are stored on our EFS drive here: /fim-home/wheeling_bathy_test.

[CarsonPruitt-NOAA]

Arash started a notes document here.

[arashmodrad]

An updated version of the dataset is now at s3://lynker-spatial/hydrofabric/v20.1/3D/ml-outputs/ml_outputs_v1.01.parquet
It has an updated metadata that can be accessed using this example script in python

`
import pandas as pd
import pyarrow.parquet as pq

table = pq.read_table('direcotry/ml_outputs_v1.01.parquet')

Read as pandas dataframe

ml_outputs = table.to_pandas()

Read metadata

schema = table.schema
for field in schema.names:
print("Column: {0}".format(field))
print("Unit: {0}".format(schema.field(field).metadata[b'unit'].decode('utf-8')))
print("Description: {0} \n".format(schema.field(field).metadata[b'description'].decode('utf-8')))
`

[hhs732]

[hhs732]

👍

[CarsonPruitt-NOAA]

I saved a flow file with some drone photos in our shared EFS folder. Can you run a test to see if the new estimated bathymetry data gets us closer to the floods shown in the pictures?

/fim-home/wheeling_bathy_test/

[hhs732]

Wheeling Island, Pittsburgh
New inundation map with bathy data added

[hhs732]

[hhs732]

Example of SRC for a nwm stream at the location after applying AI-based bathy data

[hhs732]

hydroTable_1542000004_analysis.xlsx

[CarsonPruitt-NOAA]

So, based off the drone photos, it looks like the ML-bathymetry is under-estimating FIM in this particular case.

[CarsonPruitt-NOAA]

Thank you for doing this analysis. Once you have run the full alpha domain, I would like to see those plots as well, but these are exactly what I was expecting. The right-shift of the SRC is lowering the CSI and increasing PND from an increase in false negatives (underprediction). It seems to also have a marginally greater effect on the 100-yr flood, which also aligns with our expectations.

In order to avoid possible interference with other people branching off dev with these lower skill scores, I'm going to recommend that we turn the ML-bathy "off" in a default fim_pipeline until you're able to optimize the roughness. @RyanSpies-NOAA I'd like to know your opinion on this also.

[arashmodrad]

@hhs732 Thanks for the great figures and analysis! I noticed that in my calculations and the ones we discussed for estimation of Bed Area we can directly use the flat area calculated during HAND and hydroprop generation by taking the zero stage BedArea Value as the flat area instead of previous rough estimation of the flat area (reach length * average cross sectional area). I kept the rough estimation for cases that with null zero stage. This also fixes the jump we saw in hydraulic radius calculations. Here is how I implemented it for mine

flow_cond = 'inchan' 
def get_flat_bedarea(group):
    # Find the zero stage
    stage_0_row = group[group['stage'] == 0]
    if not stage_0_row.empty:
        flat_bedarea_value = stage_0_row['BedArea (m2)'].iloc[0]
        group['Flat_BedArea (m2)'] = flat_bedarea_value
    # If null then do a rough estimation of flat part
    else:
        group['Flat_BedArea (m2)'] = (group['LENGTHKM']  * 1000 * group['owp_tw_'+flow_cond])
    return group

# Apply to each reach group to obtain flat bead area
hydroprop_table = hydroprop_table.groupby('feature_id').apply(get_flat_bedarea)
# Subtract the flat bed area from calculations 
hydroprop_table['UpdateBedArea (m2)'] = hydroprop_table['BedArea (m2)'] + hydroprop_table['ChannelBedArea (m2)'] - hydroprop_table['Flat_BedArea (m2)']

[hhs732]

Alpha domain metrics:
all0n: Calib 0n; Bathy 0n; steam_order = 7; mannN = 0.06, 0.12
all0ff: Calib 0ff; Bathy 0ff; mannN = 0.06, 0.12
Cal0ffS04: Calib 0ff, Bathy 0n; steam_order = 4; mannN = 0.06, 0.12
Cal0ffS07: Calib 0ff, Bathy 0n; steam_order = 7; mannN = 0.06, 0.12
Cal0ffS07MAI: Calib 0ff, Bathy 0n; steam_order = 7; mannN = ai-driven
Cal0ffB0ffMAI: Calib 0ff, Bathy 0ff; mannN = ai-driven

[hhs732]

[hhs732]

[hhs732]

[hhs732]

[hhs732]

[hhs732]

https://docs.google.com/presentation/d/1yAPTTEh1vM9HOcDh34tLlcHoq-5tu5N2eW0vczfV_r4/edit#slide=id.g2ef93aa8e0b_0_72