Loading saved model fails due to missing 'slice' function

[IanSaucy]

Hey!

I am trying to load the pre-trained model provided via DropBox to adapt it a bit but it fails to load. Initially it was an issue with the custom activation function but I fixed that by passing it as a custom_object to load_model in predit.py, see here for how I went about it.

But now I am running into a new problem, I get the following error(emphasis on the last line):

Traceback (most recent call last):
  File "05_prediction/src/main.py", line 22, in <module>
    ], models_path=models_path)
  File "/Users/user/Documents/spark/CS501-Liberator-Project/bbz-segment/05_prediction/src/predict.py", line 115, in load
    loaded[name] = c(name, **kwargs)
  File "/Users/user/Documents/spark/CS501-Liberator-Project/bbz-segment/05_prediction/src/predict.py", line 155, in __init__
    custom_objects={'swish': tf.nn.swish, 'FixedDropout': FixedDropout})
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/save.py", line 146, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 168, in load_model_from_hdf5
    custom_objects=custom_objects)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/model_config.py", line 55, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/layers/serialization.py", line 106, in deserialize
    printable_module_name='layer')
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/generic_utils.py", line 303, in deserialize_keras_object
    list(custom_objects.items())))
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 937, in from_config
    config, custom_objects)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 1903, in reconstruct_from_config
    process_node(layer, node_data)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py", line 1851, in process_node
    output_tensors = layer(input_tensors, **kwargs)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 773, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/tensorflow_core/python/keras/layers/core.py", line 846, in call
    result = self.function(inputs, **kwargs)
  File "/home/sc.uni-leipzig.de/bo140rasi/.local/lib/python3.7/site-packages/keras/utils/multi_gpu_utils.py", line 198, in get_slice
AttributeError: module 'tensorflow.python.keras.backend' has no attribute 'slice'

Which I can't seem to track down why it is happening. I presume it's stemming from something it is attempting to load for the pre-trained model in the HDF5 files. It seems that it's trying to find the slice function but is unable to find that function, but I'm not even able to track down where the function call is originating from! I tried updating the base_dir in the run.json file in the model directory models\v3\sep\1\run.json but this still does not fix the issue.

I am using the following:

• Macos and Linux/Ubuntu(tried both)
• Python 3.7.7
• Tensorflow 1.15.2
• Keras 2.3.1
• Segmentation-mdoels 1.0.1

Any help would be super appreciated!

[poke1024]

Hm, I've seen this before, it's some kind of pickle issue in Keras, it has also been reported (keras-team/keras-contrib#488) and you could try the fix described there (keras-team/keras-contrib#488 (comment)), however I'm not sure if this is the only issue.

Have you tried enabling the alternative loading code at https://github.com/IanSaucy/bbz-segment/blob/39be9165e5bb3866655a354cd1620c6b564c2469/05_prediction/src/predict.py#L143 ?

For my production setting for running inference, I use TensorFlow 2.1.2 with Keras 2.3.1 and haven't seen any problems, but I understand that you want to train and that needs TensorFlow 1 indeed. DL libraries are quite a mess right now.

EDIT Just rechecked that these models were trained on TensorFlow 1.15.2 with Keras 2.3.1 so we should get this to work. My best guess right now is the alternative loading code linked above.

[IanSaucy]

Thanks for getting back to me! I had seen those two issues but the libraries on both of the systems I tested didn't seem to reflect the code talked about in those two issues.

I did try enabling the alternative loader which solved the slice issue but now I'm running into the follow:

ValueError: You are trying to load a weight file containing 1 layers into a model with 208 layers.

Most of the solutions online don't really apply to this use case either. So I'm a bit stumped again. I did double check that I'm using those specific versions of Keras and TensorFlow. The only other thing I'm going to try is re-downloading the model just in case something got corrupted.

On the point of using the alternate loader, wouldn't changing the activation function from swish -> softmax affect the loaded model?

Thanks so much for taking the time to help troubleshoot this with me. It's a breath of fresh air to have such high quality code from a scientific paper.

Instructions for updating:
If using Keras pass *_constraint arguments to layers.
2020-10-28 13:40:49.210429: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-10-28 13:40:49.223306: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9d7a7895a0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-28 13:40:49.223360: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
loading models:   0%|                                                                                                                                                                                             | 0/6 [00:06<?, ?it/s]
Traceback (most recent call last):
  File "05_prediction/src/main.py", line 22, in <module>
    ], models_path=models_path)
  File "/Users/user/Documents/spark/CS501-Liberator-Project/bbz-segment/05_prediction/src/predict.py", line 115, in load
    loaded[name] = c(name, **kwargs)
  File "/Users/user/Documents/spark/CS501-Liberator-Project/bbz-segment/05_prediction/src/predict.py", line 149, in __init__
    model.load_weights(str(network_path / "model.h5"))
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/keras/engine/saving.py", line 492, in load_wrapper
    return load_function(*args, **kwargs)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/keras/engine/network.py", line 1230, in load_weights
    f, self.layers, reshape=reshape)
  File "/Users/user/.local/share/virtualenvs/bbz-segment-czho4_QW/lib/python3.7/site-packages/keras/engine/saving.py", line 1209, in load_weights_from_hdf5_group
    str(len(filtered_layers)) + ' layers.')
ValueError: You are trying to load a weight file containing 1 layers into a model with 208 layers.

[IanSaucy]

I think I figured it out! I was playing around with Origami and noticed that model loading worked fine there so I did some cross comparison. Basically it came down to using keras directly kerasmodels.load_model in place of tensorflow.keras.models.load_model. This also fixed the issue of custom objects.

here is how it looks now.

I'm currently running some test images through the model right now(which take a bit since I don't have access to a GPU locally).

Thanks so much for your help, if you want me to open a PR to fix this issue let me know. I'd also be happy to throw in an updated requirements.txt or pipFile.

[poke1024]

This is great! Having an PR for this would be really nice.

[IanSaucy]

Sorry for the bit of a delay but I've gone ahead and created PR that should resolve this issue. With a few fresh cups of coffee I noticed that issues I had where only due to conflict of package versions. Pinning packages as is done in #4 seems to resolve the issue. Also, a new bug in tensorflows dependencies caused an issue and thus was pinned as well in the PR. Here is an overview of that issue.
tensorflow/tensorflow#44467

Thanks for all you help, I'll leave future discussions to the PR!