WARNING:tensorflow:5 out of the last 13 calls

[talhaanwarch]

Hi, i am getting this warning when i used GridSearchCv. otherwise i did not. ANy idea

WARNING:tensorflow:5 out of the last 13 calls to <function Model.make_predict_function..predict_function at 0x7feb67b54940> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.

[adriangb]

Hi Talha.

I've see this as well. I believe it is because grid search passes in inputs of different lengths, which forces TensorFlow to re-do some work internally that is normally cached (re-compile parts of the execution graph or something like that). So it's not really an issue with SciKeras per se. That said, I'm happy to look into it some more if you can provide a minimal working example.

Are you seeing any issues aside from the warning?

[talhaanwarch]

Here is the code

import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], enable=True)

from sklearn.model_selection import GroupKFold,GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import precision_recall_fscore_support,accuracy_score
from sklearn.pipeline import Pipeline
import numpy as np
from tensorflow import keras
from tensorflow.keras.models import Sequential,Model
from tensorflow.keras.layers import Dense,LSTM,Flatten
from tensorflow.keras import Input
from tensorflow.keras.backend import clear_session 
from tensorflow.keras.utils import plot_model
from scikeras.wrappers import KerasClassifier
from tensorflow.keras.optimizers import Adam

def ourmodel(numberOfLSTMcells=2,fch=64,fc1h=32,n_timesteps_in=100,opt_lr=1e-3,n_features=50):
    clear_session()
    inp =Input(shape=(n_timesteps_in, n_features))
    lstm= LSTM(numberOfLSTMcells,return_sequences=True, return_state=False) (inp)
    flatten=Flatten()(lstm)
    fc=Dense(fch,activation='relu')(flatten)
    fc=Dense(fc1h,activation='relu')(fc)
    out=Dense(1,activation='sigmoid')(fc)

    model = Model(inputs=inp, outputs=out)
    model.compile(loss='binary_crossentropy', optimizer=Adam(lr=opt_lr), 
                  metrics=['accuracy'])
    return model


data=np.random.rand(500, 100, 50)
label=np.concatenate((np.zeros(250),np.ones(250)))
print(data.shape,label.shape)


from sklearn.base import TransformerMixin,BaseEstimator
from sklearn.preprocessing import StandardScaler

class StandardScaler3D(BaseEstimator,TransformerMixin):

    def __init__(self):
        self.scaler = StandardScaler()

    def fit(self,X,y=None):
        self.scaler.fit(X.reshape(-1, X.shape[2]))
        return self

    def transform(self,X):
        return self.scaler.transform(X.reshape( -1,X.shape[2])).reshape(X.shape)

def gridsearch(data_array,label_array,channels=None):
    params = {
        "classifier__model__numberOfLSTMcells": [2, 5],
        "classifier__model__fch": [8,16],
        "classifier__model__fc1h": [8,16],
        "classifier__model__opt_lr":[1e-3,1e-4]
    }
    if channels:
        data_array=data_array[:,:,channels]
    
    gkf=GroupKFold(n_splits=5)
    sk_clf = KerasClassifier(ourmodel,epochs=20,batch_size=100,verbose=False)
    pipe_clf=Pipeline([('scaler',StandardScaler3D()),('classifier',sk_clf)])
    gs = GridSearchCV(pipe_clf, params, scoring='f1_macro', verbose=True)
    gs.fit(data_array, label_array)
    return gs.best_score_, gs.best_params_

gridsearch(data,label)