Keras siamese model history is empty

[RotemTon]

Body

I am making a siamese neural network with triplet loss using keras, and have encountered an odd problem. I tried saving my history twice: once in a callback (saved as a dictionary), and once after training with model.fit(), using pickle (saved as a history object). In both instances, the code runs smoothly, but afterwards the saved files are only several hundred bytes and when I plot them with matplotlib, the plot is empty.

Here's the callback I passed to model.fit():
`
class MetricsCallback(tensorflow.keras.callbacks.Callback):
def init(self, train_data, validation_data, embedding_model, folder_path, file_name):
super(MetricsCallback, self).init()
self.validation_data = validation_data
self.train_data = train_data
self.history={}
self.embedding_model=embedding_model
self.file_name = file_name
self.folder_path = folder_path
def calc_metrics(self ,dataset):
#for accuracy
y_true_acc=[]
y_pred=[]

#for auc-roc and eer
y_true=[]
y_scores=[]

for triplet in dataset: #dataset is train_data or validation_data
  anchor, positive, negative = triplet #get the anchor, posotive and negative tensors from each triplet tuple
  a_emb=embedding_model(np.expand_dims(anchor, axis=0)) #get the embeddings for each
  p_emb=embedding_model(np.expand_dims(positive, axis=0))
  n_emb=embedding_model(np.expand_dims(negative, axis=0))

  dist_p = np.linalg.norm(a_emb - p_emb) #calculate distances
  dist_n = np.linalg.norm(a_emb - n_emb)
  #for triplet acc:
  y_true_acc. append(1) #the true label is always 1
  if dist_p +alpha<dist_n : #accurate
    y_pred.append(1)
  else:
    y_pred.append(0)

  #for auc-roc and eer:
  y_true.append(0) #for a-p
  y_true.append(1) #for a-n

  y_scores.append(1-dist_p)#if it's small like we want, it gets a high score
  y_scores.append(dist_n)# if it's big like we want, it gets a high score

#compute acc:
accuracy = accuracy_score(y_true_acc, y_pred)
#compute auc-roc and eer:
fpr, tpr, thresholds = roc_curve(y_true, y_scores)
auc_roc = auc(fpr, tpr)
eer = 1 - max(tpr - fpr)

return accuracy, auc_roc, eer, fpr, tpr

def on_epoch_end(self, epoch, logs=None):
# Compute metrics on train data
loss_train = logs.get('loss')
accuracy_train , auc_roc_train, eer_train, fpr_train, tpr_train= self.calc_metrics(self.train_data)

self.history['train_loss'] = loss_train
self.history['train_accuracy'] = accuracy_train
self.history['train_auc_roc'] = auc_roc_train
self.history['train_eer'] = eer_train
self.history['train_fpr'] = fpr_train
self.history['train_tpr'] = tpr_train

# Log the metrics
print(f'Epoch {epoch + 1} - Train Loss: {loss_train}, Accuracy: {accuracy_train}, AUC-ROC: {auc_roc_train}, EER: {eer_train}')

# Compute metrics on validation data
loss_val = logs.get('val_loss')

accuracy_val, auc_roc_val, eer_val, fpr_val, tpr_val= self.calc_metrics(self.validation_data)

self.history['val_loss'] = loss_val
self.history['val_accuracy'] = accuracy_val
self.history['val_auc_roc'] = auc_roc_val
self.history['val_eer'] = eer_val
self.history['val_fpr'] = fpr_val
self.history['val_tpr'] = tpr_val

# Log the metrics
print(f'Epoch {epoch + 1} - Validation Loss: {loss_val},  Accuracy: {accuracy_val}, AUC-ROC: {auc_roc_val}, EER: {eer_val}')

# Save the metric history to Google Drive
epoch_number = epoch + 1  # Epoch starts from 0, so we add 1 to get the actual epoch number
file_name_with_epoch = f"{self.file_name}_epoch_{epoch_number}.pkl"
file_path = os.path.join(self.folder_path, file_name_with_epoch)
with open(file_path, 'wb') as file_pi:
    pickle.dump(self.history, file_pi)
print('saved metrics')

metrics_callback = MetricsCallback(train_set, val_set,simese_model_1,folder_path, file_name)
I tried just saving the history after training like this, but this too is empty:
with open('historyPath', 'wb') as file_pi: pickle.dump(history, file_pi)
I suspect this is because I have a custom loss (triplet loss) and custom metrics in the callback. However, this is the very reason I tried saving it in the custom callback.
`
Does anyone know why this is happening and how to fix it? Thanks.

Guidelines

• I have read and understood this category's guidelines before making this post.

[markolalovic]

Hi, have you tried running simply

history = model.fit()

Also check the recorded metrics logs in history.history.

[RotemTon]

Hi, the history is still empty after running model.fit().
After training, the history is completely empty, both if I save it in this callback and if I save the history that model.fit() provides.

[markolalovic]

I think the history resets on each epoch, try training it one epoch at the time and collecting the history output. Or try to test it with:

if epoch == 0:
  history = model.fit()

[RotemTon]

Hi, thanks, this was definitely an issue!
The callback still behaves oddly: the loss is either huge or 0, and the accuracy is either 1 (perfect) or 0... here's an example:
Epoch 1 - Train Loss Logs: 402.486328125 , Accuracy: 1.0, AUC-ROC: 0.19148936170212766, EER: 0.6170212765957447
Epoch 1 - Validation Loss Logs: 0.0 , Accuracy: 1.0, AUC-ROC: 0.1515151515151515, EER: 0.696969696969697
Also, the history still doesn't save.

[ebndev]

@markolalovic thanks for being active across the different categories in Discussions this past week! It's been awesome to see 😄

@RotemTon were you able to resolve your issue / get the history to save?

[RotemTon]

Yes, I posted the solution in another comment. Thanks :)

[RotemTon]

Hey, I was eventually able to solve the issue. The history was empty because i had only trained on one epoch, not because there was an issue saving. The reason I never got past the first epoch is because of another issue I had to solve. as for the miscalculations, it turns out that this was a problem with my data generator, that made image triplets. It has to be recreated every epoch, or it doesn't generate anything and the metrics are zero.

Here's the updated code. In it I save each metric as a key in logs, and have a history list (self.logs_history) to which I add this epoch's history to the list. This is because logs resets every epoch. then i save the history (a list of directories) with pickle. Also, I added the functions I used to make my data generators and call them ever epoch to make a new generator in on_epoch_end.

`
class MetricsCallback(tensorflow.keras.callbacks.Callback):
def init(self, train_x, train_y, val_x, val_y, embedding_model,alpha, folder_path, file_name):
super(MetricsCallback, self).init()
self.train_x=train_x
self.val_x=val_x
self.train_y=train_y
self.val_y=val_y
self.embedding_model=embedding_model
self.alpha=alpha
self.file_name = file_name
self.folder_path = folder_path
self.logs_history=[]

#makes a map of the labels' indices in the labels(y) array
def get_map_indices(self,labels: np.array) -> Dict[int, np.array]: #makes map indices
    unique_label = np.unique(labels) #gets the unique labels
    map_label_indices = {label: np.flatnonzero(labels == label) for label in unique_label}
    #for every unique label, creates an array of the indices int the label array where it is, these lists are put 
into a dict
    return map_label_indices
    
#gets the indices for a single triplet from the map_label_indices directory
def get_triplets(self,labels: np.array, map_label_indices: Dict[int, np.array]
) -> Tuple[int, int, int]:# makes a triplet
    unique_label = np.unique(labels) #gives array of unique values (the classes)
    label_l, label_r = np.random.choice(unique_label, 2, replace=False) #randomly chooses two !different! 
labels
    a, p = np.random.choice(map_label_indices[label_l], 2, replace=False) # randomly chooses 2 images with 
label 1 for anchor and positive
    n = np.random.choice(map_label_indices[label_r]) # randomly chooses image with label 2 for negative
    return a, p, n

#makes a generator object of image triplets. Makes ken(labels) amount of them.
def eval_triplets_gen(self,images: np.array,labels: np.array,map_label_indices: Dict[int, np.array]):
        for _ in range(len(labels)):  #for each triplet we add its image indices to the lists
            a, p, n = self.get_triplets(labels=labels, map_label_indices=map_label_indices)
            anchor = images[a]
            positive = images[p]
            negative = images[n]
            yield anchor, positive, negative

def calc_metrics(self ,generator):
    loss=0.0
    #for  accuracy
    y_true_acc=[]
    y_pred=[]

    #for auc-roc and eer
    y_true=[]
    y_scores=[]

    for triplet in generator: #dataset is train_eval_gen or val_eval_gen
      anchor, positive, negative = triplet #get the anchor, positive and negative tensors from each triplet tuple
      a_emb=embedding_model(np.expand_dims(anchor, axis=0)) #get the embeddings for each
      p_emb=embedding_model(np.expand_dims(positive, axis=0))
      n_emb=embedding_model(np.expand_dims(negative, axis=0))

      dist_p = np.linalg.norm(a_emb - p_emb) #calculate distances
      dist_n = np.linalg.norm(a_emb - n_emb)

      loss= loss+K.maximum(dist_p - dist_n + self.alpha, 0)
      #for triplet acc:
      y_true_acc. append(1) #the true label is always 1
      if dist_p +self.alpha<dist_n : #accurate
        y_pred.append(1)
      else:
        y_pred.append(0)

      #for auc-roc and eer:
      y_true.append(1) #for a-p
      y_true.append(0) #for a-n

      y_scores.append(np.exp(-dist_p))#if it's small like we want, it gets a high score
      y_scores.append(np.exp(-dist_n))# if it's big like we want, it gets a high score

    #compute acc:
    accuracy = accuracy_score(y_true_acc, y_pred)
    #compute auc-roc and eer:
    fpr, tpr, thresholds = roc_curve(y_true, y_scores, pos_label=1)
    auc_roc = auc(fpr, tpr)
    eer = fpr[np.argmin(np.abs(fpr - (1 - tpr)))]

    return loss, accuracy, auc_roc, eer, fpr, tpr

def on_epoch_end(self, epoch, logs={}):
    # Compute metrics on train data
    loss_train_logs = logs.get('loss')
    train_label_indices=self.get_map_indices(self.train_y)
    train_eval_gen=self.eval_triplets_gen(images=self.train_x,labels=self.train_y,map_label_indices=train_label_indices)
    loss_train, accuracy_train , auc_roc_train, eer_train, fpr_train, tpr_train= self.calc_metrics(train_eval_gen)
    
    # Log the metrics
    logs["train_loss"]=self.train_loss
    logs["train_accuracy"]=self.train_accuracy
    logs["train_auc_roc"]=self.train_auc_roc
    logs["train_eer"]=self.train_eer
    logs["train_fpr"]=self.train_fpr
    logs["train_tpr"]=self.train_tpr
    
    print(f'Epoch {epoch + 1} - Train Loss Logs: {loss_train_logs} Train Loss: {loss_train}, Accuracy: {accuracy_train}, AUC-ROC: {auc_roc_train}, EER: {eer_train}')

    # Compute metrics on validation data
    loss_val_logs = logs.get('val_loss')
    val_label_indices=self.get_map_indices(self.val_y)
    val_eval_gen=self.eval_triplets_gen(images=self.val_x,labels=self.val_y,map_label_indices=val_label_indices)
    loss_val, accuracy_val, auc_roc_val, eer_val, fpr_val, tpr_val= self.calc_metrics(val_eval_gen)

    # Log the metrics
    logs["val_loss"]=self.val_loss
    logs["val_accuracy"]=self.val_accuracy
    logs["val_auc_roc"]=self.val_auc_roc
    logs["val_eer"]=self.val_eer
    logs["val_fpr"]=self.val_fpr
    logs["val_tpr"]=self.val_tpr

    print(f'Epoch {epoch + 1} - Validation Loss Logs: {loss_val_logs} Validation Loss: {loss_val},  Accuracy: {accuracy_val}, AUC-ROC: {auc_roc_val}, EER: {eer_val}')

    self.logs_history.append(logs.copy())
    # Save the metric history
    epoch_number = epoch + 1  # Epoch starts from 0, so we add 1 to get the actual epoch number
    file_name_with_epoch = f"{self.file_name}_epoch_{epoch_number}.pkl"
    file_path = os.path.join(self.folder_path, file_name_with_epoch)
    with open(file_path, 'wb') as file_pi:
        pickle.dump(self.logs_history, file_pi)
    print('saved metrics')

metrics_callback = MetricsCallback(train_x,train_y, val_x,val_y,siamese_model_1,alpha,folder_path, file_name="siamese_train_hist")
`

Alternatively, instead of using logs it is possible to create a list for every metric in the init function like this:
self.train_loss = [] self.train_accuracy = [] self.train_auc_roc = [] self.train_eer = [] self.train_fpr = [] self.train_tpr = [] self.val_loss = [] self.val_accuracy = [] self.val_auc_roc = [] self.val_eer = [] self.val_fpr = [] self.val_tpr = []

And append them in on_epoch_end(instead of inserting them to logs):
self.train_loss.append(loss_train) self.train_accuracy.append(accuracy_train) self.train_auc_roc.append(auc_roc_train) self.train_eer.append(eer_train) self.train_fpr.append(fpr_train) self.train_tpr.append(tpr_train) self.val_loss.append(loss_val) self.val_accuracy.append(accuracy_val) self.val_auc_roc.append(auc_roc_val) self.val_eer.append(eer_val) self.val_fpr.append(fpr_val) self.val_tpr.append(tpr_val)

And then add them all to a history dictionary in a separate callback:
`
class SaveHistoryCallback(tensorflow.keras.callbacks.Callback):
def init(self, folder_path, file_name, metrics_callback):
self.folder_path = folder_path
self.file_name = file_name
self.metrics_callback= metrics_callback

def on_epoch_end(self,epoch,  logs=None):
    history1 = {
        'train_loss_logs': self.metrics_callback.train_loss_logs,
        'train_loss': self.metrics_callback.train_loss,
        'train_accuracy': self.metrics_callback.train_accuracy,
        'train_auc_roc': self.metrics_callback.train_auc_roc,
        'train_eer': self.metrics_callback.train_eer,
        'train_fpr': self.metrics_callback.train_fpr,
        'train_tpr': self.metrics_callback.train_tpr,
        'val_loss_logs': self.metrics_callback.val_loss_logs,
        'val_loss': self.metrics_callback.val_loss,
        'val_accuracy': self.metrics_callback.val_accuracy,
        'val_auc_roc': self.metrics_callback.val_auc_roc,
        'val_eer': self.metrics_callback.val_eer,
        'val_fpr': self.metrics_callback.val_fpr,
        'val_tpr': self.metrics_callback.val_tpr
    }

    # Save the metric history
    epoch_number = epoch + 1  # Epoch starts from 0, so we add 1 to get the actual epoch number
    file_name_with_epoch = f"{self.file_name}_epoch_{epoch_number}.pkl"
    file_path = os.path.join(self.folder_path, file_name_with_epoch)
    with open(file_path, 'wb') as file_pi:
        pickle.dump(history1, file_pi)
    print('saved metrics (History callback)')

save_hist_callback=SaveHistoryCallback(folder_path, file_name_savehist, metrics_callback)
`

I hope this helps anyone who stumbles upon this. Huge thanks to @markolalovic for the helpful and quick response :)