Training loss growing up

[yuanc3]

Does anyone has problem like this? I just use the colorization config for my data, and the loss growing up and the output always be noisy.

23-09-23 23:32:14.272 - INFO: Begin model train.
23-09-23 23:39:51.920 - INFO: train/mse_loss: 0.5875651128242876
23-09-23 23:39:51.921 - INFO: epoch: 1
23-09-23 23:39:51.921 - INFO: iters: 6008
23-09-23 23:44:58.423 - INFO: train/mse_loss: 0.18443983706285458
23-09-23 23:47:28.679 - INFO: train/mse_loss: 0.1707256488630997
23-09-23 23:47:28.679 - INFO: epoch: 2
23-09-23 23:47:28.679 - INFO: iters: 12016
23-09-23 23:55:01.374 - INFO: train/mse_loss: 0.12716468866324218
23-09-23 23:55:01.375 - INFO: epoch: 3
23-09-23 23:55:01.375 - INFO: iters: 18024
23-09-23 23:57:34.015 - INFO: train/mse_loss: 0.11658619797452982
23-09-24 00:02:33.583 - INFO: train/mse_loss: 0.12064346502449477
23-09-24 00:02:33.584 - INFO: epoch: 4
23-09-24 00:02:33.584 - INFO: iters: 24032
23-09-24 00:10:04.679 - INFO: train/mse_loss: 0.1233654677937801
23-09-24 00:10:08.177 - INFO: train/mse_loss: 0.12338643875741927
23-09-24 00:10:08.178 - INFO: epoch: 5
23-09-24 00:10:08.178 - INFO: iters: 30040
23-09-24 00:10:08.178 - INFO:

------------------------------Validation Start------------------------------
23-09-24 00:17:50.267 - INFO: val/mae: 1.152842402458191
23-09-24 00:17:50.267 - INFO:
------------------------------Validation End------------------------------

23-09-24 00:25:21.371 - INFO: train/mse_loss: 0.1386190212509048
23-09-24 00:25:21.372 - INFO: epoch: 6
23-09-24 00:25:21.372 - INFO: iters: 36058
23-09-24 00:32:51.582 - INFO: train/mse_loss: 0.16339113016162668
23-09-24 00:32:51.584 - INFO: epoch: 7
23-09-24 00:32:51.584 - INFO: iters: 42066
23-09-24 00:40:20.440 - INFO: train/mse_loss: 0.20504292834813045
23-09-24 00:40:20.441 - INFO: epoch: 8
23-09-24 00:40:20.441 - INFO: iters: 48074
23-09-24 00:47:50.634 - INFO: train/mse_loss: 0.26625456471497144
23-09-24 00:47:50.635 - INFO: epoch: 9
23-09-24 00:47:50.635 - INFO: iters: 54082
23-09-24 00:55:21.751 - INFO: train/mse_loss: 0.3486290028425412
23-09-24 00:55:21.752 - INFO: epoch: 10
23-09-24 00:55:21.752 - INFO: iters: 60090
23-09-24 00:55:21.752 - INFO: Saving the self at the end of epoch 10
23-09-24 00:55:40.332 - INFO:

------------------------------Validation Start------------------------------
23-09-24 01:03:16.369 - INFO: val/mae: 1.0535274744033813
23-09-24 01:03:16.370 - INFO:
------------------------------Validation End------------------------------

23-09-24 01:10:41.844 - INFO: train/mse_loss: 0.4689194846169133
23-09-24 01:10:41.845 - INFO: epoch: 11
23-09-24 01:10:41.845 - INFO: iters: 66108
23-09-24 01:15:32.251 - INFO: train/mse_loss: 0.587346616041256
23-09-24 01:18:05.937 - INFO: train/mse_loss: 0.6161996504755693
23-09-24 01:18:05.937 - INFO: epoch: 12
23-09-24 01:18:05.937 - INFO: iters: 72116
23-09-24 01:25:27.781 - INFO: train/mse_loss: 0.7735867933252363
23-09-24 01:25:27.782 - INFO: epoch: 13
23-09-24 01:25:27.782 - INFO: iters: 78124
23-09-24 01:27:54.342 - INFO: train/mse_loss: 0.8663824072270505
23-09-24 01:32:54.690 - INFO: train/mse_loss: 0.9051068551372117
23-09-24 01:32:54.690 - INFO: epoch: 14
23-09-24 01:32:54.690 - INFO: iters: 84132
23-09-24 01:40:07.631 - INFO: train/mse_loss: 0.9838901592422361
23-09-24 01:40:18.302 - INFO: train/mse_loss: 0.9842592114694267
23-09-24 01:40:18.302 - INFO: epoch: 15
23-09-24 01:40:18.302 - INFO: iters: 90140
23-09-24 01:40:18.302 - INFO:

------------------------------Validation Start------------------------------
23-09-24 01:47:51.676 - INFO: val/mae: 1.1024448871612549
23-09-24 01:47:51.677 - INFO: