I was having problems with missing information in my problem. There were missing images which were read as NaNs image shape matrixes but at the end they were fed to the CNN.

Debugging the Python code I realized that the function transforms.ToPILImage() was converting the NaNs to zero values.