Coding Tips on Paco's Log

Ommiting corrupted images inside a Pytorch dataloader

Sun, 30 Jan 2022 00:00:00 +0000

Sometimes our data is not as clean as it should be. Maybe some images are corruped, or the entries inside the database were not as expected. In that case, iterating through them using the Pytorch dataloader can be hustle. Recently I learned that you can clean your data after it has been loaded, but before you are iterating throught in your training lops. In this case I am going to give an example with images, but it can be applied to any kind of data.

PIL converts NaNs images to zero value images

Sat, 07 Mar 2020 00:00:00 +0000

I was having problems with missing information in my problem. There were missing images which were read as NaNs image shape matrixes but at the end they were fed to the CNN.

Debugging the Python code I realized that the function transforms.ToPILImage() was converting the NaNs to zero values.

Count NaN values that are not NaN in Pandas DataFrame

Mon, 13 Jan 2020 00:00:00 +0000

I am working with a dataset that contains lots of missing values. However, when a value is missed it is not represented as NaN but as “–”. In order to find how many missing values there are in an specific column I cannot use typical Pandas functions. Henceforth, I have worked it out by count the number of appereances of this character in the column and substract it to the total number of entries in the column:

How to set Python3 as default version on MacOS

Fri, 10 Jan 2020 00:00:00 +0000

In order not to have to type everytime you want to run Python 3

python3

I wanted to link to the default version of Python. Then you need to use the following commands:

Install again Python 3 with brew:

brew install python

Look how it is installed:

ls -l /usr/local/bin/python*

Different versions should appear. We would create a new default python symlink to the version we want to use:

ln -s -f /user/local/bin/python3.7 /usr/local/bin/python

Exit the terminal. Now, if we type

Combine images vertically

Tue, 17 Dec 2019 00:00:00 +0000

Recently I needed to combine a bunch of images for a book. Since I didn’t want to use a software I thought it would be easier to combine them using Python with the following script (thanks to Python Programming):

import numpy as np
from PIL import Image
import glob

list_im = glob.glob('*.png')
imgs = [ Image.open(i) for i in list_im ]
# choose the minimum size and resize the other ones
min_shape = sorted( [(np.sum(i.size), i.size ) for i in imgs])[0][1]
imgs_comb = np.vstack( (np.asarray( i.resize(min_shape) ) for i in imgs ) )
imgs_comb = Image.fromarray( imgs_comb)
imgs_comb.save( 'imagename.png')

Moving from Tensorflow 2.0 to Pytorch

Tue, 17 Dec 2019 00:00:00 +0000

I was having many issues with GPU memory management in a Transfer Learning problem when using Tensorflow 2.0 with Keras module. Even though I was using Inception V3 model with all the weights of the architecture freezed, the code ran out of memory. The weights of the model occupied only 98 MB therefore that wasn’t the problem. Input shape was (299, 299, 3) so not huge images were being used, but I couldn’t use a batch size higher than 16.

Parallel training in Tensorflow 2.0

Tue, 17 Dec 2019 00:00:00 +0000

Recently I have been able to use 2 GPUs for model training. I was having problems while training with both of them (only one of them was being used).

At the end, thanks to this issue thread, I knew that with the new eager mode, there exist a bug for multi gpu training.

In order to fix it, you need to disable eager mode using the following command:

import tensorflow as tf
tf.compat.v1.disable_eager_execution()

Using glob for file retrieval

Thu, 12 Dec 2019 00:00:00 +0000

When is glob useful?

Usually I need to iterate over lots of CSV files. Imaging that you have loads of CSV files which contains the EMG (declared as an array of floats) information of different patients. You want to read them with Python’s library Pandas. Having to declare each CSV by hand and then pass it to Pandas would be utterly painfull. Insted, there exist an amazing Python library called glob that helps with this task.