python - Google Colab runs out of RAM(12GB) when trying to tf.image.resize dataset

Question

new_img = []
images = list(df['pixels'])
  
 
for image in images:
  image = image.split()
  new_img.append(image)

new_img = np.array(new_img)
new_img = new_img.astype('float32')/255
new_img = new_img.reshape(df.shape[0], 48, 48, 1)
new_img = tf.image.resize(new_img(297,297))

In the code above (last line), I am trying to resize the whole dataset (35000 elements). But Google Colab exhausts all RAM. Is there any other way to approach this problem, please?

Thank you in advance.

question from:https://stackoverflow.com/questions/65844488/google-colab-runs-out-of-ram12gb-when-trying-to-tf-image-resize-dataset

Answer 1

Try doing it iteratively with tf.data.Dataset. The pictures will only be resized while iterating through the dataset, one batch at a time. So it won't create a new dataset in memory.

new_img = np.array(new_img)

ds = tf.data.Dataset.from_tensor_slices(new_img).
    map(lambda x: tf.resize(x/255, (297, 297)))

print(next(iter(ds)))