Tensorflow矩阵过大问题的解决

it2022-05-05  184

问题:推荐系统中,用户和物品矩阵进行embedding,但矩阵过大时,超过2G时,会有如下提示

ValueError: Cannot create a tensor proto whose content is larger than 2GB.

出现问题的代码语句是:

self.user_item_embedding = tf.convert_to_tensor(matrix)

matrix是一个numpy的大型矩阵。

解决:

 matrix_init = tf.placeholder(tf.float32, shape=(self.shape[0], self.shape[1]))  matrixV = tf.Variable(matrix_init) self.user_item_embedding = tf.convert_to_tensor(matrixV)

先构建占位符,在设置变量,并在sess.run时赋予:

elf.sess.run(tf.global_variables_initializer(), feed_dict={matrix_init: matrix})

参考:

https://stackoverflow.com/questions/51470991/create-a-tensor-proto-whose-content-is-larger-than-2gb

https://stackoverflow.com/questions/35394103/initializing-tensorflow-variable-with-an-array-larger-than-2gb


最新回复(0)