I used below code to train my model in v3-8 TPU which is single device TPU, it works fine, however, the same code does not work on TPU pod v2-32. As I know v2-32 are clusters of TPU devices that are connected to each other over dedicated high-speed networks, so how to tweak the code to make it work on v2-32?
tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu="tpu-name", zone="us-central1-a", project="myproject")
tf.config.experimental_connect_to_cluster(tpu)
tf.tpu.experimental.initialize_tpu_system(tpu)
strategy = tf.distribute.TPUStrategy(tpu)
....... code to load data and define model ......
with strategy.scope():
myModel = createModel()
....
....
it works perfectly on v3-8 single TPU, but on TPU pod v2-32, it gives error: invalidargument ... unable to find the relevant tensor remote_handle .....
how to configure strategy for v2-32 as clusters of TPU in jupyter notebook?
question from:
https://stackoverflow.com/questions/65949876/tpu-pod-v2-32-for-jupyter-notebook 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…