【发布时间】:2018-12-09 13:21:39
【问题描述】:
我在 Google Cloud 上使用 ML 引擎运行 TensorFlow 模型,但检查点保护程序无法将文件保存在存储桶上。我正在使用 TensorFlow 1.4 和 tf.Estimator 与方法 tf.estimator.train_and_evaluate。
这些是日志记录,其中gs://e-trial-central1/models/1530351907.8359423 是为估算器提供的参数model_dir:
E master-replica-0 Couldn't match files for checkpoint gs://e-trial-central1/models/1530351907.8359423/.
I master-replica-0 Create CheckpointSaverHook.
I master-replica-0 Restoring parameters from gs://e-trial-central1/models/1530351907.8359423/.
W master-replica-0 Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for gs://e-trial-central1/models/1530351907.8359423/.
W master-replica-0 Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for gs://e-trial-central1/models/1530351907.8359423/.
W master-replica-0 Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for gs://e-trial-central1/models/1530351907.8359423/.
W master-replica-0 Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for gs://e-trial-central1/models/1530351907.8359423/.
W master-replica-0 Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for gs://e-trial-central1/models/1530351907.8359423/.
W master-replica-0 Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for gs://e-trial-central1/models/1530351907.8359423/.
W master-replica-0 Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for gs://e-trial-central1/models/1530351907.8359423/.
W master-replica-0 Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for gs://e-trial-central1/models/1530351907.8359423/.
根据其他帖子(here 和 here)的建议,我已经尝试过的事情:
- 保存到区域存储桶 (us-central1) 而不是多区域。这会导致同样的错误。
- 使用不包含“.”的更简单的路径在文件夹名称中。这会导致同样的错误。
- 保存到本地路径,而不是存储桶。这行得通!但我最终想要存储桶上的文件。
与其他帖子相比,这里有点奇怪的是检查点路径实际上已损坏。有 '。'在模型目录而不是 Tensorflow 模式之后 (model.ckpt)。
此外,在我查看存储桶中的模型目录失败后,实际上有文件存在 - TF 事件文件,以及 .index、.meta 和 .data... 文件,但检查点文件不存在。
有什么想法会导致这种情况吗?或者接下来要尝试什么?
不胜感激!
【问题讨论】:
标签: tensorflow google-cloud-ml tensorflow-estimator