模型文件回写失败问题

问题描述

智算GPU训练任务模型回写时报错：FPutObject(efficientnet_b0-650_142.ckpt) failed: The difference between the request time and the server's time is too large.

导致模型结果无法显示和下载。

任务类型（调试/训练/推理）

训练

任务名

efficientv1_11
任务链接

日志说明或问题截图

日志报错如下：

2022/10/10 00:07:51 start uploading model
2022/10/10 00:07:51 file name:/tmp/output/efficientnet_b0-200_142.ckpt
2022/10/10 00:16:23 file name:/tmp/output/efficientnet_b0-graph.meta
2022/10/10 00:16:50 file name:/tmp/output/efficientnet_b0-650_142.ckpt
2022/10/10 00:41:45 FPutObject(efficientnet_b0-650_142.ckpt) failed: The difference between the request time and the server's time is too large.
2022/10/10 00:41:45 finish uploading model

期望的解决方案或建议

今天又出现了同样的问题

2022/10/11 01:07:34 start uploading model
2022/10/11 01:07:34 file name:/tmp/output/efficientnet_b0-770_142.ckpt
2022/10/11 01:51:45 FPutObject(efficientnet_b0-770_142.ckpt) failed: The difference between the request time and the server's time is too large.
2022/10/11 01:51:45 finish uploading model

导致一个ckpt文件都没回写出来，这样的话就又白白训练了一次！

希望可以尽快排查问题，看看是我这边设置的原因还是服务器网络的原因。

Deleting a branch is permanent. It CANNOT be undone. Continue?

Dear OpenI User

Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.

For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》

#224 模型文件回写失败问题

问题描述

相关环境（GPU/NPU）

相关集群（启智/智算）

任务类型（调试/训练/推理）

任务名

日志说明或问题截图

期望的解决方案或建议