#4327 【智算NPU训练】2节点任务运行结束后,回传结果文件只有节点1的、没有节点2的

Closed
created 10 months ago by wangj · 4 comments
wangj commented 10 months ago
2节点的智算NPU训练任务运行结束后,等回传动作完成(通过查db确认)后查看结果下载页签:只有节点1的、没有节点2的文件。 通过在日志中打印/cache/output目录下的所有文件,可以确认训练结束后、回传开始之前节点1、节点2上面的结果文件。 测试数据: 云脑侧任务名 wjtes20230614-2t491094600
wangj added this to the V20230531 milestone 10 months ago
wangj added the
bug
label 10 months ago
lewis was assigned by wangj 10 months ago
liuzx was assigned by wangj 10 months ago
ychao_1983 was assigned by wangj 10 months ago
chenshihai was assigned by wangj 10 months ago
wangj added the
need review
label 10 months ago
tanglj was assigned by wangj 10 months ago
wangj commented 10 months ago
Owner
结果下载页签,如何展示多个节点的结果?待确认。 @tanglj
tanglj commented 10 months ago
Collaborator
> 结果下载页签,如何展示多个节点的结果?待确认。 @tanglj 在结果下载页签,展示多个节点的结果,多个节点通过文件夹区分。现有界面设计无须调整。
wangj added the
test
label 10 months ago
wangj commented 10 months ago
Owner
有时候回传成功。 有时候,会一直显示“文件同步中”,虎鲸调度已经成功。测试数据:虎鲸侧任务名 wjtes20230617-2t402339792 。
wangj removed the
test
label 10 months ago
wangj added the
test
label 10 months ago
wangj commented 10 months ago
Owner
> 有时候回传成功。 > 有时候,会一直显示“文件同步中”,虎鲸调度已经成功。测试数据:虎鲸侧任务名 wjtes20230617-2t402339792 。 obs域名解析出错了。偶发现象,可能与当时网络有关,另外提issue #4342 跟踪。
wangj closed this issue 10 months ago
Sign in to join this conversation.
No Milestone
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.