关于GCU、沐曦GPGPU、MLU、0卡V100资源4月7日恢复上架的公告>>> 关于共建具身智能开源数据集的倡议>>> 关于云脑任务中统一路径访问方式的公告>>> 关于将启智集群GPU资源迁移至智算集群的公告>>>
OpenI
/
octopus
Not watched
28
46
Fork 56
Code
Releases 26 Wiki Activity
Issues 109 Pull Requests 0 Datasets Cloudbrain
Issues
/
Detail
Labels Milestones
New Issue

#609 支持数据集缓存加速

Open
created 1 year ago by linfj · 1 comments
linfj commented 1 year ago

利用SSD/HDD等对章鱼的数据集进行缓存加速

利用SSD/HDD等对章鱼的数据集进行缓存加速
linfj added the
enhancement
label 1 year ago
linfj self-assigned this 1 year ago
denglei was assigned by linfj 1 year ago
linfj added this to the v4.3.2 milestone 11 months ago
linfj commented 11 months ago
Owner

数据集缓存主要是在任务较多或者多机任务频繁数据通信导致网络带宽不够的情况下能够加速数据集的访问，模拟这种场景可通过WonderShaper工具限制节点带宽https://www.51cto.com/article/717545.html

由于节点闲置硬盘资源有限，目前数据集缓存开启的入口放在管理端，暂时只支持缓存用户数据集不支持缓存预置数据集。开启数据集缓存时，根据实际情况填写参数，由于个别节点并没有足够的磁盘空间用来缓存，因此一般需要填写节点标签选择特定的一些节点而不是所有节点。
image

数据集开启缓存后，通过以下命令查看是否已经缓存到节点，以及查看所填写的目录是否有数据。
image

创建训练任务时，缓存数据集节点的亲和性调度策略分数较高，是否调度到缓存节点还取决于其他调度策略分数综合
image

数据集缓存主要是在任务较多或者多机任务频繁数据通信导致网络带宽不够的情况下能够加速数据集的访问，模拟这种场景可通过WonderShaper工具限制节点带宽https://www.51cto.com/article/717545.html 由于节点闲置硬盘资源有限，目前数据集缓存开启的入口放在管理端，暂时只支持缓存用户数据集不支持缓存预置数据集。开启数据集缓存时，根据实际情况填写参数，由于个别节点并没有足够的磁盘空间用来缓存，因此一般需要填写节点标签选择特定的一些节点而不是所有节点。 ![image](/attachments/487aab74-8eb5-4924-8131-288b4675e18b) 数据集开启缓存后，通过以下命令查看是否已经缓存到节点，以及查看所填写的目录是否有数据。 ![image](/attachments/c5148375-8fbd-41b6-8124-cf1eb90403ed) 创建训练任务时，缓存数据集节点的亲和性调度策略分数较高，是否调度到缓存节点还取决于其他调度策略分数综合 ![image](/attachments/639cecaf-ef87-4f25-8fce-9e11eba91a61)
image.png
9.3 KiB
image.png
8.9 KiB
image.png
62 KiB
image.png
38 KiB
linfj added the
开发完成
label 11 months ago
Sign in to join this conversation.
No Branch/Tag Specified
Clear branch or tag
Branches Tags
Labels
Apply labels
Clear labels
bug
Something is not working duplicate
This issue or pull request already exists enhancement
New feature help wanted
Need some help invalid
Something is wrong optimizing
Optimizing question
More information is needed wontfix
This won't be fixed 已解决 开发中 开发完成 新建 测试不通过 测试中 测试通过 跟进中
No Label
bug
duplicate
enhancement
help wanted
invalid
optimizing
question
wontfix
已解决
开发中
开发完成
新建
测试不通过
测试中
测试通过
跟进中
Milestone
Set milestone
Clear milestone
No items
No Milestone
v4.3.2
Assignees
Assign users
Clear assignees
No Assignees
 linfj
 denglei
1 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Write Preview
Loading…
Cancel
Save
There is no content yet.
Delete Branch '%!s(MISSING)'

Deleting a branch is permanent. It CANNOT be undone. Continue?

No
Yes
Please read the following content carefully:

Dear OpenI User

Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.

For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》

Agree and continue
Disagree, exit