关于GCU、沐曦GPGPU、MLU、0卡V100资源4月7日恢复上架的公告>>> 关于共建具身智能开源数据集的倡议>>> 关于云脑任务中统一路径访问方式的公告>>> 关于将启智集群GPU资源迁移至智算集群的公告>>>
PCL-Platform.Intelligence
/
AISynergy
Not watched
20
18
Fork 11
Code
Releases 5 Wiki Activity
Issues 9 Pull Requests 0 Datasets Model Cloudbrain
Issues
/
Detail
Labels Milestones
New Issue

#20 梯度量化方法（BinarySGD+TrinarySGD）

Open
created 1 year ago by JinWang · 1 comments
JinWang commented 1 year ago

任务目标：

  • 基于pytorch实现BinarySGD、TrinarySGD梯度量化方法，并在ViT、ResNet等模型上进行对比验证
任务目标： * 基于pytorch实现BinarySGD、TrinarySGD梯度量化方法，并在ViT、ResNet等模型上进行对比验证
JinWang commented 1 year ago
Owner

2022-06-24:

  • 1、学习Pytorch中DDP模块的通信hook实现机制，参考完成BinarySGD代码编写，存在问题是：Pytorch中没有1bit数据类型
  • 2、研究学习DeepSpeed中的1bit-Adam原理和实现源码，其中1bit数据采用了基于CUDA实现的cupy多维数组库来实现，将梯度压缩实现在优化器类中，不确定是否支持基于Bucket的梯度聚合

存在问题：
（1）1bit通信实现问题：继续参考DeepSpeed的实现方法进行
（2）目前设计的梯度量化方法中，m,v等中间变量采用的是每张卡本地的状态值，而不是全局的状态值，是否影响模型收敛还有待验证，下面是BinarySGD、TrinarySGD方法和DeepSpeed中1bit Adam的算法设计对比
图片

**2022-06-24:** * 1、学习Pytorch中DDP模块的通信hook实现机制，参考完成BinarySGD代码编写，存在问题是：Pytorch中没有1bit数据类型 * 2、研究学习DeepSpeed中的1bit-Adam原理和实现源码，其中1bit数据采用了基于CUDA实现的cupy多维数组库来实现，将梯度压缩实现在优化器类中，不确定是否支持基于Bucket的梯度聚合 **存在问题：** （1）1bit通信实现问题：继续参考DeepSpeed的实现方法进行 （2）目前设计的梯度量化方法中，m,v等中间变量采用的是每张卡本地的状态值，而不是全局的状态值，是否影响模型收敛还有待验证，下面是BinarySGD、TrinarySGD方法和DeepSpeed中1bit Adam的算法设计对比 ![图片](/attachments/722b3d9c-bd1d-4183-a95a-40e227af4e39)
图片.png
271 KiB
Sign in to join this conversation.
No Branch/Tag Specified
Clear branch or tag
Branches Tags
Labels
Apply labels
Clear labels
bug
Something is not working duplicate
This issue or pull request already exists enhancement
New feature help wanted
Need some help invalid
Something is wrong question
More information is needed wontfix
This won't be fixed
No Label
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Set milestone
Clear milestone
No items
No Milestone
Assignees
Assign users
Clear assignees
No Assignees
1 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Write Preview
Loading…
Cancel
Save
There is no content yet.
Delete Branch '%!s(MISSING)'

Deleting a branch is permanent. It CANNOT be undone. Continue?

No
Yes
Please read the following content carefully:

Dear OpenI User

Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.

For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》

Agree and continue
Disagree, exit