Deleting a branch is permanent. It CANNOT be undone. Continue?
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》
数据集cifar10
模型resnet50
优化策略:sdg,svrg,sag
两种场景:
1. iid 正常集中式训练
2. non-iid cifar10按照类别分10份,每次迭代取其中一方参与
探索目的:通过改进优化策略,减少梯度方差,加快模型收敛,尤其non-iid场景下
实验结果:http://192.168.202.124:12434/
分析:
i 改进bn层,全局bn
ii 去掉bn,用svrg,sag等优化策略
iii 联合运用改进bn和svrg,sag等策略
继续上次实验
分析:
i.bn改进group norm, instance norm, layer norm
i.观察svrg和sgd梯度方差分布,验证方差减小。
继续上次实验:
下一步:
继续上次实验,已经完成特例场景下,优化的更新策略可以加速模型收敛,继续一般性场景
一般性参数设置实验探索:
设置:共5个参与方,探索参数
sgd的baseline结果
参数A分析:
参数L分析:
餐数C分析:
完成探索三个参数对于baseline模型收敛影响
并且已经验证在non-iid下,优化后的更新策略A=0, L=1的情况下加速收敛。
下一步实验验证优化更新策略在不同的A,L,C组合下的收敛效果,验证一般性。