2.6B模型的output.mapping.weight 的 input_size和源码的不一致

你好！请教个问题。

当我加载每个*.ckpt文件的时候发现, 每个Mapping_output.weight.shape = (40, 1280)有点奇怪

...
Parameter (name=backbone.blocks.30.attention.dense3.weight, shape=(5, 2560), dtype=Float32, requires_grad=True),
Parameter (name=backbone.blocks.30.attention.dense3.bias, shape=(5,), dtype=Float32, requires_grad=True),
Parameter (name=backbone.blocks.30.output.mapping.weight, shape=(40, 1280), dtype=Float32, requires_grad=True),
Parameter (name=backbone.blocks.30.output.mapping.bias, shape=(20,), dtype=Float32, requires_grad=True)
...

按道理是不是应该是(20, 2560)?
想请问下模型参数拆分成512份的时候是怎么拆分的？

512卡：mp=8， dp=512/8=64
backbone.blocks.30.output.mapping.weight参数未拆分应该是(2560, 1280)
64路数据并行，拆分后为(2560/64=40, 1280),即(40, 1280)

Deleting a branch is permanent. It CANNOT be undone. Continue?

Dear OpenI User

Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.

For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》

#10 2.6B模型的output.mapping.weight 的 input_size和源码的不一致