AISynergy simulation提供两种类型的仿真功能:(1)小模型、大量参与者场景,模拟输出联邦训练的模型性能;(2)较大参数量模型+少量参与者,模拟输出协同训练时的计算效率。
toDO
(1)仿真实验-1:不同参数规模MLP模型的效率仿真
序号 | 模型参数量 | 任务总耗时 | 算力利用率 | 通信耗时占比 | 数据序列化及反序列化耗时占比 | |||
---|---|---|---|---|---|---|---|---|
client1 | client2 | client1 | client2 | client1 | client2 | |||
1 | 10M | 1663.35 | 87.08 | 70.23 | 10.63 | 27.51 | 0.83 | 0.84 |
2 | 20M | 1867.66 | 78.58 | 62.46 | 17.57 | 33.82 | 1.58 | 1.48 |
3 | 40M | |||||||
4 | 60M | 24743.54 | 57.47 | 44.06 | 34.82 | 48.34 | 2.96 | 2.83 |
5 | 80M | 141537.48 | 46.27 | 37.10 | 40.67 | 48.45 | 3.13 | 3.16 |
6 | 100M | 3733.16 | 45.38 | 33.75 | 42.84 | 54.19 | 3.43 | 3.35 |
总结:
(1)DistIR仿真结果测试
序号 | model_size | world_size | batch_size | dp_degree | hp_degree | pp_degree | num_microbatches | throughput |
---|---|---|---|---|---|---|---|---|
1 | gpt2 | 8 | 128 | 1 | 8 | 1 | 1 | 143.11 |
2 | gpt2-medium | 8 | 128 | 1 | 8 | 1 | 1 | 69.38 |
3 | gpt2-large | 8 | 128 | 1 | 8 | 1 | 1 | 55.06 |
4 | mlp-small | 8 | 128 | 1 | 8 | 1 | 1 | 2657.17 |
总结:
(2)DistIR仿真与Colossal-AI真实训练效率对比
序号 | 类型 | world_size | batch_size | dp_degree | hp_degree | pp_degree | num_microbatches | throughput |
---|---|---|---|---|---|---|---|---|
1 | 仿真 | 4 | 4 | 4 | 1 | 1 | 1 | 3.85 |
2 | 仿真 | 4 | 8 | 4 | 1 | 1 | 2 | 7.70 |
3 | 仿真 | 4 | 16 | 4 | 1 | 1 | 4 | 15.4 |
4 | 仿真 | 4 | 32 | 4 | 1 | 1 | 8 | 30.75 |
5 | 仿真 | 4 | 64 | 4 | 1 | 1 | 16 | 61.29 |
6 | 仿真 | 4 | 128 | 4 | 1 | 1 | 32 | 121.73 |
7 | 真实训练 | 4 | 4 | 4 | 1 | 1 | 1 | 4.5 |
8 | 真实训练 | 4 | 8 | 4 | 1 | 1 | 2 | |
9 | 真实训练 | 4 | 16 | 4 | 1 | 1 | 4 | |
10 | 真实训练 | 4 | 32 | 4 | 1 | 1 | 8 | |
11 | 真实训练 | 4 | 64 | 4 | 1 | 1 | 16 | |
12 | 真实训练 | 4 | 128 | 4 | 1 | 1 | 32 |
总结:
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》