鹏程.mPanGu-α-53基于一带一路多语言翻译场景应用,支持多语言翻译任务的预训练、迁移学习、微调,支持在NPU/GPU上基于mindspore分布式训练(最少8卡)、
推理(全精度/FP16,1卡),单模型支持53种语言任意两种语言间的互译。
相关原理和方法,请参见[项目主页]。
要点:
原生支持基于mindspore的NPU/GPU环境,通过模型转换也可支持基于pytorch的GPU环境,环境信息:请参考项目主页。
如果基于鹏程.mPanGu-α做微调或者基于鹏程.mPanGu-α直接训练,请参考如下的所有数据准备过程,基于鹏程.mPanGu-α做推理预测只需要参考‘模型预测’流程即可。
示例:
转换后数据:
单语zh.txt:'Mary Ng (伍凤仪),加拿大华裔联邦国会议员,几天前刚刚被委任加拿大小型企业及出口促进部部长。'
双语zh-en_corpus.txt:'第三,无功绩。\t third, with no success.'
每段完整的内容作为一个数据样本。
样本间用两个换行符\n\n隔开
from pcl_pangu.context import set_context
from pcl_pangu.dataset import txt2mindrecord
set_context(backend="mindspore")
txt2mindrecord(input_glob='YOUR_DATASET_TXT_DIR/*', \
output_prefix='Your_Saving_DIR/text_document', \
vocab_file='vocab_13w')
如果你想利用鹏程.mPanGu-α做训练,可以参考如下实现过程。
from pcl_pangu.context import set_context
from pcl_pangu.dataset import txt2mindrecord
from pcl_pangu.model import mPangu
set_context(backend="mindspore")
data_path = 'path/of/training/dataset'
txt2mindrecord(input_glob='your/txt/path/*.txt', \
output_prefix=data_path, \
vocab_file='vocab_13w')
config = mPangu.model_config_npu(model='2B6', save='path/to/save/ckpt', data_path=data_path)
mPangu.train(config)
如果你想基于鹏程.mPanGu-α用自己的数据做进一步的增量训练,可以参考如下实现过程。
首先需要在模型下载页面下载对应的鹏程.mPanGu-α
多语言翻译模型。
模型微调流程基本和训练的流程一致,只需要更改为调用 model.fine_tune()
。
from pcl_pangu.context import set_context
from pcl_pangu.dataset import txt2mindrecord
from pcl_pangu.model import mPangu
set_context(backend="mindspore")
data_path = 'path/of/training/dataset'
txt2mindrecord(input_glob='your/txt/path/*.txt', \
output_prefix=data_path, \
vocab_file='vocab_13w')
config = mPangu.model_config_npu(model='2B6', save='path/of/your/existing/ckpt', data_path=data_path)
mPangu.fine_tune(config)
提供了简单的推理流程,用户只需要输入需要生成的文本就可以在进行推理。首先需要在模型下载页面下载对应的模型。
from pcl_pangu.context import set_context
from pcl_pangu.model import mPangu
set_context(backend='mindspore')
config = mPangu.model_config_npu(model='2B6',save='2B6/mode/path')
mPangu.inference(config,input='四川的省会是?', src_language='zh', tag_language='kk')
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》