=====args_opt is: Namespace(data_url=None, device_id=0, device_num=1, distribute='false', embedding_size=2560, load_ckpt_name='PANGUALPHA3.ckpt', load_ckpt_path='/mnt/pangu/pangu-alpha/checkpoint_file', micro_size=1, mode='2.6B', num_heads=32, num_layers=32, per_batch_size=1, run_type='predict', seq_length=1024, stage_num=1, strategy_load_ckpt_path='/mnt/pangu/pangu-alpha/strategy_load_ckpt/pangu_alpha_2.6B_ckpt_strategy.ckpt', tensor_model_parallel_num=8, tokenizer_path='/mnt/pangu/pangu-alpha/tokenizer/', train_url=None, vocab_size=40000)
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.601.022 [mindspore/common/_decorator.py:32] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.642.878 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.643.323 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.644.994 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.645.344 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.843.766 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.845.435 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
Traceback (most recent call last):
File "/mnt/pangu/pangu-alpha/run_pangu_alpha_predict.py", line 92, in
run_predict(args_opt)
File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 228, in run_predict
run_predict_no_pipeline(args_opt)
File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 195, in run_predict_no_pipeline
pangu_alpha = PANGUALPHA(config)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 886, in init
self.backbone = PANGUALPHA_Model(config)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 720, in init
per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 459, in init
self.attention = Attention(config, scale, layer_idx)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 285, in init
self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),))
File "/mnt/anaconda3/envs/pangu/lib/python3.7/site-packages/mindspore/nn/cell.py", line 266, in getattr
raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).name, name))
AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.
模型: 2.6B
mindspore: 1.2
GPU: NVIDA V100
(pangu) root@iZ2ze15skiya5kcq1eryyuZ:/mnt/pangu/pangu-alpha# ./scripts/infer.sh
rank_id:0 rank_id str:0
local_rank:0, device id:0 start to run...
===config is: [PANGUALPHAConfig]==============================
batch_size:0
seq_length:1024
vocab_size:40000
embedding_size:2560
num_layers:32
num_heads:32
expand_ratio:4
post_layernorm_residual:False
dropout_rate:0.1
compute_dtype:Float16
use_past:False
dp:0
mp:8
self_layernorm:True
forward_reduce_scatter:True
stage_num:1
micro_size:1
word_emb_dp:True
eod_reset:False
load_ckpt_path:/mnt/pangu/pangu-alpha/checkpoint_file
==========
=====args_opt is: Namespace(data_url=None, device_id=0, device_num=1, distribute='false', embedding_size=2560, load_ckpt_name='PANGUALPHA3.ckpt', load_ckpt_path='/mnt/pangu/pangu-alpha/checkpoint_file', micro_size=1, mode='2.6B', num_heads=32, num_layers=32, per_batch_size=1, run_type='predict', seq_length=1024, stage_num=1, strategy_load_ckpt_path='/mnt/pangu/pangu-alpha/strategy_load_ckpt/pangu_alpha_2.6B_ckpt_strategy.ckpt', tensor_model_parallel_num=8, tokenizer_path='/mnt/pangu/pangu-alpha/tokenizer/', train_url=None, vocab_size=40000)
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.601.022 [mindspore/common/_decorator.py:32] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.642.878 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.643.323 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.644.994 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.645.344 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.843.766 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.845.435 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
Traceback (most recent call last):
File "/mnt/pangu/pangu-alpha/run_pangu_alpha_predict.py", line 92, in <module>
run_predict(args_opt)
File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 228, in run_predict
run_predict_no_pipeline(args_opt)
File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 195, in run_predict_no_pipeline
pangu_alpha = PANGUALPHA(config)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 886, in __init__
self.backbone = PANGUALPHA_Model(config)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 720, in __init__
per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 459, in __init__
self.attention = Attention(config, scale, layer_idx)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 285, in __init__
self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),))
File "/mnt/anaconda3/envs/pangu/lib/python3.7/site-packages/mindspore/nn/cell.py", line 266, in __getattr__
raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).__name__, name))
AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.
Hi,我在Ascend上运行也会报这个错:'Dropout' object has no attribute 'dropout_gen_mask'
INFO:Training model with standard mode:
INFO: - local_rank:0, device id:0 start to run...
INFO:Distributed Training: device_id is 0, rank_id is 0, device_num is 1
INFO:===config is: [PANGUALPHAConfig]==============================
batch_size:4
seq_length:256
vocab_size:15928
embedding_size:512
num_layers:6
num_heads:8
expand_ratio:4
post_layernorm_residual:False
dropout_rate:0.1
compute_dtype:Float16
use_past:False
dp:0
mp:4
self_layernorm:True
forward_reduce_scatter:True
stage_num:1
micro_size:16
word_emb_dp:True
eod_reset:False
load_ckpt_path:None
==========
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.269.472 [mindspore/common/_decorator.py:33] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.276.926 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.278.269 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.281.198 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.282.496 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.287.459 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.290.675 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-7-4e527a9a05d2> in <module>
----> 1 run_train(args_opt)
~/work/pangu_alpha_train.py in run_train(args_opt)
341 run_train_pipeline(args_opt)
342 else:
--> 343 run_train_no_pipeline(args_opt)
344
345
~/work/pangu_alpha_train.py in run_train_no_pipeline(args_opt)
271 word_emb_dp=True)
272 logger.info(f"===config is: {config}")
--> 273 pangu_alpha = PANGUALPHA(config)
274 loss = CrossEntropyLoss(config)
275 pangu_alpha_with_loss = PANGUALPHAWithLoss(config, pangu_alpha, loss)
~/work/pangu_alpha.py in __init__(self, config)
897 def __init__(self, config):
898 super(PANGUALPHA, self).__init__()
--> 899 self.backbone = PANGUALPHA_Model(config)
900 self.head = PANGUALPHA_Head(config)
901
~/work/pangu_alpha.py in __init__(self, config)
732
733 for i in range(num_layers):
--> 734 per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2)
735 per_block.recompute()
736 per_block.attention.dropout.dropout_gen_mask.recompute(False)
~/work/pangu_alpha.py in __init__(self, config, layer_idx)
458 self.layernorm2.layer_norm.shard(((config.dp, 1, 1), (1,), (1,)))
459
--> 460 self.attention = Attention(config, scale, layer_idx)
461 self.output = Output(config, scale)
462 self.post_layernorm_residual = config.post_layernorm_residual
~/work/pangu_alpha.py in __init__(self, config, scale, layer_idx)
284 self.use_past = config.use_past
285 self.dropout = nn.Dropout(1 - config.dropout_rate)
--> 286 self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),))
287 self.dropout.dropout_do_mask.shard(((config.dp, 1, 1),))
288 self.prob_dropout = nn.Dropout(1 - config.dropout_rate)
~/miniconda3/envs/MindSpore-python3.7-aarch64/lib/python3.7/site-packages/mindspore/nn/cell.py in __getattr__(self, name)
286 para_list = ParameterTuple(cast_list)
287 return para_list
--> 288 raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).__name__, name))
289
290 def __del__(self):
AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.
Hi,我在Ascend上运行也会报这个错:'Dropout' object has no attribute 'dropout_gen_mask'
```
INFO:Training model with standard mode:
INFO: - local_rank:0, device id:0 start to run...
INFO:Distributed Training: device_id is 0, rank_id is 0, device_num is 1
INFO:===config is: [PANGUALPHAConfig]==============================
batch_size:4
seq_length:256
vocab_size:15928
embedding_size:512
num_layers:6
num_heads:8
expand_ratio:4
post_layernorm_residual:False
dropout_rate:0.1
compute_dtype:Float16
use_past:False
dp:0
mp:4
self_layernorm:True
forward_reduce_scatter:True
stage_num:1
micro_size:16
word_emb_dp:True
eod_reset:False
load_ckpt_path:None
==========
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.269.472 [mindspore/common/_decorator.py:33] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.276.926 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.278.269 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.281.198 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.282.496 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.287.459 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.290.675 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-7-4e527a9a05d2> in <module>
----> 1 run_train(args_opt)
~/work/pangu_alpha_train.py in run_train(args_opt)
341 run_train_pipeline(args_opt)
342 else:
--> 343 run_train_no_pipeline(args_opt)
344
345
~/work/pangu_alpha_train.py in run_train_no_pipeline(args_opt)
271 word_emb_dp=True)
272 logger.info(f"===config is: {config}")
--> 273 pangu_alpha = PANGUALPHA(config)
274 loss = CrossEntropyLoss(config)
275 pangu_alpha_with_loss = PANGUALPHAWithLoss(config, pangu_alpha, loss)
~/work/pangu_alpha.py in __init__(self, config)
897 def __init__(self, config):
898 super(PANGUALPHA, self).__init__()
--> 899 self.backbone = PANGUALPHA_Model(config)
900 self.head = PANGUALPHA_Head(config)
901
~/work/pangu_alpha.py in __init__(self, config)
732
733 for i in range(num_layers):
--> 734 per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2)
735 per_block.recompute()
736 per_block.attention.dropout.dropout_gen_mask.recompute(False)
~/work/pangu_alpha.py in __init__(self, config, layer_idx)
458 self.layernorm2.layer_norm.shard(((config.dp, 1, 1), (1,), (1,)))
459
--> 460 self.attention = Attention(config, scale, layer_idx)
461 self.output = Output(config, scale)
462 self.post_layernorm_residual = config.post_layernorm_residual
~/work/pangu_alpha.py in __init__(self, config, scale, layer_idx)
284 self.use_past = config.use_past
285 self.dropout = nn.Dropout(1 - config.dropout_rate)
--> 286 self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),))
287 self.dropout.dropout_do_mask.shard(((config.dp, 1, 1),))
288 self.prob_dropout = nn.Dropout(1 - config.dropout_rate)
~/miniconda3/envs/MindSpore-python3.7-aarch64/lib/python3.7/site-packages/mindspore/nn/cell.py in __getattr__(self, name)
286 para_list = ParameterTuple(cast_list)
287 return para_list
--> 288 raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).__name__, name))
289
290 def __del__(self):
AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.
```
模型: 2.6B
mindspore: 1.2
GPU: NVIDA V100
(pangu) root@iZ2ze15skiya5kcq1eryyuZ:/mnt/pangu/pangu-alpha# ./scripts/infer.sh
rank_id:0 rank_id str:0
local_rank:0, device id:0 start to run...
===config is: [PANGUALPHAConfig]==============================
batch_size:0
seq_length:1024
vocab_size:40000
embedding_size:2560
num_layers:32
num_heads:32
expand_ratio:4
post_layernorm_residual:False
dropout_rate:0.1
compute_dtype:Float16
use_past:False
dp:0
mp:8
self_layernorm:True
forward_reduce_scatter:True
stage_num:1
micro_size:1
word_emb_dp:True
eod_reset:False
load_ckpt_path:/mnt/pangu/pangu-alpha/checkpoint_file
=====args_opt is: Namespace(data_url=None, device_id=0, device_num=1, distribute='false', embedding_size=2560, load_ckpt_name='PANGUALPHA3.ckpt', load_ckpt_path='/mnt/pangu/pangu-alpha/checkpoint_file', micro_size=1, mode='2.6B', num_heads=32, num_layers=32, per_batch_size=1, run_type='predict', seq_length=1024, stage_num=1, strategy_load_ckpt_path='/mnt/pangu/pangu-alpha/strategy_load_ckpt/pangu_alpha_2.6B_ckpt_strategy.ckpt', tensor_model_parallel_num=8, tokenizer_path='/mnt/pangu/pangu-alpha/tokenizer/', train_url=None, vocab_size=40000)
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.601.022 [mindspore/common/_decorator.py:32] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.642.878 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.643.323 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.644.994 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.645.344 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.843.766 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.845.435 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
Traceback (most recent call last):
File "/mnt/pangu/pangu-alpha/run_pangu_alpha_predict.py", line 92, in
run_predict(args_opt)
File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 228, in run_predict
run_predict_no_pipeline(args_opt)
File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 195, in run_predict_no_pipeline
pangu_alpha = PANGUALPHA(config)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 886, in init
self.backbone = PANGUALPHA_Model(config)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 720, in init
per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 459, in init
self.attention = Attention(config, scale, layer_idx)
File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 285, in init
self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),))
File "/mnt/anaconda3/envs/pangu/lib/python3.7/site-packages/mindspore/nn/cell.py", line 266, in getattr
raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).name, name))
AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.
目前只支持在Ascend设备上运行,请查看具体环境要求
Hi,我在Ascend上运行也会报这个错:'Dropout' object has no attribute 'dropout_gen_mask'
dropout_gen_mask和dropout_do_mask接口在ascend-mindspore1.3中均统一至dropout,麻烦您修改相关代码!