启智社区最近又发版了,新增模型推理功能,欢迎大家体验>>>

#7 NVIDIA V100 单卡运行推理, AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.

Closed
created 8 months ago by xuanzhanwei · 3 comments

模型: 2.6B
mindspore: 1.2
GPU: NVIDA V100

(pangu) root@iZ2ze15skiya5kcq1eryyuZ:/mnt/pangu/pangu-alpha# ./scripts/infer.sh
rank_id:0 rank_id str:0
local_rank:0, device id:0 start to run...
===config is: [PANGUALPHAConfig]==============================
batch_size:0
seq_length:1024
vocab_size:40000
embedding_size:2560
num_layers:32
num_heads:32
expand_ratio:4
post_layernorm_residual:False
dropout_rate:0.1
compute_dtype:Float16
use_past:False
dp:0
mp:8
self_layernorm:True
forward_reduce_scatter:True
stage_num:1
micro_size:1
word_emb_dp:True
eod_reset:False
load_ckpt_path:/mnt/pangu/pangu-alpha/checkpoint_file

=====args_opt is: Namespace(data_url=None, device_id=0, device_num=1, distribute='false’, embedding_size=2560, load_ckpt_name='PANGUALPHA3.ckpt’, load_ckpt_path=’/mnt/pangu/pangu-alpha/checkpoint_file’, micro_size=1, mode='2.6B’, num_heads=32, num_layers=32, per_batch_size=1, run_type='predict’, seq_length=1024, stage_num=1, strategy_load_ckpt_path=’/mnt/pangu/pangu-alpha/strategy_load_ckpt/pangu_alpha_2.6B_ckpt_strategy.ckpt’, tensor_model_parallel_num=8, tokenizer_path=’/mnt/pangu/pangu-alpha/tokenizer/', train_url=None, vocab_size=40000)
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.601.022 [mindspore/common/_decorator.py:32] ‘GatherV2’ is deprecated from version 1.1 and will be removed in a future version, use ‘Gather’ instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.642.878 [mindspore/common/_decorator.py:32] ‘TensorAdd’ is deprecated from version 1.1 and will be removed in a future version, use ‘Add’ instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.643.323 [mindspore/common/_decorator.py:32] ‘TensorAdd’ is deprecated from version 1.1 and will be removed in a future version, use ‘Add’ instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.644.994 [mindspore/common/_decorator.py:32] ‘TensorAdd’ is deprecated from version 1.1 and will be removed in a future version, use ‘Add’ instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.645.344 [mindspore/common/_decorator.py:32] ‘TensorAdd’ is deprecated from version 1.1 and will be removed in a future version, use ‘Add’ instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.843.766 [mindspore/common/_decorator.py:32] ‘TensorAdd’ is deprecated from version 1.1 and will be removed in a future version, use ‘Add’ instead.
[WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.845.435 [mindspore/common/_decorator.py:32] ‘TensorAdd’ is deprecated from version 1.1 and will be removed in a future version, use ‘Add’ instead.
Traceback (most recent call last):
File “/mnt/pangu/pangu-alpha/run_pangu_alpha_predict.py”, line 92, in
run_predict(args_opt)
File “/mnt/pangu/pangu-alpha/pangu_alpha_predict.py”, line 228, in run_predict
run_predict_no_pipeline(args_opt)
File “/mnt/pangu/pangu-alpha/pangu_alpha_predict.py”, line 195, in run_predict_no_pipeline
pangu_alpha = PANGUALPHA(config)
File “/mnt/pangu/pangu-alpha/pangu_alpha.py”, line 886, in init
self.backbone = PANGUALPHA_Model(config)
File “/mnt/pangu/pangu-alpha/pangu_alpha.py”, line 720, in init
per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2)
File “/mnt/pangu/pangu-alpha/pangu_alpha.py”, line 459, in init
self.attention = Attention(config, scale, layer_idx)
File “/mnt/pangu/pangu-alpha/pangu_alpha.py”, line 285, in init
self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),))
File “/mnt/anaconda3/envs/pangu/lib/python3.7/site-packages/mindspore/nn/cell.py”, line 266, in getattr
raise AttributeError("'{}’ object has no attribute ‘{}'.".format(type(self).name, name))
AttributeError: ‘Dropout’ object has no attribute ‘dropout_gen_mask’.

模型: 2.6B mindspore: 1.2 GPU: NVIDA V100 (pangu) root@iZ2ze15skiya5kcq1eryyuZ:/mnt/pangu/pangu-alpha# ./scripts/infer.sh rank_id:0 rank_id str:0 local_rank:0, device id:0 start to run... ===config is: [PANGUALPHAConfig]============================== batch_size:0 seq_length:1024 vocab_size:40000 embedding_size:2560 num_layers:32 num_heads:32 expand_ratio:4 post_layernorm_residual:False dropout_rate:0.1 compute_dtype:Float16 use_past:False dp:0 mp:8 self_layernorm:True forward_reduce_scatter:True stage_num:1 micro_size:1 word_emb_dp:True eod_reset:False load_ckpt_path:/mnt/pangu/pangu-alpha/checkpoint_file ========== =====args_opt is: Namespace(data_url=None, device_id=0, device_num=1, distribute='false', embedding_size=2560, load_ckpt_name='PANGUALPHA3.ckpt', load_ckpt_path='/mnt/pangu/pangu-alpha/checkpoint_file', micro_size=1, mode='2.6B', num_heads=32, num_layers=32, per_batch_size=1, run_type='predict', seq_length=1024, stage_num=1, strategy_load_ckpt_path='/mnt/pangu/pangu-alpha/strategy_load_ckpt/pangu_alpha_2.6B_ckpt_strategy.ckpt', tensor_model_parallel_num=8, tokenizer_path='/mnt/pangu/pangu-alpha/tokenizer/', train_url=None, vocab_size=40000) [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.601.022 [mindspore/common/_decorator.py:32] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.642.878 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.643.323 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.644.994 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.645.344 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.843.766 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.845.435 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. Traceback (most recent call last): File "/mnt/pangu/pangu-alpha/run_pangu_alpha_predict.py", line 92, in <module> run_predict(args_opt) File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 228, in run_predict run_predict_no_pipeline(args_opt) File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 195, in run_predict_no_pipeline pangu_alpha = PANGUALPHA(config) File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 886, in __init__ self.backbone = PANGUALPHA_Model(config) File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 720, in __init__ per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2) File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 459, in __init__ self.attention = Attention(config, scale, layer_idx) File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 285, in __init__ self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),)) File "/mnt/anaconda3/envs/pangu/lib/python3.7/site-packages/mindspore/nn/cell.py", line 266, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).__name__, name)) AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.
taoht commented 8 months ago
Owner

目前只支持在Ascend设备上运行,请查看具体环境要求

目前只支持在Ascend设备上运行,请查看具体环境要求
xuanzhanwei closed this issue 8 months ago

Hi,我在Ascend上运行也会报这个错:‘Dropout’ object has no attribute ‘dropout_gen_mask’

INFO:Training model with standard mode:
INFO:	- local_rank:0, device id:0 start to run...
INFO:Distributed Training: device_id is 0, rank_id is 0, device_num is 1
INFO:===config is: [PANGUALPHAConfig]==============================
batch_size:4
seq_length:256
vocab_size:15928
embedding_size:512
num_layers:6
num_heads:8
expand_ratio:4
post_layernorm_residual:False
dropout_rate:0.1
compute_dtype:Float16
use_past:False
dp:0
mp:4
self_layernorm:True
forward_reduce_scatter:True
stage_num:1
micro_size:16
word_emb_dp:True
eod_reset:False
load_ckpt_path:None
==========
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.269.472 [mindspore/common/_decorator.py:33] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.276.926 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.278.269 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.281.198 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.282.496 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.287.459 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.290.675 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-4e527a9a05d2> in <module>
----> 1 run_train(args_opt)

~/work/pangu_alpha_train.py in run_train(args_opt)
    341         run_train_pipeline(args_opt)
    342     else:
--> 343         run_train_no_pipeline(args_opt)
    344 
    345 

~/work/pangu_alpha_train.py in run_train_no_pipeline(args_opt)
    271         word_emb_dp=True)
    272     logger.info(f"===config is: {config}")
--> 273     pangu_alpha = PANGUALPHA(config)
    274     loss = CrossEntropyLoss(config)
    275     pangu_alpha_with_loss = PANGUALPHAWithLoss(config, pangu_alpha, loss)

~/work/pangu_alpha.py in __init__(self, config)
    897     def __init__(self, config):
    898         super(PANGUALPHA, self).__init__()
--> 899         self.backbone = PANGUALPHA_Model(config)
    900         self.head = PANGUALPHA_Head(config)
    901 

~/work/pangu_alpha.py in __init__(self, config)
    732 
    733         for i in range(num_layers):
--> 734             per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2)
    735             per_block.recompute()
    736             per_block.attention.dropout.dropout_gen_mask.recompute(False)

~/work/pangu_alpha.py in __init__(self, config, layer_idx)
    458             self.layernorm2.layer_norm.shard(((config.dp, 1, 1), (1,), (1,)))
    459 
--> 460         self.attention = Attention(config, scale, layer_idx)
    461         self.output = Output(config, scale)
    462         self.post_layernorm_residual = config.post_layernorm_residual

~/work/pangu_alpha.py in __init__(self, config, scale, layer_idx)
    284         self.use_past = config.use_past
    285         self.dropout = nn.Dropout(1 - config.dropout_rate)
--> 286         self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),))
    287         self.dropout.dropout_do_mask.shard(((config.dp, 1, 1),))
    288         self.prob_dropout = nn.Dropout(1 - config.dropout_rate)

~/miniconda3/envs/MindSpore-python3.7-aarch64/lib/python3.7/site-packages/mindspore/nn/cell.py in __getattr__(self, name)
    286                 para_list = ParameterTuple(cast_list)
    287                 return para_list
--> 288         raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).__name__, name))
    289 
    290     def __del__(self):

AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.
Hi,我在Ascend上运行也会报这个错:'Dropout' object has no attribute 'dropout_gen_mask' ``` INFO:Training model with standard mode: INFO: - local_rank:0, device id:0 start to run... INFO:Distributed Training: device_id is 0, rank_id is 0, device_num is 1 INFO:===config is: [PANGUALPHAConfig]============================== batch_size:4 seq_length:256 vocab_size:15928 embedding_size:512 num_layers:6 num_heads:8 expand_ratio:4 post_layernorm_residual:False dropout_rate:0.1 compute_dtype:Float16 use_past:False dp:0 mp:4 self_layernorm:True forward_reduce_scatter:True stage_num:1 micro_size:16 word_emb_dp:True eod_reset:False load_ckpt_path:None ========== [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.269.472 [mindspore/common/_decorator.py:33] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.276.926 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.278.269 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.281.198 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.282.496 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.287.459 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.290.675 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-7-4e527a9a05d2> in <module> ----> 1 run_train(args_opt) ~/work/pangu_alpha_train.py in run_train(args_opt) 341 run_train_pipeline(args_opt) 342 else: --> 343 run_train_no_pipeline(args_opt) 344 345 ~/work/pangu_alpha_train.py in run_train_no_pipeline(args_opt) 271 word_emb_dp=True) 272 logger.info(f"===config is: {config}") --> 273 pangu_alpha = PANGUALPHA(config) 274 loss = CrossEntropyLoss(config) 275 pangu_alpha_with_loss = PANGUALPHAWithLoss(config, pangu_alpha, loss) ~/work/pangu_alpha.py in __init__(self, config) 897 def __init__(self, config): 898 super(PANGUALPHA, self).__init__() --> 899 self.backbone = PANGUALPHA_Model(config) 900 self.head = PANGUALPHA_Head(config) 901 ~/work/pangu_alpha.py in __init__(self, config) 732 733 for i in range(num_layers): --> 734 per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2) 735 per_block.recompute() 736 per_block.attention.dropout.dropout_gen_mask.recompute(False) ~/work/pangu_alpha.py in __init__(self, config, layer_idx) 458 self.layernorm2.layer_norm.shard(((config.dp, 1, 1), (1,), (1,))) 459 --> 460 self.attention = Attention(config, scale, layer_idx) 461 self.output = Output(config, scale) 462 self.post_layernorm_residual = config.post_layernorm_residual ~/work/pangu_alpha.py in __init__(self, config, scale, layer_idx) 284 self.use_past = config.use_past 285 self.dropout = nn.Dropout(1 - config.dropout_rate) --> 286 self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),)) 287 self.dropout.dropout_do_mask.shard(((config.dp, 1, 1),)) 288 self.prob_dropout = nn.Dropout(1 - config.dropout_rate) ~/miniconda3/envs/MindSpore-python3.7-aarch64/lib/python3.7/site-packages/mindspore/nn/cell.py in __getattr__(self, name) 286 para_list = ParameterTuple(cast_list) 287 return para_list --> 288 raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).__name__, name)) 289 290 def __del__(self): AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'. ```
imyzx commented 1 month ago
Owner

dropout_gen_mask和dropout_do_mask接口在ascend-mindspore1.3中均统一至dropout,麻烦您修改相关代码!

dropout_gen_mask和dropout_do_mask接口在ascend-mindspore1.3中均统一至dropout,麻烦您修改相关代码!
Sign in to join this conversation.
No Label
No Milestone
No Assignees
4 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.