#7 NVIDIA V100 单卡运行推理, AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.

Closed
created 2 years ago by xuanzhanwei · 3 comments
模型: 2.6B mindspore: 1.2 GPU: NVIDA V100 (pangu) root@iZ2ze15skiya5kcq1eryyuZ:/mnt/pangu/pangu-alpha# ./scripts/infer.sh rank_id:0 rank_id str:0 local_rank:0, device id:0 start to run... ===config is: [PANGUALPHAConfig]============================== batch_size:0 seq_length:1024 vocab_size:40000 embedding_size:2560 num_layers:32 num_heads:32 expand_ratio:4 post_layernorm_residual:False dropout_rate:0.1 compute_dtype:Float16 use_past:False dp:0 mp:8 self_layernorm:True forward_reduce_scatter:True stage_num:1 micro_size:1 word_emb_dp:True eod_reset:False load_ckpt_path:/mnt/pangu/pangu-alpha/checkpoint_file ========== =====args_opt is: Namespace(data_url=None, device_id=0, device_num=1, distribute='false', embedding_size=2560, load_ckpt_name='PANGUALPHA3.ckpt', load_ckpt_path='/mnt/pangu/pangu-alpha/checkpoint_file', micro_size=1, mode='2.6B', num_heads=32, num_layers=32, per_batch_size=1, run_type='predict', seq_length=1024, stage_num=1, strategy_load_ckpt_path='/mnt/pangu/pangu-alpha/strategy_load_ckpt/pangu_alpha_2.6B_ckpt_strategy.ckpt', tensor_model_parallel_num=8, tokenizer_path='/mnt/pangu/pangu-alpha/tokenizer/', train_url=None, vocab_size=40000) [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.601.022 [mindspore/common/_decorator.py:32] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.642.878 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.643.323 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.644.994 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.645.344 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.843.766 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(24520:140708633995072,MainProcess):2021-04-28-16:12:46.845.435 [mindspore/common/_decorator.py:32] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. Traceback (most recent call last): File "/mnt/pangu/pangu-alpha/run_pangu_alpha_predict.py", line 92, in <module> run_predict(args_opt) File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 228, in run_predict run_predict_no_pipeline(args_opt) File "/mnt/pangu/pangu-alpha/pangu_alpha_predict.py", line 195, in run_predict_no_pipeline pangu_alpha = PANGUALPHA(config) File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 886, in __init__ self.backbone = PANGUALPHA_Model(config) File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 720, in __init__ per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2) File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 459, in __init__ self.attention = Attention(config, scale, layer_idx) File "/mnt/pangu/pangu-alpha/pangu_alpha.py", line 285, in __init__ self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),)) File "/mnt/anaconda3/envs/pangu/lib/python3.7/site-packages/mindspore/nn/cell.py", line 266, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).__name__, name)) AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'.
taoht commented 2 years ago
Owner
目前只支持在Ascend设备上运行,请查看具体环境要求
xuanzhanwei closed this issue 2 years ago
Hi,我在Ascend上运行也会报这个错:'Dropout' object has no attribute 'dropout_gen_mask' ``` INFO:Training model with standard mode: INFO: - local_rank:0, device id:0 start to run... INFO:Distributed Training: device_id is 0, rank_id is 0, device_num is 1 INFO:===config is: [PANGUALPHAConfig]============================== batch_size:4 seq_length:256 vocab_size:15928 embedding_size:512 num_layers:6 num_heads:8 expand_ratio:4 post_layernorm_residual:False dropout_rate:0.1 compute_dtype:Float16 use_past:False dp:0 mp:4 self_layernorm:True forward_reduce_scatter:True stage_num:1 micro_size:16 word_emb_dp:True eod_reset:False load_ckpt_path:None ========== [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.269.472 [mindspore/common/_decorator.py:33] 'GatherV2' is deprecated from version 1.1 and will be removed in a future version, use 'Gather' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.276.926 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.278.269 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.281.198 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.282.496 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.287.459 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. [WARNING] ME(850:281473686031968,MainProcess):2021-11-24-06:18:15.290.675 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead. --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-7-4e527a9a05d2> in <module> ----> 1 run_train(args_opt) ~/work/pangu_alpha_train.py in run_train(args_opt) 341 run_train_pipeline(args_opt) 342 else: --> 343 run_train_no_pipeline(args_opt) 344 345 ~/work/pangu_alpha_train.py in run_train_no_pipeline(args_opt) 271 word_emb_dp=True) 272 logger.info(f"===config is: {config}") --> 273 pangu_alpha = PANGUALPHA(config) 274 loss = CrossEntropyLoss(config) 275 pangu_alpha_with_loss = PANGUALPHAWithLoss(config, pangu_alpha, loss) ~/work/pangu_alpha.py in __init__(self, config) 897 def __init__(self, config): 898 super(PANGUALPHA, self).__init__() --> 899 self.backbone = PANGUALPHA_Model(config) 900 self.head = PANGUALPHA_Head(config) 901 ~/work/pangu_alpha.py in __init__(self, config) 732 733 for i in range(num_layers): --> 734 per_block = Block(config, i + 1).set_comm_fusion(int(i / fusion_group_size) + 2) 735 per_block.recompute() 736 per_block.attention.dropout.dropout_gen_mask.recompute(False) ~/work/pangu_alpha.py in __init__(self, config, layer_idx) 458 self.layernorm2.layer_norm.shard(((config.dp, 1, 1), (1,), (1,))) 459 --> 460 self.attention = Attention(config, scale, layer_idx) 461 self.output = Output(config, scale) 462 self.post_layernorm_residual = config.post_layernorm_residual ~/work/pangu_alpha.py in __init__(self, config, scale, layer_idx) 284 self.use_past = config.use_past 285 self.dropout = nn.Dropout(1 - config.dropout_rate) --> 286 self.dropout.dropout_gen_mask.shard(((config.dp, 1, 1),)) 287 self.dropout.dropout_do_mask.shard(((config.dp, 1, 1),)) 288 self.prob_dropout = nn.Dropout(1 - config.dropout_rate) ~/miniconda3/envs/MindSpore-python3.7-aarch64/lib/python3.7/site-packages/mindspore/nn/cell.py in __getattr__(self, name) 286 para_list = ParameterTuple(cast_list) 287 return para_list --> 288 raise AttributeError("'{}' object has no attribute '{}'.".format(type(self).__name__, name)) 289 290 def __del__(self): AttributeError: 'Dropout' object has no attribute 'dropout_gen_mask'. ```
imyzx commented 2 years ago
Owner
dropout_gen_mask和dropout_do_mask接口在ascend-mindspore1.3中均统一至dropout,麻烦您修改相关代码!
Sign in to join this conversation.
No Label
No Milestone
No Assignees
4 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.