|
- Log file created at: 2021/12/02 09:40:06
- Running on machine: notebook-a4290948-256e-4acd-b052-bf1559e42e09
- Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
- [WARNING] ANALYZER(70859,ffffa8242190,python):2021-12-02-09:40:06.831.602 [mindspore/ccsrc/pipeline/jit/static_analysis/program_specialize.cc:607] BuildFromBroadedArgsVal] Cannot find joined argvals in cache, run with broaded argsvals: 1, [const vector][AbstractTensor(shape: (), element: AbstractScalar(Type: Float32, Value: AnyValue, Shape: NoShape), value_ptr: 0xaaaaec220520, value: AnyValue)]
- [WARNING] ANALYZER(70859,ffffa8242190,python):2021-12-02-09:40:06.880.236 [mindspore/ccsrc/pipeline/jit/static_analysis/program_specialize.cc:607] BuildFromBroadedArgsVal] Cannot find joined argvals in cache, run with broaded argsvals: 1, [const vector][AbstractTensor(shape: (), element: AbstractScalar(Type: Float32, Value: AnyValue, Shape: NoShape), value_ptr: 0xaaaaec220520, value: AnyValue)]
- [WARNING] CORE(70859,ffffa8242190,python):2021-12-02-09:40:18.930.548 [mindspore/core/ir/anf_extends.cc:62] fullname_with_scope] Input 0 of cnode is not a value node, its type is CNode.
- [ERROR] COMMON(70859,fffe03fff1e0,python):2021-12-02-09:40:36.318.949 [mindspore/ccsrc/common/duplex_pipe.cc:87] Write] [DuplexPipe] write failed, errno: 32
- [ERROR] SESSION(70859,fffe03fff1e0,python):2021-12-02-09:40:36.319.462 [mindspore/ccsrc/backend/session/kernel_build_client.h:104] Response] Try to get response before Open()
- [WARNING] MD(70859,ffffa8242190,python):2021-12-02-09:40:44.136.420 [mindspore/ccsrc/minddata/dataset/util/task.cc:159] Join] DeviceQueueOp(ID:0) Thread ID 281466957181408 is not responding. Interrupt again
- [WARNING] MD(70859,ffffa8242190,python):2021-12-02-09:40:45.136.744 [mindspore/ccsrc/minddata/dataset/util/task.cc:159] Join] DeviceQueueOp(ID:0) Thread ID 281466957181408 is not responding. Interrupt again
- [WARNING] MD(70859,ffffa8242190,python):2021-12-02-09:40:46.137.049 [mindspore/ccsrc/minddata/dataset/util/task.cc:159] Join] DeviceQueueOp(ID:0) Thread ID 281466957181408 is not responding. Interrupt again
- [WARNING] MD(70859,ffffa8242190,python):2021-12-02-09:40:47.137.351 [mindspore/ccsrc/minddata/dataset/util/task.cc:159] Join] DeviceQueueOp(ID:0) Thread ID 281466957181408 is not responding. Interrupt again
- [WARNING] MD(70859,ffffa8242190,python):2021-12-02-09:40:48.137.642 [mindspore/ccsrc/minddata/dataset/util/task.cc:159] Join] DeviceQueueOp(ID:0) Thread ID 281466957181408 is not responding. Interrupt again
- [WARNING] MD(70859,ffffa8242190,python):2021-12-02-09:40:49.137.928 [mindspore/ccsrc/minddata/dataset/util/task.cc:159] Join] DeviceQueueOp(ID:0) Thread ID 281466957181408 is not responding. Interrupt again
- [WARNING] MD(70859,ffffa8242190,python):2021-12-02-09:40:49.138.109 [mindspore/ccsrc/minddata/dataset/util/task.cc:165] Join] Wait 6 seconds, the task: DeviceQueueOp(ID:0) will be destroyed by TdtHostDestory.
- [ERROR] MD(70859,fffe21ffb1e0,python):2021-12-02-09:40:49.167.483 [mindspore/ccsrc/minddata/dataset/util/task_manager.cc:217] InterruptMaster] Task is terminated with err msg(more detail in info level log):TDT Push data into device Failed, check the first error or TraceBack first, following are several possible checking way: 1) if training is not ready, still in network graph compiling stage, check error raised by Network used operator or environment configuration. 2) if interrupt in middle process of training, may check whether dataset sending num and network training num mismatch. 3) if this error raised in end of training, ignore this. 4) other cases, try find ascend host log or checking info log ects or search this in mindspore's FAQ.
- [WARNING] MD(70859,ffffa8242190,python):2021-12-02-09:40:49.238.125 [mindspore/ccsrc/minddata/dataset/engine/datasetops/device_queue_op.cc:73] ~DeviceQueueOp] preprocess_batch: 48; batch_queue: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0; push_start_time: 2021-12-02-09:39:37.396.060, 2021-12-02-09:39:37.817.748, 2021-12-02-09:39:38.030.110, 2021-12-02-09:39:38.143.866, 2021-12-02-09:39:38.349.215, 2021-12-02-09:39:38.459.539, 2021-12-02-09:39:38.721.484, 2021-12-02-09:39:38.871.583, 2021-12-02-09:39:39.049.211, 2021-12-02-09:39:39.104.172; push_end_time: 2021-12-02-09:39:37.181.839, 2021-12-02-09:39:37.404.139, 2021-12-02-09:39:37.825.709, 2021-12-02-09:39:38.038.322, 2021-12-02-09:39:38.152.015, 2021-12-02-09:39:38.357.974, 2021-12-02-09:39:38.467.561, 2021-12-02-09:39:38.725.727, 2021-12-02-09:39:38.879.722, 2021-12-02-09:39:39.054.525.
|