|
- train dataset length is: 16
- [WARNING] ME(69851:281473593811344,MainProcess):2021-11-15-03:56:34.518.231 [mindspore/nn/loss/loss.py:103] '_Loss' is deprecated from version 1.3 and will be removed in a future version, use 'LossBase' instead.
- ## epoch start ##
- [WARNING] ME(70521:281473593811344,ForkPoolWorker-48):2021-11-15-03:56:38.372.116 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70518:281473593811344,ForkPoolWorker-45):2021-11-15-03:56:38.438.425 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70515:281473593811344,ForkPoolWorker-42):2021-11-15-03:56:38.504.909 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70511:281473593811344,ForkPoolWorker-38):2021-11-15-03:56:38.556.414 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70510:281473593811344,ForkPoolWorker-37):2021-11-15-03:56:38.620.571 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70512:281473593811344,ForkPoolWorker-39):2021-11-15-03:56:38.749.153 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70516:281473593811344,ForkPoolWorker-43):2021-11-15-03:56:38.751.377 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70513:281473593811344,ForkPoolWorker-40):2021-11-15-03:56:39.515.614 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70514:281473593811344,ForkPoolWorker-41):2021-11-15-03:56:39.866.275 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70519:281473593811344,ForkPoolWorker-46):2021-11-15-03:56:39.948.266 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70517:281473593811344,ForkPoolWorker-44):2021-11-15-03:56:40.105.467 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70520:281473593811344,ForkPoolWorker-47):2021-11-15-03:56:40.230.612 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70518:281473593811344,ForkPoolWorker-45):2021-11-15-03:56:42.485.965 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70521:281473593811344,ForkPoolWorker-48):2021-11-15-03:56:43.887.170 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-03:56:54.285.51 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-03:56:54.387.48 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-03:56:54.633.94 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-03:56:54.147.074 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-03:56:54.153.801 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-03:56:54.224.584 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-03:56:54.289.574 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-03:56:55.272.507 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-03:56:55.445.934 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-03:56:55.828.086 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-03:56:55.909.664 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-03:56:56.189.88 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-03:56:57.867.545 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-03:56:58.107.494 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-03:56:58.662.294 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-03:56:59.555.87 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-03:56:59.634.588 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] SESSION(69851,fffe3f7fe1e0,python3):2021-11-15-03:57:07.884.684 [mindspore/ccsrc/backend/session/ascend_session.cc:1377] SelectKernel] There are 17 node/nodes used raise precision to selected the kernel!
- epoch: 1 step: 16, loss is 2.1121953
- epoch time: 158266.882 ms, per step time: 9891.680 ms
- ## epoch start ##
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-03:59:13.638.155 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-03:59:14.185.439 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-03:59:14.529.161 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-03:59:14.607.424 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-03:59:14.682.194 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-03:59:14.781.950 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-03:59:15.644.33 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-03:59:15.594.871 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-03:59:16.192.930 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 2 step: 16, loss is 2.0301929
- epoch time: 5063.594 ms, per step time: 316.475 ms
- ## epoch start ##
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-03:59:18.617.458 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-03:59:18.767.951 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-03:59:19.766.704 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 3 step: 16, loss is 1.9224644
- epoch time: 4719.864 ms, per step time: 294.992 ms
- ## epoch start ##
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-03:59:24.105.897 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 4 step: 16, loss is 1.7708204
- epoch time: 4697.225 ms, per step time: 293.577 ms
- ## epoch start ##
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-03:59:27.627.677 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-03:59:28.452.200 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 5 step: 16, loss is 1.5607431
- epoch time: 2959.202 ms, per step time: 184.950 ms
- ## epoch start ##
- epoch: 6 step: 16, loss is 1.4205363
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-03:59:35.355.801 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-03:59:35.515.526 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 5413.691 ms, per step time: 338.356 ms
- ## epoch start ##
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-03:59:35.887.825 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-03:59:35.927.454 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 7 step: 16, loss is 1.2467297
- epoch time: 7627.032 ms, per step time: 476.689 ms
- ## epoch start ##
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-03:59:44.709.213 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-03:59:44.762.242 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 8 step: 16, loss is 1.2085274
- epoch time: 4063.034 ms, per step time: 253.940 ms
- ## epoch start ##
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-03:59:51.448.2 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-03:59:51.180.098 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 9 step: 16, loss is 0.89852846
- epoch time: 6266.387 ms, per step time: 391.649 ms
- ## epoch start ##
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-03:59:54.495.327 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-03:59:55.202.891 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 10 step: 16, loss is 0.7565096
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-03:59:55.816.822 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 2618.863 ms, per step time: 163.679 ms
- ## epoch start ##
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-03:59:56.763.930 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-03:59:59.366.773 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-03:59:59.482.154 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 11 step: 16, loss is 0.7589711
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-03:59:59.637.106 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 5239.403 ms, per step time: 327.463 ms
- ## epoch start ##
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-04:00:02.615.72 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:00:02.257.419 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-04:00:03.363.317 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:00:05.148.999 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 12 step: 16, loss is 0.6773382
- epoch time: 5001.608 ms, per step time: 312.600 ms
- ## epoch start ##
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-04:00:07.455.574 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 13 step: 16, loss is 0.50971293
- epoch time: 3200.921 ms, per step time: 200.058 ms
- ## epoch start ##
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-04:00:11.284.149 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:00:11.442.196 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 14 step: 16, loss is 0.44407284
- epoch time: 3322.802 ms, per step time: 207.675 ms
- ## epoch start ##
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-04:00:14.283.801 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-04:00:14.646.396 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-04:00:14.727.407 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:00:14.743.780 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-04:00:16.795.211 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 15 step: 16, loss is 0.46908158
- epoch time: 5604.451 ms, per step time: 350.278 ms
- ## epoch start ##
- epoch: 16 step: 16, loss is 0.7655868
- epoch time: 4686.962 ms, per step time: 292.935 ms
- ## epoch start ##
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:00:23.451.285 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 17 step: 16, loss is 0.39310265
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:00:26.166.180 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-04:00:26.185.868 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 2768.790 ms, per step time: 173.049 ms
- ## epoch start ##
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-04:00:26.454.770 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 18 step: 16, loss is 0.3484941
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:00:29.937.117 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-04:00:30.134.892 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 5235.005 ms, per step time: 327.188 ms
- ## epoch start ##
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-04:00:33.105.857 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:00:34.158.509 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-04:00:34.712.971 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 19 step: 16, loss is 0.578479
- epoch time: 5225.246 ms, per step time: 326.578 ms
- ## epoch start ##
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-04:00:37.351.041 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-04:00:37.766.897 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:00:39.421.221 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:00:40.721.82 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 20 step: 16, loss is 0.48471653
- epoch time: 5439.597 ms, per step time: 339.975 ms
- ## epoch start ##
- epoch: 21 step: 16, loss is 0.29555073
- epoch time: 2727.660 ms, per step time: 170.479 ms
- ## epoch start ##
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:00:45.725.519 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-04:00:45.860.052 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-04:00:48.875.997 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 22 step: 16, loss is 0.25148422[WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:00:49.186.697 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
-
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-04:00:49.518.924 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-04:00:50.537.008 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 6291.191 ms, per step time: 393.199 ms
- ## epoch start ##
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-04:00:51.759.897 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-04:00:53.841.782 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 23 step: 16, loss is 0.22353278
- epoch time: 4059.356 ms, per step time: 253.710 ms
- ## epoch start ##
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:00:56.973.205 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-04:00:57.488.388 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:00:57.845.777 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-04:01:00.260.446 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-04:01:01.569.827 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 24 step: 16, loss is 0.6072957
- epoch time: 8670.223 ms, per step time: 541.889 ms
- ## epoch start ##
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-04:01:04.269.391 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-04:01:05.857.459 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:01:06.778.019 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 25 step: 16, loss is 0.18336233
- epoch time: 3978.614 ms, per step time: 248.663 ms
- ## epoch start ##
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-04:01:12.191.977 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 26 step: 16, loss is 0.27192593
- epoch time: 5900.774 ms, per step time: 368.798 ms
- ## epoch start ##
- epoch: 27 step: 16, loss is 0.64456326
- epoch time: 2376.215 ms, per step time: 148.513 ms
- ## epoch start ##
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:01:17.257.938 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-04:01:18.630.511 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 28 step: 16, loss is 0.2529453
- epoch time: 4481.776 ms, per step time: 280.111 ms
- ## epoch start ##
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-04:01:22.959.790 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 29 step: 16, loss is 0.43137172
- epoch time: 4900.215 ms, per step time: 306.263 ms
- ## epoch start ##
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-04:01:25.757.556 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-04:01:27.108.695 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 30 step: 16, loss is 0.27498126
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-04:01:27.652.971 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 3061.989 ms, per step time: 191.374 ms
- ## epoch start ##
- epoch: 31 step: 16, loss is 0.31642294
- epoch time: 6246.838 ms, per step time: 390.427 ms
- ## epoch start ##
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-04:01:35.548.963 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-04:01:35.787.596 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-04:01:35.992.791 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 32 step: 16, loss is 0.16016106
- epoch time: 3479.516 ms, per step time: 217.470 ms
- ## epoch start ##
- epoch: 33 step: 16, loss is 0.1726416
- epoch time: 4511.251 ms, per step time: 281.953 ms
- ## epoch start ##
- epoch: 34 step: 16, loss is 0.18398736
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-04:01:47.635.152 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 5326.006 ms, per step time: 332.875 ms
- ## epoch start ##
- epoch: 35 step: 16, loss is 0.23071945
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-04:01:51.696.874 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 3709.291 ms, per step time: 231.831 ms
- ## epoch start ##
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:01:52.715.863 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:01:54.711.760 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 36 step: 16, loss is 0.47783816
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-04:01:58.822.905 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 8352.385 ms, per step time: 522.024 ms
- ## epoch start ##
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-04:02:03.926.4 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 37 step: 16, loss is 0.6507052
- epoch time: 3601.112 ms, per step time: 225.069 ms
- ## epoch start ##
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-04:02:05.474.299 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 38 step: 16, loss is 0.87698126
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-04:02:08.862.890 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 5949.713 ms, per step time: 371.857 ms
- ## epoch start ##
- epoch: 39 step: 16, loss is 0.09047725
- epoch time: 1218.622 ms, per step time: 76.164 ms
- ## epoch start ##
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-04:02:12.375.526 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:02:14.176.533 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 40 step: 16, loss is 0.26150244
- epoch time: 4347.296 ms, per step time: 271.706 ms
- ## epoch start ##
- epoch: 41 step: 16, loss is 0.34678084
- epoch time: 4854.054 ms, per step time: 303.378 ms
- ## epoch start ##
- epoch: 42 step: 16, loss is 0.45514312
- epoch time: 2809.983 ms, per step time: 175.624 ms
- ## epoch start ##
- epoch: 43 step: 16, loss is 0.17431776
- epoch time: 5806.932 ms, per step time: 362.933 ms
- ## epoch start ##
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-04:02:29.388.373 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:02:30.474.683 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-04:02:30.517.872 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-04:02:30.577.648 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 44 step: 16, loss is 0.12957168[WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-04:02:31.862.936 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
-
- epoch time: 4061.733 ms, per step time: 253.858 ms
- ## epoch start ##
- epoch: 45 step: 16, loss is 0.45466778
- epoch time: 4870.551 ms, per step time: 304.409 ms
- ## epoch start ##
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-04:02:39.653.783 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 46 step: 16, loss is 0.27201018
- epoch time: 6135.321 ms, per step time: 383.458 ms
- ## epoch start ##
- epoch: 47 step: 16, loss is 0.24136896
- epoch time: 5566.408 ms, per step time: 347.900 ms
- ## epoch start ##
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-04:02:52.956.121 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 48 step: 16, loss is 0.057390522
- epoch time: 5837.539 ms, per step time: 364.846 ms
- ## epoch start ##
- epoch: 49 step: 16, loss is 0.20367375
- epoch time: 2849.762 ms, per step time: 178.110 ms
- ## epoch start ##
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-04:03:01.429.902 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-04:03:01.802.344 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:03:02.415.307 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 50 step: 16, loss is 0.7958206
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-04:03:04.248.97 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 6057.910 ms, per step time: 378.619 ms
- ## epoch start ##
- epoch: 51 step: 16, loss is 0.13677244
- epoch time: 1391.708 ms, per step time: 86.982 ms
- ## epoch start ##
- epoch: 52 step: 16, loss is 0.11924967
- epoch time: 4298.414 ms, per step time: 268.651 ms
- ## epoch start ##
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-04:03:10.640.712 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:03:11.961.146 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 53 step: 16, loss is 0.24753949
- epoch time: 6103.080 ms, per step time: 381.442 ms
- ## epoch start ##
- epoch: 54 step: 16, loss is 0.16768044
- epoch time: 3942.876 ms, per step time: 246.430 ms
- ## epoch start ##
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-04:03:23.981.47 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 55 step: 16, loss is 0.13550459
- epoch time: 6150.149 ms, per step time: 384.384 ms
- ## epoch start ##
- epoch: 56 step: 16, loss is 0.08188738
- epoch time: 3333.193 ms, per step time: 208.325 ms
- ## epoch start ##
- epoch: 57 step: 16, loss is 1.0381557
- epoch time: 2659.042 ms, per step time: 166.190 ms
- ## epoch start ##
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-04:03:33.422.16 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70893:281473593811344,ForkPoolWorker-61):2021-11-15-04:03:33.323.539 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-04:03:33.797.096 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-04:03:37.152.094 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 58 step: 16, loss is 0.69063514
- epoch time: 6803.190 ms, per step time: 425.199 ms
- ## epoch start ##
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-04:03:39.474.905 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:03:40.915.18 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 59 step: 16, loss is 0.36312252
- epoch time: 4442.744 ms, per step time: 277.672 ms
- ## epoch start ##
- epoch: 60 step: 16, loss is 0.15213734
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:03:50.235.580 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 8061.952 ms, per step time: 503.872 ms
- ## epoch start ##
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-04:03:52.977.276 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70889:281473593811344,ForkPoolWorker-57):2021-11-15-04:03:55.224.03 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 61 step: 16, loss is 0.70357925
- epoch time: 5081.133 ms, per step time: 317.571 ms
- ## epoch start ##
- [WARNING] ME(70887:281473593811344,ForkPoolWorker-55):2021-11-15-04:03:57.154.817 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 62 step: 16, loss is 0.6113806
- epoch time: 3291.211 ms, per step time: 205.701 ms
- ## epoch start ##
- epoch: 63 step: 16, loss is 0.18709968
- epoch time: 2673.332 ms, per step time: 167.083 ms
- ## epoch start ##
- epoch: 64 step: 16, loss is 0.09821298
- epoch time: 3061.861 ms, per step time: 191.366 ms
- ## epoch start ##
- epoch: 65 step: 16, loss is 0.04322882
- epoch time: 4188.973 ms, per step time: 261.811 ms
- ## epoch start ##
- [WARNING] ME(70896:281473593811344,ForkPoolWorker-64):2021-11-15-04:04:09.769.692 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 66 step: 16, loss is 0.31466237
- epoch time: 5258.301 ms, per step time: 328.644 ms
- ## epoch start ##
- [WARNING] ME(70895:281473593811344,ForkPoolWorker-63):2021-11-15-04:04:15.320.564 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-04:04:16.955.84 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 67 step: 16, loss is 0.31410116
- epoch time: 5236.601 ms, per step time: 327.288 ms
- ## epoch start ##
- [WARNING] ME(70885:281473593811344,ForkPoolWorker-53):2021-11-15-04:04:21.103.004 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- [WARNING] ME(70891:281473593811344,ForkPoolWorker-59):2021-11-15-04:04:22.580.966 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 68 step: 16, loss is 0.16892433
- epoch time: 4900.175 ms, per step time: 306.261 ms
- ## epoch start ##
- epoch: 69 step: 16, loss is 0.25871655
- [WARNING] ME(70894:281473593811344,ForkPoolWorker-62):2021-11-15-04:04:30.292.632 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch time: 5298.005 ms, per step time: 331.125 ms
- ## epoch start ##
- epoch: 70 step: 16, loss is 0.15960878
- epoch time: 4916.316 ms, per step time: 307.270 ms
- ## epoch start ##
- epoch: 71 step: 16, loss is 0.14576481
- epoch time: 4300.076 ms, per step time: 268.755 ms
- ## epoch start ##
- epoch: 72 step: 16, loss is 0.048070516
- epoch time: 5254.318 ms, per step time: 328.395 ms
- ## epoch start ##
- [WARNING] ME(70886:281473593811344,ForkPoolWorker-54):2021-11-15-04:04:47.291.898 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 73 step: 16, loss is 0.43805355
- epoch time: 4078.904 ms, per step time: 254.932 ms
- ## epoch start ##
- epoch: 74 step: 16, loss is 0.20705777
- epoch time: 8598.917 ms, per step time: 537.432 ms
- ## epoch start ##
- epoch: 75 step: 16, loss is 0.087900326
- epoch time: 3601.473 ms, per step time: 225.092 ms
- ## epoch start ##
- epoch: 76 step: 16, loss is 0.17150873
- epoch time: 3905.045 ms, per step time: 244.065 ms
- ## epoch start ##
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-04:05:06.779.60 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 77 step: 16, loss is 0.2378177
- epoch time: 2324.871 ms, per step time: 145.304 ms
- ## epoch start ##
- epoch: 78 step: 16, loss is 0.40027475
- epoch time: 3334.192 ms, per step time: 208.387 ms
- ## epoch start ##
- epoch: 79 step: 16, loss is 0.38522345
- epoch time: 7345.164 ms, per step time: 459.073 ms
- ## epoch start ##
- epoch: 80 step: 16, loss is 0.37516826
- epoch time: 4852.039 ms, per step time: 303.252 ms
- ## epoch start ##
- [WARNING] ME(70892:281473593811344,ForkPoolWorker-60):2021-11-15-04:05:23.346.967 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 81 step: 16, loss is 0.15315205
- epoch time: 6552.588 ms, per step time: 409.537 ms
- ## epoch start ##
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:05:30.351.274 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 82 step: 16, loss is 0.20525634
- epoch time: 3377.734 ms, per step time: 211.108 ms
- ## epoch start ##
- epoch: 83 step: 16, loss is 0.31164607
- epoch time: 4453.762 ms, per step time: 278.360 ms
- ## epoch start ##
- epoch: 84 step: 16, loss is 0.44258845
- epoch time: 6636.577 ms, per step time: 414.786 ms
- ## epoch start ##
- epoch: 85 step: 16, loss is 0.05534556
- epoch time: 1534.598 ms, per step time: 95.912 ms
- ## epoch start ##
- epoch: 86 step: 16, loss is 0.4599887
- epoch time: 7996.444 ms, per step time: 499.778 ms
- ## epoch start ##
- epoch: 87 step: 16, loss is 0.21285146
- epoch time: 3093.145 ms, per step time: 193.322 ms
- ## epoch start ##
- epoch: 88 step: 16, loss is 0.33244067
- epoch time: 3002.677 ms, per step time: 187.667 ms
- ## epoch start ##
- epoch: 89 step: 16, loss is 0.025271753
- epoch time: 2507.255 ms, per step time: 156.703 ms
- ## epoch start ##
- epoch: 90 step: 16, loss is 0.18199705
- epoch time: 3150.898 ms, per step time: 196.931 ms
- ## epoch start ##
- epoch: 91 step: 16, loss is 0.25922903
- epoch time: 6258.555 ms, per step time: 391.160 ms
- ## epoch start ##
- epoch: 92 step: 16, loss is 0.30310813
- epoch time: 2538.785 ms, per step time: 158.674 ms
- ## epoch start ##
- epoch: 93 step: 16, loss is 0.25664172
- epoch time: 6359.242 ms, per step time: 397.453 ms
- ## epoch start ##
- epoch: 94 step: 16, loss is 0.03523473
- epoch time: 5654.168 ms, per step time: 353.385 ms
- ## epoch start ##
- epoch: 95 step: 16, loss is 0.11133428
- epoch time: 3851.239 ms, per step time: 240.702 ms
- ## epoch start ##
- epoch: 96 step: 16, loss is 0.42098278
- epoch time: 8870.972 ms, per step time: 554.436 ms
- ## epoch start ##
- epoch: 97 step: 16, loss is 0.41124684
- epoch time: 5663.276 ms, per step time: 353.955 ms
- ## epoch start ##
- epoch: 98 step: 16, loss is 0.14253937
- epoch time: 7261.960 ms, per step time: 453.872 ms
- ## epoch start ##
- epoch: 99 step: 16, loss is 0.30959213
- epoch time: 4895.043 ms, per step time: 305.940 ms
- ## epoch start ##
- epoch: 100 step: 16, loss is 0.17766282
- epoch time: 4577.015 ms, per step time: 286.063 ms
- ## epoch start ##
- epoch: 101 step: 16, loss is 0.5119715
- epoch time: 1530.827 ms, per step time: 95.677 ms
- ## epoch start ##
- epoch: 102 step: 16, loss is 0.25580448
- epoch time: 1463.820 ms, per step time: 91.489 ms
- ## epoch start ##
- epoch: 103 step: 16, loss is 0.18950208
- epoch time: 7224.804 ms, per step time: 451.550 ms
- ## epoch start ##
- epoch: 104 step: 16, loss is 0.6553689
- epoch time: 3048.381 ms, per step time: 190.524 ms
- ## epoch start ##
- epoch: 105 step: 16, loss is 0.05219055
- epoch time: 4756.369 ms, per step time: 297.273 ms
- ## epoch start ##
- epoch: 106 step: 16, loss is 0.019803634
- epoch time: 4260.307 ms, per step time: 266.269 ms
- ## epoch start ##
- epoch: 107 step: 16, loss is 0.15288424
- epoch time: 2754.751 ms, per step time: 172.172 ms
- ## epoch start ##
- epoch: 108 step: 16, loss is 0.3362367
- epoch time: 5333.003 ms, per step time: 333.313 ms
- ## epoch start ##
- epoch: 109 step: 16, loss is 0.22804566
- epoch time: 2825.735 ms, per step time: 176.608 ms
- ## epoch start ##
- epoch: 110 step: 16, loss is 0.14764245
- epoch time: 4900.069 ms, per step time: 306.254 ms
- ## epoch start ##
- epoch: 111 step: 16, loss is 0.49019098
- epoch time: 2229.157 ms, per step time: 139.322 ms
- ## epoch start ##
- epoch: 112 step: 16, loss is 0.12158593
- epoch time: 3655.755 ms, per step time: 228.485 ms
- ## epoch start ##
- epoch: 113 step: 16, loss is 0.14602199
- epoch time: 4258.095 ms, per step time: 266.131 ms
- ## epoch start ##
- epoch: 114 step: 16, loss is 0.09077976
- epoch time: 4601.167 ms, per step time: 287.573 ms
- ## epoch start ##
- epoch: 115 step: 16, loss is 0.01772577
- epoch time: 7262.990 ms, per step time: 453.937 ms
- ## epoch start ##
- epoch: 116 step: 16, loss is 0.076904126
- epoch time: 4612.915 ms, per step time: 288.307 ms
- ## epoch start ##
- epoch: 117 step: 16, loss is 0.311674
- epoch time: 4489.161 ms, per step time: 280.573 ms
- ## epoch start ##
- epoch: 118 step: 16, loss is 0.1530404
- epoch time: 5730.263 ms, per step time: 358.141 ms
- ## epoch start ##
- epoch: 119 step: 16, loss is 0.28203177
- epoch time: 3070.338 ms, per step time: 191.896 ms
- ## epoch start ##
- epoch: 120 step: 16, loss is 0.021746732
- epoch time: 6316.588 ms, per step time: 394.787 ms
- ## epoch start ##
- epoch: 121 step: 16, loss is 0.3223167
- epoch time: 3006.140 ms, per step time: 187.884 ms
- ## epoch start ##
- epoch: 122 step: 16, loss is 0.08896692
- epoch time: 5013.338 ms, per step time: 313.334 ms
- ## epoch start ##
- epoch: 123 step: 16, loss is 0.62394494
- epoch time: 2346.405 ms, per step time: 146.650 ms
- ## epoch start ##
- epoch: 124 step: 16, loss is 0.3033384
- epoch time: 4304.609 ms, per step time: 269.038 ms
- ## epoch start ##
- epoch: 125 step: 16, loss is 0.09365121
- epoch time: 5961.485 ms, per step time: 372.593 ms
- ## epoch start ##
- epoch: 126 step: 16, loss is 0.095688805
- epoch time: 3541.080 ms, per step time: 221.317 ms
- ## epoch start ##
- epoch: 127 step: 16, loss is 0.014012268
- epoch time: 6478.790 ms, per step time: 404.924 ms
- ## epoch start ##
- epoch: 128 step: 16, loss is 0.21668851
- epoch time: 3013.840 ms, per step time: 188.365 ms
- ## epoch start ##
- epoch: 129 step: 16, loss is 0.2050103
- epoch time: 4547.455 ms, per step time: 284.216 ms
- ## epoch start ##
- epoch: 130 step: 16, loss is 0.025045596
- epoch time: 6901.523 ms, per step time: 431.345 ms
- ## epoch start ##
- epoch: 131 step: 16, loss is 0.23314242
- epoch time: 3092.015 ms, per step time: 193.251 ms
- ## epoch start ##
- epoch: 132 step: 16, loss is 0.016626334
- epoch time: 3766.664 ms, per step time: 235.417 ms
- ## epoch start ##
- epoch: 133 step: 16, loss is 0.44821197
- epoch time: 3830.820 ms, per step time: 239.426 ms
- ## epoch start ##
- epoch: 134 step: 16, loss is 0.09155351
- epoch time: 6864.114 ms, per step time: 429.007 ms
- ## epoch start ##
- epoch: 135 step: 16, loss is 0.018434204
- epoch time: 3476.609 ms, per step time: 217.288 ms
- ## epoch start ##
- [WARNING] ME(70888:281473593811344,ForkPoolWorker-56):2021-11-15-04:09:32.900.137 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 136 step: 16, loss is 0.8548652
- epoch time: 5350.840 ms, per step time: 334.427 ms
- ## epoch start ##
- epoch: 137 step: 16, loss is 0.25054267
- epoch time: 7153.077 ms, per step time: 447.067 ms
- ## epoch start ##
- epoch: 138 step: 16, loss is 0.23651522
- epoch time: 2644.346 ms, per step time: 165.272 ms
- ## epoch start ##
- epoch: 139 step: 16, loss is 0.012309359
- epoch time: 8295.763 ms, per step time: 518.485 ms
- ## epoch start ##
- epoch: 140 step: 16, loss is 0.15201907
- epoch time: 2777.281 ms, per step time: 173.580 ms
- ## epoch start ##
- epoch: 141 step: 16, loss is 0.59351355
- epoch time: 4011.167 ms, per step time: 250.698 ms
- ## epoch start ##
- epoch: 142 step: 16, loss is 0.8691423
- epoch time: 5352.596 ms, per step time: 334.537 ms
- ## epoch start ##
- epoch: 143 step: 16, loss is 0.04046496
- epoch time: 2878.096 ms, per step time: 179.881 ms
- ## epoch start ##
- epoch: 144 step: 16, loss is 0.27447906
- epoch time: 4743.220 ms, per step time: 296.451 ms
- ## epoch start ##
- epoch: 145 step: 16, loss is 0.17781769
- epoch time: 5843.911 ms, per step time: 365.244 ms
- ## epoch start ##
- epoch: 146 step: 16, loss is 0.28637806
- epoch time: 4620.959 ms, per step time: 288.810 ms
- ## epoch start ##
- epoch: 147 step: 16, loss is 0.28344277
- epoch time: 5672.633 ms, per step time: 354.540 ms
- ## epoch start ##
- epoch: 148 step: 16, loss is 0.5005207
- epoch time: 1688.520 ms, per step time: 105.532 ms
- ## epoch start ##
- [WARNING] ME(70890:281473593811344,ForkPoolWorker-58):2021-11-15-04:10:33.186.632 [mindspore/dataset/engine/queue.py:106] Using shared memory queue, but rowsize is larger than allocated memory max_rowsize 16777216 current rowwize 70778880
- epoch: 149 step: 16, loss is 0.3126863
- epoch time: 8078.401 ms, per step time: 504.900 ms
- ## epoch start ##
- epoch: 150 step: 16, loss is 0.011890976
- epoch time: 3449.116 ms, per step time: 215.570 ms
- ## epoch start ##
- epoch: 151 step: 16, loss is 0.6752943
- epoch time: 4493.861 ms, per step time: 280.866 ms
- ## epoch start ##
- epoch: 152 step: 16, loss is 0.2774931
- epoch time: 3591.135 ms, per step time: 224.446 ms
- ## epoch start ##
- epoch: 153 step: 16, loss is 0.11882457
- epoch time: 5805.592 ms, per step time: 362.849 ms
- ## epoch start ##
- epoch: 154 step: 16, loss is 0.44703194
- epoch time: 7501.127 ms, per step time: 468.820 ms
- ## epoch start ##
- epoch: 155 step: 16, loss is 0.06839551
- epoch time: 3650.032 ms, per step time: 228.127 ms
- ## epoch start ##
- epoch: 156 step: 16, loss is 0.030028002
- epoch time: 2565.122 ms, per step time: 160.320 ms
- ## epoch start ##
- epoch: 157 step: 16, loss is 0.26867822
- epoch time: 3977.932 ms, per step time: 248.621 ms
- ## epoch start ##
- epoch: 158 step: 16, loss is 0.1557419
- epoch time: 4805.508 ms, per step time: 300.344 ms
- ## epoch start ##
- epoch: 159 step: 16, loss is 0.12895256
- epoch time: 4939.835 ms, per step time: 308.740 ms
- ## epoch start ##
- epoch: 160 step: 16, loss is 0.4562061
- epoch time: 8331.983 ms, per step time: 520.749 ms
- ## epoch start ##
- epoch: 161 step: 16, loss is 0.021792155
- epoch time: 4905.850 ms, per step time: 306.616 ms
- ## epoch start ##
- epoch: 162 step: 16, loss is 0.0411106
- epoch time: 2782.466 ms, per step time: 173.904 ms
- ## epoch start ##
- epoch: 163 step: 16, loss is 0.17244342
- epoch time: 5408.653 ms, per step time: 338.041 ms
- ## epoch start ##
- epoch: 164 step: 16, loss is 0.26380825
- epoch time: 3090.978 ms, per step time: 193.186 ms
- ## epoch start ##
- epoch: 165 step: 16, loss is 0.39362073
- epoch time: 5936.741 ms, per step time: 371.046 ms
- ## epoch start ##
- epoch: 166 step: 16, loss is 0.44121274
- epoch time: 2988.180 ms, per step time: 186.761 ms
- ## epoch start ##
- epoch: 167 step: 16, loss is 0.19607039
- epoch time: 3219.017 ms, per step time: 201.189 ms
- ## epoch start ##
- epoch: 168 step: 16, loss is 0.22761425
- epoch time: 2513.344 ms, per step time: 157.084 ms
- ## epoch start ##
- epoch: 169 step: 16, loss is 0.38768533
- epoch time: 3635.924 ms, per step time: 227.245 ms
- ## epoch start ##
- epoch: 170 step: 16, loss is 0.018489813
- epoch time: 5763.229 ms, per step time: 360.202 ms
- ## epoch start ##
- epoch: 171 step: 16, loss is 0.24710506
- epoch time: 4448.955 ms, per step time: 278.060 ms
- ## epoch start ##
- epoch: 172 step: 16, loss is 0.05865738
- epoch time: 8811.442 ms, per step time: 550.715 ms
- ## epoch start ##
- epoch: 173 step: 16, loss is 0.11476419
- epoch time: 3127.123 ms, per step time: 195.445 ms
- ## epoch start ##
- epoch: 174 step: 16, loss is 0.3032296
- epoch time: 1974.395 ms, per step time: 123.400 ms
- ## epoch start ##
- epoch: 175 step: 16, loss is 0.039852887
- epoch time: 7158.626 ms, per step time: 447.414 ms
- ## epoch start ##
- epoch: 176 step: 16, loss is 0.19121106
- epoch time: 3654.833 ms, per step time: 228.427 ms
- ## epoch start ##
- epoch: 177 step: 16, loss is 0.21714853
- epoch time: 5451.816 ms, per step time: 340.738 ms
- ## epoch start ##
- epoch: 178 step: 16, loss is 0.36425525
- epoch time: 3745.590 ms, per step time: 234.099 ms
- ## epoch start ##
- epoch: 179 step: 16, loss is 0.123812385
- epoch time: 4849.585 ms, per step time: 303.099 ms
- ## epoch start ##
- epoch: 180 step: 16, loss is 0.017458163
- epoch time: 1270.407 ms, per step time: 79.400 ms
- ## epoch start ##
- epoch: 181 step: 16, loss is 0.009092916
- epoch time: 4237.526 ms, per step time: 264.845 ms
- ## epoch start ##
- epoch: 182 step: 16, loss is 0.24586578
- epoch time: 5513.875 ms, per step time: 344.617 ms
- ## epoch start ##
- epoch: 183 step: 16, loss is 0.06566836
- epoch time: 3329.665 ms, per step time: 208.104 ms
- ## epoch start ##
- epoch: 184 step: 16, loss is 0.36559245
- epoch time: 5749.669 ms, per step time: 359.354 ms
- ## epoch start ##
- epoch: 185 step: 16, loss is 0.34545788
- epoch time: 4508.922 ms, per step time: 281.808 ms
- ## epoch start ##
- epoch: 186 step: 16, loss is 0.3114282
- epoch time: 3941.226 ms, per step time: 246.327 ms
- ## epoch start ##
- epoch: 187 step: 16, loss is 0.4888051
- epoch time: 9060.180 ms, per step time: 566.261 ms
- ## epoch start ##
- epoch: 188 step: 16, loss is 0.27146503
- epoch time: 3595.826 ms, per step time: 224.739 ms
- ## epoch start ##
- epoch: 189 step: 16, loss is 0.22621642
- epoch time: 6530.596 ms, per step time: 408.162 ms
- ## epoch start ##
- epoch: 190 step: 16, loss is 0.021492077
- epoch time: 5288.157 ms, per step time: 330.510 ms
- ## epoch start ##
- epoch: 191 step: 16, loss is 0.0077163544
- epoch time: 3131.752 ms, per step time: 195.734 ms
- ## epoch start ##
- epoch: 192 step: 16, loss is 0.20035438
- epoch time: 3628.661 ms, per step time: 226.791 ms
- ## epoch start ##
- epoch: 193 step: 16, loss is 0.19348726
- epoch time: 1410.504 ms, per step time: 88.157 ms
- ## epoch start ##
- epoch: 194 step: 16, loss is 0.33835796
- epoch time: 9016.966 ms, per step time: 563.560 ms
- ## epoch start ##
- epoch: 195 step: 16, loss is 0.010586217
- epoch time: 2950.192 ms, per step time: 184.387 ms
- ## epoch start ##
- epoch: 196 step: 16, loss is 0.37651068
- epoch time: 5121.262 ms, per step time: 320.079 ms
- ## epoch start ##
- epoch: 197 step: 16, loss is 0.0302212
- epoch time: 4336.264 ms, per step time: 271.017 ms
- ## epoch start ##
- epoch: 198 step: 16, loss is 0.486463
- epoch time: 2901.956 ms, per step time: 181.372 ms
- ## epoch start ##
- epoch: 199 step: 16, loss is 0.38680035
- epoch time: 5312.186 ms, per step time: 332.012 ms
- ## epoch start ##
- epoch: 200 step: 16, loss is 0.17961273
- epoch time: 3564.106 ms, per step time: 222.757 ms
- ## epoch start ##
- epoch: 201 step: 16, loss is 0.107239194
- epoch time: 6436.762 ms, per step time: 402.298 ms
- ## epoch start ##
- epoch: 202 step: 16, loss is 0.86747086
- epoch time: 2138.700 ms, per step time: 133.669 ms
- ## epoch start ##
- epoch: 203 step: 16, loss is 0.019132622
- epoch time: 3718.871 ms, per step time: 232.429 ms
- ## epoch start ##
- epoch: 204 step: 16, loss is 0.10860398
- epoch time: 2039.723 ms, per step time: 127.483 ms
- ## epoch start ##
- epoch: 205 step: 16, loss is 0.1038439
- epoch time: 5764.392 ms, per step time: 360.275 ms
- ## epoch start ##
- epoch: 206 step: 16, loss is 0.016625583
- epoch time: 6466.340 ms, per step time: 404.146 ms
- ## epoch start ##
- epoch: 207 step: 16, loss is 0.17890982
- epoch time: 3337.939 ms, per step time: 208.621 ms
- ## epoch start ##
- epoch: 208 step: 16, loss is 0.0075300923
- epoch time: 7025.264 ms, per step time: 439.079 ms
- ## epoch start ##
- epoch: 209 step: 16, loss is 0.25811794
- epoch time: 5365.341 ms, per step time: 335.334 ms
- ## epoch start ##
- epoch: 210 step: 16, loss is 0.008427001
- epoch time: 5514.683 ms, per step time: 344.668 ms
- ## epoch start ##
- epoch: 211 step: 16, loss is 0.53260046
- epoch time: 6342.005 ms, per step time: 396.375 ms
- ## epoch start ##
- epoch: 212 step: 16, loss is 0.08255746
- epoch time: 3144.433 ms, per step time: 196.527 ms
- ## epoch start ##
- epoch: 213 step: 16, loss is 0.007997545
- epoch time: 6317.815 ms, per step time: 394.863 ms
- ## epoch start ##
- epoch: 214 step: 16, loss is 0.20630485
- epoch time: 3987.783 ms, per step time: 249.236 ms
- ## epoch start ##
- epoch: 215 step: 16, loss is 0.27148873
- epoch time: 4137.884 ms, per step time: 258.618 ms
- ## epoch start ##
- epoch: 216 step: 16, loss is 0.19641285
- epoch time: 2742.171 ms, per step time: 171.386 ms
- ## epoch start ##
- epoch: 217 step: 16, loss is 0.064706646
- epoch time: 3200.733 ms, per step time: 200.046 ms
- ## epoch start ##
- epoch: 218 step: 16, loss is 0.01730727
- epoch time: 6548.923 ms, per step time: 409.308 ms
- ## epoch start ##
- epoch: 219 step: 16, loss is 0.018825088
- epoch time: 3224.566 ms, per step time: 201.535 ms
- ## epoch start ##
- epoch: 220 step: 16, loss is 0.109316744
- epoch time: 5382.680 ms, per step time: 336.417 ms
- ## epoch start ##
- epoch: 221 step: 16, loss is 0.045225788
- epoch time: 2894.027 ms, per step time: 180.877 ms
- ## epoch start ##
- epoch: 222 step: 16, loss is 0.026947362
- epoch time: 4093.485 ms, per step time: 255.843 ms
- ## epoch start ##
- epoch: 223 step: 16, loss is 0.33908182
- epoch time: 7765.610 ms, per step time: 485.351 ms
- ## epoch start ##
- epoch: 224 step: 16, loss is 0.06614789
- epoch time: 2358.911 ms, per step time: 147.432 ms
- ## epoch start ##
- epoch: 225 step: 16, loss is 0.05800104
- epoch time: 6953.325 ms, per step time: 434.583 ms
- ## epoch start ##
- epoch: 226 step: 16, loss is 0.053182323
- epoch time: 3379.903 ms, per step time: 211.244 ms
- ## epoch start ##
- epoch: 227 step: 16, loss is 0.030626388
- epoch time: 2722.781 ms, per step time: 170.174 ms
- ## epoch start ##
- epoch: 228 step: 16, loss is 0.019706933
- epoch time: 3120.081 ms, per step time: 195.005 ms
- ## epoch start ##
- epoch: 229 step: 16, loss is 0.04432093
- epoch time: 5168.117 ms, per step time: 323.007 ms
- ## epoch start ##
- epoch: 230 step: 16, loss is 0.31380168
- epoch time: 3119.580 ms, per step time: 194.974 ms
- ## epoch start ##
- epoch: 231 step: 16, loss is 0.18344173
- epoch time: 3801.853 ms, per step time: 237.616 ms
- ## epoch start ##
- epoch: 232 step: 16, loss is 0.014367152
- epoch time: 5657.373 ms, per step time: 353.586 ms
- ## epoch start ##
- epoch: 233 step: 16, loss is 0.32220563
- epoch time: 3605.326 ms, per step time: 225.333 ms
- ## epoch start ##
- epoch: 234 step: 16, loss is 0.07081256
- epoch time: 3578.839 ms, per step time: 223.677 ms
- ## epoch start ##
- epoch: 235 step: 16, loss is 0.006741004
- epoch time: 6971.741 ms, per step time: 435.734 ms
- ## epoch start ##
- epoch: 236 step: 16, loss is 0.17815149
- epoch time: 2977.459 ms, per step time: 186.091 ms
- ## epoch start ##
- epoch: 237 step: 16, loss is 0.30675945
- epoch time: 5297.096 ms, per step time: 331.069 ms
- ## epoch start ##
- epoch: 238 step: 16, loss is 0.26379755
- epoch time: 5367.441 ms, per step time: 335.465 ms
- ## epoch start ##
- epoch: 239 step: 16, loss is 0.12287341
- epoch time: 2417.374 ms, per step time: 151.086 ms
- ## epoch start ##
- epoch: 240 step: 16, loss is 0.005289639
- epoch time: 6455.719 ms, per step time: 403.482 ms
- ## epoch start ##
- epoch: 241 step: 16, loss is 0.26942047
- epoch time: 5952.733 ms, per step time: 372.046 ms
- ## epoch start ##
- epoch: 242 step: 16, loss is 0.13692664
- epoch time: 2779.086 ms, per step time: 173.693 ms
- ## epoch start ##
- epoch: 243 step: 16, loss is 0.30606896
- epoch time: 2194.820 ms, per step time: 137.176 ms
- ## epoch start ##
- epoch: 244 step: 16, loss is 0.31039804
- epoch time: 7246.237 ms, per step time: 452.890 ms
- ## epoch start ##
- epoch: 245 step: 16, loss is 0.46505535
- epoch time: 4272.224 ms, per step time: 267.014 ms
- ## epoch start ##
- epoch: 246 step: 16, loss is 0.023084698
- epoch time: 4408.043 ms, per step time: 275.503 ms
- ## epoch start ##
- epoch: 247 step: 16, loss is 0.18886891
- epoch time: 6796.743 ms, per step time: 424.796 ms
- ## epoch start ##
- epoch: 248 step: 16, loss is 0.045625497
- epoch time: 4142.080 ms, per step time: 258.880 ms
- ## epoch start ##
- epoch: 249 step: 16, loss is 0.013390799
- epoch time: 4712.801 ms, per step time: 294.550 ms
- ## epoch start ##
- epoch: 250 step: 16, loss is 0.057721116
- epoch time: 4605.258 ms, per step time: 287.829 ms
- ## epoch start ##
- epoch: 251 step: 16, loss is 0.082561724
- epoch time: 2134.651 ms, per step time: 133.416 ms
- ## epoch start ##
- epoch: 252 step: 16, loss is 0.0738865
- epoch time: 3692.693 ms, per step time: 230.793 ms
- ## epoch start ##
- epoch: 253 step: 16, loss is 0.3001319
- epoch time: 3579.263 ms, per step time: 223.704 ms
- ## epoch start ##
- epoch: 254 step: 16, loss is 0.017892938
- epoch time: 4800.050 ms, per step time: 300.003 ms
- ## epoch start ##
- epoch: 255 step: 16, loss is 0.31705126
- epoch time: 3738.538 ms, per step time: 233.659 ms
- ## epoch start ##
- epoch: 256 step: 16, loss is 0.098797
- epoch time: 4836.151 ms, per step time: 302.259 ms
- ## epoch start ##
- epoch: 257 step: 16, loss is 0.32294837
- epoch time: 4522.501 ms, per step time: 282.656 ms
- ## epoch start ##
- epoch: 258 step: 16, loss is 0.06009613
- epoch time: 3530.817 ms, per step time: 220.676 ms
- ## epoch start ##
- epoch: 259 step: 16, loss is 0.114593886
- epoch time: 5343.226 ms, per step time: 333.952 ms
- ## epoch start ##
- epoch: 260 step: 16, loss is 0.18096296
- epoch time: 4771.656 ms, per step time: 298.228 ms
- ## epoch start ##
- epoch: 261 step: 16, loss is 0.32310984
- epoch time: 7838.242 ms, per step time: 489.890 ms
- ## epoch start ##
- epoch: 262 step: 16, loss is 0.35726342
- epoch time: 4566.536 ms, per step time: 285.408 ms
- ## epoch start ##
- epoch: 263 step: 16, loss is 0.08725352
- epoch time: 3714.171 ms, per step time: 232.136 ms
- ## epoch start ##
- epoch: 264 step: 16, loss is 0.007024201
- epoch time: 3595.681 ms, per step time: 224.730 ms
- ## epoch start ##
- epoch: 265 step: 16, loss is 0.28223723
- epoch time: 3190.950 ms, per step time: 199.434 ms
- ## epoch start ##
- epoch: 266 step: 16, loss is 0.0049783303
- epoch time: 5404.251 ms, per step time: 337.766 ms
- ## epoch start ##
- epoch: 267 step: 16, loss is 0.018768575
- epoch time: 2905.467 ms, per step time: 181.592 ms
- ## epoch start ##
- epoch: 268 step: 16, loss is 0.17560215
- epoch time: 6981.869 ms, per step time: 436.367 ms
- ## epoch start ##
- epoch: 269 step: 16, loss is 0.084460184
- epoch time: 5193.456 ms, per step time: 324.591 ms
- ## epoch start ##
- epoch: 270 step: 16, loss is 0.06499186
- epoch time: 3723.654 ms, per step time: 232.728 ms
- ## epoch start ##
- epoch: 271 step: 16, loss is 0.1841403
- epoch time: 7629.350 ms, per step time: 476.834 ms
- ## epoch start ##
- epoch: 272 step: 16, loss is 0.12520975
- epoch time: 3619.576 ms, per step time: 226.224 ms
- ## epoch start ##
- epoch: 273 step: 16, loss is 0.04506164
- epoch time: 7793.394 ms, per step time: 487.087 ms
- ## epoch start ##
- epoch: 274 step: 16, loss is 0.05944519
- epoch time: 4526.474 ms, per step time: 282.905 ms
- ## epoch start ##
- epoch: 275 step: 16, loss is 0.41126552
- epoch time: 1907.963 ms, per step time: 119.248 ms
- ## epoch start ##
- epoch: 276 step: 16, loss is 0.031720813
- epoch time: 5542.874 ms, per step time: 346.430 ms
- ## epoch start ##
- epoch: 277 step: 16, loss is 0.3106036
- epoch time: 2782.462 ms, per step time: 173.904 ms
- ## epoch start ##
- epoch: 278 step: 16, loss is 0.003954835
- epoch time: 4478.983 ms, per step time: 279.936 ms
- ## epoch start ##
- epoch: 279 step: 16, loss is 0.29309815
- epoch time: 4387.920 ms, per step time: 274.245 ms
- ## epoch start ##
- epoch: 280 step: 16, loss is 0.20294933
- epoch time: 5391.746 ms, per step time: 336.984 ms
- ## epoch start ##
- epoch: 281 step: 16, loss is 0.0947876
- epoch time: 6931.444 ms, per step time: 433.215 ms
- ## epoch start ##
- epoch: 282 step: 16, loss is 0.23856147
- epoch time: 3337.418 ms, per step time: 208.589 ms
- ## epoch start ##
- epoch: 283 step: 16, loss is 0.10864732
- epoch time: 3506.036 ms, per step time: 219.127 ms
- ## epoch start ##
- epoch: 284 step: 16, loss is 0.016124386
- epoch time: 5926.233 ms, per step time: 370.390 ms
- ## epoch start ##
- epoch: 285 step: 16, loss is 0.3664138
- epoch time: 6053.478 ms, per step time: 378.342 ms
- ## epoch start ##
- epoch: 286 step: 16, loss is 0.016400686
- epoch time: 4648.421 ms, per step time: 290.526 ms
- ## epoch start ##
- epoch: 287 step: 16, loss is 0.0076649133
- epoch time: 4180.234 ms, per step time: 261.265 ms
- ## epoch start ##
- epoch: 288 step: 16, loss is 0.10613684
- epoch time: 2216.822 ms, per step time: 138.551 ms
- ## epoch start ##
- epoch: 289 step: 16, loss is 0.102973506
- epoch time: 3404.207 ms, per step time: 212.763 ms
- ## epoch start ##
- epoch: 290 step: 16, loss is 0.04191219
- epoch time: 5581.358 ms, per step time: 348.835 ms
- ## epoch start ##
- epoch: 291 step: 16, loss is 0.22343607
- epoch time: 7557.431 ms, per step time: 472.339 ms
- ## epoch start ##
- epoch: 292 step: 16, loss is 0.06334201
- epoch time: 3049.454 ms, per step time: 190.591 ms
- ## epoch start ##
- epoch: 293 step: 16, loss is 0.014619174
- epoch time: 6356.583 ms, per step time: 397.286 ms
- ## epoch start ##
- epoch: 294 step: 16, loss is 0.25744635
- epoch time: 4136.508 ms, per step time: 258.532 ms
- ## epoch start ##
- epoch: 295 step: 16, loss is 0.008828595
- epoch time: 3387.619 ms, per step time: 211.726 ms
- ## epoch start ##
- epoch: 296 step: 16, loss is 0.01615433
- epoch time: 3069.192 ms, per step time: 191.824 ms
- ## epoch start ##
- epoch: 297 step: 16, loss is 0.090798154
- epoch time: 5127.011 ms, per step time: 320.438 ms
- ## epoch start ##
- epoch: 298 step: 16, loss is 0.07692989
- epoch time: 6555.573 ms, per step time: 409.723 ms
- ## epoch start ##
- epoch: 299 step: 16, loss is 0.076306134
- epoch time: 3203.884 ms, per step time: 200.243 ms
- ## epoch start ##
- epoch: 300 step: 16, loss is 0.1202646
- epoch time: 7568.252 ms, per step time: 473.016 ms
- ## epoch start ##
- epoch: 301 step: 16, loss is 0.19342947
- epoch time: 3029.830 ms, per step time: 189.364 ms
- ## epoch start ##
- epoch: 302 step: 16, loss is 0.250191
- epoch time: 3038.526 ms, per step time: 189.908 ms
- ## epoch start ##
- epoch: 303 step: 16, loss is 0.1791828
- epoch time: 7348.928 ms, per step time: 459.308 ms
- ## epoch start ##
- epoch: 304 step: 16, loss is 0.23345011
- epoch time: 3901.319 ms, per step time: 243.832 ms
- ## epoch start ##
- epoch: 305 step: 16, loss is 0.09694685
- epoch time: 4365.182 ms, per step time: 272.824 ms
- ## epoch start ##
- epoch: 306 step: 16, loss is 0.36277875
- epoch time: 2979.450 ms, per step time: 186.216 ms
- ## epoch start ##
- epoch: 307 step: 16, loss is 0.9216372
- epoch time: 5613.105 ms, per step time: 350.819 ms
- ## epoch start ##
- epoch: 308 step: 16, loss is 0.059622787
- epoch time: 1968.658 ms, per step time: 123.041 ms
- ## epoch start ##
- epoch: 309 step: 16, loss is 0.034529984
- epoch time: 1893.556 ms, per step time: 118.347 ms
- ## epoch start ##
- epoch: 310 step: 16, loss is 0.11637698
- epoch time: 4954.502 ms, per step time: 309.656 ms
- ## epoch start ##
- epoch: 311 step: 16, loss is 0.22871852
- epoch time: 5073.084 ms, per step time: 317.068 ms
- ## epoch start ##
- epoch: 312 step: 16, loss is 0.019811967
- epoch time: 6934.611 ms, per step time: 433.413 ms
- ## epoch start ##
- epoch: 313 step: 16, loss is 0.07858948
- epoch time: 3557.635 ms, per step time: 222.352 ms
- ## epoch start ##
- epoch: 314 step: 16, loss is 0.02050751
- epoch time: 3661.747 ms, per step time: 228.859 ms
- ## epoch start ##
- epoch: 315 step: 16, loss is 0.059927747
- epoch time: 6509.351 ms, per step time: 406.834 ms
- ## epoch start ##
- epoch: 316 step: 16, loss is 0.03504432
- epoch time: 2107.920 ms, per step time: 131.745 ms
- ## epoch start ##
- epoch: 317 step: 16, loss is 0.04474555
- epoch time: 7091.744 ms, per step time: 443.234 ms
- ## epoch start ##
- epoch: 318 step: 16, loss is 0.08514886
- epoch time: 3892.007 ms, per step time: 243.250 ms
- ## epoch start ##
- epoch: 319 step: 16, loss is 0.07219324
- epoch time: 5166.548 ms, per step time: 322.909 ms
- ## epoch start ##
- epoch: 320 step: 16, loss is 0.29019424
- epoch time: 3347.414 ms, per step time: 209.213 ms
- ## epoch start ##
- epoch: 321 step: 16, loss is 0.008934712
- epoch time: 3084.647 ms, per step time: 192.790 ms
- ## epoch start ##
- epoch: 322 step: 16, loss is 0.3022961
- epoch time: 5530.609 ms, per step time: 345.663 ms
- ## epoch start ##
- epoch: 323 step: 16, loss is 0.14572878
- epoch time: 4705.701 ms, per step time: 294.106 ms
- ## epoch start ##
- epoch: 324 step: 16, loss is 0.101337604
- epoch time: 7001.767 ms, per step time: 437.610 ms
- ## epoch start ##
- epoch: 325 step: 16, loss is 0.030377388
- epoch time: 2898.931 ms, per step time: 181.183 ms
- ## epoch start ##
- epoch: 326 step: 16, loss is 0.23527455
- epoch time: 1650.103 ms, per step time: 103.131 ms
- ## epoch start ##
- epoch: 327 step: 16, loss is 0.08423759
- epoch time: 8453.104 ms, per step time: 528.319 ms
- ## epoch start ##
- epoch: 328 step: 16, loss is 0.15855472
- epoch time: 4428.650 ms, per step time: 276.791 ms
- ## epoch start ##
- epoch: 329 step: 16, loss is 0.11552229
- epoch time: 5930.142 ms, per step time: 370.634 ms
- ## epoch start ##
- epoch: 330 step: 16, loss is 0.30753702
- epoch time: 2974.239 ms, per step time: 185.890 ms
- ## epoch start ##
- epoch: 331 step: 16, loss is 0.07106604
- epoch time: 4498.032 ms, per step time: 281.127 ms
- ## epoch start ##
- epoch: 332 step: 16, loss is 0.33072257
- epoch time: 1940.030 ms, per step time: 121.252 ms
- ## epoch start ##
- epoch: 333 step: 16, loss is 0.13592836
- epoch time: 4212.010 ms, per step time: 263.251 ms
- ## epoch start ##
- epoch: 334 step: 16, loss is 0.16007796
- epoch time: 4655.059 ms, per step time: 290.941 ms
- ## epoch start ##
- epoch: 335 step: 16, loss is 0.052131165
- epoch time: 3503.384 ms, per step time: 218.961 ms
- ## epoch start ##
- epoch: 336 step: 16, loss is 0.036394473
- epoch time: 4549.290 ms, per step time: 284.331 ms
- ## epoch start ##
- epoch: 337 step: 16, loss is 0.06319896
- epoch time: 3193.507 ms, per step time: 199.594 ms
- ## epoch start ##
- epoch: 338 step: 16, loss is 0.04176037
- epoch time: 5556.049 ms, per step time: 347.253 ms
- ## epoch start ##
- epoch: 339 step: 16, loss is 0.3655789
- epoch time: 7919.066 ms, per step time: 494.942 ms
- ## epoch start ##
- epoch: 340 step: 16, loss is 0.38390946
- epoch time: 2329.934 ms, per step time: 145.621 ms
- ## epoch start ##
- epoch: 341 step: 16, loss is 0.011550763
- epoch time: 4628.975 ms, per step time: 289.311 ms
- ## epoch start ##
- epoch: 342 step: 16, loss is 0.0037276405
- epoch time: 3009.729 ms, per step time: 188.108 ms
- ## epoch start ##
- epoch: 343 step: 16, loss is 0.005608469
- epoch time: 4079.026 ms, per step time: 254.939 ms
- ## epoch start ##
- epoch: 344 step: 16, loss is 0.060359016
- epoch time: 3301.337 ms, per step time: 206.334 ms
- ## epoch start ##
- epoch: 345 step: 16, loss is 0.015159009
- epoch time: 4149.662 ms, per step time: 259.354 ms
- ## epoch start ##
- epoch: 346 step: 16, loss is 0.1129925
- epoch time: 6997.084 ms, per step time: 437.318 ms
- ## epoch start ##
- epoch: 347 step: 16, loss is 0.16210544
- epoch time: 4488.019 ms, per step time: 280.501 ms
- ## epoch start ##
- epoch: 348 step: 16, loss is 0.11634672
- epoch time: 5450.305 ms, per step time: 340.644 ms
- ## epoch start ##
- epoch: 349 step: 16, loss is 0.09943172
- epoch time: 4154.010 ms, per step time: 259.626 ms
- ## epoch start ##
- epoch: 350 step: 16, loss is 0.23706679
- epoch time: 3115.623 ms, per step time: 194.726 ms
- ## epoch start ##
- epoch: 351 step: 16, loss is 0.07148523
- epoch time: 6515.685 ms, per step time: 407.230 ms
- ## epoch start ##
- epoch: 352 step: 16, loss is 0.095657684
- epoch time: 2852.204 ms, per step time: 178.263 ms
- ## epoch start ##
- epoch: 353 step: 16, loss is 0.0032210662
- epoch time: 4167.713 ms, per step time: 260.482 ms
- ## epoch start ##
- epoch: 354 step: 16, loss is 0.030926036
- epoch time: 5943.821 ms, per step time: 371.489 ms
- ## epoch start ##
- epoch: 355 step: 16, loss is 0.17861317
- epoch time: 3491.060 ms, per step time: 218.191 ms
- ## epoch start ##
- epoch: 356 step: 16, loss is 0.011590699
- epoch time: 3018.636 ms, per step time: 188.665 ms
- ## epoch start ##
- epoch: 357 step: 16, loss is 0.036917377
- epoch time: 6613.235 ms, per step time: 413.327 ms
- ## epoch start ##
- epoch: 358 step: 16, loss is 0.115331076
- epoch time: 4071.598 ms, per step time: 254.475 ms
- ## epoch start ##
- epoch: 359 step: 16, loss is 0.32052746
- epoch time: 3161.599 ms, per step time: 197.600 ms
- ## epoch start ##
- epoch: 360 step: 16, loss is 0.12451971
- epoch time: 5991.168 ms, per step time: 374.448 ms
- ## epoch start ##
- epoch: 361 step: 16, loss is 0.024503494
- epoch time: 3261.601 ms, per step time: 203.850 ms
- ## epoch start ##
- epoch: 362 step: 16, loss is 0.07594081
- epoch time: 3003.649 ms, per step time: 187.728 ms
- ## epoch start ##
- epoch: 363 step: 16, loss is 0.008076238
- epoch time: 5897.225 ms, per step time: 368.577 ms
- ## epoch start ##
- epoch: 364 step: 16, loss is 0.0040808474
- epoch time: 4968.646 ms, per step time: 310.540 ms
- ## epoch start ##
- epoch: 365 step: 16, loss is 0.006933076
- epoch time: 4852.374 ms, per step time: 303.273 ms
- ## epoch start ##
- epoch: 366 step: 16, loss is 0.16841665
- epoch time: 4189.366 ms, per step time: 261.835 ms
- ## epoch start ##
- epoch: 367 step: 16, loss is 0.0025190637
- epoch time: 7373.798 ms, per step time: 460.862 ms
- ## epoch start ##
- epoch: 368 step: 16, loss is 0.0038588967
- epoch time: 2980.039 ms, per step time: 186.252 ms
- ## epoch start ##
- epoch: 369 step: 16, loss is 0.004195106
- epoch time: 3737.497 ms, per step time: 233.594 ms
- ## epoch start ##
- epoch: 370 step: 16, loss is 0.18526584
- epoch time: 7879.766 ms, per step time: 492.485 ms
- ## epoch start ##
- epoch: 371 step: 16, loss is 0.20599943
- epoch time: 4737.515 ms, per step time: 296.095 ms
- ## epoch start ##
- epoch: 372 step: 16, loss is 0.077733174
- epoch time: 3478.734 ms, per step time: 217.421 ms
- ## epoch start ##
- epoch: 373 step: 16, loss is 0.011389979
- epoch time: 3940.006 ms, per step time: 246.250 ms
- ## epoch start ##
- epoch: 374 step: 16, loss is 0.057972647
- epoch time: 2724.152 ms, per step time: 170.260 ms
- ## epoch start ##
- epoch: 375 step: 16, loss is 0.20566921
- epoch time: 5346.030 ms, per step time: 334.127 ms
- ## epoch start ##
- epoch: 376 step: 16, loss is 0.22621137
- epoch time: 4801.739 ms, per step time: 300.109 ms
- ## epoch start ##
- epoch: 377 step: 16, loss is 0.0052918475
- epoch time: 6778.404 ms, per step time: 423.650 ms
- ## epoch start ##
- epoch: 378 step: 16, loss is 0.05043868
- epoch time: 3932.216 ms, per step time: 245.764 ms
- ## epoch start ##
- epoch: 379 step: 16, loss is 0.0054065073
- epoch time: 3399.699 ms, per step time: 212.481 ms
- ## epoch start ##
- epoch: 380 step: 16, loss is 0.33646953
- epoch time: 5644.514 ms, per step time: 352.782 ms
- ## epoch start ##
- epoch: 381 step: 16, loss is 0.117869265
- epoch time: 3947.035 ms, per step time: 246.690 ms
- ## epoch start ##
- epoch: 382 step: 16, loss is 0.1797716
- epoch time: 3116.631 ms, per step time: 194.789 ms
- ## epoch start ##
- epoch: 383 step: 16, loss is 0.07002587
- epoch time: 6087.591 ms, per step time: 380.474 ms
- ## epoch start ##
- epoch: 384 step: 16, loss is 0.020861758
- epoch time: 6137.841 ms, per step time: 383.615 ms
- ## epoch start ##
- epoch: 385 step: 16, loss is 0.061959878
- epoch time: 3649.454 ms, per step time: 228.091 ms
- ## epoch start ##
- epoch: 386 step: 16, loss is 0.03326116
- epoch time: 8208.359 ms, per step time: 513.022 ms
- ## epoch start ##
- epoch: 387 step: 16, loss is 0.07964668
- epoch time: 5023.009 ms, per step time: 313.938 ms
- ## epoch start ##
- epoch: 388 step: 16, loss is 0.04431426
- epoch time: 4587.513 ms, per step time: 286.720 ms
- ## epoch start ##
- epoch: 389 step: 16, loss is 0.21142706
- epoch time: 5651.473 ms, per step time: 353.217 ms
- ## epoch start ##
- epoch: 390 step: 16, loss is 0.12780343
- epoch time: 4412.296 ms, per step time: 275.768 ms
- ## epoch start ##
- epoch: 391 step: 16, loss is 0.09035108
- epoch time: 5206.133 ms, per step time: 325.383 ms
- ## epoch start ##
- epoch: 392 step: 16, loss is 0.11832934
- epoch time: 3137.463 ms, per step time: 196.091 ms
- ## epoch start ##
- epoch: 393 step: 16, loss is 0.009158075
- epoch time: 5522.171 ms, per step time: 345.136 ms
- ## epoch start ##
- epoch: 394 step: 16, loss is 0.07393381
- epoch time: 3289.277 ms, per step time: 205.580 ms
- ## epoch start ##
- epoch: 395 step: 16, loss is 0.06417333
- epoch time: 3644.721 ms, per step time: 227.795 ms
- ## epoch start ##
- epoch: 396 step: 16, loss is 0.028627696
- epoch time: 6408.070 ms, per step time: 400.504 ms
- ## epoch start ##
- epoch: 397 step: 16, loss is 0.12611517
- epoch time: 4516.794 ms, per step time: 282.300 ms
- ## epoch start ##
- epoch: 398 step: 16, loss is 0.025906585
- epoch time: 6950.024 ms, per step time: 434.376 ms
- ## epoch start ##
- epoch: 399 step: 16, loss is 0.10733242
- epoch time: 3156.878 ms, per step time: 197.305 ms
- ## epoch start ##
- epoch: 400 step: 16, loss is 0.0940124
- epoch time: 2532.533 ms, per step time: 158.283 ms
- ## epoch start ##
- epoch: 401 step: 16, loss is 0.13671851
- epoch time: 6265.509 ms, per step time: 391.594 ms
- ## epoch start ##
- epoch: 402 step: 16, loss is 0.13359387
- epoch time: 4272.231 ms, per step time: 267.014 ms
- ## epoch start ##
- epoch: 403 step: 16, loss is 0.0867184
- epoch time: 5523.998 ms, per step time: 345.250 ms
- ## epoch start ##
- epoch: 404 step: 16, loss is 0.007402207
- epoch time: 5122.374 ms, per step time: 320.148 ms
- ## epoch start ##
- epoch: 405 step: 16, loss is 0.10627076
- epoch time: 4643.825 ms, per step time: 290.239 ms
- ## epoch start ##
- epoch: 406 step: 16, loss is 0.0041180453
- epoch time: 4340.743 ms, per step time: 271.296 ms
- ## epoch start ##
- epoch: 407 step: 16, loss is 0.11032787
- epoch time: 1891.422 ms, per step time: 118.214 ms
- ## epoch start ##
- epoch: 408 step: 16, loss is 0.16229457
- epoch time: 6655.718 ms, per step time: 415.982 ms
- ## epoch start ##
- epoch: 409 step: 16, loss is 0.029949421
- epoch time: 3944.314 ms, per step time: 246.520 ms
- ## epoch start ##
- epoch: 410 step: 16, loss is 0.102188006
- epoch time: 6357.163 ms, per step time: 397.323 ms
- ## epoch start ##
- epoch: 411 step: 16, loss is 0.054575644
- epoch time: 3718.508 ms, per step time: 232.407 ms
- ## epoch start ##
- epoch: 412 step: 16, loss is 0.08860176
- epoch time: 3482.048 ms, per step time: 217.628 ms
- ## epoch start ##
- epoch: 413 step: 16, loss is 0.0035920476
- epoch time: 7703.164 ms, per step time: 481.448 ms
- ## epoch start ##
- epoch: 414 step: 16, loss is 0.104002155
- epoch time: 4201.634 ms, per step time: 262.602 ms
- ## epoch start ##
- epoch: 415 step: 16, loss is 0.1553461
- epoch time: 6155.704 ms, per step time: 384.732 ms
- ## epoch start ##
- epoch: 416 step: 16, loss is 0.11470117
- epoch time: 3689.870 ms, per step time: 230.617 ms
- ## epoch start ##
- epoch: 417 step: 16, loss is 0.13089345
- epoch time: 4936.815 ms, per step time: 308.551 ms
- ## epoch start ##
- epoch: 418 step: 16, loss is 0.044403266
- epoch time: 3457.835 ms, per step time: 216.115 ms
- ## epoch start ##
- epoch: 419 step: 16, loss is 0.037987337
- epoch time: 2206.846 ms, per step time: 137.928 ms
- ## epoch start ##
- epoch: 420 step: 16, loss is 0.037058167
- epoch time: 7792.473 ms, per step time: 487.030 ms
- ## epoch start ##
- epoch: 421 step: 16, loss is 0.09027269
- epoch time: 3843.170 ms, per step time: 240.198 ms
- ## epoch start ##
- epoch: 422 step: 16, loss is 0.17314671
- epoch time: 5857.809 ms, per step time: 366.113 ms
- ## epoch start ##
- epoch: 423 step: 16, loss is 0.092219695
- epoch time: 5975.301 ms, per step time: 373.456 ms
- ## epoch start ##
- epoch: 424 step: 16, loss is 0.06684129
- epoch time: 2180.365 ms, per step time: 136.273 ms
- ## epoch start ##
- epoch: 425 step: 16, loss is 0.012808695
- epoch time: 4927.389 ms, per step time: 307.962 ms
- ## epoch start ##
- epoch: 426 step: 16, loss is 0.012071252
- epoch time: 4971.949 ms, per step time: 310.747 ms
- ## epoch start ##
- epoch: 427 step: 16, loss is 0.0349284
- epoch time: 6313.775 ms, per step time: 394.611 ms
- ## epoch start ##
- epoch: 428 step: 16, loss is 0.07546385
- epoch time: 3663.217 ms, per step time: 228.951 ms
- ## epoch start ##
- epoch: 429 step: 16, loss is 0.008433548
- epoch time: 1935.547 ms, per step time: 120.972 ms
- ## epoch start ##
- epoch: 430 step: 16, loss is 0.050872687
- epoch time: 2723.336 ms, per step time: 170.208 ms
- ## epoch start ##
- epoch: 431 step: 16, loss is 0.0032088032
- epoch time: 5188.861 ms, per step time: 324.304 ms
- ## epoch start ##
- epoch: 432 step: 16, loss is 0.20843813
- epoch time: 3436.617 ms, per step time: 214.789 ms
- ## epoch start ##
- epoch: 433 step: 16, loss is 0.12014855
- epoch time: 4105.155 ms, per step time: 256.572 ms
- ## epoch start ##
- epoch: 434 step: 16, loss is 0.1156838
- epoch time: 6577.254 ms, per step time: 411.078 ms
- ## epoch start ##
- epoch: 435 step: 16, loss is 0.11992125
- epoch time: 1684.829 ms, per step time: 105.302 ms
- ## epoch start ##
- epoch: 436 step: 16, loss is 0.003422587
- epoch time: 2953.244 ms, per step time: 184.578 ms
- ## epoch start ##
- epoch: 437 step: 16, loss is 0.019854814
- epoch time: 6739.430 ms, per step time: 421.214 ms
- ## epoch start ##
- epoch: 438 step: 16, loss is 0.004109321
- epoch time: 4992.192 ms, per step time: 312.012 ms
- ## epoch start ##
- epoch: 439 step: 16, loss is 0.13793714
- epoch time: 5773.294 ms, per step time: 360.831 ms
- ## epoch start ##
- epoch: 440 step: 16, loss is 0.1736365
- epoch time: 3011.968 ms, per step time: 188.248 ms
- ## epoch start ##
- epoch: 441 step: 16, loss is 0.005344572
- epoch time: 4672.439 ms, per step time: 292.027 ms
- ## epoch start ##
- epoch: 442 step: 16, loss is 0.089897655
- epoch time: 1372.751 ms, per step time: 85.797 ms
- ## epoch start ##
- epoch: 443 step: 16, loss is 0.03389459
- epoch time: 4773.022 ms, per step time: 298.314 ms
- ## epoch start ##
- epoch: 444 step: 16, loss is 0.113034554
- epoch time: 4268.971 ms, per step time: 266.811 ms
- ## epoch start ##
- epoch: 445 step: 16, loss is 0.055552255
- epoch time: 8134.103 ms, per step time: 508.381 ms
- ## epoch start ##
- epoch: 446 step: 16, loss is 0.16002665
- epoch time: 6566.333 ms, per step time: 410.396 ms
- ## epoch start ##
- epoch: 447 step: 16, loss is 0.11217331
- epoch time: 3623.740 ms, per step time: 226.484 ms
- ## epoch start ##
- epoch: 448 step: 16, loss is 0.005989112
- epoch time: 6368.402 ms, per step time: 398.025 ms
- ## epoch start ##
- epoch: 449 step: 16, loss is 0.04406446
- epoch time: 4661.526 ms, per step time: 291.345 ms
- ## epoch start ##
- epoch: 450 step: 16, loss is 0.153882
- epoch time: 6037.799 ms, per step time: 377.362 ms
- ## epoch start ##
- epoch: 451 step: 16, loss is 0.19830942
- epoch time: 2394.101 ms, per step time: 149.631 ms
- ## epoch start ##
- epoch: 452 step: 16, loss is 0.14568762
- epoch time: 4337.612 ms, per step time: 271.101 ms
- ## epoch start ##
- epoch: 453 step: 16, loss is 0.10796914
- epoch time: 2539.137 ms, per step time: 158.696 ms
- ## epoch start ##
- epoch: 454 step: 16, loss is 0.038898975
- epoch time: 3216.165 ms, per step time: 201.010 ms
- ## epoch start ##
- epoch: 455 step: 16, loss is 0.07610659
- epoch time: 7002.351 ms, per step time: 437.647 ms
- ## epoch start ##
- epoch: 456 step: 16, loss is 0.13620293
- epoch time: 3611.602 ms, per step time: 225.725 ms
- ## epoch start ##
- epoch: 457 step: 16, loss is 0.10428237
- epoch time: 7910.181 ms, per step time: 494.386 ms
- ## epoch start ##
- epoch: 458 step: 16, loss is 0.11610938
- epoch time: 3673.362 ms, per step time: 229.585 ms
- ## epoch start ##
- epoch: 459 step: 16, loss is 0.00998856
- epoch time: 3970.707 ms, per step time: 248.169 ms
- ## epoch start ##
- epoch: 460 step: 16, loss is 0.027998831
- epoch time: 7166.968 ms, per step time: 447.936 ms
- ## epoch start ##
- epoch: 461 step: 16, loss is 0.12192317
- epoch time: 3076.875 ms, per step time: 192.305 ms
- ## epoch start ##
- epoch: 462 step: 16, loss is 0.13169146
- epoch time: 6627.917 ms, per step time: 414.245 ms
- ## epoch start ##
- epoch: 463 step: 16, loss is 0.10143636
- epoch time: 2066.839 ms, per step time: 129.177 ms
- ## epoch start ##
- epoch: 464 step: 16, loss is 0.0453278
- epoch time: 4917.871 ms, per step time: 307.367 ms
- ## epoch start ##
- epoch: 465 step: 16, loss is 0.085412115
- epoch time: 1414.974 ms, per step time: 88.436 ms
- ## epoch start ##
- epoch: 466 step: 16, loss is 0.016131138
- epoch time: 4824.983 ms, per step time: 301.561 ms
- ## epoch start ##
- epoch: 467 step: 16, loss is 0.07947416
- epoch time: 6335.907 ms, per step time: 395.994 ms
- ## epoch start ##
- epoch: 468 step: 16, loss is 0.07334787
- epoch time: 4728.066 ms, per step time: 295.504 ms
- ## epoch start ##
- epoch: 469 step: 16, loss is 0.10918109
- epoch time: 4208.540 ms, per step time: 263.034 ms
- ## epoch start ##
- epoch: 470 step: 16, loss is 0.0686169
- epoch time: 2712.151 ms, per step time: 169.509 ms
- ## epoch start ##
- epoch: 471 step: 16, loss is 0.05655846
- epoch time: 4619.260 ms, per step time: 288.704 ms
- ## epoch start ##
- epoch: 472 step: 16, loss is 0.11451168
- epoch time: 8279.750 ms, per step time: 517.484 ms
- ## epoch start ##
- epoch: 473 step: 16, loss is 0.0824986
- epoch time: 3820.164 ms, per step time: 238.760 ms
- ## epoch start ##
- epoch: 474 step: 16, loss is 0.20840414
- epoch time: 5964.777 ms, per step time: 372.799 ms
- ## epoch start ##
- epoch: 475 step: 16, loss is 0.028441817
- epoch time: 3677.520 ms, per step time: 229.845 ms
- ## epoch start ##
- epoch: 476 step: 16, loss is 0.25228074
- epoch time: 3900.301 ms, per step time: 243.769 ms
- ## epoch start ##
- epoch: 477 step: 16, loss is 0.12670949
- epoch time: 1433.743 ms, per step time: 89.609 ms
- ## epoch start ##
- epoch: 478 step: 16, loss is 0.17274973
- epoch time: 3041.274 ms, per step time: 190.080 ms
- ## epoch start ##
- epoch: 479 step: 16, loss is 0.029060049
- epoch time: 6362.199 ms, per step time: 397.637 ms
- ## epoch start ##
- epoch: 480 step: 16, loss is 0.15791577
- epoch time: 4287.066 ms, per step time: 267.942 ms
- ## epoch start ##
- epoch: 481 step: 16, loss is 0.057811048
- epoch time: 7794.064 ms, per step time: 487.129 ms
- ## epoch start ##
- epoch: 482 step: 16, loss is 0.10344069
- epoch time: 4111.831 ms, per step time: 256.989 ms
- ## epoch start ##
- epoch: 483 step: 16, loss is 0.04061574
- epoch time: 2163.064 ms, per step time: 135.192 ms
- ## epoch start ##
- epoch: 484 step: 16, loss is 0.15692303
- epoch time: 5758.493 ms, per step time: 359.906 ms
- ## epoch start ##
- epoch: 485 step: 16, loss is 0.11569163
- epoch time: 2949.998 ms, per step time: 184.375 ms
- ## epoch start ##
- epoch: 486 step: 16, loss is 0.120636284
- epoch time: 7851.669 ms, per step time: 490.729 ms
- ## epoch start ##
- epoch: 487 step: 16, loss is 0.062426183
- epoch time: 2289.520 ms, per step time: 143.095 ms
- ## epoch start ##
- epoch: 488 step: 16, loss is 0.1645184
- epoch time: 4242.769 ms, per step time: 265.173 ms
- ## epoch start ##
- epoch: 489 step: 16, loss is 0.022097262
- epoch time: 8591.555 ms, per step time: 536.972 ms
- ## epoch start ##
- epoch: 490 step: 16, loss is 0.062303297
- epoch time: 3467.696 ms, per step time: 216.731 ms
- ## epoch start ##
- epoch: 491 step: 16, loss is 0.09494007
- epoch time: 5620.253 ms, per step time: 351.266 ms
- ## epoch start ##
- epoch: 492 step: 16, loss is 0.0030579471
- epoch time: 2599.544 ms, per step time: 162.471 ms
- ## epoch start ##
- epoch: 493 step: 16, loss is 0.006718177
- epoch time: 4627.024 ms, per step time: 289.189 ms
- ## epoch start ##
- epoch: 494 step: 16, loss is 0.007304805
- epoch time: 1304.113 ms, per step time: 81.507 ms
- ## epoch start ##
- epoch: 495 step: 16, loss is 0.060515456
- epoch time: 4592.060 ms, per step time: 287.004 ms
- ## epoch start ##
- epoch: 496 step: 16, loss is 0.23645665
- epoch time: 5270.331 ms, per step time: 329.396 ms
- ## epoch start ##
- epoch: 497 step: 16, loss is 0.13258564
- epoch time: 5203.848 ms, per step time: 325.241 ms
- ## epoch start ##
- epoch: 498 step: 16, loss is 0.008449696
- epoch time: 6742.108 ms, per step time: 421.382 ms
- ## epoch start ##
- epoch: 499 step: 16, loss is 0.20675562
- epoch time: 3300.349 ms, per step time: 206.272 ms
- ## epoch start ##
- epoch: 500 step: 16, loss is 0.07841814
- epoch time: 2684.056 ms, per step time: 167.754 ms
- ## epoch start ##
- epoch: 501 step: 16, loss is 0.010832556
- epoch time: 8503.158 ms, per step time: 531.447 ms
- ## epoch start ##
- epoch: 502 step: 16, loss is 0.034103632
- epoch time: 4102.489 ms, per step time: 256.406 ms
- ## epoch start ##
- epoch: 503 step: 16, loss is 0.032436173
- epoch time: 7675.268 ms, per step time: 479.704 ms
- ## epoch start ##
- epoch: 504 step: 16, loss is 0.27409503
- epoch time: 3870.961 ms, per step time: 241.935 ms
- ## epoch start ##
- epoch: 505 step: 16, loss is 0.095045015
- epoch time: 2994.508 ms, per step time: 187.157 ms
- ## epoch start ##
- epoch: 506 step: 16, loss is 0.036163855
- epoch time: 3547.872 ms, per step time: 221.742 ms
- ## epoch start ##
- epoch: 507 step: 16, loss is 0.08993706
- epoch time: 3878.773 ms, per step time: 242.423 ms
- ## epoch start ##
- epoch: 508 step: 16, loss is 0.12810193
- epoch time: 7659.477 ms, per step time: 478.717 ms
- ## epoch start ##
- epoch: 509 step: 16, loss is 0.24714255
- epoch time: 4212.922 ms, per step time: 263.308 ms
- ## epoch start ##
- epoch: 510 step: 16, loss is 0.041269567
- epoch time: 4812.991 ms, per step time: 300.812 ms
- ## epoch start ##
- epoch: 511 step: 16, loss is 0.013865722
- epoch time: 4040.497 ms, per step time: 252.531 ms
- ## epoch start ##
- epoch: 512 step: 16, loss is 0.116782896
- epoch time: 3165.188 ms, per step time: 197.824 ms
- ## epoch start ##
- epoch: 513 step: 16, loss is 0.15571612
- epoch time: 5165.578 ms, per step time: 322.849 ms
- ## epoch start ##
- epoch: 514 step: 16, loss is 0.24646606
- epoch time: 3302.310 ms, per step time: 206.394 ms
- ## epoch start ##
- epoch: 515 step: 16, loss is 0.08860675
- epoch time: 6187.737 ms, per step time: 386.734 ms
- ## epoch start ##
- epoch: 516 step: 16, loss is 0.17805865
- epoch time: 2921.788 ms, per step time: 182.612 ms
- ## epoch start ##
- epoch: 517 step: 16, loss is 0.0046912306
- epoch time: 4421.654 ms, per step time: 276.353 ms
- ## epoch start ##
- epoch: 518 step: 16, loss is 0.12193041
- epoch time: 3127.515 ms, per step time: 195.470 ms
- ## epoch start ##
- epoch: 519 step: 16, loss is 0.44036594
- epoch time: 4012.318 ms, per step time: 250.770 ms
- ## epoch start ##
- epoch: 520 step: 16, loss is 0.0053014983
- epoch time: 5700.089 ms, per step time: 356.256 ms
- ## epoch start ##
- epoch: 521 step: 16, loss is 0.150331
- epoch time: 5899.637 ms, per step time: 368.727 ms
- ## epoch start ##
- epoch: 522 step: 16, loss is 0.067040116
- epoch time: 4504.696 ms, per step time: 281.544 ms
- ## epoch start ##
- epoch: 523 step: 16, loss is 0.0834959
- epoch time: 5729.601 ms, per step time: 358.100 ms
- ## epoch start ##
- epoch: 524 step: 16, loss is 0.09282332
- epoch time: 5509.986 ms, per step time: 344.374 ms
- ## epoch start ##
- epoch: 525 step: 16, loss is 0.095110394
- epoch time: 4477.590 ms, per step time: 279.849 ms
- ## epoch start ##
- epoch: 526 step: 16, loss is 0.06949193
- epoch time: 3189.323 ms, per step time: 199.333 ms
- ## epoch start ##
- epoch: 527 step: 16, loss is 0.072623536
- epoch time: 7287.010 ms, per step time: 455.438 ms
- ## epoch start ##
- epoch: 528 step: 16, loss is 0.2638216
- epoch time: 4688.088 ms, per step time: 293.005 ms
- ## epoch start ##
- epoch: 529 step: 16, loss is 0.041845374
- epoch time: 3727.138 ms, per step time: 232.946 ms
- ## epoch start ##
- epoch: 530 step: 16, loss is 0.049906254
- epoch time: 3589.177 ms, per step time: 224.324 ms
- ## epoch start ##
- epoch: 531 step: 16, loss is 0.0033808902
- epoch time: 3615.132 ms, per step time: 225.946 ms
- ## epoch start ##
- epoch: 532 step: 16, loss is 0.10107186
- epoch time: 5421.268 ms, per step time: 338.829 ms
- ## epoch start ##
- epoch: 533 step: 16, loss is 0.0043776864
- epoch time: 4483.621 ms, per step time: 280.226 ms
- ## epoch start ##
- epoch: 534 step: 16, loss is 0.04377621
- epoch time: 7287.944 ms, per step time: 455.497 ms
- ## epoch start ##
- epoch: 535 step: 16, loss is 0.0074575515
- epoch time: 5117.967 ms, per step time: 319.873 ms
- ## epoch start ##
- epoch: 536 step: 16, loss is 0.045615494
- epoch time: 1887.417 ms, per step time: 117.964 ms
- ## epoch start ##
- epoch: 537 step: 16, loss is 0.0057726605
- epoch time: 6470.435 ms, per step time: 404.402 ms
- ## epoch start ##
- epoch: 538 step: 16, loss is 0.039240506
- epoch time: 4870.066 ms, per step time: 304.379 ms
- ## epoch start ##
- epoch: 539 step: 16, loss is 0.09494556
- epoch time: 5680.588 ms, per step time: 355.037 ms
- ## epoch start ##
- epoch: 540 step: 16, loss is 0.09845539
- epoch time: 4787.597 ms, per step time: 299.225 ms
- ## epoch start ##
- epoch: 541 step: 16, loss is 0.0063031907
- epoch time: 3004.621 ms, per step time: 187.789 ms
- ## epoch start ##
- epoch: 542 step: 16, loss is 0.16502678
- epoch time: 3940.138 ms, per step time: 246.259 ms
- ## epoch start ##
- epoch: 543 step: 16, loss is 0.08008573
- epoch time: 2155.572 ms, per step time: 134.723 ms
- ## epoch start ##
- epoch: 544 step: 16, loss is 0.09346569
- epoch time: 5359.239 ms, per step time: 334.952 ms
- ## epoch start ##
- epoch: 545 step: 16, loss is 0.020571325
- epoch time: 5684.039 ms, per step time: 355.252 ms
- ## epoch start ##
- epoch: 546 step: 16, loss is 0.028909665
- epoch time: 6061.203 ms, per step time: 378.825 ms
- ## epoch start ##
- epoch: 547 step: 16, loss is 0.13101415
- epoch time: 3238.071 ms, per step time: 202.379 ms
- ## epoch start ##
- epoch: 548 step: 16, loss is 0.22522503
- epoch time: 2444.610 ms, per step time: 152.788 ms
- ## epoch start ##
- epoch: 549 step: 16, loss is 0.030512318
- epoch time: 4945.916 ms, per step time: 309.120 ms
- ## epoch start ##
- epoch: 550 step: 16, loss is 0.11068125
- epoch time: 4260.815 ms, per step time: 266.301 ms
- ## epoch start ##
- epoch: 551 step: 16, loss is 0.09286743
- epoch time: 7580.812 ms, per step time: 473.801 ms
- ## epoch start ##
- epoch: 552 step: 16, loss is 0.022085268
- epoch time: 5542.495 ms, per step time: 346.406 ms
- ## epoch start ##
- epoch: 553 step: 16, loss is 0.0022463154
- epoch time: 2152.098 ms, per step time: 134.506 ms
- ## epoch start ##
- epoch: 554 step: 16, loss is 0.020700531
- epoch time: 3177.176 ms, per step time: 198.574 ms
- ## epoch start ##
- epoch: 555 step: 16, loss is 0.06769657
- epoch time: 3101.641 ms, per step time: 193.853 ms
- ## epoch start ##
- epoch: 556 step: 16, loss is 0.25841218
- epoch time: 6282.712 ms, per step time: 392.669 ms
- ## epoch start ##
- epoch: 557 step: 16, loss is 0.0031835868
- epoch time: 3393.732 ms, per step time: 212.108 ms
- ## epoch start ##
- epoch: 558 step: 16, loss is 0.050049044
- epoch time: 6542.687 ms, per step time: 408.918 ms
- ## epoch start ##
- epoch: 559 step: 16, loss is 0.058709353
- epoch time: 5001.666 ms, per step time: 312.604 ms
- ## epoch start ##
- epoch: 560 step: 16, loss is 0.0027338373
- epoch time: 4274.555 ms, per step time: 267.160 ms
- ## epoch start ##
- epoch: 561 step: 16, loss is 0.06701562
- epoch time: 9528.261 ms, per step time: 595.516 ms
- ## epoch start ##
- epoch: 562 step: 16, loss is 0.0031235847
- epoch time: 4684.240 ms, per step time: 292.765 ms
- ## epoch start ##
- epoch: 563 step: 16, loss is 0.09706271
- epoch time: 7895.236 ms, per step time: 493.452 ms
- ## epoch start ##
- epoch: 564 step: 16, loss is 0.06067055
- epoch time: 5042.775 ms, per step time: 315.173 ms
- ## epoch start ##
- epoch: 565 step: 16, loss is 0.029923167
- epoch time: 3661.979 ms, per step time: 228.874 ms
- ## epoch start ##
- epoch: 566 step: 16, loss is 0.16610283
- epoch time: 1761.124 ms, per step time: 110.070 ms
- ## epoch start ##
- epoch: 567 step: 16, loss is 0.042695455
- epoch time: 5212.997 ms, per step time: 325.812 ms
- ## epoch start ##
- epoch: 568 step: 16, loss is 0.01888522
- epoch time: 4828.108 ms, per step time: 301.757 ms
- ## epoch start ##
- epoch: 569 step: 16, loss is 0.049555223
- epoch time: 3379.910 ms, per step time: 211.244 ms
- ## epoch start ##
- epoch: 570 step: 16, loss is 0.090840235
- epoch time: 7044.168 ms, per step time: 440.260 ms
- ## epoch start ##
- epoch: 571 step: 16, loss is 0.15223944
- epoch time: 3589.596 ms, per step time: 224.350 ms
- ## epoch start ##
- epoch: 572 step: 16, loss is 0.0025729805
- epoch time: 3816.193 ms, per step time: 238.512 ms
- ## epoch start ##
- epoch: 573 step: 16, loss is 0.007698009
- epoch time: 5522.482 ms, per step time: 345.155 ms
- ## epoch start ##
- epoch: 574 step: 16, loss is 0.054480944
- epoch time: 2191.291 ms, per step time: 136.956 ms
- ## epoch start ##
- epoch: 575 step: 16, loss is 0.05018974
- epoch time: 9182.541 ms, per step time: 573.909 ms
- ## epoch start ##
- epoch: 576 step: 16, loss is 0.08208209
- epoch time: 4705.915 ms, per step time: 294.120 ms
- ## epoch start ##
- epoch: 577 step: 16, loss is 0.011642734
- epoch time: 3313.709 ms, per step time: 207.107 ms
- ## epoch start ##
- epoch: 578 step: 16, loss is 0.036427103
- epoch time: 2200.886 ms, per step time: 137.555 ms
- ## epoch start ##
- epoch: 579 step: 16, loss is 0.023341
- epoch time: 3401.455 ms, per step time: 212.591 ms
- ## epoch start ##
- epoch: 580 step: 16, loss is 0.003960892
- epoch time: 6787.790 ms, per step time: 424.237 ms
- ## epoch start ##
- epoch: 581 step: 16, loss is 0.024213972
- epoch time: 4317.768 ms, per step time: 269.861 ms
- ## epoch start ##
- epoch: 582 step: 16, loss is 0.008063064
- epoch time: 7026.534 ms, per step time: 439.158 ms
- ## epoch start ##
- epoch: 583 step: 16, loss is 0.110130616
- epoch time: 3739.025 ms, per step time: 233.689 ms
- ## epoch start ##
- epoch: 584 step: 16, loss is 0.047186837
- epoch time: 3327.890 ms, per step time: 207.993 ms
- ## epoch start ##
- epoch: 585 step: 16, loss is 0.06359781
- epoch time: 7326.466 ms, per step time: 457.904 ms
- ## epoch start ##
- epoch: 586 step: 16, loss is 0.113720655
- epoch time: 2156.050 ms, per step time: 134.753 ms
- ## epoch start ##
- epoch: 587 step: 16, loss is 0.065203436
- epoch time: 6514.750 ms, per step time: 407.172 ms
- ## epoch start ##
- epoch: 588 step: 16, loss is 0.041729882
- epoch time: 3236.924 ms, per step time: 202.308 ms
- ## epoch start ##
- epoch: 589 step: 16, loss is 0.35402554
- epoch time: 4599.653 ms, per step time: 287.478 ms
- ## epoch start ##
- epoch: 590 step: 16, loss is 0.005285474
- epoch time: 1534.069 ms, per step time: 95.879 ms
- ## epoch start ##
- epoch: 591 step: 16, loss is 0.08613648
- epoch time: 3949.193 ms, per step time: 246.825 ms
- ## epoch start ##
- epoch: 592 step: 16, loss is 0.047263622
- epoch time: 5043.278 ms, per step time: 315.205 ms
- ## epoch start ##
- epoch: 593 step: 16, loss is 0.034702994
- epoch time: 3095.919 ms, per step time: 193.495 ms
- ## epoch start ##
- epoch: 594 step: 16, loss is 0.0044206497
- epoch time: 6476.953 ms, per step time: 404.810 ms
- ## epoch start ##
- epoch: 595 step: 16, loss is 0.002584256
- epoch time: 3482.388 ms, per step time: 217.649 ms
- ## epoch start ##
- epoch: 596 step: 16, loss is 0.13604516
- epoch time: 3601.942 ms, per step time: 225.121 ms
- ## epoch start ##
- epoch: 597 step: 16, loss is 0.122254655
- epoch time: 5821.329 ms, per step time: 363.833 ms
- ## epoch start ##
- epoch: 598 step: 16, loss is 0.09867697
- epoch time: 3806.237 ms, per step time: 237.890 ms
- ## epoch start ##
- epoch: 599 step: 16, loss is 0.13943313
- epoch time: 6058.400 ms, per step time: 378.650 ms
- ## epoch start ##
- epoch: 600 step: 16, loss is 0.0022921981
- epoch time: 2778.368 ms, per step time: 173.648 ms
|