ImportError: cannot import name 'create_dataset' from 'dataset'

问题描述

开始日志报错

Traceback (most recent call last):
File "/cache/code/2023naic/Main_Mindspore.py", line 17, in
from dataset import create_dataset
ModuleNotFoundError: No module named 'dataset'

在启动文件中增加
os.system('pip install dataset') 后

报错

Successfully installed Mako-1.2.4 alembic-1.12.0 banal-1.0.6 dataset-1.6.2 greenlet-3.0.0 importlib-resources-5.12.0 sqlalchemy-1.4.49
Traceback (most recent call last):
File "/cache/code/2023naic/Main_Mindspore.py", line 18, in
from dataset import create_dataset
ImportError: cannot import name 'create_dataset' from 'dataset' (/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/dataset/init.py)
INFO:root:List OBS time cost: 0.13 seconds.

任务类型（调试/训练/推理）

训练任务

任务名

deepl202310061266003

日志说明或问题截图

[ModelArts Service Log]2023-10-06 12:20:55,126 - INFO - Route plan ends for env ROUTE_PLAN = false. Route plan acceleration service may not be available in this region
[ModelArts Service Log]2023-10-06 12:20:55,127 - INFO - env already exists. env_name: HCCL_CONNECT_TIMEOUT, env_value: 1800
[ModelArts Service Log]2023-10-06 12:20:55,128 - INFO - bootstrap proc-rank-0-device-0
[ModelArts Service Log]2023-10-06 12:20:55,134 - INFO - proc-rank-0-device-0 (pid: 129)
INFO:root:Listing OBS: 1000
INFO:root:Listing OBS: 2000
Looking in indexes: http://pip.modelarts.private.com:8888/repository/pypi/simple
Collecting dataset
Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/dataset/1.6.2/dataset-1.6.2-py2.py3-none-any.whl (18 kB)
Collecting sqlalchemy<2.0.0,>=1.3.2 (from dataset)
Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/sqlalchemy/1.4.49/SQLAlchemy-1.4.49-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 55.6 MB/s eta 0:00:00
Collecting alembic>=0.6.2 (from dataset)
Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/alembic/1.12.0/alembic-1.12.0-py3-none-any.whl (226 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 226.0/226.0 kB 34.8 MB/s eta 0:00:00
Collecting banal>=1.0.1 (from dataset)
Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/banal/1.0.6/banal-1.0.6-py2.py3-none-any.whl (6.1 kB)
Collecting Mako (from alembic>=0.6.2->dataset)
Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/mako/1.2.4/Mako-1.2.4-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 7.6 MB/s eta 0:00:00
Requirement already satisfied: typing-extensions>=4 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from alembic>=0.6.2->dataset) (4.3.0)
Requirement already satisfied: importlib-metadata in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from alembic>=0.6.2->dataset) (4.12.0)
Collecting importlib-resources (from alembic>=0.6.2->dataset)
Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/importlib-resources/5.12.0/importlib_resources-5.12.0-py3-none-any.whl (36 kB)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='pip.modelarts.private.com', port=8888): Read timed out. (read timeout=15)")': /repository/pypi/simple/greenlet/
Collecting greenlet!=0.4.17 (from sqlalchemy<2.0.0,>=1.3.2->dataset)
Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/greenlet/3.0.0/greenlet-3.0.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (610 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 610.1/610.1 kB 52.8 MB/s eta 0:00:00
Requirement already satisfied: zipp>=0.5 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from importlib-metadata->alembic>=0.6.2->dataset) (3.8.1)
Requirement already satisfied: MarkupSafe>=0.9.2 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from Mako->alembic>=0.6.2->dataset) (2.1.1)
DEPRECATION: moxing-framework 2.1.7.dc1f3d0b has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of moxing-framework or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: banal, importlib-resources, greenlet, sqlalchemy, Mako, alembic, dataset
Successfully installed Mako-1.2.4 alembic-1.12.0 banal-1.0.6 dataset-1.6.2 greenlet-3.0.0 importlib-resources-5.12.0 sqlalchemy-1.4.49
Traceback (most recent call last):
File "/cache/code/2023naic/Main_Mindspore.py", line 18, in
from dataset import create_dataset
ImportError: cannot import name 'create_dataset' from 'dataset' (/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/dataset/init.py)
INFO:root:List OBS time cost: 0.13 seconds.

期望的解决方案或建议

给出可以在该环境训练的训练数据读取命令。或者解决dataset包相关问题。
无法找到是因为版本问题么？

### 问题描述开始日志报错 Traceback (most recent call last): File "/cache/code/2023naic/Main_Mindspore.py", line 17, in <module> from dataset import create_dataset ModuleNotFoundError: No module named 'dataset' 在启动文件中增加 os.system('pip install dataset') 后报错 Successfully installed Mako-1.2.4 alembic-1.12.0 banal-1.0.6 dataset-1.6.2 greenlet-3.0.0 importlib-resources-5.12.0 sqlalchemy-1.4.49 Traceback (most recent call last): File "/cache/code/2023naic/Main_Mindspore.py", line 18, in <module> from dataset import create_dataset ImportError: cannot import name 'create_dataset' from 'dataset' (/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/dataset/__init__.py) INFO:root:List OBS time cost: 0.13 seconds. ### 相关环境（GPU/NPU） NPU ### 相关集群（启智/智算）智算 ### 任务类型（调试/训练/推理）训练任务 ### 任务名 deepl202310061266003 ### 日志说明或问题截图 [ModelArts Service Log]2023-10-06 12:20:55,126 - INFO - Route plan ends for env ROUTE_PLAN = false. Route plan acceleration service may not be available in this region [ModelArts Service Log]2023-10-06 12:20:55,127 - INFO - env already exists. env_name: HCCL_CONNECT_TIMEOUT, env_value: 1800 [ModelArts Service Log]2023-10-06 12:20:55,128 - INFO - bootstrap proc-rank-0-device-0 [ModelArts Service Log]2023-10-06 12:20:55,134 - INFO - proc-rank-0-device-0 (pid: 129) INFO:root:Listing OBS: 1000 INFO:root:Listing OBS: 2000 Looking in indexes: http://pip.modelarts.private.com:8888/repository/pypi/simple Collecting dataset Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/dataset/1.6.2/dataset-1.6.2-py2.py3-none-any.whl (18 kB) Collecting sqlalchemy<2.0.0,>=1.3.2 (from dataset) Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/sqlalchemy/1.4.49/SQLAlchemy-1.4.49-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 55.6 MB/s eta 0:00:00 Collecting alembic>=0.6.2 (from dataset) Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/alembic/1.12.0/alembic-1.12.0-py3-none-any.whl (226 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 226.0/226.0 kB 34.8 MB/s eta 0:00:00 Collecting banal>=1.0.1 (from dataset) Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/banal/1.0.6/banal-1.0.6-py2.py3-none-any.whl (6.1 kB) Collecting Mako (from alembic>=0.6.2->dataset) Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/mako/1.2.4/Mako-1.2.4-py3-none-any.whl (78 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 7.6 MB/s eta 0:00:00 Requirement already satisfied: typing-extensions>=4 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from alembic>=0.6.2->dataset) (4.3.0) Requirement already satisfied: importlib-metadata in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from alembic>=0.6.2->dataset) (4.12.0) Collecting importlib-resources (from alembic>=0.6.2->dataset) Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/importlib-resources/5.12.0/importlib_resources-5.12.0-py3-none-any.whl (36 kB) WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='pip.modelarts.private.com', port=8888): Read timed out. (read timeout=15)")': /repository/pypi/simple/greenlet/ Collecting greenlet!=0.4.17 (from sqlalchemy<2.0.0,>=1.3.2->dataset) Downloading http://pip.modelarts.private.com:8888/repository/pypi/packages/greenlet/3.0.0/greenlet-3.0.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (610 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 610.1/610.1 kB 52.8 MB/s eta 0:00:00 Requirement already satisfied: zipp>=0.5 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from importlib-metadata->alembic>=0.6.2->dataset) (3.8.1) Requirement already satisfied: MarkupSafe>=0.9.2 in /home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages (from Mako->alembic>=0.6.2->dataset) (2.1.1) DEPRECATION: moxing-framework 2.1.7.dc1f3d0b has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of moxing-framework or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063 Installing collected packages: banal, importlib-resources, greenlet, sqlalchemy, Mako, alembic, dataset Successfully installed Mako-1.2.4 alembic-1.12.0 banal-1.0.6 dataset-1.6.2 greenlet-3.0.0 importlib-resources-5.12.0 sqlalchemy-1.4.49 Traceback (most recent call last): File "/cache/code/2023naic/Main_Mindspore.py", line 18, in <module> from dataset import create_dataset ImportError: cannot import name 'create_dataset' from 'dataset' (/home/ma-user/anaconda3/envs/MindSpore/lib/python3.7/site-packages/dataset/__init__.py) INFO:root:List OBS time cost: 0.13 seconds. ### 期望的解决方案或建议给出可以在该环境训练的训练数据读取命令。或者解决dataset包相关问题。无法找到是因为版本问题么？

屏幕截图 2023-10-06 123148.png

168 KiB

image.png

168 KiB

#1146 ImportError: cannot import name 'create_dataset' from 'dataset'

问题描述

相关环境（GPU/NPU）

相关集群（启智/智算）

任务类型（调试/训练/推理）

任务名

日志说明或问题截图

期望的解决方案或建议