你确认删除该任务么？此任务一旦删除不可恢复。
您最多选择25个标签
标签必须以中文、字母或数字开头，可以包含连字符 (-)，并且长度不得超过35个字符
15665118915@139.com
47fd48fa3e
|
1年前 | |
|---|---|---|
| scripts | 1年前 | |
| src | 1年前 | |
| README.en.md | 1年前 | |
| README.md | 1年前 | |
| default_config.yaml | 1年前 | |
| default_config_all.yaml | 1年前 | |
| default_config_yuanshi.yaml | 1年前 | |
| eval.py | 1年前 | |
| export.py | 1年前 | |
| pip-requirements.txt | 1年前 | |
| requirements.txt | 1年前 | |
| test_with_grad.py | 1年前 | |
| test_with_grad_copy.py | 1年前 | |
| train.py | 1年前 | |
README.md
Contents
- Contents
- Flownet2 Description
- Model Architecture
- Dataset
- Environment Requirements
- Quick Start
- Script Description
- Model Description
- ModelZoo Homepage
Flownet2 Description
FlowNet2.0, a deep network proposed in 2017, which performs end-to-end learning on optical flow data.
It is optimized based on the FlowNet network , The large improvements in quality and
speed are caused by three major contributions: first, it
focus on the training data and show that the schedule of
presenting data during training is very important. Second,
it develop a stacked architecture that includes warping
of the second image with intermediate optical flow. Third,
it elaborate on small displacements by introducing a subnetwork specializing on small motions.
Compared with the FLownet network, FlowNet 2.0 is only
marginally slower than the original FlowNet but decreases
the estimation error by more than 50%.
FlowNet2 paper：Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox
FlowNet paper：Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox
Model Architecture
The FlowNet2 network is stacked by multiple flownet sub-modules. After the output of the previous network is processed by warp, it is used as the input of the second network.
The model structure is flowNet2CSS and FlowNet2SD two sub-networks fuse the output through the FlownetFusion network, and the entire large network structure formed is FLowNet2
The FlowNet2CSS network is a stack of FLowNet2C and two FLowNet2S. The specific structure can be further understood according to the paper
This source code provides the following model structure, which can be configured and used in the yaml file:
- FlowNet2S
- FlowNet2C
- FlowNet2CS
- FlowNet2CSS
- FlowNet2SD
- FlowNet2
Dataset
Dataset used: FlyingChairs
- Dataset size：31GB，22,872 pairs 512*384 colorful images
- Data format：PPM
- Note：Data will be processed in src/dataset.py
Dataset used: ChairsSDHom
- Dataset size: 51GB, 21,668 pairs 512*384 colorful images
- Train：20,965 pairs image
- Test： 703 pairs image
- Data format：PNG
- Note: Data will be processed in src/dataset.py
Dataset used: MpiSintel
- Dataset size: 536M, 1024 x 436 colorful images in 23 classes
- MpiSintelClean：1150 images
- MpiSintelFinal： 1150 images
- Data format：PNG
- Note: Data will be processed in src/dataset.py
Environment Requirements
- Hardware（Ascend/GPU/CPU）
- Prepare hardware environment with Ascend/GPU/CPU processor.
- Framework
- For more information, please check the resources below：
Quick Start
After installing MindSpore via the official website, you can start training and evaluation as follows:
-
download pretrained parameter
FlowNet2 620MB
FlowNet2-C 149MB
FlowNet2-CS 297MB
FlowNet2-CSS 445MB
FlowNet2-CSS-ft-sd 445MB
FlowNet2-S 148MB
FlowNet2-SD 173MB
-
convert pretrained parameter (from pytorch pretrained parameter to mindspore pretained parameter,so the env should both installed torch and mindspore)
convert pytorch pretrained parameter to mindspore pretrained parameterbash run_ckpt_convert.sh [PYTORCH_FILE_PATH] [MINDSPORE_FILE_PATH] # example: bash run_ckpt_convert.sh /path/to/flownet2.pth.tar /path/to/flownet2.ckpt -
config pretrained parameter path in yaml file
pre_trained: # whether use pretrained parameter file 'True' or 'False' pre_trained_ckpt_path: # pretrained checkpoint file path # 实例： pre_trained: True pre_trained_ckpt_path: /path/checkpoint/flownet2.ckpt -
config dataset name and path in yaml file
train_data: [DATASET_NAME] # Name of dataset, 'FlyingChairs' or 'MpiSintelFinal' or 'MpiSintelClean' train_data_path：[DATASET_PATH] # path of dataset # example： train_data: FlyingChairs train_data_path: /path/to/FlyingChairs_release/data -
running on GPU
For running on GPU, please change
device_targetfromAscendtoGPUin configuration file default_config.yaml# run training example export CUDA_VISIBLE_DEVICES=0 python train.py > train.log 2>&1 & # run distributed training example bash scripts/run_train_gpu.sh 8 0,1,2,3,4,5,6,7 # run evaluation example python eval.py --crop_type="Center" --eval_checkpoint_path=[EVAL_CHECKPOINT_PATH] > eval.log 2>&1 & OR bash scripts/run_eval_gpu.sh [MpiSintelClean/MpiSintelFinal] [DATA_PATH] [MODEL_NAME] [CKPT_PATH] [DEVICE_ID]
We use FlyingChairs dataset by default. Your can also pass $dataset_type to the scripts so that select different datasets. For more details, please refer the specify script.
Script Description
Script and Sample Code
├── model_zoo
├── README.md // descriptions about all the models
├── flownet2
├── README.md // descriptions about flownet2
├── scripts
│ ├── run_ckpt_convert.sh // shell script for converting pytorch ckpt file to pickle file on GPU
│ ├── run_eval_gpu.sh // shell script for eval on GPU
│ └── run_train_gpu.sh // shell script for training on GPU
├── src
│ ├── dataset.py // creating dataset
│ ├── eval_callback.py // eval callback when training
│ ├── metric.py // metric to calculate mean error
│ ├── model_utils
│ │ ├── ckpt_convert.py // convert pytorch ckpt file to pickle file
│ │ ├── config.py // parameter configuration
│ │ ├── device_adapter.py // device adapter
│ │ ├── local_adapter.py // local adapter
│ │ ├── moxing_adapter.py // moxing adapter
│ │ ├── frame_utils.py // utils to read files of dataset
│ │ └── tools.py // tools to match class with paratmeter from config
│ ├── models.py // FlowNet2/FlowNet2CSS/FlowNet2CS/FlowNet2C/FlowNet2S/FlowNet2SD model
│ └── submodels
│ ├── FlowNetC.py // FlowNetC model
│ ├── FlowNetFusion.py // FlowNetFusion model
│ ├── FlowNetS.py // FlowNetS model
│ ├── FlowNetSD.py // FlowNetSD model
│ └── submodules.py // submodules used in flownet model
├── default_config.yaml // parameter configuration
├── requirements.txt // requirements configuration
├── export.py // convert mindspore ckpt file to MINDIR file
├── eval.py // evaluation script
└── train.py // training script
Script Parameters
Parameters for both training and evaluation can be set in config.py
-
config for FLowNet2
# ============================================================================== # Device device_target: "GPU" device_id: 0 # Dataset Setup crop_type: Random # Type of cropping operation (Random and Center) crop_size: [384, 512] # (Height, Width) of image when training eval_size: [256, 256] # (Height, Width) of image when eval # Experiment Setup model: "FlowNet2" # Name of model to be loaded rgb_max: 255 # rgb channel used batchNorm: False # boolean switch to whether add batchnorm before conv lr: 1e-6 # Learning rate num_parallel_workers: 2 # Number of CPU worker used to load data max_rowsize: 2 # Number of max rowsize used to load data batch_size: 2 # Numbers of image pairs in a mini-batch epoch_size: 100 # Total number of epochs pre_trained: 1 # Load pretrained network pre_trained_ckpt_path: "/path/flownet2.ckpt" # Pretrained ckpt path seed: 1 # Seed for reproducibility is_dynamicLoss_scale: 0 # Using dynamicLoss scale or fix scale scale: 1024 # Fix scale value weight_decay: 0.00001 # Weight decay train_data: "FlyingChairs" # Train Dataset name train_data_path: "/path/ds/FlyingChairs_release/data" # Train Dataset path # Train Setup run_distribute: 1 # Distributed training or not is_save_on_master: 1 # Only save ckpt on master device save_checkpoint: 1 # Is save ckpt while training save_ckpt_interval: 2 # Saving ckpt interval keep_checkpoint_max: 20 # Max ckpt file number save_checkpoint_path: "/path/ckpt/" # Ckpt save path # eval Setup eval_data: "MpiSintelClean" # Eval Dataset name eval_data_path: "/home/shm/ds/training" # Eval Dataset path eval_checkpoint_path: "/path/flownet2.ckpt" # Ckpt path used to eval run_evalCallback: 1 # Is run evalCallBack while training eval_start_epoch: 1 # EvalCallback start epoch eval_interval: 5 # EvalCallback running interval save_best_ckpt: 1 # Is save best ckpt # Export Setup mindir_file_name: "Flownet2" # Save file path file_format: "MINDIR" # Save file format # Modelarts Setup enable_modelarts: 0 # Is training on modelarts
For more configuration details, please refer the script config.py.
Training Process
Training
-
running on GPU
export CUDA_VISIBLE_DEVICES=0 python train.py > train.log 2>&1 &bash scripts/run_train_gpu.sh 1 0The python command above will run in the background, you can view the results through the file
train.log.After training, you'll get some checkpoint files under the folder
${save_checkpoint_path}/ckpt_0/by default. -
train.log for flyingchairs
to be added
Distributed Training
-
running on GPU
bash scripts/run_train_gpu.sh 8 0,1,2,3,4,5,6,7The above shell script will run distribute training in the background. You can view the results through the file
train.log. -
train.log for flyingchairs
to be added
Evaluation Process
Evaluation
-
evaluation on MpiSintelClean dataset when running on GPU
Before running the command below, please check the checkpoint path used for evaluation. Please set the checkpoint path to be the absolute full path, e.g., "path/flownet2/ckpt/flownet2-125_390.ckpt".
python eval.py --eval_data=[DATASET_NAME] --eval_data_path=[DATASET_PATH]/ --model=[MODEL_NAME] --eval_checkpoint_path=[CHECKPOINT_PATH] --crop_type=Center> eval.log 2>&1 &The above python command will run in the background. You can view the results through the file "eval.log". The accuracy of the test dataset will be as follows:
# grep "mean error: " eval.log flownet2 mean error: 2.12OR,
bash run_eval_gpu.sh run_eval_gpu.sh [MpiSintelClean/MpiSintelFinal] [DATA_PATH] [MODEL_NAME] [CKPT_PATH] [DEVICE_ID]The above python command will run in the background. You can view the results through the file "eval/eval.log". The accuracy of the test dataset will be as follows:
# grep "mean error: " eval.log flownet2 mean error: 2.12
Export Process
Export
Before export model, you must modify the config file,
The config items you should modify are batch_size and pre_trained_ckpt_path.
python export.py --config_path [CONFIG_PATH]
Model Description
Performance
Training Performance
FlowNet2 train on FlyingChairs
| Parameters | GPU |
|---|---|
| Model Version | Inception V1 |
| Resource | NV SMX2 V100-32G |
| uploaded Date | 04/05/2021 (month/day/year) |
| MindSpore Version | 1.7.0 |
| Dataset | FlyingChairs |
| Training Parameters | epoch=50, steps=5000, batch_size=8, lr=1e-6 |
| Optimizer | Adam |
| Loss Function | L1loss |
| outputs | flow |
| Speed | 1pc: 152 ms/step; 8pcs: 171 ms/step |
| Total time | 8pcs: 8.8 hours |
| Parameters | 162,518,834 |
| Checkpoint for Fine tuning | 260M (.ckpt file) |
| Scripts | flownet2 script |
Inference Performance
FlowNet2 infer on MpiSintelClean
| Parameters | GPU |
|---|---|
| Model Version | Inception V1 |
| Resource | NV SMX2 V100-32G |
| Uploaded Date | 04/05/2022 (month/day/year) |
| MindSpore Version | 1.7.0 |
| Dataset | MpiSintelClean |
| batch_size | 8 |
| outputs | flow |
| Mean Error | 2.12 |
ModelZoo Homepage
Please check the official homepage.
