You can not select more than 25 topics Topics must start with a chinese character,a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
 
liupengfei ee3c266bd1 222 1 week ago
scripts init 2 months ago
src 222 1 week ago
README.en.md init 2 months ago
README.md init 2 months ago
default_config.yaml 111 1 week ago
default_config_all.yaml 222 1 week ago
default_config_yuanshi.yaml 修改配置1 2 months ago
eval.py 222 1 week ago
export.py init 2 months ago
pip-requirements.txt 配置1 2 months ago
requirements.txt init 2 months ago
test_with_grad.py init 2 months ago
test_with_grad_copy.py init 2 months ago
train.py 2卡,改了float16 1 month ago

README.md

Contents

Flownet2 Description

FlowNet2.0, a deep network proposed in 2017, which performs end-to-end learning on optical flow data.
It is optimized based on the FlowNet network , The large improvements in quality and
speed are caused by three major contributions: first, it
focus on the training data and show that the schedule of
presenting data during training is very important. Second,
it develop a stacked architecture that includes warping
of the second image with intermediate optical flow. Third,
it elaborate on small displacements by introducing a subnetwork specializing on small motions.

Compared with the FLownet network, FlowNet 2.0 is only
marginally slower than the original FlowNet but decreases
the estimation error by more than 50%.

FlowNet2 paper:Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, Thomas Brox

FlowNet paper:Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox

Model Architecture

The FlowNet2 network is stacked by multiple flownet sub-modules. After the output of the previous network is processed by warp, it is used as the input of the second network.

The model structure is flowNet2CSS and FlowNet2SD two sub-networks fuse the output through the FlownetFusion network, and the entire large network structure formed is FLowNet2
The FlowNet2CSS network is a stack of FLowNet2C and two FLowNet2S. The specific structure can be further understood according to the paper

This source code provides the following model structure, which can be configured and used in the yaml file:

  • FlowNet2S
  • FlowNet2C
  • FlowNet2CS
  • FlowNet2CSS
  • FlowNet2SD
  • FlowNet2

Dataset

Dataset used: FlyingChairs

  • Dataset size:31GB,22,872 pairs 512*384 colorful images
  • Data format:PPM
    • Note:Data will be processed in src/dataset.py

Dataset used: ChairsSDHom

  • Dataset size: 51GB, 21,668 pairs 512*384 colorful images
    • Train:20,965 pairs image
    • Test: 703 pairs image
  • Data format:PNG
    • Note: Data will be processed in src/dataset.py

Dataset used: MpiSintel

  • Dataset size: 536M, 1024 x 436 colorful images in 23 classes
    • MpiSintelClean:1150 images
    • MpiSintelFinal: 1150 images
  • Data format:PNG
    • Note: Data will be processed in src/dataset.py

Environment Requirements

Quick Start

After installing MindSpore via the official website, you can start training and evaluation as follows:

  • download pretrained parameter

    FlowNet2 620MB

    FlowNet2-C 149MB

    FlowNet2-CS 297MB

    FlowNet2-CSS 445MB

    FlowNet2-CSS-ft-sd 445MB

    FlowNet2-S 148MB

    FlowNet2-SD 173MB

  • convert pretrained parameter (from pytorch pretrained parameter to mindspore pretained parameter,so the env should both installed torch and mindspore)
    convert pytorch pretrained parameter to mindspore pretrained parameter

    bash run_ckpt_convert.sh [PYTORCH_FILE_PATH] [MINDSPORE_FILE_PATH]
    # example:
    bash run_ckpt_convert.sh /path/to/flownet2.pth.tar /path/to/flownet2.ckpt
    
  • config pretrained parameter path in yaml file

    pre_trained:     # whether use pretrained parameter file 'True' or 'False'
    pre_trained_ckpt_path: # pretrained checkpoint file path
    # 实例:
    pre_trained: True
    pre_trained_ckpt_path: /path/checkpoint/flownet2.ckpt
    
  • config dataset name and path in yaml file

    train_data: [DATASET_NAME]  # Name of dataset, 'FlyingChairs' or 'MpiSintelFinal' or 'MpiSintelClean'
    train_data_path:[DATASET_PATH] # path of dataset
    # example:
    train_data: FlyingChairs
    train_data_path: /path/to/FlyingChairs_release/data
    
  • running on GPU

    For running on GPU, please change device_target from Ascend to GPU in configuration file default_config.yaml

    # run training example
    export CUDA_VISIBLE_DEVICES=0
    python train.py > train.log 2>&1 &
    
    # run distributed training example
    bash scripts/run_train_gpu.sh 8 0,1,2,3,4,5,6,7
    
    # run evaluation example
    python eval.py --crop_type="Center" --eval_checkpoint_path=[EVAL_CHECKPOINT_PATH] > eval.log 2>&1 &  
    OR
    bash scripts/run_eval_gpu.sh [MpiSintelClean/MpiSintelFinal] [DATA_PATH] [MODEL_NAME] [CKPT_PATH] [DEVICE_ID]
    

We use FlyingChairs dataset by default. Your can also pass $dataset_type to the scripts so that select different datasets. For more details, please refer the specify script.

Script Description

Script and Sample Code

├── model_zoo
    ├── README.md                          // descriptions about all the models
    ├── flownet2
        ├── README.md                      // descriptions about flownet2
        ├── scripts
        │   ├── run_ckpt_convert.sh        // shell script for converting pytorch ckpt file to pickle file on GPU
        │   ├── run_eval_gpu.sh            // shell script for eval on GPU
        │   └── run_train_gpu.sh           // shell script for training on GPU
        ├── src
        │   ├── dataset.py                 // creating dataset
        │   ├── eval_callback.py           // eval callback when training
        │   ├── metric.py                  // metric to calculate mean error
        │   ├── model_utils
        │   │   ├── ckpt_convert.py         // convert pytorch ckpt file to pickle file
        │   │   ├── config.py               // parameter configuration
        │   │   ├── device_adapter.py       // device adapter
        │   │   ├── local_adapter.py        // local adapter
        │   │   ├── moxing_adapter.py       // moxing adapter
        │   │   ├── frame_utils.py          // utils to read files of dataset
        │   │   └── tools.py                // tools to match class with paratmeter from config
        │   ├── models.py                   // FlowNet2/FlowNet2CSS/FlowNet2CS/FlowNet2C/FlowNet2S/FlowNet2SD model
        │   └── submodels
        │       ├── FlowNetC.py             // FlowNetC model
        │       ├── FlowNetFusion.py        // FlowNetFusion model
        │       ├── FlowNetS.py             // FlowNetS model
        │       ├── FlowNetSD.py            // FlowNetSD model
        │       └── submodules.py           // submodules used in flownet model
        ├── default_config.yaml             // parameter configuration
        ├── requirements.txt                // requirements configuration
        ├── export.py                       // convert mindspore ckpt file to MINDIR file
        ├── eval.py                         // evaluation script
        └── train.py                        // training script

Script Parameters

Parameters for both training and evaluation can be set in config.py

  • config for FLowNet2

      # ==============================================================================
      # Device
      device_target:          "GPU"
      device_id:              0
    
      # Dataset Setup
      crop_type:              Random                   # Type of cropping operation (Random and Center)
      crop_size:              [384, 512]               # (Height, Width) of image when training
      eval_size:              [256, 256]               # (Height, Width) of image when eval
    
      # Experiment Setup
      model:                  "FlowNet2"                # Name of model to be loaded
      rgb_max:                255                       # rgb channel used
      batchNorm:              False                     # boolean switch to whether add batchnorm before conv
      lr:                     1e-6                      # Learning rate
      num_parallel_workers:   2                         # Number of CPU worker used to load data
      max_rowsize:            2                         # Number of max rowsize used to load data
      batch_size:             2                         # Numbers of image pairs in a mini-batch
      epoch_size:             100                       # Total number of epochs
      pre_trained:            1                         # Load pretrained network
      pre_trained_ckpt_path:  "/path/flownet2.ckpt"     # Pretrained ckpt path
      seed:                   1                         # Seed for reproducibility
      is_dynamicLoss_scale:   0                         # Using dynamicLoss scale or fix scale
      scale:                  1024                      # Fix scale value
      weight_decay:           0.00001                   # Weight decay
      train_data:             "FlyingChairs"            # Train Dataset name
      train_data_path:        "/path/ds/FlyingChairs_release/data"       # Train Dataset path
    
      # Train Setup
      run_distribute:         1                         # Distributed training or not
      is_save_on_master:      1                         # Only save ckpt on master device
      save_checkpoint:        1                         # Is save ckpt while training
      save_ckpt_interval:     2                         # Saving ckpt interval
      keep_checkpoint_max:    20                        # Max ckpt file number
      save_checkpoint_path:   "/path/ckpt/"             # Ckpt save path
    
      # eval Setup
      eval_data:              "MpiSintelClean"           # Eval Dataset name
      eval_data_path:         "/home/shm/ds/training"    # Eval Dataset path
      eval_checkpoint_path:   "/path/flownet2.ckpt"      # Ckpt path used to eval
      run_evalCallback:       1                          # Is run evalCallBack while training
      eval_start_epoch:       1                          # EvalCallback start epoch
      eval_interval:          5                          # EvalCallback running interval
      save_best_ckpt:         1                          # Is save best ckpt
    
      # Export Setup
      mindir_file_name:        "Flownet2"                 # Save file path
      file_format:             "MINDIR"                   # Save file format
    
      # Modelarts Setup
      enable_modelarts:       0                           # Is training on modelarts
    

For more configuration details, please refer the script config.py.

Training Process

Training

  • running on GPU

    export CUDA_VISIBLE_DEVICES=0
    python train.py > train.log 2>&1 &
    
    bash scripts/run_train_gpu.sh 1 0
    

    The python command above will run in the background, you can view the results through the file train.log.

    After training, you'll get some checkpoint files under the folder ${save_checkpoint_path}/ckpt_0/ by default.

  • train.log for flyingchairs
    to be added

Distributed Training

  • running on GPU

    bash scripts/run_train_gpu.sh 8 0,1,2,3,4,5,6,7
    

    The above shell script will run distribute training in the background. You can view the results through the file train.log.

  • train.log for flyingchairs
    to be added

Evaluation Process

Evaluation

  • evaluation on MpiSintelClean dataset when running on GPU

    Before running the command below, please check the checkpoint path used for evaluation. Please set the checkpoint path to be the absolute full path, e.g., "path/flownet2/ckpt/flownet2-125_390.ckpt".

    python eval.py --eval_data=[DATASET_NAME] --eval_data_path=[DATASET_PATH]/
    --model=[MODEL_NAME] --eval_checkpoint_path=[CHECKPOINT_PATH] --crop_type=Center> eval.log 2>&1 &  
    

    The above python command will run in the background. You can view the results through the file "eval.log". The accuracy of the test dataset will be as follows:

    # grep "mean error: " eval.log
    flownet2 mean error: 2.12
    

    OR,

    bash run_eval_gpu.sh run_eval_gpu.sh [MpiSintelClean/MpiSintelFinal] [DATA_PATH] [MODEL_NAME] [CKPT_PATH] [DEVICE_ID]
    

    The above python command will run in the background. You can view the results through the file "eval/eval.log". The accuracy of the test dataset will be as follows:

    # grep "mean error: " eval.log
    flownet2 mean error: 2.12
    

Export Process

Export

Before export model, you must modify the config file,
The config items you should modify are batch_size and pre_trained_ckpt_path.

python export.py --config_path [CONFIG_PATH]

Model Description

Performance

Training Performance

FlowNet2 train on FlyingChairs

Parameters GPU
Model Version Inception V1
Resource NV SMX2 V100-32G
uploaded Date 04/05/2021 (month/day/year)
MindSpore Version 1.7.0
Dataset FlyingChairs
Training Parameters epoch=50, steps=5000, batch_size=8, lr=1e-6
Optimizer Adam
Loss Function L1loss
outputs flow
Speed 1pc: 152 ms/step; 8pcs: 171 ms/step
Total time 8pcs: 8.8 hours
Parameters 162,518,834
Checkpoint for Fine tuning 260M (.ckpt file)
Scripts flownet2 script

Inference Performance

FlowNet2 infer on MpiSintelClean

Parameters GPU
Model Version Inception V1
Resource NV SMX2 V100-32G
Uploaded Date 04/05/2022 (month/day/year)
MindSpore Version 1.7.0
Dataset MpiSintelClean
batch_size 8
outputs flow
Mean Error 2.12

ModelZoo Homepage

Please check the official homepage.

简介

光流估计网络

Python Cuda Shell other

贡献者 (1)