Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
Larry-u 5755bdce0a | 3 years ago | |
---|---|---|
media | 3 years ago | |
src | 3 years ago | |
test | 3 years ago | |
third_party/neural_renderer | 3 years ago | |
train | 3 years ago | |
utils | 3 years ago | |
LICENSE | 3 years ago | |
README.md | 3 years ago | |
environment.yml | 3 years ago | |
front_facial.json | 3 years ago | |
head.json | 3 years ago | |
mapper.txt | 3 years ago | |
options.py | 3 years ago | |
requirements.txt | 3 years ago | |
smpl_faces.npy | 3 years ago | |
smpl_part_info.json | 3 years ago |
This repository contains official code for the paper
JAFPro: Joint Appearance Fusion and Propagation for Human Video Motion Transfer from Multiple Reference Images.
You can find detailed usage instructions for training from scratch and testing pre-trained model below.
If you find our code or paper useful, please consider citing
@inproceedings{10.1145/3394171.3414001,
author = {Yu, Xianggang and Liu, Haolin and Han, Xiaoguang and Li, Zhen and Xiong, Zixiang and Cui, Shuguang},
title = {JAFPro: Joint Appearance Fusion and Propagation for Human Video Motion Transfer from Multiple Reference Images},
year = {2020},
booktitle = {Proceedings of the 28th ACM International Conference on Multimedia},
}
First you have to make sure that you have all dependencies in place.
The simplest way to do so, is to use anaconda.
You can create an anaconda environment called JAFPro
using
conda env create -f environment.yml
conda activate JAFPro
Our DanceVideo dataset consists of 1651 dance videos in the wild with static background. Each video has a length of two seconds with 15FPS (30 frames in total). The resolution of all videos is 256x256. You can download the whole dataset through https://drive.google.com/file/d/1xlybcxZXcQUCgUuq1K2FLaVGSr8d_NJW/view?usp=sharing
Once you have installed all dependencies and downloaded the dataset, you are ready to test our pre-trained model or train a model from scratch.
Except code directories, the scripts assume the following file structure.
JAFPro
├── checkpoints # opt['model_save_dir'], where checkpoints placed
└── test_results # opt["test_save_dir"], path to save test results
At first, download the pretrained model from https://drive.google.com/file/d/1q4Lu6ayHG3-dAl5mE4b6tr6e-O3SHbye/view?usp=sharing, and unzip it into checkpoints
directory.
JAFPro
├── checkpoints
├── $model_name # e.g. appearance_propagation_interval
├── Accu_iter_xx.pth
├── bg_iter_xx.pth
├── inpaint_iter_xx.pth
├── pro_iter_xx.pth
├── refine_iter_xx.pth
Then, you have to setup the data root in options.py
opt['smpl_root'] = '/your/path/to/DanceVideo/smpls'
opt['mask_root'] = '/your/path/to/DanceVideo/masks'
opt['data_root'] = '/your/path/to/DanceVideo/images'
To test the pretrained model, run
cd test/
python conv_pro_test.py --gpu 0 -e appearance_propagation_interval -n 4
arguments explanation:
--gpu: specify gpu device
-e: experiment name (model_name). The program will load checkpoints from JAFPro/checkpoints/$model_name
-n: number of frames
And the results will be saved to test_results/$model_name
After testing a pretrained model, you can calculate metrics by running
python video_evaluation.py --pred /path/to/test/results/dir --gt /path/to/DanceVideo/images/test --gpu 0 --type densepose
such that the metric for each video and for whole test set will be printed to terminal.
First, setup data root path in options.py
opt['smpl_root'] = '/your/path/to/DanceVideo/smpls'
opt['mask_root'] = '/your/path/to/DanceVideo/masks'
opt['data_root'] = '/your/path/to/DanceVideo/images'
Then,
xxx
You can monitor on http://localhost:6006 the training process using tensorboard:
cd OUTPUT_DIR
tensorboard --logdir ./monitoring --port 6006
Official code and dataset release for "JAFPro: Joint Appearance Fusion and Propagation for Human Video Motion Transfer from Multiple Reference Images"
Text Python Cuda C++ other
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》