在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:DAIN开源软件地址:https://gitee.com/mirrors/DAIN开源软件介绍:DAIN (Depth-Aware Video Frame Interpolation)Wenbo Bao,Wei-Sheng Lai,Chao Ma,Xiaoyun Zhang,Zhiyong Gao,and Ming-Hsuan Yang IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CVPR 2019 This work is developed based on our TPAMI work MEMC-Net, where we propose the adaptive warping layer. Please also consider referring to it. Table of Contents
IntroductionWe propose the Depth-Aware video frame INterpolation (DAIN) model to explicitly detect the occlusion by exploring the depth cue.We develop a depth-aware flow projection layer to synthesize intermediate flows that preferably sample closer objects than farther ones.Our method achieves state-of-the-art performance on the Middlebury dataset.We provide videos here. CitationIf you find the code and datasets useful in your research, please cite: @inproceedings{DAIN, author = {Bao, Wenbo and Lai, Wei-Sheng and Ma, Chao and Zhang, Xiaoyun and Gao, Zhiyong and Yang, Ming-Hsuan}, title = {Depth-Aware Video Frame Interpolation}, booktitle = {IEEE Conference on Computer Vision and Pattern Recognition}, year = {2019}}@article{MEMC-Net, title={MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement}, author={Bao, Wenbo and Lai, Wei-Sheng, and Zhang, Xiaoyun and Gao, Zhiyong and Yang, Ming-Hsuan}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, doi={10.1109/TPAMI.2019.2941941}, year={2018}} Requirements and Dependencies
InstallationDownload repository: $ git clone https://github.com/baowenbo/DAIN.git Before building Pytorch extensions, be sure you have $ python -c "import torch; print(torch.__version__)" Generate our PyTorch extensions: $ cd DAIN$ cd my_package $ ./build.sh Generate the Correlation package required by PWCNet: $ cd ../PWCNet/correlation_package_pytorch1_0$ ./build.sh Testing Pre-trained ModelsMake model weights dir and Middlebury dataset dir: $ cd DAIN$ mkdir model_weights$ mkdir MiddleBurySet Download pretrained models, $ cd model_weights$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/best.pth and Middlebury dataset: $ cd ../MiddleBurySet$ wget http://vision.middlebury.edu/flow/data/comp/zip/other-color-allframes.zip$ unzip other-color-allframes.zip$ wget http://vision.middlebury.edu/flow/data/comp/zip/other-gt-interp.zip$ unzip other-gt-interp.zip$ cd .. preinstallations: $ cd PWCNet/correlation_package_pytorch1_0$ sh build.sh$ cd ../my_package$ sh build.sh$ cd .. We are good to go by: $ CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury.py The interpolated results are under Downloading ResultsOur DAIN model achieves the state-of-the-art performance on the UCF101, Vimeo90K, and Middlebury (eval and other).Download our interpolated results with: $ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/UCF101_DAIN.zip$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/Vimeo90K_interp_DAIN.zip$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/Middlebury_eval_DAIN.zip$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/Middlebury_other_DAIN.zip Slow-motion GenerationOur model is fully capable of generating slow-motion effect with minor modification on the network architecture.Run the following code by specifying $ CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury_slowmotion.py --netName DAIN_slowmotion --time_step 0.25 or set $ CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury_slowmotion.py --netName DAIN_slowmotion --time_step 0.125$ CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury_slowmotion.py --netName DAIN_slowmotion --time_step 0.1 to generate x8 and x10 slow-motion respectively. Or if you would like to have x100 slow-motion for a little fun. $ CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury_slowmotion.py --netName DAIN_slowmotion --time_step 0.01 You may also want to create gif animations by: $ cd MiddleBurySet/other-result-author/[random number]/Beanbags$ convert -delay 1 *.png -loop 0 Beanbags.gif //1*10ms delay Have fun and enjoy yourself! Training New ModelsDownload the Vimeo90K triplet dataset for video frame interpolation task, also see here by Xue et al., IJCV19. $ cd DAIN$ mkdir /path/to/your/dataset & cd /path/to/your/dataset $ wget http://data.csail.mit.edu/tofu/dataset/vimeo_triplet.zip$ unzip vimeo_triplet.zip$ rm vimeo_triplet.zip Download the pretrained MegaDepth and PWCNet models $ cd MegaDepth/checkpoints/test_local$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/best_generalization_net_G.pth$ cd ../../../PWCNet$ wget http://vllab1.ucmerced.edu/~wenbobao/DAIN/pwc_net.pth.tar$ cd .. Run the training script: $ CUDA_VISIBLE_DEVICES=0 python train.py --datasetPath /path/to/your/dataset --batch_size 1 --save_which 1 --lr 0.0005 --rectify_lr 0.0005 --flow_lr_coe 0.01 --occ_lr_coe 0.0 --filter_lr_coe 1.0 --ctx_lr_coe 1.0 --alpha 0.0 1.0 --patience 4 --factor 0.2 The optimized models will be saved to the Replace the pre-trained $ CUDA_VISIBLE_DEVICES=0 python demo_MiddleBury.py Google Colab DemoThis is a modification of DAIN that allows the usage of Google Colab and is able to do a full demo interpolation from a source video to a target video. Original Notebook File by btahir can be found here. To use the Colab, follow these steps:
Colab file authors: Styler00Dollar and Alpha. ContactWenbo Bao; Wei-Sheng (Jason) Lai LicenseSee MIT License |
请发表评论