a798da6b32
File "/home/openxs/jj/roma_unsb/train.py", line 47, in <module>
model.optimize_parameters() # calculate loss functions, get gradients, update network weights
File "/home/openxs/jj/roma_unsb/models/roma_unsb_model.py", line 315, in optimize_parameters
self.forward()
File "/home/openxs/jj/roma_unsb/models/roma_unsb_model.py", line 445, in forward
Xt_1 = self.netG(Xt, self.time, z)
File "/home/openxs/anaconda3/envs/I2V/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/openxs/anaconda3/envs/I2V/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 169, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/openxs/anaconda3/envs/I2V/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/openxs/jj/roma_unsb/models/networks.py", line 980, in forward
feat = layer(feat)
File "/home/openxs/anaconda3/envs/I2V/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/openxs/anaconda3/envs/I2V/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/openxs/anaconda3/envs/I2V/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 1, 1734, 774] to have 3 channels, but got 1 channels instead
ROMA
This repository is the official Pytorch implementation for ACM MM'22 paper "ROMA: Cross-Domain Region Similarity Matching for Unpaired Nighttime Infrared to Daytime Visible Video Translation".[Arxiv]
Examples of Object Detection:
Examples of Video Fusion
More experimental results can be obtained by contacting us.
Introduction
Method
- The domain gaps between unpaired nighttime infrared and daytime visible videos are even huger than paired ones that captured at the same time, establishing an effective translation mapping will greatly contribute to various fields.
- Our proposed cross-similarity, which are calculated across domains, could make the generative process focus on learning the content of structural correspondence between real and synthesized frames, getting rid of the negative effects of different styles.
Training
The following is the required structure of dataset. For the video mode, the input of a single data is the result of concatenating two adjacent frames; for the image mode, the input of a single data is a single image.
Video/Image mode:
trainA: \Path\of\trainA
trainB: \Path\of\trainB
Concrete examples of the training and testing are shown in the script files ./scripts/train.sh and ./scripts/test.sh, respectively.
InfraredCity and InfraredCity-Lite Dataset
| InfraredCity | Total Frame | ||||
|---|---|---|---|---|---|
| Nighttime Infrared | 201,856 | ||||
| Nighttime Visible | 178,698 | ||||
| Daytime Visible | 199,430 | ||||
| InfraredCity-Lite | Infrared Train |
Infrared Test |
Visible Train |
Total | |
| City | clearday | 5,538 | 1,000 | 5360 | 15,180 |
| overcast | 2,282 | 1,000 | |||
| Highway | clearday | 4,412 | 1,000 | 6,463 | 15,853 |
| overcast | 2,978 | 1,000 | |||
| Monitor | 5,612 | 500 | 4,194 | 10,306 | |
The datasets and their more details are available in InfiRay.
Citation
If you find our work useful in your research or publication, please cite our work:
@inproceedings{ROMA2022,
title = {ROMA: Cross-Domain Region Similarity Matching for Unpaired Nighttime Infrared to Daytime Visible Video Translation},
author = {Zhenjie Yu and Kai Chen and Shuang Li and Bingfeng Han and Chi Harold Liu and Shuigen Wang},
booktitle = {ACM MM},
pages = {5294--5302},
year = {2022}
}
Acknowledgements
This code borrows heavily from the PyTorch implementation of Cycle-GAN and Pix2Pix and CUT.
A huge thanks to them!
@inproceedings{CycleGAN2017,
title = {Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkss},
author = {Zhu, Jun-Yan and Park, Taesung and Isola, Phillip and Efros, Alexei A},
booktitle = {ICCV},
year = {2017}
}
@inproceedings{CUT2020,
author = {Taesung Park and Alexei A. Efros and Richard Zhang and Jun{-}Yan Zhu},
title = {Contrastive Learning for Unpaired Image-to-Image Translation},
booktitle = {ECCV},
pages = {319--345},
year = {2020},
}