SSL-RGB2IR: Semi-supervised RGB-to-IR Image-to-Image Translation for Enhancing Visual Task Training in Semantic Segmentation and Object Detection

The scarcity of annotated infrared (IR) image datasets limits deep learning networks from achieving per- formances comparable to those achieved with RGB data. To address this, we introduce a novel semi-supervised RGB- to-IR Image-to-Image Translation model (SSL-RGB2IR) that generates synthetic IR data from RGB images. Our model effectively preserves the IR characteristics in the generated images from both synthetic and real-world data. Compared to existing image-to-image translation techniques, training models on this generated IR data significantly improves per- formance in downstream tasks like segmentation and de- tection. Notably, in sim-to-real transfer, the segmentation model trained on SSL-RGB2IR generated IR images out- performs baselines and other Image-to-Image (I2I) models. Furthermore, for real-world applications utilizing EO/IR fu- sion images, this approach solves the well-known challenge of co-registering EO and IR images, which often have in- herent misalignment’s due to differing sensor characteristics. Our code is available at https://github.com/prahlad-anand/ssl- rgb2irhttps://github.com/prahlad-anand/ssl-rgb2ir.