A team of researchers led by Prof. XIE Chengjun and Associate Prof. ZHANG Jie from the Hefei Institutes of Physical Science of the Chinese Academy of Sciences has presented Fourier-Image Signal Processing (ISP), a novel deep learning-based framework for RAW-to-sRGB image conversion.

This approach was accepted for publication in the 2024 Proceedings of the Association for the Advancement of Artificial Intelligence.

Converting RAW images to standard red-green-blue (sRGB) images enhances the visual appeal and usability of smartphone photography. However, current methods struggle with color and spatial structure accuracy, especially with resolution and image type variations. Combining color mapping and spatial structure produces suboptimal results, due to the complex interplay between style and structure within the images.

To overcome these challenges, the researchers developed the framework, Fourier-ISP. Inspired by the ISP pipeline, this approach separates the style and structure of the image in the frequency domain.

"It enables independent optimization," said ZHANG Jie, a member of the team.

Fourier-ISP consists of three subnetworks: one for refining the structural details, one for learning accurate colors, and one for blending these elements seamlessly. This decoupling of style and structure enables improved image conversion performance, resulting in sharper and more accurate color and structural details.

Extensive evaluations on diverse datasets confirm that Fourier-ISP achieves state-of-the-art results in qualitative and quantitative assessments, surpassing existing methods in accuracy and detail reproduction. It demonstrates robust transferability and effectiveness in handling both structural and stylistic information, ensuring improved color reproduction and texture preservation.

Specially, Fourier-ISP achieved an impressive Peak Signal to Noise Ratio (PSNR) improvement of 0.17dB in Zurich-area collected RAW and RGB paired image dataset (ZRR-dataset).

This framework introduces a new insight into the field of image processing, showcasing the potential of style-structure decoupling to achieve high fidelity image conversion, especially in mobile photography, according to the team.

Fourier-ISP Framework. (Image by ZHANG Jie)

The results image from ZRRdataset. The last row showcases the color histogram of the image. (Image by ZHANG Jie)