Figure 1: Electromagnetic spetrum.
Silicon-based digital camera sensors exhibit significant sensitivity beyond the visible spectrum (400-700nm). They are able to capture wavelengths up to 1100 nm, i.e., they are sensitive to near-infrared (NIR) radiation. This additional information is conventionally treated as noise and is absorbed by a NIR blocking filter affixed to the sensor.
Figure 1: Electromagnetic spetrum.
In this demo and paper, we we show that retaining instead of removing NIR information can significantly improve portrait photography. By fusing RGB and NIR images, we can automatically obtain "smoothed" skin that still looks physically realistic.
Figure 2: From left to right: visible image, near-infrared image, and enhanced image. Click to open a full-resolution image.
This is just one application that shows the benefit of combining RGB and NIR images. Below (Related Visible/NIR Papers) you can find other applications where we show that retaining instead of removing NIR information can significantly improve certain computational photography and computer vision tasks.
Skin tones, portraits in particular, are of tremendous importance in photography and video, but a number of factors, such as pigmentation irregularities (e.g., moles, freckles), irritation, roughness, or wrinkles can reduce their appeal. Moreover, such "defects" are oftentimes enhanced by lighting conditions.
Starting with the observations that melanin and hemoglobin, the key components of skin color, have little absorption in the near-infrared (NIR) part of the spectrum, and that the depth of light penetration in the epidermis is proportional to the incident light's wavelength, we show that near-infrared images provide information that can be used to automatically smooth skin tones in a physically realistic manner. Download the paper describing the method in more detail.
We developed a prototype camera system that consists of capturing a pair of visible/near-infrared images and separating both of them into base and detail layers (akin to a low/high frequency decomposition) with the fast bilateral filter. Smooth, realistic, output images are obtained by fusing the base layer of the visible image with the near-infrared detail layer. The proposed method delivers consistently good results across various skin types.
The prototype system is currently in use at the Swiss Camera Museum in Vevey, Switzerland, where the visitors can take their pictures and e-mail themselves the results. In the process, we are collecting the user's preference for either the "original" (visible) image or the "smoothed" (visible and NIR fused) image. Statistics indicate that 76% prefer the smoothed image.
A prototype of the system was built and deployed in a public setting, namely as an installation in an exhibit entitled Digital Revolution at the Swiss Camera Museum in Vevey, Switzerland. The installation is set-up to resemble a photo booth (Fig. 3). The visitor sits down on a stool in front of the camera and presses a button to have his portrait taken. The "normal" (i.e. visible) and "enhanced" (i.e. fused visible and NIR) pictures are shown on a display (Fig. 4). The visitor can compare the images. He is then asked which picture he prefers and can have both pictures sent to his email address. Pictures and preference choices are stored in a database. Fig. 2 shows an example of image pairs.
Figure 3: Installation views of the system at the Swiss Camera Museum in Vevey. Click to enlarge.
Figure 4: User interface of the system as presented to the end user with the "normal" image on the left and the "enhanced" image on the right.
To acquire visible and NIR images simultaneously, we use a JAI AD-080GE camera. This 2CCD multi-spectral camera can simultaneously capture both color and NIR images in one camera housing and through a single lens (Fig. 5). It uses a multi-faceted prism in the optical path with bandpass filters on each spectral axis to capture RGB through a Bayer Filter Aray on one sensor and NIR on the other. The camera, primarily intended for industrial quality control, has a resolution of 1024x768 pixels and is able to capture up to 30 frames per second in full frame mode. An SDK is provided that enables control of every feature of this GigE-compliant camera. Among the limitations of this camera system are the low resolution, the maximum exposure time of 1/30 second, and the need for a powerfull light source.
The processing of images is written in C++ and takes about 500ms for a 1024x768 image pair on a 2.4GHz Core2Duo processor using only one core. Most of the processing time is used to apply the bilateral filter that separates the image into its base and its detail layer. Current values for the bilateral filter are a spatial sigma of 16 and a range sigma of 0.1.
Figure 5: JAI AD-080GE 2CCD camera simultaneously captures visible and NIR images.
This work was supported in part by the Swiss National Science Foundation under grant number 200021-124796/1.
Following are other papers where we explore the benefits of retaining instead of removing NIR information to enhance photography or to extract more accurate information about the visible scene.
S. Süsstrunk and C. Fredembach, Enhancing the Visible with the Invisible: Exploiting Near-Infrared to Advance Computational Photography and Computer Vision, SID International Symposium Digest, 2010.
[detailed record] [bibtex]
C. Fredembach, N. Barbuscia and S. Süsstrunk, Combining visible and near-infrared images for realistic skin smoothing, Proc. IS&T/SID 17th Color Imaging Conference (CIC), 2009.
[detailed record] [bibtex]
L. Schaul, C. Fredembach and S. Süsstrunk, Color Image Dehazing using the Near-Infrared, Proc. IEEE International Conference on Image Processing, 2009.
[detailed record] [bibtex]
C. Fredembach and S. Süsstrunk, Illuminant estimation and detection using near infrared, IS&T/SPIE Electronic Imaging, Digital Photography V, Vol. 7250, 2009.
[detailed record] [bibtex]
N. Salamati, C. Fredembach and S. Süsstrunk, Material Classification Using Color and NIR Images, Proc. IS&T/SID 17th Color Imaging Conference (CIC), 2009.
[detailed record] [bibtex]
N. Salamati and S. Süsstrunk, Material-Based Object Segmentation Using Near-Infrared Information, IS&T/SID 18th Color Imaging Conference (CIC), 2010.
[detailed record] [bibtex]
Y.M. Lu, C. Fredembach, M. Vetterli and S. Süsstrunk, Designing color filter arrays for the joint capture of visible and near-infrared images, Proc. IEEE International Conference on Image Processing, 2009.
[detailed record] [bibtex]
C. Fredembach and S. Süsstrunk, Colouring the near infrared, Proceedings of the IS&T/SID 16th Color Imaging Conference, pp. 176-182, 2008.
[detailed record] [bibtex]