The photodiodes employed in an image sensor are color-blind by nature: they can only record shades of grey. To get color into the picture, they are covered with different color filters: red, green and blue (RGB) according to the pattern designated by the Bayer filter. As each photodiode records the color information for exactly one pixel of the image, without an image processor there would be a green pixel next to each red and blue pixel.
This process, however, is quite complex, and involves a number of different operations. Its quality depends largely on the effectiveness of the algorithms applied to the raw data coming from the sensor. The mathematically manipulated data becomes the recorded photo file.
Demosaicing
As stated above, the image processor evaluates the color and brightness data of a given pixel, compares them with the data from neighboring pixels, and then uses a demosaicing algorithm to produce an appropriate color and brightness value for the pixel. The image processor also assesses the whole picture to guess at the correct distribution of contrast. By adjusting the gamma value (heightening or lowering the contrast range of an image's mid-tones), subtle tonal gradations, such as in human skin or the blue of the sky, become much more realistic.
Noise reduction
Noise is a phenomenon found in any electronic circuitry. In digital photography its effect is often visible as random spots of obviously wrong color in an otherwise smoothly-colored area. Noise increases with temperature and exposure times. When higher ISO settings are chosen the electronic signal in the image sensor is amplified, which at the same time increases the noise level, leading to a lower signal-to-noise ratio. The image processor attempts to separate the noise from the image information and to remove it. This can be quite a challenge, as the image may contain areas with fine textures which, if treated as noise, may lose some of their definition.
Image sharpening
As the color and brightness values for each pixel are interpolated some image sharpening is applied to even out any fuzziness that has occurred. To preserve the impression of depth, clarity and fine details, the image processor must sharpen edges and contours. It therefore must detect edges correctly and reproduce them smoothly and without over-sharpening.
THine - THP series [1] with compatible SDK Kit for developing firmware [2]
UNISOC - Vivimagic
Speed
With the ever-higher pixel count in image sensors, the image processor's speed becomes more critical: photographers don't want to wait for the camera's image processor to complete its job before they can carry on shooting - they don't even want to notice some processing is going on inside the camera. Therefore, image processors must be optimised to cope with more data in the same or even a shorter period of time.
Software
libcamera is a software library that supports using image signal processors for the capture of pictures.
