Temperature reconstruction of infrared images with motion deblurring

Infrared images of an uncooled microbolometer camera can show significant blurring effects while recording a moving object. The electrical signal in the pixel of a microbolometer detector decays exponentially; hence, the moving object is mapped to more pixels resulting in a blurred image. Not only the contrast is corrupted by the motion, but also the temperature of the object seems to be significantly lower. In this paper, it is shown how such images can be deblurred and the true temperature with a good approximation restored. Since the detection mechanism of a microbolometer camera is different from complementary metal–oxide–semiconductor (CMOS) or charge-coupled device (CCD) cameras, also the point-spread function (PSF) needed for the deblurring restoration is different. It is shown how the exponential coefficient of the PSF can be calculated if the motion speed and the camera resolution are known, or otherwise how it can be estimated from the image itself. Experimental examples are presented for motion deblurring used to restore images with linear or rotational motion.


Introduction
The elimination of motion blurring in complementary metaloxide-semiconductor (CMOS) and charge-coupled device (CCD) camera images is an extensively investigated topic (Rajagopalan and Chellappa, 2014).Even in the last years, much research has been inspired by problems, e.g.incidental shake in hand-held cameras.
Assuming only one motion direction, the problem can be reduced to a simple image deconvolution.If the blurring kernel (the point-spread function -PSF) is known, then this is called a non-blind deconvolution.On the other hand, if the PSF is not known, it has to be estimated first from the image itself, which is called blind deconvolution.Additional techniques, e.g.Wiener filter, have been developed for suppressing the noise, which could be strongly amplified by the deconvolution itself (Rajagopalan and Chellappa, 2014;Gonzalez et al., 2009).
Motion blurring can be also observed in images of infrared cameras, as it is described also by Vollmer and Möllmann (Vollmer and Möllmann, 2010).Cooled cameras with high sensitive photonic detectors have usually a very short integration time that is typically 1-1.5 ms for room temperature measurements, and the motion blurring can be mostly ne-glected.In contrast, the electrical signal in the pixel of a microbolometer camera decays with a time constant of 10-15 ms; therefore, a significant blurring effect can be observed by recording moving objects.
The PSF of motion blurring in a CMOS camera is mainly a linear function with a necessary length and direction (Rajagopalan and Chellappa, 2014;Gonzalez et al., 2009).In contrast, the signal of a microbolometer camera decays with an exponential function, and therefore it requires a PSF with an exponential decay (Oswald-Tranta et al., 2010a).
Furthermore, by the deblurring of a CMOS camera image, the main goal is to obtain an image with high contrast, while the recorded value of each pixel itself has no significance.In contrast, by restoring infrared images, it is also expected that the recorded infrared (IR) radiation for each pixel and, in a further step, the correct temperature of the objects should be also reconstructed.
In this paper, it is investigated how in infrared images of a microbolometer camera the motion distortion can be deblurred.Part of the results have been published earlier (Oswald-Tranta et al., 2010a;Oswald-Tranta, 2012, 2017).
Here, investigations are shown on how well the temperature of moving objects can be restored.Different experimental re- sults for the usage of image deblurring are presented, and it is shown how non-blind and blind deblurring can be carried out.

Motion blurring in microbolometer images
Blurring of an image is described generally by the equation: where f is the true image recorded under perfect conditions and h is the so-called PSF representing the distortion of the acquisition.The convolution of these two (h ⊗ f ) results in the distorted image.n denotes the additional noise and g the raw image recorded by the camera.The PSF of a microbolometer camera for a moving object is an exponential function (Oswald-Tranta et al., 2010a;Oswald-Tranta, 2012): The decay factor τ im depends on the relative speed between camera and the moving object (v), and further on the resolution (r) of the recorded image given in pixel mm −1 .The product vr represents the speed in pixel s −1 .τ im depends also on the exponential decay constant of the electrical signal of the microbolometer detector (τ camera ) which has a typical value of about 10 ms: In this way, τ im represents approximately the number of pixels that the object moves during the recording time of one image.As cameras with cooled semiconductor detectors usually work with an integration time of 1-2 ms, τ im is about 1 order of magnitude less than for microbolometer cameras; hence, motion blurring of the images is in most of the cases negligibly small.However, for microbolometer cameras, the distortion of the images is often recognisable but can be well restored by the method described here.
If the temperature difference between object and background is denoted by T true , then the temperature difference in the blurred image ( T blurred ) has the following form (Oswald-Tranta et al., 2010a): where a denotes the size of the object in pixels.Introducing where d is the real size of the object, the blurred temperature at x = a is equal to Due to the blurring, small objects with high speed therefore seem to have much lower temperatures than they really have, because the recorded infrared radiation is smeared over more pixels during the object motion.

Motion experiments
In the experiments to test motion blurring, a small conveyor belt is moving at a specified speed up to 1.5 m s −1 in the  1.   field of view of the camera (see Fig. 1).For comparison, two cameras have been used, a microbolometer camera (left side) (DIAS, 2011) and a camera with a cooled indium antimonide (InSb) detector (right side) (FLIR, 2010).For the cooled camera, one may select the integration time from a wide range; for the experiment, it was set to 1.5 ms, which allows temperature measurements up to 70 • C. Because of the short integration time, the blurring due to motion is negligible in the images of this camera.
A small warm plate (65 × 10 × 3 mm 3 ) has been moved with different speeds in the field of view of both cameras.Table 1 summarizes the data of four measurements, and Fig. 2 shows the corresponding microbolometer images.It is well visible that the higher the speed, the more the image is blurred.

Deblurring of the images
It is well known that the Fourier transformation changes the convolution into a multiplication of the spectra.Therefore, if G, H , F and N denote the Fourier transformation of the functions g, h, f and n, then Eq. ( 1) can be written as In order to obtain an image f close to the true image, the inverse Fourier transformation of G/H has to be calculated: If there were no noise, then f = f , which means the image could be perfectly restored.H , the Fourier transformation of the PSF, is the so-called optical transfer function (OTF) and it www.j-sens-sens-syst.net/7/13/2018/can be calculated from Eq. ( 2) (Oswald-Tranta et al., 2010a): where p is the frequency.Since H becomes small for high frequencies, the high-frequency noise gets strongly amplified.Therefore, a kind of low-pass filter is necessary for the deblurring.A good possibility is to use a parametrized Wiener filter (Gonzalez et al., 2009;Oswald-Tranta et al., 2010a): where k is a small non-negative number.If k = 0, then Eq. ( 10) is reduced to Eq. ( 8).In Fig. 3, temperature profiles across the four blurred images are shown.Also the reconstructed temperature profiles are displayed in the diagrams, where k was set to 0.005.The temperature of the plate was not the same for each measurement as it was slowly cooling down from one measurement to the next one.However, the reconstructed temperature values have been compared with the temperatures measured simultaneously with the cooled IR camera and a very good correspondence has been found.In Table 1, the observed T blurred / T true ratios are compared with 1−exp (−m); see also Eq. ( 6).For the speeds of 1 and 1.5 m s −1 , the blurred temperature is significantly less than the true temperature, but it is very well reconstructed with the proposed Wiener filter, giving the same value as was measured by the cooled IR camera.Nevertheless, the higher the speed, the more enhanced the noise after the reconstruction, which results in an error in the reconstruction.

Influence of the Wiener filter parameter
A signal with the frequency p after dividing by the OTF and after the inverse Fourier transformation is amplified by Since, according to the Nyquist theorem, the shortest wavelength has to have at least the length of two pixels, the highest frequency after Fourier transform of a digital image is p = π .
If, e.g. the motion speed v = 1 m s −1 and the resolution r = 4 pixels mm −1 , then τ im = 40; thereby, if k = 0, the highest frequency noise is amplified by 1 + p 2 τ 2 im ≈ π τ im ≈ 125.Using the parametrized Wiener filter, the amplification of a signal with the frequency p is If pτ im 1, Figure 4 demonstrates this behaviour for τ im = 40; the larger the value of k is, the stronger the high-frequency signals are damped.For small k values, the signal is amplified.If k =1/(π τ im ) = 0.008, then for the highest frequency p = π the amplification A = 1.
As it has been shown (Oswald-Tranta et al., 2010a), also the restored temperature depends on the k value: where T restored = T restored,object − T restored,background and T true = T true,object − T true,background .The larger the k value is, the lower the reconstructed temperature.With large k values, not only the high-frequency noise is damped but also the high-frequency components at the edges of an object; thus, the object becomes smeared and its reconstructed temperature is lower than the true one.This is demonstrated in Figs. 5 and 6.A small warm ball with a diameter of 7 mm was recorded during a motion with 1 m s −1 (see Fig. 5a).The image is strongly blurred and shows a typical comet-like shape.The resolution is 4.4 pixel mm −1 and the images are reconstructed with τ im = 44.According to the simultaneously recorded images, by the cooled camera, the temperature of the ball is about 45 • C.
-If k = 0, then the image is very noisy; the highfrequency noise is amplified by approximately 2 orders of magnitude (Figs.5b and 6).
-If k = 0.005, the noise amplification is well suppressed due to the Wiener filtering, the deblurred image is sharp and the temperature is well restored (Figs.5c and 6), which corresponds well with the result of Eq. ( 14): T restored / T true = 0.99.
-If k = 0.05, then the edges of the restored image are smoothed out due to the strong damping of high frequencies, causing an elongation of the image (Figs.5d  and 6).According to Eq. ( 14), T restored / T true = 0.8, which corresponds very well with the measured data.
To our experience, k values around 0.005 are a good compromise.If k is too low, then the restored image is too noisy.On the other hand, if it is too high, then due to its low-pass filtering the edges of the object become smoothed, causing a blurring in the motion direction, which consequently results also in an apparently lower temperature.

Influence of the deviation in τ im
The same image as shown in Fig. 5a has been restored with different τ im values (see Figs. 7 and 8).If τ im is too low, then the image is not fully deblurred and a slight distortion remains in the restored image (see Figs. 7a and 8).On the other hand, if τ im is too high, then a kind of negative "shadow" appears behind the object (see Figs. 7b and 8).Since the deconvolution keeps the total intensity of the whole image, the restored temperature with a too-low τ im value is also too low, and with a too-high one the restored temperature becomes too high.It is important to note that the used τ im values in Figs.
where τ 2 is the used coefficient, instead of the correct τ im .
For the example of Figs. 7 and 8, with a = 30 pixels, τ im = 44, τ 2 = 0.5 τ im and T true = 22 • C results in a temperature of about T restored ≈ 0.75 If the used coefficient is closer to the correct value, e.g.τ 2 = 0.9 τ im , then the deviation of the reconstructed temperature from the true one would be about 1.1 • C. On the other hand, it is also important to note that only small objects with high speed lose temperature due to incorrect deblurring; large objects can be well restored even if τ im is not exactly known.

Sharpness of restored images
In many cases, e.g. in thermographic non-destructive testing, non-calibrated cameras are used, because the temperature value itself is not important but only its distribution and a good contrast in the images (Maldague, 2001;Oswald-Tranta et al., 2010b;Oswald-Tranta, 2012).
Figure 9a and b demonstrates how well the sharpness of an image can be restored with the proposed deblurring algorithm.A 10 × 10 cm 2 metallic square ruler has been moved with the conveyor belt with 1 m s −1 speed in the field of view of the camera.Due to their different emissivity, the digits www.j-sens-sens-syst.net/7/13/2018/can be well read in a static infrared image, but after motion distortion they cannot be recognized anymore.However, after reconstruction, the digits become readable again (see Fig. 9b).It is important to note that the object in Fig. 9 has a square shape, but the recorded and also the restored images are squeezed.

Distortion due to continuous read-out
As the microbolometer camera is not a snapshot camera and it reads out the rows of pixels continuously, the bottom part of the object moves further, before its values are read out in the detector.This is causing the squeezing of the object (Fig. 9a and b), which can be corrected by a projective transformation.
The camera works with 50 Hz, which means every 20 ms the recording of a new image is triggered.In our camera, the read-out of all the n y = 480 rows takes about τ readout = 15 ms.Therefore, the shift of the object during the read-out from one row to the next one is Using this shifting value, a transformation matrix can be set up The application of this transformation before deblurring the image corrects the shearing effect, as can be seen in Fig. 9c.
It is important to note that this effect occurs always in the x direction.Since the object may move any direction in the image, v x in Eq. ( 16) and v in Eq. ( 3) are not necessarily identical -only in the case, if the relative motion between camera and object takes place in the x direction as shown in Fig. 9. Otherwise, v x is the x component of the speed v, showing in the x direction, which means along the pixel rows of the IR image.This squeezing effect can be also well observed in the images of Fig. 2. The higher the speed is, the stronger the lower part of the image is shifted.However, the small plate in each experiment was parallel to the y axis.The restored images  by applying first the shearing transformation according to Eqs. ( 16) and ( 17), and then the motion deblurring, are shown in Fig. 10.The images again become parallel to the y axis and the shape of the plates is well restored.
Figure 11 demonstrates a further example for the advantage of this reconstruction technique.Even if the different objects are hardly distinguishable in the infrared image, after restoration they become well visible.Figure 11 shows two plates moving with 1.5 m s −1 from the left to the right.The plate on the right side has higher temperature than the one on the left side.In the recorded infrared image, the two plates are blurred and show almost the same temperature (Fig. 11a), but after reconstruction their shape and temperature are correctly restored (Fig. 11b).
Infrared cameras can be well used to detect failures, e.g.surface cracks in objects.In the so-called active thermographic inspection, first heat is introduced to the object in a specific way, and due to the distribution and the temporal change of the temperature the failures can be localized in the infrared images (Maldague, 2001).One of the heating techniques is to apply inductive heating, which is very efficient for ferromagnetic materials, e.g. for steel.It can be excellently carried out in a scanning way, where the objects are moved continuously below or through the induction coil and the camera records the temperature distribution shortly afterwards (Oswald-Tranta et al., 2010b;Oswald-Tranta and Sorger, 2012).Figure 12 shows a small bell, which is moved below a coil, positioned on the left side, already outside of  the image.The inductive heating causes a high temperature increase around a surface crack, which becomes very well visible and detectable in the deblurred image.Also, in this case, the transformation with the matrix of Eq. ( 17) has been used and the shape of the bell is very well reconstructed.

Blind deblurring
In all the previous examples, the exponential coefficient of the PSF and the motion direction has been determined once for the measurement setup and this has been used for restoring the images.This technique works in many cases, e.g. in process control or in non-destructive testing, if the objects always move with the same speed into the same direction.However, it is also possible to determine the PSF from the image itself, provided there is a small hot point in the image somewhere.Theoretically, the PSF is the answer to a Dirac delta perturbation, which means a small hot spot in the ideal image becomes the shape of the PSF in the blurred image.Figure 5a shows the deblurred image of the small moving ball.By extracting a profile in the motion direction, one can fit an exponential function (see Fig. 13) to it, and the obtained coefficient can be used as τ im .
If the motion is linear but its direction is not known, this can be also determined for many cases from the image itself (Oswald-Tranta et al., 2010a).Figure 14 demonstrates this situation: in the first step, the gradient image is calculated, in which the edges show the motion direction.After rotating the  image, the PSF can be determined by fitting an exponential function.This technique works not only for such a simple case as a small ball in the image but also for images where several spots have significantly higher temperature than their surroundings.
The technique even works if the motion itself is not linear, but its path is known.Using this a priori knowledge, an additional step is necessary to transform the image first into a linear motion.For example, a rotation can be transformed into a linear motion (Oswald-Tranta and Sorger, 2012), which can be deblurred, and then by applying the inverse transformation an image with high contrast is obtained.Figure 15 shows the infrared image of two small balls with the same temperature, fixed to a stick which is rotating around one of its ends.The circle of the ball on the right side has a larger radius and thus a larger speed.Due to the larger blurring, its temperature seems to be lower than the left ball's temperature.Figure 15 shows also the intermediate steps for determining the rotation centre and the transformed image to a linear motion.After the deblurring and the inverse transformation, both balls have again a circular shape and their temperature is also restored to show the same value.
Even if the motion direction is not exactly a straight line or a precise rotation, very good results can be achieved with an estimated direction and speed.Figure 16a shows a hand moving in front of the camera.Its path is arbitrary but almost goes along a straight line.With an approximated linear motion, the image has been deblurred and its result is shown in Fig. 16b.The image is sharp, even the ring on the finger becomes well visible.

Summary and conclusion
It has been shown that in many cases the motion blurring in the image of a microbolometer camera can be eliminated or strongly reduced.If the correct PSF is used, a good estimation of the correct temperature can be calculated.Several experimental examples have been presented and the main factors influencing the temperature reconstruction have been identified.
If the speed and the direction of the motion are well known, then the PSF needs to be determined once, and it can be used for deblurring of the subsequent images.If the speed and the direction are not known, then the blurred image of a small hot spot which is present in the image corresponds to the PSF, which can be then determined with a good approximation from the image itself.
This motion deblurring technique can be excellently used, e.g. in industrial applications, where the objects are moving always with the same speed, like on a conveyor belt or rotating always on the same trajectory.In these cases, the reconstruction parameters can be determined once and used for the deblurring of the images.
This method is also well applicable if there is only one object moving in the field of view of the camera.Even if its speed or moving direction is not known, the blind algorithm for reconstruction delivers usually very good results.
If many objects with different speeds and different directions are moving in front of the camera, and also their trajectories are crossing each other, then the blurred images of the individual objects will also cross each other, causing a problem for this reconstruction technique.

BFigure 1 .
Figure 1.Experimental setup in the laboratory with a microbolometer and with a cooled photon detector, recording objects in motion on the conveyor belt.

Figure 2 .
Figure 2. Microbolometer images of a small plate moving from the left to the right with different speed: 0.3 (a), 0.5 (b), 1 (c) and 1.5 m s −1 (d).

Figure 3 .
Figure 3. Temperature profiles through the blurred and the reconstructed images of Fig. 2. The corresponding data are summarized in Table1.

Figure 4 .
Figure 4. Amplification factor depending on the frequency due to deconvolution with the parametrized Wiener filter.

Figure 6 .
Figure 6.Temperature profiles through the three restored images compared with the original blurred one in Fig. 5.

Figure 8 .
Figure 8. Temperature profiles through the three restored images of Figs. 7 and 5c compared with the original one in Fig. 5a.
7 and 8 are 50 and 150 % of the correct value.The ratio of the restored temperature to the true one is approximately(Oswald-Tranta et al., 2010a;Oswald-Tranta, 2012)

Figure 9 .
Figure 9.A square-shaped metallic ruler during the motion (a), the deblurred image (b) and the deblurred image applying also the shearing transformation of Eq. (17) (c).

Figure 10 .
Figure 10.Restored images of Fig. 2 moving a small plate with different speeds.

Figure 11 .
Figure 11.Infrared image of two plates moving with a speed of 1.5 m s −1 (a); after deblurring the objects (b), their shape and real temperature are well visible.

Figure 12 .
Figure 12.A small, inductively heated bell with a motion speed 1 m s −1 (a) and the deblurred image (b).

Figure 13 .
Figure 13.Blurred temperature profile from Fig. 5a and the fitted exponential function.

Figure 14 .
Figure 14.Infrared image of a small warm ball (a), gradient image with marked motion direction (b), edges of the gradient image (c); restored image (d).

Figure 15 .
Figure 15.Infrared image of two small balls circularly moving (a), gradient image to determine the rotation centre (b), image transformed to linear motion (c) and the restored image (d).

Figure 16 .
Figure 16.Infrared image of a hand with motion blurring (a) and the reconstructed image (b).

Table 1 .
Data of the four measurements in Figs. 2 and 3 are summarized in the table.