Modeling the unified measurement uncertainty of deflectometric and plenoptic 3-D sensors

In this paper we propose new models of two complementary optical sensors to obtain 2.5-D measurements of opaque surfaces: a deflectometric and a plenoptic sensor. The deflectometric sensor uses active triangulation and works best on specular surfaces, while the plenoptic sensor uses passive triangulation and works best on textured, diffusely reflecting surfaces. We propose models to describe the measurement uncertainties of the sensors for specularly to diffusely reflecting surfaces under consideration of typical disturbances like ambient light or vibration. The predicted measurement uncertainties of both sensors can be used to obtain optimized measurements uncertainties for varying surface properties on the basis of a combined sensor system. The models are validated exemplarily based on real measurements.


Introduction
Automated quality inspection of product surfaces requires a fast and robust sensor, capable of detecting all relevant defects without damaging the surface.Optical measurement techniques fulfill these requirements but are highly dependent on the surface properties.For example, pattern projection and passive stereoscopic methods require diffuse reflectance, while deflectometric methods depend on specular reflectance of the inspected surface.Many surfaces are partially specular or a mixture of diffusely and specularly reflecting parts and cannot be robustly measured with only one method.By combining several measurement methods into a single sensor system that adapts its algorithms to exploit the advantages of the single methods, we are capable of measuring surfaces with a large variety of surface properties.To demonstrate the principle, we propose uncertainty models for plenoptic and deflectometric sensors, and based on the models we simulate both sensors under similar circumstances on varying partially specular surfaces.
1.1 Related work Tutsch et al. (2011) give a good overview of optical 3-D measurement techniques with structured illumination that includes passive triangulation and deflectometry.The measurement uncertainty for the deflectometric sensor is based on the phase noise model from Fischer et al. (2012), which is itself based on parameters defined in the EMVA 1288 measurement standard by the European Machine Vision Association.Fiete and Paul (2014) describe a systematic approach to model the imaging chain using the optical transfer function, that will be used in Sect. 2.
Plenoptic cameras have been used in computational imaging for several decades now.However, Perwaß and Wietzke (2012) proposed the first plenoptic camera for industrial applications.They commercialized their camera in the company Raytrix.In the recent years, several papers on the metric calibration of Raytrix cameras were published; see Johannsen et al. (2013), Heinze et al. (2016), Zeller et al. (2016), Strobl and Lingenauber (2016) and Zeller et al. (2017).Furthermore, the potential of plenoptic cameras with respect to 3-D measurement was derived analytically by Perwaß and Wietzke (2012) as well as Zeller et al. (2016) and Published by Copernicus Publications on behalf of the AMA Association for Sensor Technology.demonstrated experimentally by Heinze et al. (2016), Zeller et al. (2016), and Sardemann and Maas (2016).Nevertheless, all existing analytical evaluations considered the optical system only from a purely geometrical perspective, ignoring the effects of real optical systems as is done in this paper.

Outline
First, in Sect. 2 we introduce the photometric properties of the measurement system by means of the spatial distribution of light and the modulation transfer function.In Sects. 3 and 4 we describe the deflectometric sensor and the plenoptic sensor and derive their measurement uncertainties.Then in Sect. 5 we compare the uncertainties for both methods.All plots shown in this paper use parameters from an exemplary setup given in Table 1.Finally, in Sect.6 we present, by way of example, real measurements to validate the proposed models.

Photometry
Deflectometry, as well as the plenoptic method, relies on the recognition of spatial light patterns to identify unique positions that can be triangulated.The reliability of this recognition depends on the pattern contrast.When the pattern contrast is very low, noise introduced by the camera dominates the pattern.In the following section we introduce a systematic approach to describe the pattern contrast and a reduction of this contrast depending on its spatial frequencies.Despite the 2-D nature of image-based measurements we decided to describe our approach in 1-D, since the direction has no impact on the results.

Deflectometry
In deflectometry the camera integrates light emitted at some point x by the screen L scr (x) (radiance) and reflected by the surface (ignoring spectral dependency).The light L cam (m) reaches the camera image plane at some point m "smeared" by the screen itself, the reflection at the surface, the refraction and diffraction by the camera lens and the sensor itself.The smearing can be mathematically described as a convolution of the incoming light with the point spread function (PSF).We assume that the PSF is translation-invariant to the object and the sensor point: Now, the image radiance is the convolution of the screen radiance and the point spread functions of the screen (PSF scr (x)), the surface (PSF srf (x)) and the camera (PSF cam (x)): Instead of using PSFs in the spatial domain, the imaging properties of optical systems can be described in the Fourier domain, depending on the spatial frequency k.Then the optical transfer function (OTF), which is the Fourier transform of the PSF, is used instead.As we are only interested in modulation changes, we separate the OTF into modulation transfer function (MTF) and phase transfer function (PTF): (k) . (3) Additionally, we normalize the MTF to 1 for the DC component: We will introduce the patterns shown on the screen I scr (x) (radiant intensity) in Sect.3.1.The radiant intensity from the screen is linked with the radiance from Eq. ( 2) by multiplication with the pixel size A scr and the cosine of the incident angle θ scr of the optical axis to the screen normal direction (which is approximately the same for all points on the screen for a small screen size compared to the screen distance): I scr (x scr ) = L scr (x scr )A scr cos(θ scr ). (5) The radiance distribution γ scr (k scr ) with the spatial frequency k scr on the screen is the spectrum of the radiance (F{•} denotes the Fourier transform): The irradiance on the camera sensor is linked with the radiance from Eq. ( 2) by multiplication with the solid angle of the sensor pixel cam and the cosine of the incident angle θ cam of the optical axis to the sensor normal direction (which again is approximately the same for all points on the sensor for a small sensor size compared to the camera distance): Now we obtain the irradiance distribution on the camera sensor, as a product of the screen radiance distribution γ scr (k scr ) and the MTFs: Note the usage of different spatial frequencies k cam and k scr .We will introduce the correspondence between camera and screen spatial frequencies for simple surface shapes in Sect.3.4.In Sect. 5 we will discuss the optimum screen pattern, minimizing the measurement uncertainty.The screen and camera MTF can be measured in advance using a plane first-surface mirror with M srf (k cam ) ≈ 1.If necessary, the camera MTF can be measured separately in advance using calibrated MTF targets and subsequently the screen MTF can be measured with the calibrated camera at close range to the screen; see Triantaphillidou and Jacobson (2004).The surface MTF M srf depends on the surface roughness and position between screen and camera, so M srf has to be measured for a fixed setup.

Plenoptic
Things change for the passive plenoptic setup, where the surface itself is considered as an emitter of structured light.We assume that the surface itself reflects unstructured light from the environment depending on the surface texture γ srf (k srf ).However, for surfaces which do not show perfect Lambertian reflectance, that surface texture is superimposed by the specular reflections on the surface and imaged and sensed by the plenoptic camera (described by M cam (k cam )) and lead to a radiance distribution γ cam (k cam ) on the camera sensor.The superposition of the surface texture showing Lambertian reflectance with the specular reflections of unstructured light from the environment can be considered as a reduction of the signal-to-noise ratio (SNR) as it increases the noise level on the surface.Instead of increasing the noise level, we model this superposition as a spatial-frequency-dependent attenuation of the surface texture γ srf (k srf ).Anyway, on the image sensor no absolute signal levels can be measured since they are equalized by the automatic exposure control of the camera.The relative attenuation of the surface texture γ srf (k srf ) due to specular reflectance is modeled by a function that we call the Lambertian surface MTF M srf .Since this Lambertian surface MTF M srf (k srf ) models the deviation from a perfect Lambertian reflectance it behaves reciprocally to the surface MTF M srf (k srf ) defined for the deflectometric setup.Hence, similar to Eq. ( 7) we obtain the following: In the best case for plenoptic measurements, the surface pattern consists of intensity steps with an modulation of 1 over a wide spatial frequency spectrum: This pattern is superimposed by the specular reflection of pattern in the environment.Hence, in contrast to deflectometry, the specular reflection decreases the measurable patterns contrast for the plenoptic setup.A detailed definition of the surface MTF will be discussed in Sect.2.5.

Camera MTF
Following Fiete and Paul (2014) the camera OTF consists of the lens OTF and the sensor OTF.In the best case of a diffraction-limited optical system, the lens OTF is given by the following: with k = k cam and F is the ratio of the focal length f and the diameter D of the entrance pupil and is commonly known in photography as the "f number".Ignoring signal sampling, noise, quantization and anisotropy, the OTF of the sensor is given (with k = k cam ) by the following: The bandwidth of the sinc function depends on the aperture of a pixel on the sensor and is assumed to be equal to the pixel pitch c.The camera MTF is the composition of the lens OTF in Eq. ( 10) and the sensor OTF in Eq. ( 11) as follows: In Fig. 1 we show the camera MTFs for the deflectometric and the plenoptic camera with lens MTFs as given in the manufacturer data sheet and sensor MTF derived from parameters in Table 1 in comparison with camera MTF for the same sensor and a diffraction-limited lens.

Defocus MTF
The OTF introduced in the previous section describes a camera in focus; in this section we will discuss a camera out of focus.Let the camera be in focus at some object distance g Using the thin lens equation 1 f = 1 g + 1 b the image distance b can be eliminated and we get the size for a point out of focus on the sensor: The size of a point on the image plane c cannot be smaller than the camera pixel size c, and thus we define it as follows: The OTF for an image out of focus depends on the size of this point (see Beyerer et al., 2012) and replaces the sensor OTF in Eq. ( 11), where J 1 is the Bessel function of first kind and order: Figure 2 shows the camera MTF of screen points with 1.0 m distance for several focus distances g from the surface (g = 0.5 m) to the screen (g = 1.0 m).

Motion blur
In many situations the camera is shaking during exposure due to vibrations caused by heavy machines, etc. Fiete and Paul (2014) give an OTF for motion blur caused by a random movement of the camera during the exposure with standard deviation σ mot : Of course, the measured surface may also be subject to vibrations, but due to the complexity of the implications of changing surface normals during exposure it is not covered here.The influence of translational camera motion blur on the MTF according to Eq. ( 17) is shown in Fig. 3.Note that the motion blur is related to the pixel pitch (here 6.45 µm), so motions during exposure smaller than half the pixel pitch in image space have almost no impact on the modulation.

Roughness
With more surface roughness the reflectance decreases from specular to diffuse; see Fig. 4. Harvey et al. (2012) and Harvey (2013) describe the relation of surface roughness measured as root mean squared (RMS) surface height and the optical properties of the surface using total integrated scatter and the MTF.They observe that the RMS value has to be calculated within the relevant scale.However, this model seems to be valid only for very smooth surfaces, so in the following we propose a simple parametric model for the surface MTF: This matches the MTF measurements of the surfaces shown in Fig. 4, but has no dependency on physical parameters like the surface roughness.For each surface type, the parameter c has to be estimated.The five surfaces shown in Fig. 4 have parameters in the range c = 2.5. ..3.8.As M srf (k cam ) depends on the camera distance s and the screen distance r, it has to be estimated again, if the setup changes.Figure 5 shows the surface MTF for different gloss factors c.
While deflectometry utilizes the specularity of the surface, passive triangulation approaches like plenoptic-camerabased methods rely on Lambertian reflectance.Hence, for the plenoptic camera we consider the specular component as an additional noise component and therefore model the Lambertian surface MTF M srf as follows: Figure 6 shows the Lambertian surface MTF for different gloss factors c.

Ambient light
In the previous section we described how surface roughness increases the amount of light scattered by the surface.If more ambient light is present in the scene, the light scattered in the direction of the camera also increases.This can be measured as the Michelson contrast, i.e., the ratio of the difference and the sum of the maximum and minimum radiance (L min , L max ).To visualize the influence of (constant, unstructured) ambient light, we assume that the radiance of the pattern, received by the camera, is increased by a constant offset L amb , which leads to a modified surface MTF: If the ambient radiance reflected into the camera equals half the radiance of the maximum pattern intensity, the pattern contrast decreases to one-half of the original contrast.Calculating the influence of ambient radiance emitted by the surface requires knowing the location of the ambient light sources and the surface BRDF (bidirectional reflectance distribution function).On specular reflecting surfaces, ambient light does not influence the contrast of a reflected pattern, but more diffuse reflection increases the amount of ambient radiance reflected into the camera.The contrast of a surface texture is influenced by the amount of specularly reflected light.

Deflectometry
Deflectometry (see Werling et al., 2009), is a relatively inexpensive but powerful method for inspection and measurement of specular surfaces.A generic setup is shown in Fig. 7.The basic idea is to measure the surface normals by identifying the origin x scr of each camera ray (going through x cam and the optical center) that is reflected at the surface.A phase-shifting algorithm is used to identify this origin on the screen x scr .We use M > 3 shifts of a cosine fringe pattern (as seen in Fig. 4) with frequency k scr on the screen to identify the location of x scr with subpixel uncertainty in one direction.The result of this algorithm is the phase φ at the current screen point x scr : Hence, the measurement uncertainty of the camera ray origin on the screen σ x scr is given by the following: In the following section we will look at the phase-shifting algorithm and the phase noise model.

Phase shifting
Let the origin x scr of each camera ray be a point on a flat screen showing a sequence of i = 1. ..M cosine patterns captured by the camera with intensity I at the position x cam on the image plane with relative mean exposure β and relative pattern contrast γ cam (k cam ): β and γ cam are both constrained to 0. ..1 and β(1 The maximum exposure is the saturation capacity of the sensor I sat .Each pattern In the case of a four-step phase shifting (M = 4) the phase is obtained from the captured intensities as follows: To get the absolute position on the screen x scr the phase φ ∈ (0. ..2π ) has to be unwrapped by determining m in Eq. ( 26).
One popular phase-unwrapping method is the heterodyne method, which uses two different pattern frequencies.See Zuo et al. (2016) for comparison of phase-unwrapping methods.

Phase noise
A model describing the phase noise of Eq. ( 25) for a symmetric M-step algorithm was derived by Fischer et al. (2012): The model is based on the EMVA 1288 camera noise model by the European Machine Vision Association with the parameters of saturation capacity µ e.sat , dark noise variance σ 2 d and overall system gain K. Hence, the authors show a way to estimate the EMVA 1288 camera parameters using a deflectometric setup.Since β and γ depend on local surface reflection properties, the authors propose a way to estimate the measurement uncertainty in Eq. ( 27) based on the observed intensities for each pixel separately: Here y i = I i − µ d denotes the physically correct intensities, µ d is the mean dark noise and R 4 the noise term, defined as follows: The two parameters influenced by the environment and the surface are β and γ .We can ensure β = 0.5, if the camera exposure is chosen appropriately.The pattern contrast on the sensor γ cam is given as product of the MTF functions for the screen, surface, ambient light, camera and motion blur in Sect. 2. The rest of the parameters can be either chosen (M) or are known from the camera data sheet (EMVA 1288 parameters).Alternatively σ φ can be estimated directly from a measurement using Eq.(28).

Geometric properties
In the following section the uncertainty of the surface height σ z will be derived.Let the surface be acquired in many small mirror segments.The width of each segment is the lateral uncertainty σ x , and the slant uncertainty is σ α ; see Fig. 7.The uncertainty in surface height is then obtained by the following: Measurement uncertainty σ z of a deflectometric system with lateral uncertainty σ x caused by discretization σ c on the camera sensor and uncertainty of the origin of incident rays from the screen σ x scr caused by the phase-shifting algorithm.
The lateral uncertainty σ x is determined by the angular uncertainty σ θ and the distance s of the camera to the surface (assuming only small angles between viewing and surface normal direction): The angular uncertainty of the camera σ θ is determined by the discretization on the sensor σ c = c and the focal length f : Either the size of one projected pixel from Eqs. ( 31) and (32) or the defocus on the surface caused by focal plane distances g ∈ [s, r] and the aperture size D ( D σ x = g s−g ) set the lower limit of lateral uncertainty: The slant uncertainty of each surface segment σ α is determined by the screen σ x scr from Eq. ( 23) and the screen distance r (assuming only small angles between direction of reflected ray and screen normal): Combining the above equations, the uncertainty of the surface height σ z (with tan( σ α 2 ) ≈ tan σ α 2 ) is as follows:

Surface shape
Assume that the specular surface has a convex spherical shape.Then the screen appears smaller, which results in a smaller fringe period.In the simple case of a plane mirror, the effective fringe pattern frequency depends only on the distance from the camera to the screen: If the surface is curved (at least piecewise) like a sphere with radius R, the imaging properties can be described by the imaging properties of a convex (positive radius) or concave (negative radius) mirror with focal length f s = − R 2 : This equation has a pole at f s = (f r − rs)/(f − r − s) for concave surfaces, where the screen is located in a caustic.Near this pole, even infinitely small changes of the screen pattern would lead to changes in the image plane, but this configuration is impractical for a real inspection task.

Plenoptic camera
A plenoptic camera is a single sensor system which records a 4-D light-field representation of a scene in a single image.That means a point in the object space does not only correspond to a single image point in the image, as it would be for a regular camera, but to multiple image points.In other words, a plenoptic camera does not only capture a single ray emitted from a certain point in the object space but multiple light rays with different incident angles.Hence, the four dimensions describe two spatial dimensions and two angular dimensions.Even though plenoptic sensors for industrial applications are still expensive (see Raytrix GmbH, 2016), they rely on a quite simple idea.In general any industrial camera can be transformed into a plenoptic camera by placing a micro lens array (MLA) in front of the image sensor.
The 4-D light field recorded by a plenoptic camera enables tasks like 3-D measurement or software-based refocusing after an image is captured.Industrial tasks for plenoptic cameras may include 3-D microscopy or the inspection of production parts.
Here we describe the principle of plenoptic depth measurement based on the concept of a focused plenoptic camera developed by Lumsdaine and Georgiev (2009).As well as this, we formulate an uncertainty prediction model for the plenoptic depth measurement that relies on this concept.
Figure 8 shows the imaging process of a focused plenoptic camera in the Galilean mode (see Lumsdaine and Georgiev, 2009).The main lens produces a virtual image of the real object at distance b L behind the main lens.Due to the MLA a  The image distance b L results from the object distance s and the main lens focal length f as defined by the thin lens equation: Based on disparities µ, which can be measured in the micro images, one is able to calculate the image distance b L and from that the respective object distance s of a certain point, under the assumption that all intrinsic camera parameters are known; see Zeller et al. (2016).

Uncertainty prediction model
In contrast to deflectometry, the plenoptic camera is a passive measurement system that relies on high-contrast patterns on the surface to be measured.Besides, the surface has to have Lambertian reflectance to obtain correct measurements.
In the following we define a model to predict the measurement accuracy of the plenoptic camera for a certain measurement setup.This measurement setup is shown in Fig. 10.
In analogy to deflectometry and to obtain a general definition of the surface contrast we define the surface structure as a fringe pattern similar to Eq. ( 24).
Of course, the pattern on the surface which is captured by the camera will never be a perfect fringe pattern but can always be modeled as a mixture of frequencies.However, this formulation gives us the possibility to model the camera response dependent on the frequency k srf of the surface pattern.In general, the structures to be measured on the surface are intensity steps and thus contain the complete frequency spectrum.
In a local region one can consider the imaging process just as a scaling of the fringe pattern on the surface in combination with a frequency-dependent attenuation of the intensity modeled by the MTF of the imaging system.Therefore, by applying the assumption of being in a local region around a certain point, perspective distortion does not have to be considered.

Imaging scale
In contrast to a regular camera, which can be defined mathematically by a pinhole camera model, the imaging scale of a plenoptic camera is not proportional to the distance between main lens and object s.Instead, a plenoptic camera basically performs two perspective projections, one by the main lens from object space to the virtual image and one from the virtual image on the sensor.The scaling between object space and virtual image s L is given by the following: Furthermore, one gets the scaling between virtual image and the micro images on the sensor as given by the following: Here, b L0 is the distance between MLA and main lens and B is the distance between image sensor and MLA (see Fig. 8).
Both scaling factors (s L and s ML ) behave reciprocally to each other.While an increasing object distance s results in an increasing scaling factor s L , the scaling factor s ML decreases at the same time since the virtual image distance b L is also decreasing.
Besides the scaling of the pattern due to perspective projection, the fringe pattern is compressed if the viewing angle of the plenoptic camera α is not 90 • to the surface (see Fig. 10).This results in an additional scaling of the pattern s α defined as follows: with 0 < α < π.
Based on the defined scaling factors one can define the following relations between surface x srf , virtual image x vim and sensor coordinates x cam : Hence, we obtain the following frequencies of the fringe pattern in the virtual image k vim and on the sensor k cam , respectively: Following the Shannon-Nyquist sampling theorem we can calculate from Eq. ( 45) an upper boundary for the fringe pattern frequency on the surface.Thus, to avoid aliasing, the following condition has to hold: Here, c is the pixel pitch of the sensor.After inserting Eq. ( 45) into Eq.( 47) and rearranging it, one receives the following condition for the frequency of the fringe pattern on the surface to assure aliasing-free sampling:

Attenuation
Similarly to in deflectometry, we can describe the imaging properties of the plenoptic camera by its MTF.In a plenoptic camera, we have a sequence of two optical systems: the main lens and the MLA.This results in an MTF of the complete plenoptic camera that depends on the distance to the surface.This can be formulated as the sequence of two MTFs with a nonlinear connection between k vim and k cam : For simplification, we approximate the complete MTF M cam (k cam ) as follows: By this approximation we neglect the depth scaling from the virtual image on the sensor and therefore consider M L and M ML to be defined on a common frequency axis.However, the error made by the approximation is negligible for object distances that are large with respect to the focal length f L and short depth ranges, which have to be measured.An alternative would be to define and measure a depth-dependent MTF.
For a Raytrix camera the MLA consists of three different types of micro lenses to increase the depth of field of the camera.Therefore, strictly speaking one has to define three different MTFs for the respective micro lens types.
For the simulations in Sect. 5 we assume a plenoptic camera MTF M cam (k cam ) as defined in Eq. ( 12).However, it might be worth investigating differences in the camera MTF of a plenoptic camera in comparison to a regular monocular camera.

Measurement uncertainty
Depth measurements are obtained based on disparities µ, which are estimated from the recorded micro images.The accuracy of the estimated disparity mainly relies on the intensity gradient along the epipolar line as well as the additive signal noise of the image sensor.The signal noise already combines noise sources like dark noise and quantization noise.
For a given fringe pattern I srf (x srf ), as defined in Eq. ( 24), on the surface to be measured one obtains a maximum absolute intensity gradient on the image sensor as follows: The MLA in a plenoptic camera is in most cases arranged on a hexagonal grid.Therefore, for each micro lens multiple epipolar lines in all possible directions are obtained.Figure 11 exemplarily shows the epipolar lines in a hexagonally arranged MLA.Thus, in the following we equate the maximum absolute gradient g cam with the maximum absolute gradient along one specific epipolar line.Zeller et al. (2016) show that one can approximate the variance of the estimated disparity σ 2 µ based on the variance of the signal noise σ 2 I and the maximum gradient along the epipolar line g cam as follows: As defined in the EMVA1288 standard, the signal noise σ I results from the system gain K, the dark noise σ d , the quantization noise σ q , the gray value µ y and the dark signal µ y.dark as follows: However, for later simulations we use an average value derived from the camera specification as given in Table 1.
Eq. ( 52) states that for a high-intensity gradient along the epipolar line the disparity can be determined more accurately than for a low-intensity gradient.
For multifocus plenoptic cameras (see Perwaß and Wietzke, 2012), like those from the manufacturer Raytrix, the MLA consists of different types of micro lenses with different focal lengths.Therefore, for a certain object distance one type of micro lenses will produce focused micro images while the micro images of another type will be out of focus.This effect of differently focused micro images also has to be considered when estimating the disparity µ.How this focus disparity error can be modeled is shown by Zeller et al. (2017).
For simplification, we do not consider the focus disparity error here.Besides, by choosing an appropriate camera setup one can assure that a pair of focused micro images is always present for a given object point.
Based on the theory of propagation of uncertainties, one is able to calculate the standard deviation of the measured object distance σ s from the disparity standard deviation σ µ .The relationship between σ µ and σ s is given by the following: Equation ( 54) was derived by Zeller et al. (2017).Here D M defines the diameter of a micro lens, while κ is a normalized distance factor between the two micro lenses used for disparity estimation, i.e., κ is one for two directly adjacent micro lenses.In a hexagonal arrangement of the MLA the smallest 10 values of κ are 1.00, 1.73, 2.00, 2.65, 3.00, 3.46, 3.61, 4.00, 4.36 and 4.58. Perwaß and Wietzke (2012) showed that, depending on the distance between virtual image and MLA plane v, it can be guaranteed that micro lenses further apart still see the same object point and therefore a larger value of κ can be chosen.

Simulation
In the following section, we present simulation results for the measurement uncertainty of both the deflectometric and the plenoptic sensor.They are derived for an exemplary setup shown in Table 1.We chose similar settings for both systems to make realistic comparisons.All simulation results that depend on the spatial frequency of some pattern are shown as a function of the spatial frequency on the image plane k cam .The frequency limits have been chosen according to the Shannon-Nyquist sampling theorem: the highest detectable frequency is limited by the pixel pitch, i.e., k cam < 1 2 c .The surface MTF parameters for the low and high gloss and mirror were estimated from real measurements, the value for the ideally specular surface should be c → ∞, but for numerical reasons we chose c = 8.For plots containing results of both sensors, we plotted results in blue for the deflectometric sensor and results in red for the plenoptic sensor.

Simulation results -deflectometry
We assume that the overall shape of the reflecting surface is flat.Thus we can apply Eq. ( 36) to estimate the spatial frequency on the sensor k cam from the frequency on the screen k scr .
The first result, depicted in Fig. 12, shows the uncertainty of the phase noise model using Eq. ( 27) for different surface gloss parameters.It can be seen that the uncertainty increases rapidly with the pattern frequency when the surface is not perfectly specularly reflecting.
In contrast, higher spatial frequencies (and thereby shorter period lengths) in Eq. ( 23) decrease the uncertainty in estimating the screen position x scr .These two opposite effects lead to a unique pattern frequency, where the resulting measurement uncertainty on the screen reaches a minimum for some frequency k scr opt and a given roughness.As the uncertainty of the surface slant σ α given in Eq. ( 34) directly depends on the uncertainty of the screen position, we can also see these minima in Fig. 13.Here the measurement uncertainty of the surface normal dependent on the pattern frequency k cam is shown for different focus distances g as well as different surface gloss factors c.For focus distances g < 1 m the image of the screen is defocused and the Bessel function from Eq. ( 16) starts to appear.Please note that the curves in Fig. 13 with g ≤ 0.8 m are subject to aliasing artifacts.
It can be seen that k scr opt changes with the surface properties.With higher surface gloss c and the focus on the screen (g → 1 m), higher frequencies can still be resolved on the sensor, and hence k scr opt increases and leads to lower measurement uncertainties.Of course, the camera sensor still limits the highest resolvable frequency on the sensor: In Fig 14 we show the measurement uncertainty from Eq. ( 35) in z direction next to the results of the plenoptic sensor.Note the triangle model shown in Fig. 7 that connects the uncertainties of surface height σ z , position on the surface σ x and angle of the reflected ray σ α .Since both uncertainties σ x and σ α are very small, the final measurement uncertainty σ z is very small as well (for plane surface mirrors in the nanometer range).Also note that the surface height is not measured directly, so σ z is only valid for local changes.The whole surface can be reconstructed by integration over the surface gradients, but along with the integration, the uncertainties sum up to larger errors for a global surface shape.

Simulation results -plenoptic camera
Similar to Sect.5.1 we performed different simulations based on the measurement uncertainty model defined in Sect.4.1.We chose a plenoptic sensor similar to the Raytrix R5 camera (see Table 1).
In Table 1 the measurement range s defines the range around the surface distance s for which measurements can be obtained.To obtain depth measurements from a plenoptic camera it has to be assured that a point is visible in at least two micro images.This is the case for virtual depth v ≥ v min = 2.0 (v := b L0 −b L B ; see Fig. 9) and thus for b L ≥ b Lmin = b L0 + 2.0B for a hexagonally arranged MLA (see Perwaß and Wietzke, 2012).Hence, b L0 is chosen such that s + s results in an image distance b L ≥ b Lmin .Therefore, for all distances larger than s + s no depth measurements can be obtained.By minimizing the measurement range s one is able to improve the measurement accuracy.

Focal length
Figure 15 shows simulation results for a perfect Lambertian surface.Here we varied the focal length f while all other parameters are as given in Table 1.As can be seen from the figure, the depth uncertainty of the camera can be significantly improved by increasing the focal length.However, there exists an optimum focal length around f = 50 mm for which the best measurement uncertainty is obtained.The reason for that is the reciprocal behavior between image distance b L and object distance s, which is dependent on the focal length f .Hence, as long as the distance s is much larger than the focal length f the depth uncertainty can be improved by increas- ing the focal length.However, at approximately s ≈ 10•f the best accuracy is obtained, as can be seen from the figure.
For the following simulations, which model the effect of motion blur as well as different surface roughnesses, the focal length was set to f = 16 mm.

Simulation results -motion blur
Figure 16 shows the effect of different amounts of camera vibration for both systems.The resulting motion blur behaves as a low-pass filter which is modeled as given in Sect.2.4.Due to the implicit low-pass filtering the optimal pattern is shifted to a lower frequency.Thus, the introduced blur significantly degrades the measurement results.

Simulation results -surface roughness
While increasing surface roughness has a negative effect on the deflectometric measurement results, due to less specular reflectance, it has the opposite effect on the plenoptic measurements.For the plenoptic setup best results are expected at a completely Lambertian reflectance (surface gloss c → 0).Noise is introduced by the specular reflection component.Figure 14 shows the expected measurement results for surfaces with different specularity.
Finally, in Fig. 17 we show the measurement uncertainty as a function of the surface gloss c and the focus distance g, assuming (for deflectometry) that the optimal pattern k scr = k scr opt is chosen for each case.
For high-gloss surfaces the deflectometric sensor is most accurate with the camera focused on the screen.For lowgloss surfaces it is preferable to focus on the surface and use patterns with low spatial frequencies.

Experiments
In this paper we present quite complex and complete mathematical models for the measurement process of a deflectometric and a plenoptic sensor system, although for such complex models it is almost impossible to validate them entirely.Hence, we validate our models on the basis of only two distinct setups: one for deflectometry and one for plenoptic.
For both measurement systems we use the same configurations as given in Table 1.However, one cannot guarantee that all parameters match exactly, due to manufacturing tolerances as well as manual adjustments.
For both sensor systems we obtain measurements based on the case of a thermometer that consists of surfaces with different reflectance properties.This case is shown in Fig. 18.
Since for the plenoptic camera we cannot influence the contrast of the surface pattern on the case, we performed a second experiment, in which we generated a fringe pattern on a screen and recorded this pattern with the plenoptic camera.Here, we measured the measurement uncertainty for different fringe frequencies.This setup is shown in Fig. 24.

Experimental results -deflectometry
We used the deflectometric sensor shown in Fig. 19 to validate the measurement uncertainty of the surface slant σ α .
The measurements of the thermometer surface were taken using 24 different pattern frequencies in the range k cam = 0.127-32.7 mm −1 with a logarithmic step size and under dark and bright illumination conditions.For dark illumination conditions there is no ambient light, apart from stray light emitted by the screen.For bright conditions there is additional light from fluorescent ceiling lamps.Fringe modulation γ cam was obtained using the captured images under both illumination conditions.Figure 20 shows the fringe modulation for one low and one high pattern frequency.
The red boxes mark areas on the surface with different reflection properties: case and display.Measurements are taken per pixel and then averaged over these two areas.On the one hand, we calculated a spline interpolation γ cam (k cam ) from the fringe modulation measurements, as shown in Fig. 21.Using this continuous function of γ cam (k cam ) and the sen-  sor model we are able to predict a continuous function for σ α (k cam ).
On the other hand, we estimated σφ for each pattern frequency from the standard deviation over nine phase measurements.Two phase measurements are shown in Fig. 22.
Hence, using Eq. ( 34), we calculated the standard deviation of the surface slant σα and compared it to the predicted measurement uncertainty as shown in Fig. 23.
The predicted uncertainties for σ α slightly underestimate the measurement standard deviation, especially for the surface case.This may be caused by inaccurate measurements or interpolation of γ cam .At k cam ≈ 1e−3 mm −1 the measurement noise is dropping, because the standard deviation of the phase σφ is limited to 2π , while the frequency in Eq. ( 23) is increasing.Apart from this, the model approximates the uncertainty of the display surface slant very well.

Experimental results -plenoptic camera
Using the setup which is shown in Fig. 24 we recorded sets of images at different fringe frequencies.We recorded images at 10 different frequencies starting at a screen frequency k scr = e−3pixel −1 up to k scr = e−1pixel −1 with a logarithmic step size.For each frequency we recorded 20 images, which we used to calculate the measurement statistics empirically.To obtain the frequencies k cam of the fringe pattern in the camera image we calculated the discrete Fourier transform (DFT) of a single image and detected the peak of the first harmonic.This way a scaling factor from k scr to k cam is obtained.
Figure 25 shows the mean empirical standard deviation σs of the measured distance s for a pixel in the depth map.Different than expected, we obtain a very low standard deviation for the smaller frequency.Furthermore, for the highest  frequency the standard deviation seems to increase slightly.However, both effects are plausible.For low frequencies, the gradient in the image is just not high enough to obtain reliable depth estimates, and hence no measurements at all or only a few measurements are obtained.Thus, too few measurements do not allow us to calculate reliable statistics.Due to the periodic pattern, for high frequencies one obtains ambiguous stereo matches in neighboring micro images.Hence, the depth estimation has a higher number of outliers.However, the standard deviation σs behaves similarly to that predicted by the presented model, as shown in Fig. 15.Due to the fact that we have now accurate MTF for the screen used in this experiment and the ambient light can only be  influenced to some degree, we cannot validate the model on a more quantitative basis.
Figure 26 shows the reconstructed intensity image of the case recorded by the plenoptic camera.Here, one can still slightly see the borders of the micro images, which are a little bit darker than the center.
In contrast to the deflectometric setup, we are not able to validate our model based on the recordings of this case, since we are not able to influence the pattern on the surface of the case.However, we still measured the empirical standard deviation for two different positions on the case.Here, the mean standard deviation is calculated based on a set of 40 images for all valid points seen in Fig. 26b.We used all valid points on the display and only those valid points on the case around the "Min/Max" inscription.
For the display we measured a mean standard deviation of 8.8 mm and for the inscription a mean standard deviation of 12.8 mm.Intuitively one would expect to obtain a higher uncertainty for the display than for the inscription on the case.However, the depth estimation is already filtering out uncertain estimates, which leads to a sparser depth map on the display.This sparsity must also be taken into account when rating the results.However, as can be seen from deflectometry, both the case and its display are not perfect Lambertian surfaces.Hence, the obtained accuracy conforms quite well to the simulations shown in Fig. 17.

Conclusions
In this paper we proposed two models to predict the measurement uncertainty of a deflectometric and a plenoptic sensor.Based on our introduced models, we have shown that, for a given measurement setup, there exists an optimum fringe pattern that results in the lowest measurement uncertainty.In case of the deflectometric sensor, the achieved height measurement uncertainty ranges between 1 and 100 nm if the surface is at least partially specular.In contrast the measurement uncertainty of the plenoptic sensor on a perfect diffusely reflecting and textured surface can be as low as several micrometers, depending on the measurement setup.
While the deflectometric sensor has much lower uncertainty for surface changes (3 orders of magnitude for partially specular surfaces), it measures surface normals instead of distances, which have to be integrated to obtain the surface height.The plenoptic measurement could help to regularize this integration by providing a relatively rough but robust distance measure.
While the simulation shows plausible results for the proposed models, we furthermore were able to validate the models exemplarily by real measurements.
Data availability.The raw image data can be made available upon request.
Author contributions.MZ and his supervisor MH developed the theory and evaluation for deflectometry, and NZ and his supervisor FQ did the same for the plenoptic sensor.
Competing interests.The authors declare that they have no conflict of interest.Special issue statement.This article is part of the special issue "Evaluating measurement data and uncertainty".It is not associated with a conference.

Figure 2 .
Figure 2. Deflectometric camera MTF for defocused image of screen points with 1.0 m distance and focal plane distances g.

Figure 3 .
Figure 3. Motion blur MTF caused by camera motion in image space.

Figure 4 .Figure 5 .
Figure 4. Image of a reflected pattern for five different surfaces ranging from high to low gloss for a pattern on the screen with spatial frequency k cam = 1 mm −1 .Note the modulation change from left (high gloss, c = 3.8) to right (low gloss, c = 2.5), where the pattern is almost invisible.

Figure 6 .
Figure 6.Lambertian surface MTF for different surface gloss parameters.

Figure 8 .
Figure 8. Image projection of a focused plenoptic camera in the Galilean mode.One object point is projected to multiple micro images on the sensor.

Figure 9 .Figure 10 .
Figure9.Raw image recorded by a focused plenoptic camera.Different to a regular camera, a plenoptic camera has a micro lens array (MLA) placed in front of the sensor (see Fig.8).Therefore, the raw image recorded by the camera is not one consistent central perspective image, but consists of thousands of circular micro images where each micro images shows only a small portion of the complete scene, as can be seen in the figure.From the magnified section, one can see that the same object point is projected to multiple neighboring micro images.

Figure 11 .
Figure11.Epipolar lines in a hexagonally arranged MLA.Due to the reason that the micro images are rectified to each other by nature, the epipolar line for a pair of micro images is defined by the vector between the respective principal points.The figure exemplarily shows one epipolar line (blue) for the five shortest stereo baseline distances (red).

Figure 12 .
Figure 12.Deflectometric measurement uncertainty of the phase on the screen, showing a pattern with fringe frequency translating to k cam when reflected onto the sensor.

Figure 13 .
Figure 13.Deflectometric measurement uncertainty of the surface normal for several focus distances and surface MTF parameters.

Figure 14 .
Figure 14.Measurement uncertainty of plenoptic and deflectometric sensor for several surface MTF parameters.The pattern is shown on the surface and also shown on the screen.Both pattern frequencies are given relative to the camera sensor image frequencies k cam .

Figure 15 .
Figure 15.Measurement uncertainty of the plenoptic sensor for different main lens focal lengths f and a perfect diffusely reflecting surface.

Figure 16 .Figure 17 .
Figure 16.Measurement uncertainty of plenoptic and deflectometric sensor with motion blur σ mot .The pattern is shown on the surface and also on the screen.Both pattern frequencies are given relative to the camera sensor k cam .

Figure 18 .
Figure 18.Case of a thermometer used to validate the proposed sensor models.The case consists of different surfaces with different reflectance properties.

Figure 19 .
Figure 19.Setup used to measure the measurement uncertainty of the deflectometric sensor for different frequencies of the fringe pattern on the surface.

Figure 20 .
Figure 20.Measurements of γ cam for different pattern frequencies under dark conditions.

Figure 21 .
Figure 21.Measurements of the surface MTF (points) and extrapolated data (lines) for two areas on the surface (display and case) and two illumination conditions (dark and bright surrounding).

Figure 22 .
Figure 22.Measurements of φ for different pattern frequencies under dark conditions.Measurement noise increases with the pattern frequency, depending on the surface reflectance.

Figure 23 .
Figure 23.Comparison of predicted measurement uncertainty (lines) and standard deviation (points) of σ α using the deflectometric measurement model.

Figure 24 .
Figure 24.Setup used to measure the measurement uncertainty of the plenoptic camera for different frequencies of the fringe pattern on the surface.

Figure 25 .
Figure 25.Measurement uncertainty of the plenoptic sensor for different main lens focal lengths f and a perfect diffusely reflecting surface.

Figure 26 .
Figure 26.Case recorded by the plenoptic camera.Intensity (a) image and depth map (b) calculated from the recordings of the plenoptic camera.