It is difficult to obtain the distance ground truth to determine our error because of our high point density and accuracy. Matching the range, resolution, and density of our point cloud would require multiple expensive survey-grade lidar scanners.
Example of a flat target board…
… but find out that foam board is not that flat.
Instead, we used a flat-plane target to measure our pixel error. Specifically, we applied RANSAC to fit a plane to points in the image that corresponded to a flat surface. These flat surfaces were manually labeled in the image, such as walls, signs, or car license plates. We extract two key values from the fit: the distance of the fitted plane (used as the x-values) and the median distance between the points and the flat-plane target (used as the y-values). For example:
Fitting plane: a x + b y + c z + d = 0
Plane equation: -0.051x + 0.047y + 0.998z + -56.429 = 0
relative error 0.86% mean distance 0.485 median_distance 0.182
For example, we obtained an x-value of -56.43 meters (plane distance) and a y-value of 0.182 meters (median distance). This pair represents one data point for a specific condition (x = -56.43, y = 0.182 [m]). We repeated this process for multiple parts within a single frame (e.g., wall, sign, etc.) and across different frames captured at varying distances to the targets. For each distance, we recorded up to 10 frames. The resulting x-y data pairs were then used to fit parabolas, allowing us to calculate the disparity error using the relationship:
$\Delta R = \Delta \text{Disp} \cdot R^2 + \text{offset}$
where the offset was negligible. Below are the results obtained using this methodology:
Relative range estimation error <~ 0.5 % even in adverse weather conditions
The disparity error is < 0.1 pixels for good weather conditions and < 0.3 pixels for bad weather conditions
In the datasheet, we report the disparity error as ±median. Therefore, a median error of 0.1 pixels is reported as ±0.1 pixels on the datasheet.
Internal Notes for NODAR: