An extended database of annotated skylight polarization images covering a period of two months

Abstract

Objectives

Recent advances in bio-inspired navigation have sparked interest in the phenomenon of skylight polarization. This interest stems from the potential of skylight-based orientation sensors, which performance can be simulated using physical models. However, the effectiveness of machine learning algorithms in this domain relies heavily on access to large datasets for training. Although there are several databases of simulated images in literature, there remains a lack of publicly available annotated real-world color polarimetric images of the sky across various weather conditions.

Data description

We present here a dataset obtained from a long-term experimental setup designed to collect polarimetric images from a stand-alone camera. The setup utilizes a Division-of-Focal-Plane polarization camera equipped with a fisheye lens mounted on a rotative telescope mount. Furthermore, we obtained the sensor’s orientation within the East-North-Up (ENU) frame from a geometrical calibration and an algorithm provided with the database. To facilitate further research in this area, the present sample dataset spanning two months has been made available on a public archive with manual annotations as required by deep learning algorithms. The images were acquired at 10 min intervals and were taken with various exposure times ranging from 33µs to 300ms.

Objective

Skylight polarization has garnered increasing attention across various fields, including cloud detection, realistic outdoor scene rendering, and bio-inspired navigation. By measuring the polarization pattern of sunlight scattered by the atmosphere, researchers can estimate the sun’s position in the sky dome and derive directional cues, such as heading with respect to the true north [1]. In the natural world, many insects, like the desert ant Cataglyphis, navigate using skylight polarization [2]. These insects employ a navigation strategy called path integration (PI), relying on odometry and an optical sky compass. While Rayleigh’s single-scattering model describe the basic atmospheric scattering phenomena, more complex models like those considering “neutral points” in the sky such as the Berry model [3], have been proposed to simulate patterns with multiple scattering and under uniform sky conditions. However, real-world scenarios encompass varying weather conditions, such as clouds or fog, requiring more sophisticated modeling approaches. Enhancing the robustness of polarimetric image processing [1] has been the objective of recent studies based on neural networks (NN). However, NN-based methods applied to polarimetric images are not robust in practical applications when they rely on simulated images only for training. In addition, training deep learning algorithms requires a large quantity of data. While simulated datasets have been employed before, real-world data remains scarce. Compared to [4], we present here an improved and enriched dataset captured from the same experimental skylight polarization installation, on a laboratory rooftop in Marseille, France (43.286990365824785°N, 5.403361407820939°E). Data is manually annotated to help researchers to select the weather condition, and two months of data are provided instead of only one in [4]. Some of these data was used in [5] to compare real images with simulated skylight polarization images, and in [6] to check the accuracy of a method for finding true north without ephemerids.

Data description

Polarimetric images were acquired using the experimental setup described in [4], consisting of a PHX050S-QC camera with an IMX250MYR Division-of-Focal-Plane (DoFP) color-polarimetric sensor paired with a Fujinon FE185C057HA-1 fish-eye lens (185° field of view, selected f/4 aperture).

Between 2022-07-06 and 2022-08-29, ~ 40 000 images were captured at 10-minute intervals, with various fixed or auto-adaptative exposure times, assuming a constant camera orientation.

Clear sky images feature a typical skylight polarization pattern, with a greater or lesser cloud density depending on the weather conditions.

Orientation calibration involved aligning the camera to an approximate vertical towards the skydome and determining its orientation in the local East-North-Up (ENU) frame: Images with lowest exposure times revealed the sun’s projection through the fisheye lens as a spot, providing a set of sun orientations in the camera frame by using a lens distortion function. Concomitantly, we also had another set of sun ephemerids in the ENU frame at the same time provided by the Astropy community-developed Python package [7]. We therefore obtained a rotation matrix for each day between these two sets of sun coordinates by means of a least-squares numerical method [8]. As the calibration varied over time, due to a clock drift, we applied a linear compensation to the acquisition time in order to get a stable calibration. The averages of focal and Euler angle values over clear days was then saved.

Detailed methodology is included in the Python codes shared on the GitHub repository: https://github.com/mol-1/A-2-month-long-annotated-skylight-polarization-images-database---associated-code.

First, camera orientation was inferred, then images were cropped to reduce images to a region of interest (mostly the sky here). Then, each acquisition was annotated with a code letter depending on observed weather:

Dark Sky: n,

Clear sky: c,

Mist: m,

Few clouds (11–25% coverage): f,

Scattered clouds (25–50%): s,

Broken clouds (51–84%): b,

Overcast clouds (85–100%): o

Only raw data is provided to end users, who can process it on their own computers to reduce the amount of data to download.

Finally, a code is provided to extract useful polarimetric information [5, 9] from DoFP images and to show how to select specific weather conditions in the images. As camera distortion is also provided (using Scaramuzza’s Matlab toolbox [10]) with sun ephemerids, a simple Rayleigh model can also simulate equivalent clear-sky images.

Database is publicly available at https://doi.org/10.57745/9L2YUB [11] with license CC BY 4.0.

All provided files are Numpy data files (“.npy” extension).

Table 1 Overview of data files/data sets

Limitations

Polarimetric images have been acquired over only two months. However, even if a wide range of weather conditions were observed during the exploitation of this data collection setup, a longer dataset may add a greater variety of weather conditions (snow, rain, thunderstorm, thick fog …). As the camera has a fixed orientation, the sun is only followed on a small subset of paths (but images could be numerically rotated afterwards to increase the range of the sun’s positions). In addition, as images were acquired in France with ~ 43° latitude, the sun’s maximum elevation was no more than ~ 70°. A similar experiment might be run closer to equatorial line or nearer to the poles in order to get images with the sun rising higher in the sky or following a lower trajectory. We could also have chosen a smaller acquisition periodicity to get more images in the same time frame, or also to calibrate image sensor with a polarimetric reference. Finally, the clock drift of about 5 min over 2 months could be compensated for by means of an external time reference (e.g., GPS or universal time available on internet) to improve the temporal accuracy of the acquisition.