Algorithm descriptions

EnMAP Level 1B data reader

The EnPT reader module allows you to read in EnMAP Level-1B data in the official format as provided by the EnMAP Ground Segment. After data extraction EnPT expects a folder that includes the following files:

Filename

Description

ENMAP*L1B*-HISTORY.XML

EnMAP ground segment history file for L1B processing

ENMAP*L1B*-LOG.XML

EnMAP ground segment log file

ENMAP*L1B*-METADATA.XML

L1B metadata file

ENMAP*L1B*-QL_PIXELMASK_SWIR.TIF

Defective pixel mask SWIR

ENMAP*L1B*-QL_PIXELMASK_VNIR.TIF

Defective pixel mask VNIR

ENMAP*L1B*-QL_QUALITY_CIRRUS.TIF

Quality cirrus

ENMAP*L1B*-QL_QUALITY_CLASSES.TIF

Quality classes

ENMAP*L1B*-QL_QUALITY_CLOUD.TIF

Quality cloud

ENMAP*L1B*-QL_QUALITY_CLOUDSHADOW.TIF

Quality cloud shadow

ENMAP*L1B*-QL_QUALITY_HAZE.TIF

Quality haze

ENMAP*L1B*-QL_QUALITY_SNOW.TIF

Quality snow

ENMAP*L1B*-QL_QUALITY_TESTFLAGS_SWIR.TIF

Quality test flags SWIR

ENMAP*L1B*-QL_QUALITY_TESTFLAGS_VNIR.TIF

Quality test flags VNIR

ENMAP*L1B*-QL_SWIR.TIF

SWIR quicklook image

ENMAP*L1B*-QL_VNIR.TIF

VNIR quicklook image

ENMAP*L1B*-SPECTRAL_IMAGE_SWIR.TIF

EnMAP L1B SWIR spectral image

ENMAP*L1B*-SPECTRAL_IMAGE_VNIR.TIF

EnMAP L1B VNIR spectral image

EnPT reads the raster files lazily / on-demand to save memory during execution. Metadata read from the ENMAP*-METADATA.XML file are stored in memory and used at multiple stages of the pre-processing pipeline.

Conversion to top-of-atmosphere radiance

The image data is directly transformed from digital numbers (DNs, as provided by the DLR) to top-of-atmosphere radiance in mW/m2/sr/nm.

Filling the gap between adjacent EnMAP images

There is a gap of around 20 pixels in along-track-direction between the VNIR and SWIR image of the official Level-1B product and up to 1 pixel in across-track direction. Since the Level-2A processing in EnPT requires overlapping data, the data is cut to the overlap area, which in turn leads to blank spaces between two adjacent Level-2A tiles. To create overlapping Level-2A tiles in along-track direction for a further image mosaicking, EnPT offers the opportunity to consider information from neighboring Level-1B image tiles which is then appended to the actual EnMAP image tile to be processed.

Dead pixel correction

The EnPT dead pixel correction uses the pixel masks provided by the official Level-1B product and interpolates the EnMAP image data at the indicated dead pixel positions. It supports two interpolation algorithms:

  1. spectral interpolation (default)

    • Interpolates the data in the spectral domain.

    • Points outside the data range are extrapolated.

    • possible interpolation methods: linear, nearest, zero, slinear, quadratic, cubic, etc.

  2. spatial interpolation

    • Interpolates the data spatially.

    • Remaining missing data positions (e.g., outermost columns) are spectrally interpolated.

    • possible interpolation methods: linear, bilinear, cubic, spline

Along-track destriping

EnPT optionally applies an along-track destriping step to reduce horizontal striping artifacts in the EnMAP Level-1B VNIR and SWIR image data. Such stripes may be caused by small detector-related radiometric inconsistencies that are stable over the along-track direction and can become visible as band-wise horizontal patterns. In EnMAP SWIR data, along-track striping may occur in certain bands and scenes and is observed mainly at wavelengths close to atmospheric water absorption bands. A detailed description of this effect is provided in Chapter 5.15 of the EnMAP FAQ document.

The correction can be enabled or disabled separately for the VNIR and SWIR detectors, allowing it to be adapted to the different characteristics of the observed artefacts. In the SWIR, along-track striping is primarily associated with microvibrations affecting the sensor. The VNIR detector is not affected by the same microvibration effect, although applying the correction may still improve some scenes, for example over water, due to the noise-reduction effect of the algorithm.

The destriping is performed on top-of-atmosphere radiance data in sensor geometry. For each detector, EnPT estimates the striping pattern band by band in the spatial domain and subtracts the detected high-frequency component from the radiance data. The correction is applied in along-track direction and is designed to preserve the broad radiometric image content while reducing narrow stripe patterns.

The current implementation is based on an approach by Maximilian Brell and focuses on high-frequency stripe correction. Low-frequency destriping and spectral-domain destriping are not applied by default.

The correction should be applied to EnMAP Level-1B products at the user’s discretion and is not systematically recommended for all scenes. Although it can effectively reduce along-track striping, it may introduce other artefacts. Users should therefore evaluate its effect on a scene-by-scene basis. Particular care is advised for image rows containing both land and water surfaces, where correction artefacts may occur.

Note that across-track destriping (correcting for vertical image stripes) is already applied during the Level-1 processing pipeline at EnMAP ground segment, i.e., the input data to EnPT should already be destriped in this sense. Therefore, EnPT does not implement a destriping in across-track direction.

Atmospheric correction

EnPT supports atmospheric correction over land and water surfaces, optionally by applying separate atmospheric correction algorithms. There are different modes to determine which atmospheric correction is applied at which surfaces.

Mode

Description

land

The land atmospheric correction approach is applied to all surfaces

water

The water atmospheric correction approach is applied to water, land surfaces are cleared out

combined

The land atmospheric correction approach is applied to land and the water atmospheric correction approach is applied to water

over land surfaces

For atmospheric correction over land, i.e., for the conversion of TOA-(top-of-atmosphere) radiance to BOA- (bottom-of-atmosphere / surface) reflectance, users can choose between two different algorithms: SICOR and ISOFIT.

ISOFIT

ISOFIT (Imaging Spectrometer Optimal FITting) is a Python based open-source package developed at NASA’s Jet Propulsion Laboratory in Pasadena, CA, USA. ISOFIT contains a set of routines and utilities for fitting surface, atmosphere, and instrument models to imaging spectrometer data. It is written primarily in Python, with JSON format configuration files and some dependencies on widely available numerical and scientific libraries such as scipy, numpy, and scikit-learn. It is designed for maximum flexibility, so that users can swap in and evaluate model components based on different radiative transfer models (RTMs) and various statistical descriptions of surface, instrument, and atmosphere. It can run on individual radiance spectra in text format, or imaging spectrometer data cubes. For more details on the underlying algorithm, please refer to the documentation pages of ISOFIT.

advantages:

  • high BOA reflectance quality (well within the EnMAP mission requirements)

  • water vapor and aerosol optical depth retrieval (retrieval maps may be added to L2A output)

  • estimation of BOA reflectance uncertainty (optional output)

  • consideration of surface prior spectra which helps to reduce noise in the output spectra

drawbacks:

  • computationally expensive (optimized for server execution and workstations)

over water surfaces

ACwater/Polymer

For atmospheric correction over water, EnPT can optionally use the ACwater algorithm, which is a wrapper module around Polymer (Steinmetz et al. (2011)) to generate normalized water leaving reflectance from TOA radiance. ACwater was developed at AWI Bremerhaven in cooperation with GFZ and makes the Polymer algorithm applicable to EnMAP Level 1B data in EnPT.

Polymer is a spectral matching algorithm in which atmospheric and oceanic signals are obtained simultaneously using the fully available visible spectrum. The algorithm was developed by Hygeos (https://www.hygeos.com/); it is available as a Python package and it has been largely applied to ocean colour sensors. For details on the underlying algorithm, please refer to Steinmetz et al. (2011) and Soppa et al. (2021).

The following ACwater/Polymer outputs can be included into the EnMAP Level 2A product generated by EnPT (added in version 0.19.0):

  • normalized water leaving reflectance

  • chlorophyll-a concentration (logchl)

  • fb coefficient that scales the backscattering coefficient of particles (logfb)

  • reflectance of the sun glint (Rgli)

  • TOA reflectance at 865 nm corrected for Rayleigh scattering (Rnir)

  • quality flags (bitmask).

Which of these outputs are written is controlled by the polymer_output_level option:

  • 'basic': the normalized water leaving reflectance only

  • 'additional' (default): the above plus logchl, logfb, Rgli, Rnir and the bitmask

  • 'extended': the above plus the four products listed below

At level 'extended' (requires ACwater >= 0.5.0), the following additional outputs are included:

  • Rayleigh-corrected reflectance (Rprime), the input to the Polymer spectral fit (VNIR bands)

  • uncertainty of the normalized water leaving reflectance (rho_w_unc, a standard deviation; VNIR bands)

  • uncertainty of the chlorophyll-a concentration (logchl_unc, a variance of log10(chl))

  • uncertainty of the fb coefficient (logfb_unc, a variance of log10(fb)).

The per-band products only cover the VNIR, since Polymer computes them in the VNIR only. Rprime and rho_w_unc are ~90-band cubes and considerably increase the output size and the processing time, which is why this level is not the default. The uncertainties of logchl and logfb are derived from the parameter covariance of the Polymer fit and hence reflect the goodness of fit.

Spatial Co-Registration

AROSICS Logo

EnPT enables the geometric adaptation of EnMAP data to a user-provided image scene (e.g. Sentinel-2). Spatial misregistrations are detected using the open-source Python package AROSICS (An Automated and Robust Open-Source Image Co-Registration Software for Multi-Sensor Satellite Data). It has been developed at the GFZ Helmholtz Centre for Geosciences, Germany. For detailed algorithm description and use cases refer to the corresponding (open-access) paper that can be found here: Scheffler D, Hollstein A, Diedrich H, Segl K, Hostert P. AROSICS: An Automated and Robust Open-Source Image Co-Registration Software for Multi-Sensor Satellite Data. Remote Sensing. 2017; 9(7):676.

In EnPT, AROSICS is used to automatically compute thousands of tie points between a selected EnMAP band the user-provided reference image. The computed shifts are considered in the orthorectification step.

Orthorectification

EnMAP Level 1B data are provided in sensor geometry, i.e., the image data don’t have map coordinates but only image coordinates. For the ortho-rectification of the data EnPT uses a set of Rational Polynomial Coefficients (RPCs) provided for each band of the two EnMAP subsystems (VNIR and SWIR). Together with a user provided digital elevation model these RPC coefficients enable a highly accurate assignment of map coordinates to each pixel of the EnMAP Level-1B images. The RPC coefficients already include the official information about detector coregistration and keystone. This way image map coordinates are calculated internally for each pixel and band considering the spatial misregistrations estimated by AROSICS on demand. Resampling is done using a fast KDTree gaussian weighting neighbour approach implemented in the Python library pyresample (find the documentation here).

In this processing step, the EnMAP VNIR is merged with the SWIR subsystem and from now on stored in a single 3D array.

EnMAP Level 2A data writer

The EnPT writer module writes the computed EnMAP Level-2A data (orthorectified bottom-of-atmosphere reflectance for land surfaces or normalized water-leaving reflectance for water surfaces if the atmospheric correction runs in water or combined mode) to disk after finishing the processing pipeline. The data format produced by EnPT is based on the official Level-2A format of the ground segment. However, due to differences in the underlying algorithms, EnPT also produces a slightly different Level-2A data format. The current differences are summarized below:

Filename

official L2A format

EnPT

Description

ENMAP*L2A*-ENPT_LOG.log

no

yes

EnPT log file

ENMAP*L2A*-HISTORY.XML

yes

no

EnMAP ground segment history file for L2A processing

ENMAP*L2A*-LOG.XML

yes

no

EnMAP ground segment log file

ENMAP*L2A*-METADATA.XML

yes

yes

L2A metadata file

ENMAP*L2A*-QL_PIXELMASK.TIF

yes

planned

Defective pixel mask

ENMAP*L2A*-QL_QUALITY_CIRRUS.TIF

yes

yes

Quality cirrus

ENMAP*L2A*-QL_QUALITY_CLASSES.TIF

yes

yes

Quality classes

ENMAP*L2A*-QL_QUALITY_CLOUD.TIF

yes

yes

Quality cloud

ENMAP*L2A*-QL_QUALITY_CLOUDSHADOW.TIF

yes

yes

Quality cloud shadow

ENMAP*L2A*-QL_QUALITY_HAZE.TIF

yes

yes

Quality haze

ENMAP*L2A*-QL_QUALITY_SNOW.TIF

yes

yes

Quality snow

ENMAP*L2A*-QL_QUALITY_TESTFLAGS.TIF

yes

no

Quality test flags

ENMAP*L2A*-QL_SWIR.TIF

yes

yes

SWIR quicklook image

ENMAP*L2A*-QL_VNIR.TIF

yes

yes

VNIR quicklook image

ENMAP*L2A*-SPECTRAL_IMAGE.TIF

yes

yes

EnMAP L2A bottom-of-atmosphere reflectance (land) or normalized water leaving reflectance (water)

ENMAP*L2A*-ACOUT_SICOR_CWV.TIF

no

yes

SICOR’s column water vapor output (estimated through three-phases-of-water-retrieval)

ENMAP*L2A*-ACOUT_SICOR_LIQ.TIF

no

yes

SICOR’s liquid/canopy water content output (estimated through three-phases-of-water-retrieval)

ENMAP*L2A*-ACOUT_SICOR_ICE.TIF

no

yes

SICOR’s ice content output (estimated through three-phases-of-water-retrieval)

ENMAP*L2A*-ACOUT_ISOFIT_ATM_STATE.TIF

no

yes

ISOFIT’s atmospheric state output (estimated aerosol optical depth and water vapour)

ENMAP*L2A*-ACOUT_ISOFIT_UNCERTAINTY.TIF

no

yes

ISOFIT’s uncertainty layer providing an estimate of the BOA reflectance uncertainty

ENMAP*L2A*-ACOUT_POLYMER_*RNIR.TIF

no

optional

TOA reflectance at 863 nm corrected for Rayleigh scattering

ENMAP*L2A*-ACOUT_POLYMER_*RGLI.TIF

no

optional

Reflectance of the sun glint predicted from ECMWF wind speed

ENMAP*L2A*-ACOUT_POLYMER_*LOGCHL.TIF

no

optional

Chlorophyll-a concentration (mg/m3, in 10-based logarithm)

ENMAP*L2A*-ACOUT_POLYMER_*LOGFB.TIF

no

optional

Particle scattering factor fb in Park & Ruddick (2005) (in 10-based logarithm)

ENMAP*L2A*-ACOUT_POLYMER_*BITMASK.TIF

no

optional

Polymer quality flags (more information below)

ENMAP*L2A*-ACOUT_POLYMER_*RPRIME.TIF

no

optional

Rayleigh-corrected reflectance, input to the Polymer fit (VNIR bands only)

ENMAP*L2A*-ACOUT_POLYMER_*RHO_W_UNC.TIF

no

optional

Uncertainty of the normalized water leaving reflectance (standard deviation, VNIR bands only)

ENMAP*L2A*-ACOUT_POLYMER_*LOGCHL_UNC.TIF

no

optional

Uncertainty of the chlorophyll-a concentration (variance of log10(chl))

ENMAP*L2A*-ACOUT_POLYMER_*LOGFB_UNC.TIF

no

optional

Uncertainty of the particle scattering factor fb (variance of log10(fb))

The Polymer quality flags bitmask represents a bit-encoded product with the following flag values:

Flag name

Flag value

Description

LAND

1

Land mask

CLOUD_BASE

2

Polymer’s basic cloud mask

L1_INVALID

4

Invalid level1 pixel

NEGATIVE_BB

8

(deprecated flag)

OUT_OF_BOUNDS

16

Retrieved marine parameters are outside valid bounds

EXCEPTION

32

A processing error was encountered

THICK_AEROSOL

64

Thick aerosol flag

HIGH_AIR_MASS

128

Air mass exceeds 5

EXTERNAL_MASK

512

Pixel was masked using external mask

CASE2

1024

Pixel was processed in “case2” mode

INCONSISTENCY

2048

Inconsistent result was detected (atmospheric reflectance out of bounds)

ANOMALY_RWMOD_BLUE

4096

Excessive difference was found at 412nm between Rw and Rwmod

Value 0 represents water (all fine, no flags), value -9999 represents no-data.