sicor package

Subpackages

Submodules

sicor.sicor module

Base module of SICOR, contains most IO and data handling code.

class sicor.sicor.Figs(s2rgb=None, s2cwv=None, s2tau=None, s2msk_rgb=None, s2dem=None, s2spr=None, logger=None, fs=16, ss=500)[source]

Bases: object

General plotting of internal ac products.

_to_mpl_(inp)[source]

Tyop conversion to numpy types that matplotlib can use.

aot(ax)[source]

Plot AOT map.

cwv(ax)[source]

Plot water vapour map.

static dat_2show(data, nodata)[source]

Convert data to float and set nodata as nan -> useful when plotting with imshow :param data: :param nodata: value which will be replaced with NaN :return:

dem(ax)[source]

Plot elevation map.

mak_rgb(ax)[source]

Plot Mask.

plot(figs, export_html=None, export_jpg=None, return_figure=False, n_cols=1, dpi=150)[source]

Internal plotting on a larger canvas.

rgb(ax)[source]

Plot pseude rgb of scene.

spr(ax)[source]

Plot surface pressure map.

class sicor.sicor.IO[source]

Bases: object

SICOR Input / Output handler.

static and_and_check_output_fns(options, logger=None, create_output_dirs=True, run_suffix=True)[source]

Check output path in options for existence, create if needed.

static err_out_arr(data, v_max=254, data_dd=10, hist_th=0.94, nodata_value=255)[source]
static get_mx(data, nbins=100, hist_th=0.95, data_dd=100)[source]
Parameters

  • data – numpy array, 2D with error values

  • nbins – number of bins for cumulative histogram

  • hist_th – cumulative histogram threshold

  • data_dd – stride value for data to spped things up

Returns

static rename_tmp_product(outputs, run_suffix, logger=None)[source]
static wait_jobs_dict(jobs, logger=None, sleep_interval=1.0, max_time_minutes=20)[source]

Wait until all jobs un jobs are done. Print status if requested.

static write_l2a(granule_path, band_fns, projection, data=None, uncert=None, msk=None, driver='JP2OpenJPEG', logger=None, options_lossless={'JP2OpenJPEG': ['QUALITY=100', 'REVERSIBLE=YES']}, clear_areas=None, options_lossy={'JP2OpenJPEG': ['QUALITY=30', 'REVERSIBLE=NO']}, options_mask={'JP2OpenJPEG': ['QUALITY=100', 'REVERSIBLE=YES', 'WRITE_METADATA=YES']}, n_cores=3, max_time_minutes=20, mask_ss=[10.0, 20.0, 60.0], mask_geo_band='B11', level_1c_xml=None, nodata_value_data=65535, max_value_uncert=0.2, nodata_value_mask=255, tau=None, cwv=None, ss_other=(20.0,), output_bands={10.0: ['B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B11', 'B12', 'B8A'], 20.0: ['B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B11', 'B12', 'B8A'], 60.0: ['B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B11', 'B12', 'B8A']})[source]

Write Sentinel-2 Level-2A product to file system.

static write_metadata(opts, options, logger=None)[source]

Write metadata to logfile.

static write_results(s2img, options, logger=None)[source]

Write all in options requested output formats.

static write_rgb_jpeg(s2img, opts, logger=None)[source]

Write jpeg rgm image of image according to options in opts.

class sicor.sicor.StrippedInstance(inp, cls, attributes=[], metadata_keys=[], methods=[], logger=None, verbose=False)[source]

Bases: object

Generic stripped down version of any object - used to debug interfaces with GeoMultiSens.

static pp(inp)[source]

Create sane description strings including numpy arrays.

sicor.sicor.ac_interpolation(iband, band, s2img, s2spr, s2cwv, s2tau, clear_areas, points, rhos, rfls, max_pixel_processing, reduce_lut=True)[source]

Atmospheric correction based on interpolation in unstructured grid.

sicor.sicor.barometric_formula(h, p0=1013.0, t0=288.15)[source]

Surface pressure for standard conditions :param t0: Surface temperature in K :param h: height in m :param p0: surface pressure :return: surface pressure in units of p0

sicor.sicor.conv(scale, dtype)[source]

Function to create custom data conversion functions :param scale: Scaling factor, scalar value :param dtype: Numpy dtype of output :returns: Functions which applys this scaling to numpy arrays

sicor.sicor.get_ecmwf_data(variable, image, options, logger=None)[source]

Wrapper for sicor.ECMWF.ECMWF_variable. For a given observation date, go back in time day by day in ECMWF database (given in options[“ECMWF”][‘path_db’]) until a valid file is found. This time is set in options[“ECMWF”][“max_delta_day”]. Adjust ECMWF step according to the days needed to go back in time until data was available.

Parameters

  • variable – Name of ECMWF variable, needs to be present in database root directory

  • image – object derived from sicor.sensors.RSImage

  • options – dictionary with options. Minimum is: {“ECMWF”: {“path_db”: [path to ecmwf db], “max_delta_day”: [number of days], “target_resolution”: 20}}

  • logger – None or instance of logging

Returns

numpy array with interpolated variable at given [target_resolution] for [variable] and [image] geo position

sicor.sicor.get_egl_lut(fo, spr_lim, dim_spr, dim_wv)[source]

Interpolation of solar downward irradiance for a given atmospheric state to build a small cwv and spr LUT. :param fo: Forward operator to use :param spr_lim: Lower and upper limit of spr in given data. :param dim_spr: Number of spr values. :param dim_wv: Number of cwv values. :return: E_DN LUT for different spr and cwv.

sicor.sicor.get_stats(img, stat, dd)[source]

Collect some statistical metrics for data in img.data and img.data_ac.

sicor.sicor.include_missing_spatial_samplings(data, samplings, order=0, base_key=None)[source]

Add data of additional spatial samplings to data dict

Parameters

  • data – dict with spatial_sampling:data

  • samplings – additional spatial samplings to be included in data

  • order – interpolation order

  • base_key – if None, first key:value pairs in data is used, otherwise this one

Returns

sicor.sicor.nan_gaussian_filter(data, sigma)[source]

A gaussian filter that works on array with NaN values.

sicor.sicor.prepare_s2_ac(s2spr, s2cwv, s2tau, clear_areas, n_smpl, bands, fo, target_resolution, parameter_bounds, logger=None, **kwarg)[source]

Generate needed input for atmospheric correction.

sicor.sicor.prepare_s2_wv(n_smpl, bands, fo, rho_bounds)[source]

Prepare unstructured table for interpolation of water vapour. :param rho_bounds: :param bands: :param n_smpl: :param fo: instance of RtFo :returns: dictionary {“points”:points,”values”:values} to be directly used by griddata

sicor.sicor.pretty_print(obj)[source]

Pretty print object :param obj: Any python object.

sicor.sicor.quality_check_data_ac_vs_clear_areas(s2img, logger=None)[source]

Use s2img.data_ac and s2img.clear_areas for a quality check. Creates a copy of s2img.clear_areas and updates areas with False which are orignally True but data_ac is none. This means that these areas were not successfully processed with the ac algorithm

Parameters

  • s2img – Should be S2MSI.S2Image instance. At least s2img.data_ac and s2img.clear_areas are neeeed

  • logger – logger instance

Returns

clear_areas, dictionary with {[spatial sampling]:[boolean array]}

sicor.sicor.slp2spr(slp, t, h, dt_dh=-0.006, m=0.02896, r=8.314, g=9.807)[source]

Convert Sea level pressure to given height using barometric formula and linear temperature profile: T = T0 + dTdH * h, in general: dTdH is negative.

Parameters

  • r –

  • slp – seal level pressure

  • m – molar weight of air in kg / mol

  • g – gravitation in m/ s^2

  • t – temperature at height h

  • h – height in m

  • dt_dh – temperature height gradient

Returns

pressure in same units as slp

sicor.sicor.wv(fo, s2img, clear_areas, s2spr, s2tau, cloud_fraction, nanmean, sigma, rho_bounds, bands, target_resolution, int_order, n_smpl, n_std, logger=None, **kwargs)[source]

Generic two-band water vapor retrieval

Parameters

  • fo – forward operator to use

  • s2img – RSImage derived at-sensor reflectance

  • clear_areas – dict of {spatial_sampling:clear_area_map}

  • s2spr – dict {spatial_sampling:surface_pressure_map}

  • s2tau – dict {spatial_sampling:aerosol_optical_thickness_map}

  • cloud_fraction – fraction of cloud covered image pixels

  • nanmean – TODO

  • sigma – TODO

  • rho_bounds – lower und upper reflectance bounds

  • bands – TODO

  • target_resolution – value of spatial sampling in meter to retrieve the product

  • int_order – order of polynomial interpolation for output product

  • n_smpl – TODO

  • n_std – TODO

  • logger – None or logging instance

Returns

retrieved water vapor map.

sicor.sicor.wv_band_ratio(fo, s2img, clear_areas, s2spr, nanmean, sigma, bands, target_resolution, int_order, n_std, logger=None, **kwargs)[source]

Two-band water vapor retrieval based on the logarithm of the water vapor transmittance at 940 nm. :param fo: Forward operator to use :param s2img: RSImage derived at-sensor reflectance :param clear_areas: Dict of {spatial_sampling:clear_area_map} :param s2spr: Dict {spatial_sampling:surface_pressure_map} :param nanmean: :param sigma: :param bands: :param target_resolution: Value of spatial sampling in meter to retrieve the product :param int_order: Order of polynomial interpolation for output product :param n_std: :param logger: None or logging instance :return: Retrieved water vapor map.

sicor.sicor_ac module

Sensor Independent Atmospheric CORrection (SICOR) AC module.

The module exposes methods for atmospheric correction of multi and hyper spectral Earth observation data.

class sicor.sicor_ac.Status[source]

Bases: object

Collecting exceptions along a program, use .status so signal overall ok’ness self.status = 0 -> all is ok (like in Unix) self.status != 0 -> we have a problem Exceptions can be collected using the self.add_exception method and are stored in the list self.exceptions = [([class of error],[traceback]),]

add_exception(exception)[source]
sicor.sicor_ac.ac(granule_path, out_dir, settings, aerosol_type='aerosol_1', logger=None, raise_on_memory_exception=False, catch_all_exceptions=True, logdir=None, persist_result=False, coregistration=None, ignore_coreg_status=False)[source]

SICOR Atmospheric Correction on file system level Defines logger settings, gets sensor specific options, sets outpath and -filename, loads input image, performs AC using function ‘ac_gms’. :param granule_path: path to Sentinel-2 granule :param out_dir: path to output directory :param settings: path to settings file (json) :param aerosol_type: default aerosol type, fallback if ECMWF is not available :param logger: None or logging instance :param raise_on_memory_exception: True / False :param catch_all_exceptions: True / False :param logdir: path to logging directory :param persist_result: whether to write result to file system :param coregistration: dictionary of options for coreg_s2 (None for omitting coreg_s2) :param ignore_coreg_status: True / False :return: None

sicor.sicor_ac.ac_gms(s2img, options, logger=None, script=True)[source]

In memory interface to SICOR. Core function of AC. :param s2img: Object derived from RSImage -> should be at-sensor reflectance image :param options: Dictionary with needed options :param logger: None or logging instance :param script: True / False: if True, call sys.exit() if cloud coverage is above threshold :returns Surface reflectance image.

sicor.sicor_ac.ac_mp(iband, band)[source]

Bare ac in multiprocessing mode, all is global, only specify iband and band.

sicor.sicor_ac.arguments(ignore=('logger',))[source]

Return a tuple containing dictionary of calling function’s, named arguments and a list of calling function’s unnamed positional arguments.

sicor.sicor_ac.check_if_processed(orig_image_fn, cor_image_fn, threshold=0.2)[source]

checks output jp2 for size if in the range of +-10% of the original image, continue, otherwise delete corrected image

sicor.sicor_ac.cleanup_coreg(granule, logger=None)[source]

If an error occurs, delete all files containing “coreg” in the granule path.

Parameters

  • granule – str

  • logger – logger

sicor.sicor_ac.coreg_s2(granule='', ref_image_fn=None, ref_image_basepath=None, window_size=(256, 256), force=False, suffix='_coreg', n_local_points=30, settings=None, image_pattern='IMG_DATA/**/*B[0-9][0-9A]_[1,2,6]0m.jp2', logdir=None, logger=None, fmt_out='JP2KAK', msk_pattern='*MSK_*_[1,2,6]0m.jp2', msk_proxy={'10m': 'B02_10m', '20m': 'B05_20m', '60m': 'B01_60m'}, ref_raster_band_for_s2_band={'B01': 3, 'B02': 3, 'B03': 2, 'B04': 1, 'B05': 1, 'B06': 1, 'B07': 1, 'B08': 1, 'B11': 1, 'B12': 1, 'B8A': 1})[source]

Add coregistered images to a Sentinel product, naming is based on [suffix], images are placed in the same location

Parameters

  • settings –

  • ref_image_basepath –

:param fmt_out :param ref_raster_band_for_s2_band: :param suffix: :param logdir: :param logger: :param n_local_points: :param msk_proxy: :param msk_pattern: :param granule: path to granule :param ref_image_fn: base path for reference image, if None, ref_image is searched in ref_image_basepath :param window_size: coreg option :param force: True / False, if True, overwrite existing images :param image_pattern: tuple how to find images in products, glob pattern :return: processing status (0:successful, 1:errors)

sicor.sicor_ac.d2d(dic)[source]

Recurse to dict and replace None with “None” and slices with str(slice) - only a hack

sicor.sicor_ac.get_cloud_mask(image_fn)[source]

Gets the cloudmask from corresponding MSK*.jp2 file param: image_fn (str) : filename of the data file .jp2 return: GeoArray of cloudmask (boolean) baddata(cloud)=True, clear=False

sicor.sicor_ac.is_interactive()[source]

Check whether code is executed in an interactive interpreter.

sicor.sicor_ac.load_product(options, logger=None)[source]

Load Earth Observation product, currently supported is Sentinel-2. :param options: Options dictionary :param logger: None or logging instance :returns Object derived from RSImage

sicor.sicor_enmap module

sicor.sicor_enmap.make_ac_enmap(data, enmap_l1b, fo, cwv, pt, surf_res, logger=None)[source]

Perform atmospheric correction for enmap_l1b product, based on given and retrieved parameters. Instead of returning an object, the function adds a ‘data_l2a’ attribute to each detector. This numpy array holds the retrieved surface reflectance map.

Parameters

  • data – array containing measured TOA radiance of VNIR and SWIR

  • enmap_l1b – EnMAP Level-1B object

  • fo – forward operator object

  • cwv – CWV retrieval maps for VNIR and SWIR

  • pt – forward model parameter vector for VNIR and SWIR

  • surf_res – dictionary of retrieved surface parameters (‘intercept’, ‘slope’, ‘liquid water’, ‘ice’)

  • logger – None or logging instance

Returns

None

sicor.sicor_enmap.sicor_ac_enmap(enmap_l1b, options, unknowns=False, logger=None)[source]

Atmospheric correction for EnMAP Level-1B products, including a three phases of water retrieval.

Parameters

  • enmap_l1b – EnMAP Level-1B object

  • options – dictionary with EnMAP specific options

  • unknowns – if True, uncertainties due to unknown forward model parameters are added to S_epsilon; default: False

  • logger – None or logging instance

Returns

surface reflectance for EnMAP VNIR and SWIR detectors as well as dictionary containing estimated three phases of water maps and several retrieval uncertainty measures

sicor.sicor_generic module

sicor.sicor_generic.make_ac_generic(data_l1b, fo, xx, logger=None)[source]
Perform atmospheric correction for sensor-independent hyperspectral L1 products, based on given and retrieved

parameters.

Parameters

  • data_l1b – Level-1B data object

  • fo – Forward operator

  • xx – Solution state vector, must be in the order [cwv, cwc, ice, lai, dasf, p_max, k, b]

  • logger – None or logging instance

Returns

Level-2A data object

sicor.sicor_generic.sicor_ac_generic(data_l1b, options, dem=None, unknowns=False, logger=None)[source]
Atmospheric correction for sensor-independent imaging spectroscopy L1 products, including a three phases of water

retrieval.

Parameters

  • data_l1b – Level-1B data object in units of TOA radiance

  • options – Dictionary with pre-defined specific instrument options

  • dem – Digital elevation model to be provided in advance ; default: None

  • unknowns – If True, uncertainties due to unknown forward model parameters are added to S_epsilon; default: False

  • logger – None or logging instance

Returns

Surface reflectance, water vapor, liquid water and ice maps, as well as fitted TOA radiances and retrieval uncertainties

sicor.sicor_snow module

sicor.sicor_snow.sicor_ac_snow(data_l1b, options, dem=None, unknowns=False, logger=None)[source]

Wrapper function for a simultaneous optimal estimation of atmosphere and surface state focusing on snow and glacier ice properties.

Parameters

  • data_l1b – Level-1B data object in units of TOA radiance

  • options – dictionary with pre-defined specific instrument options

  • dem – digital elevation model to be optionally provided; default: None

  • unknowns – if True, uncertainties due to unknown forward model parameters are added to S_epsilon; default: False

  • logger – None or logging instance

Returns

atmosphere and surface solution state including water vapor, aot, surface reflectance, as well as estimated snow and glacier ice surface properties; optional output: several measures of retrieval uncertainties

sicor.version module

Module contents

Sensor Independent Atmospheric CORrection (SICOR)

sicor.ac(granule_path, out_dir, settings, aerosol_type='aerosol_1', logger=None, raise_on_memory_exception=False, catch_all_exceptions=True, logdir=None, persist_result=False, coregistration=None, ignore_coreg_status=False)[source]

SICOR Atmospheric Correction on file system level Defines logger settings, gets sensor specific options, sets outpath and -filename, loads input image, performs AC using function ‘ac_gms’. :param granule_path: path to Sentinel-2 granule :param out_dir: path to output directory :param settings: path to settings file (json) :param aerosol_type: default aerosol type, fallback if ECMWF is not available :param logger: None or logging instance :param raise_on_memory_exception: True / False :param catch_all_exceptions: True / False :param logdir: path to logging directory :param persist_result: whether to write result to file system :param coregistration: dictionary of options for coreg_s2 (None for omitting coreg_s2) :param ignore_coreg_status: True / False :return: None

sicor.ac_gms(s2img, options, logger=None, script=True)[source]

In memory interface to SICOR. Core function of AC. :param s2img: Object derived from RSImage -> should be at-sensor reflectance image :param options: Dictionary with needed options :param logger: None or logging instance :param script: True / False: if True, call sys.exit() if cloud coverage is above threshold :returns Surface reflectance image.

sicor.get_options(target, validation=True)[source]

return dictionary will all options :param validation: True / False, whether to validate options read from files ot not :param target: if path to file, then json is used to load, otherwise the default template is used :return: dictionary with options

sicor.sicor_ac_enmap(enmap_l1b, options, unknowns=False, logger=None)[source]

Atmospheric correction for EnMAP Level-1B products, including a three phases of water retrieval.

Parameters

  • enmap_l1b – EnMAP Level-1B object

  • options – dictionary with EnMAP specific options

  • unknowns – if True, uncertainties due to unknown forward model parameters are added to S_epsilon; default: False

  • logger – None or logging instance

Returns

surface reflectance for EnMAP VNIR and SWIR detectors as well as dictionary containing estimated three phases of water maps and several retrieval uncertainty measures