sicor.Tools package

Subpackages

Submodules

sicor.Tools.SolarIrradiance module

Derive Solar Irradiance from various data sources.

class sicor.Tools.SolarIrradiance.SolarIrradiance(dataset='Thuillier2002', used_date=datetime.date(2024, 4, 5))[source]

Bases: object

General object for solar constant computations dataset: Irradiance dataset to use, either: Thuillier2002 or Fontenla used_date: datetime object, for sun earth distance correction

sicor.Tools.Tools module

sicor.Tools.Tools._fmt_funktion_id(data, nodata_value=None)[source]

Format function for write_raster_gdal :param data: :param nodata_value: All non-finite in data will be replaced by nodada_value. :return: data like array, with non finite values replaced with nodata_value

sicor.Tools.Tools.cl(inp, dec)[source]

Ceil inp by number of decimals as given by dec

Parameters:

  • inp – input, float

  • dec – integer, number of decimals

Returns:

ceiled value

sicor.Tools.Tools.fl(inp, dec)[source]

Floor inp by number of decimals as given by dec

Parameters:

  • inp – input, float

  • dec – integer, number of decimals

Returns:

floored value

sicor.Tools.Tools.get_data_file(module_name, file_basename)[source]

Find path to data files either locally or within installed module. :param module_name: name od the module to look for data :param file_basename: basename of the file to locate :raises FileNotFoundError if file can not be found

sicor.Tools.Tools.read_image_gdal(filename)[source]

read image file using gdal, driver is determined by gdal

Parameters:

filename –

Returns:

numpy array with image data

sicor.Tools.Tools.write_raster_gdal(data, filename, gdal_type=2, driver=None, output_ss=None, order=1, metadata=None, projection=None, geotransform=None, logger=None, options=None, nodata_value=None, fmt_funktion=<function _fmt_funktion_id>)[source]

Write numpy array to an image file using gdal.

Parameters:

  • nodata_value –

  • data – numpy array

  • filename – destination filename

  • gdal_type – GDAL type, e.g. gdal.GDT_Int16

  • driver – either None or string with internal gdal name for file writing, if None, a default driver is selected

  • projection – e.g. ‘PROJCS[“WGS 84 / UTM zone 31N”,[…],UNIT[“metre”,1,AUTHORITY[“EPSG”,”9001”]], AUTHORITY[“EPSG”,”32631”]]’

  • geotransform – e.g. (600000.0, 60.0, 0.0, 4900020.0, 0.0, -60.0)

  • options – list or tuple with option stings for gdal.create, e.g. jp2,losless: [“QUALITY=100”, “REVERSIBLE=YES”] or jp2 lossy:[“QUALITY=10”,”REVERSIBLE=NO”]

  • logger – either None or logging instance, if None, a generic logger is created

  • output_ss – changed spatial sampling, if given gt is updated in resampling is performed (used order)

  • order – order of resampling is output_ss is different from geotransform

  • metadata – dictionary with metadata, writen for raster band

  • fmt_funktion – function to apply on data, e.g. lambda x: x as identity

Returns:

None

sicor.Tools.get_memory_use module

sicor.Tools.get_memory_use.get_memory_use()[source]

sicor.Tools.inpaint module

class sicor.Tools.inpaint.SL(mx, extend=1.1)[source]

Bases: object

returns slices for given center Y and radius D but makes sure borders are within [0,mx]

Parameters:

  • mx – maximum allowed index

  • extend – multiply D by extend to over- oder under-sample the slice

Returns:

function like object, to be called like f((Y,D))

sicor.Tools.inpaint.fill_nan(arr)[source]
Parameters:

arr – 2D numpy array, NaN’s are filled by brooming

sicor.Tools.inpaint.inpaint(array, sigma=None, logger=None, update_in_place=False, fill_corners=True, fill_remaining=True, extend=1.15, max_allowed_good_points=50000, mask=None, max_interpolation_points=1500000, interpolation_method='griddata', post_processing=None)[source]

Fill nan’s in array with meaningful data. :param array: array to be inpainted :param sigma: if not None, the final result will be smoothed by a gaussian kernel with [sigma] :param logger: logger for logging :param update_in_place: if True, perform inpainting directly on array, if False, return result :param fill_corners: True / False, assign values to corner blocks to increase number of pixels which can be interpolated wel :param fill_remaining: True / False, if True, all remaining nan’s after interpolation are replaced with nearest neighbour median value, if set, no nan’s remain in the image, is so, a ValueError is raised :param extend: oversampling parameter for block splitting of array, should be chosen between 0.8 and 1.2 :param max_allowed_good_points: max input points for griddata, determines performance and number of block splits :param max_interpolation_points: either None or integer, max number of points which are interpolated by griddata if too much, might eat up all memory :param mask, either None or boolean array of shape array, only those nan’s in array where mask is True will filled with meaningful data :param interpolation_method, None,”griddata”,”spline” :param post_processing: None,”spline”,”gaussian_filter” :return: array filled with meaningful data if update_in_place=False, otherwise None

sicor.Tools.inpaint.itt2d(shape, dd, extend=1.0)[source]

Separate 2d array into 2D blocks based in iterator returning slice tuple

returns an iterator with ((slice,slice),…,(slice,slice)) which separates an 2D array with [shape] into dd^2 smaller blocks. Blocks can over- and under-sampled using the extend factor

Parameters:

  • shape – shape of 2D array

  • dd – number of desired blocks in each axis

  • extend – over-sampling factor

Returns:

iterator with ((slice,slice),…,(slice,slice))

sicor.Tools.inpaint.nangaussian_filter(uu, sigma)[source]
sicor.Tools.inpaint.nanmean_filter(inp, dd)[source]

Filter input array with mean.

Parameters:

  • inp – 2D array

  • dd – radius of mean

Returns:

filtered array

sicor.Tools.linear_error_moddeling module

Replace any complex model by a linear one, and compute model uncertainties from them.

sicor.Tools.linear_error_moddeling.linear_error_modeling(data_dict, s1, s2, mdl_vars, logger=None, inp_model=None, n_max_test=500, n_max_fit=1000, min_samples=100)[source]
Parameters:

  • min_samples –

  • n_max_fit –

  • n_max_test –

  • inp_model –

  • data_dict – dict with keys: “tau”, “sp”, “wv”, “I”, “dtau”, “dsp”, “dwv”, “dI” , “r”, either ndarrays or scalar (only for differentials)

  • s1 – tuple e.g. ((1,1,1,1),)

  • s2 – tuple e.g. ((0,0,0,0),(0,0,0,0),(0,0,0,0),(0,0,0,0))

  • mdl_vars – list of variable names used for the modeling, e.g. [“tau”, “sp”,”wv”,”I”]

  • logger – either None or logging instance

Returns:

dictionary: {“error”:[ndarray],”coef”:[list],”f”:[str, function definition], “fit_quality”:[tuple, (intercept, offset, norm)]}

sicor.Tools.majority_mask_filter module

sicor.Tools.majority_mask_filter._tuplify_majority_mask_filter(in_function)[source]

Make sure that “block_replace_params” is tuple of tuple in order to keep jit alive, where python lists are not jey implemented.

sicor.Tools.majority_mask_filter.majority_mask_filter(mask, majority_width=2, block_replace=True, block_replace_params=((50, 5),), nodata_value=255)[source]

Post processing of noisy masks of integer values. The set of values in mask are the same as in the result. Two filters are performed:

1.) Majority filter: Take a window of size [majority_width] around each pixel (i,j) and replace the the local value

with the most occuring one within that window. The nodata value is not taken. Filterung takes place if [majority_width] is larger than zero.

2.) Block replace: If the pixel (i,j) is within [block_replace_params=((([id of classs],[area to be filled up]),)]

then replace all pixels in a window of range [area to be filled up] with [id of classs]. The order is taken from the provided tuple ot list.

Parameters:

  • nodata_value – for mask, should be integer

  • mask – 2D numpy array of integer type

  • majority_width – with of the 2D area for the majority search

  • block_replace – whether to do a block replace

  • block_replace_params – tuple of 2 parameters: (([id of class],[area to be filled up]),)

Returns:

filtered mask

sicor.Tools.ram module

class sicor.Tools.ram.RAM(unit='GB')[source]

Bases: object

helper util to manage RAM usage of this process

Parameters:

unit – string as coded in self.units

free()[source]

return free memory

get_limit()[source]

limit ram usage of process, either by rlimit or hardware memory

set_limit(limit)[source]

set soft limit for ram usage :param limit:

used()[source]

currently used ram by process

sicor.Tools.rsf_functions module

sicor.Tools.rsf_functions.box_rspf(wvl_center, wvl_sol_irr, width)[source]

Returns normalized box response function with respect to given wavelenght grid.

sicor.Tools.rsf_functions.gauss_rspf(wvl_center, wvl_sol_irr, sigma=None, fwhm=None)[source]

Returns normalized Gaussian response function with respect to given wavelenght grid. You should either give FWHM or sigma, an ValueError is raised if not

sicor.Tools.sharedndarray module

class sicor.Tools.sharedndarray.SharedNdarray(dims, typecode='f', default_value=None)[source]

Bases: object

wrapper class, which collect all nessesary instances to make a numpy ndarray accessible as shared memory when using multiprocessing, it exposed the numpy array via three different views which can be used to access it globally

_init provides the mechanism to make this array available in each worker, best used using the provided __initializer__

dims : tuple of dimensions which is used to instantiate a ndarray using np.zero

Type code, from here: https://docs.python.org/3/library/array.html#module-array C Type, Python Type, Minimum size in bytes ‘b’ signed char int 1 ‘B’ unsigned char int 1 ‘u’ Py_UNICODE Unicode character 2 (1) ‘h’ signed short int 2 ‘H’ unsigned short int 2 ‘i’ signed int int 2 ‘I’ unsigned int int 2 ‘l’ signed long int 4 ‘L’ unsigned long int 4 ‘q’ signed long long int 8 (2) ‘Q’ unsigned long long int 8 (2) ‘f’ float float 4 ‘d’ double float 8

_initializer(globals_dict, name)[source]

This adds to globals while using the ctypes library view of [shared_ndaray instance].sh to make the numpy view of [shared_ndaray instance] globally available

sicor.Tools.sharedndarray.initializer(globals_dict, add_to_globals)[source]

globs shall be dict with name:value pairs, when executed value will be added to globals under the name name, if value provides a _initializer attribute this one is called instead.

This makes most sense when called as initializer in a multiprocessing pool, e.g.: Pool(initializer=__initializer__,initargs=(globs,))

sicor.Tools.tqdm_joblist module

sicor.Tools.tqdm_joblist.tqdm_joblist(jobs, update_intervall=1.0, max_update_intervall=10, n_status_prints=10, modify_jobs=True, sleep_time=1.0, apply_on_done=None, status_prints=True, overprint=True, extra_args={})[source]

Monitor how a list of jobs submitted to a Pool becomes smaller as jobs become ready.

Parameters:

  • overprint – TODO

  • status_prints – True / False

  • jobs – list og multiprocessing jobs

  • apply_on_done – callable object, is called as apply_on_done(*job.get(),**extra_args)

  • extra_args – should be dictionary of extra args which is passed to apply_on_done

  • modify_jobs – if True, then jobs is changed in place and is empty on return if false, the list remains unchanged -> this increases memory consumption (!)

  • n_status_prints – number of wanted prints, wont be reached exactly if runtime ob jobs varies

  • update_intervall – initial update time

  • sleep_time – if fiven, additional sleep time in seconds

Returns:

if apply_on_done given, then a list of results from apply_on_done is returned, order is nor preserved

Raises:

Nothing19

Module contents

class sicor.Tools.RAM(unit='GB')[source]

Bases: object

helper util to manage RAM usage of this process

Parameters:

unit – string as coded in self.units

free()[source]

return free memory

get_limit()[source]

limit ram usage of process, either by rlimit or hardware memory

set_limit(limit)[source]

set soft limit for ram usage :param limit:

used()[source]

currently used ram by process

class sicor.Tools.SharedNdarray(dims, typecode='f', default_value=None)[source]

Bases: object

wrapper class, which collect all nessesary instances to make a numpy ndarray accessible as shared memory when using multiprocessing, it exposed the numpy array via three different views which can be used to access it globally

_init provides the mechanism to make this array available in each worker, best used using the provided __initializer__

dims : tuple of dimensions which is used to instantiate a ndarray using np.zero

Type code, from here: https://docs.python.org/3/library/array.html#module-array C Type, Python Type, Minimum size in bytes ‘b’ signed char int 1 ‘B’ unsigned char int 1 ‘u’ Py_UNICODE Unicode character 2 (1) ‘h’ signed short int 2 ‘H’ unsigned short int 2 ‘i’ signed int int 2 ‘I’ unsigned int int 2 ‘l’ signed long int 4 ‘L’ unsigned long int 4 ‘q’ signed long long int 8 (2) ‘Q’ unsigned long long int 8 (2) ‘f’ float float 4 ‘d’ double float 8

_initializer(globals_dict, name)[source]

This adds to globals while using the ctypes library view of [shared_ndaray instance].sh to make the numpy view of [shared_ndaray instance] globally available

class sicor.Tools.SolarIrradiance(dataset='Thuillier2002', used_date=datetime.date(2024, 4, 5))[source]

Bases: object

General object for solar constant computations dataset: Irradiance dataset to use, either: Thuillier2002 or Fontenla used_date: datetime object, for sun earth distance correction

sicor.Tools.box_rspf(wvl_center, wvl_sol_irr, width)[source]

Returns normalized box response function with respect to given wavelenght grid.

sicor.Tools.cl(inp, dec)[source]

Ceil inp by number of decimals as given by dec

Parameters:

  • inp – input, float

  • dec – integer, number of decimals

Returns:

ceiled value

sicor.Tools.fill_nan(arr)[source]
Parameters:

arr – 2D numpy array, NaN’s are filled by brooming

sicor.Tools.fl(inp, dec)[source]

Floor inp by number of decimals as given by dec

Parameters:

  • inp – input, float

  • dec – integer, number of decimals

Returns:

floored value

sicor.Tools.gauss_rspf(wvl_center, wvl_sol_irr, sigma=None, fwhm=None)[source]

Returns normalized Gaussian response function with respect to given wavelenght grid. You should either give FWHM or sigma, an ValueError is raised if not

sicor.Tools.get_data_file(module_name, file_basename)[source]

Find path to data files either locally or within installed module. :param module_name: name od the module to look for data :param file_basename: basename of the file to locate :raises FileNotFoundError if file can not be found

sicor.Tools.get_memory_use()[source]
sicor.Tools.initializer(globals_dict, add_to_globals)[source]

globs shall be dict with name:value pairs, when executed value will be added to globals under the name name, if value provides a _initializer attribute this one is called instead.

This makes most sense when called as initializer in a multiprocessing pool, e.g.: Pool(initializer=__initializer__,initargs=(globs,))

sicor.Tools.inpaint(array, sigma=None, logger=None, update_in_place=False, fill_corners=True, fill_remaining=True, extend=1.15, max_allowed_good_points=50000, mask=None, max_interpolation_points=1500000, interpolation_method='griddata', post_processing=None)[source]

Fill nan’s in array with meaningful data. :param array: array to be inpainted :param sigma: if not None, the final result will be smoothed by a gaussian kernel with [sigma] :param logger: logger for logging :param update_in_place: if True, perform inpainting directly on array, if False, return result :param fill_corners: True / False, assign values to corner blocks to increase number of pixels which can be interpolated wel :param fill_remaining: True / False, if True, all remaining nan’s after interpolation are replaced with nearest neighbour median value, if set, no nan’s remain in the image, is so, a ValueError is raised :param extend: oversampling parameter for block splitting of array, should be chosen between 0.8 and 1.2 :param max_allowed_good_points: max input points for griddata, determines performance and number of block splits :param max_interpolation_points: either None or integer, max number of points which are interpolated by griddata if too much, might eat up all memory :param mask, either None or boolean array of shape array, only those nan’s in array where mask is True will filled with meaningful data :param interpolation_method, None,”griddata”,”spline” :param post_processing: None,”spline”,”gaussian_filter” :return: array filled with meaningful data if update_in_place=False, otherwise None

sicor.Tools.itt2d(shape, dd, extend=1.0)[source]

Separate 2d array into 2D blocks based in iterator returning slice tuple

returns an iterator with ((slice,slice),…,(slice,slice)) which separates an 2D array with [shape] into dd^2 smaller blocks. Blocks can over- and under-sampled using the extend factor

Parameters:

  • shape – shape of 2D array

  • dd – number of desired blocks in each axis

  • extend – over-sampling factor

Returns:

iterator with ((slice,slice),…,(slice,slice))

sicor.Tools.linear_error_modeling(data_dict, s1, s2, mdl_vars, logger=None, inp_model=None, n_max_test=500, n_max_fit=1000, min_samples=100)[source]
Parameters:

  • min_samples –

  • n_max_fit –

  • n_max_test –

  • inp_model –

  • data_dict – dict with keys: “tau”, “sp”, “wv”, “I”, “dtau”, “dsp”, “dwv”, “dI” , “r”, either ndarrays or scalar (only for differentials)

  • s1 – tuple e.g. ((1,1,1,1),)

  • s2 – tuple e.g. ((0,0,0,0),(0,0,0,0),(0,0,0,0),(0,0,0,0))

  • mdl_vars – list of variable names used for the modeling, e.g. [“tau”, “sp”,”wv”,”I”]

  • logger – either None or logging instance

Returns:

dictionary: {“error”:[ndarray],”coef”:[list],”f”:[str, function definition], “fit_quality”:[tuple, (intercept, offset, norm)]}

sicor.Tools.majority_mask_filter(mask, majority_width=2, block_replace=True, block_replace_params=((50, 5),), nodata_value=255)[source]

Post processing of noisy masks of integer values. The set of values in mask are the same as in the result. Two filters are performed:

1.) Majority filter: Take a window of size [majority_width] around each pixel (i,j) and replace the the local value

with the most occuring one within that window. The nodata value is not taken. Filterung takes place if [majority_width] is larger than zero.

2.) Block replace: If the pixel (i,j) is within [block_replace_params=((([id of classs],[area to be filled up]),)]

then replace all pixels in a window of range [area to be filled up] with [id of classs]. The order is taken from the provided tuple ot list.

Parameters:

  • nodata_value – for mask, should be integer

  • mask – 2D numpy array of integer type

  • majority_width – with of the 2D area for the majority search

  • block_replace – whether to do a block replace

  • block_replace_params – tuple of 2 parameters: (([id of class],[area to be filled up]),)

Returns:

filtered mask

sicor.Tools.nanmean_filter(inp, dd)[source]

Filter input array with mean.

Parameters:

  • inp – 2D array

  • dd – radius of mean

Returns:

filtered array

sicor.Tools.read_image_gdal(filename)[source]

read image file using gdal, driver is determined by gdal

Parameters:

filename –

Returns:

numpy array with image data

sicor.Tools.tqdm_joblist(jobs, update_intervall=1.0, max_update_intervall=10, n_status_prints=10, modify_jobs=True, sleep_time=1.0, apply_on_done=None, status_prints=True, overprint=True, extra_args={})[source]

Monitor how a list of jobs submitted to a Pool becomes smaller as jobs become ready.

Parameters:

  • overprint – TODO

  • status_prints – True / False

  • jobs – list og multiprocessing jobs

  • apply_on_done – callable object, is called as apply_on_done(*job.get(),**extra_args)

  • extra_args – should be dictionary of extra args which is passed to apply_on_done

  • modify_jobs – if True, then jobs is changed in place and is empty on return if false, the list remains unchanged -> this increases memory consumption (!)

  • n_status_prints – number of wanted prints, wont be reached exactly if runtime ob jobs varies

  • update_intervall – initial update time

  • sleep_time – if fiven, additional sleep time in seconds

Returns:

if apply_on_done given, then a list of results from apply_on_done is returned, order is nor preserved

Raises:

Nothing19

sicor.Tools.write_raster_gdal(data, filename, gdal_type=2, driver=None, output_ss=None, order=1, metadata=None, projection=None, geotransform=None, logger=None, options=None, nodata_value=None, fmt_funktion=<function _fmt_funktion_id>)[source]

Write numpy array to an image file using gdal.

Parameters:

  • nodata_value –

  • data – numpy array

  • filename – destination filename

  • gdal_type – GDAL type, e.g. gdal.GDT_Int16

  • driver – either None or string with internal gdal name for file writing, if None, a default driver is selected

  • projection – e.g. ‘PROJCS[“WGS 84 / UTM zone 31N”,[…],UNIT[“metre”,1,AUTHORITY[“EPSG”,”9001”]], AUTHORITY[“EPSG”,”32631”]]’

  • geotransform – e.g. (600000.0, 60.0, 0.0, 4900020.0, 0.0, -60.0)

  • options – list or tuple with option stings for gdal.create, e.g. jp2,losless: [“QUALITY=100”, “REVERSIBLE=YES”] or jp2 lossy:[“QUALITY=10”,”REVERSIBLE=NO”]

  • logger – either None or logging instance, if None, a generic logger is created

  • output_ss – changed spatial sampling, if given gt is updated in resampling is performed (used order)

  • order – order of resampling is output_ss is different from geotransform

  • metadata – dictionary with metadata, writen for raster band

  • fmt_funktion – function to apply on data, e.g. lambda x: x as identity

Returns:

None