preprocessing

Pre-processing

Requirements

Pre-process data to be able to register them in EOxServer and/or optimize performance of WMS.

At the moment mainly driven by ngEO Browse Server.

Which data shall be able to be pre-processed:

• Rectified data
  • GeoTIFF
  • Jpeg, Jpeg2000, or PNG (BMP?) + corner coordinates in x/y or lat/lon (4 values!)
• Non-rectified data
  • Jpeg, Jpeg2000, or PNG (BMP?) + footprint coordinates in x/y or lat/lon
  • Jpeg, Jpeg2000, or PNG (BMP?) + regularGrid

ngEO defines geometric models but their usage is unclear

• Footprint
• Affine
• RPC
• Picewise linear grid

Metadata needed for registration in EOxServer shall be generated. This includes the calculation of footprints i.e. delineation of no-data from data.

Design

Python module (eoxserver.processing.preprocess) plus 2 invocation methods:

• eoxserver_preprocess.py (Script in /tools directory of EOxServer.)
• ngeo_preprocess.py (Outside of EOxServer but possibly extended to be installable as EOxServer/Django app and ingest command.)

Use argparse although CentOS 6 uses still Python 2.6. Install via pip install argparse.

Future

Once internal caching for performance optimization is implemented the eoxs_register command might be extended to support non-rectified data.

Possible configuration options in eoxserver.conf:

• optimization_strategy = [service,storage] -> ?
• default_crs -> [services.ows.wms]
• default_crs -> [services.ows.wcs]

Possible changes to database model:

• CRS per DatasetSeries to be used for WMS optimization of DatasetSeries layer

eoxserver_preprocess.py

Takes raster data and pre-processes it into <outfiles_basename>.tif, a GeoTIFF converted to RGB using default internal tiling, internal overviews, no compression, and 0 as no-data value, and <outfiles_basename>.xml, a EOxServer simple XML EO metadata file.

The outfiles are ready to be used with the eoxs_register command of an EOxServer instance.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!Caution, you will loose data quality!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The data quality of <outfiles_basename>.tif is most likely different from because of the pre-processing applied. Reasons include re-projection, compression, conversion to RGB, indexing, etc.

eoxserver_preprocess.py

    --begin-time time --end-time time --coverage-id id | --no-metadata

    
    [--extent minx,miny,maxx,maxy[,srid] | --footprint polygon_wkt | --gcp pixel,line,easting,northing[,elevation]* ]
    --georef-srid ...


    [--crs CRS] CRS used for outfile. Default same as used in geo-referencing information.
    [--no-tiling]
    [--no-overview]

    [--orig-bands | --rgba] Don't convert to RGB but leave bands as in infile.
    [--bands no[:low:high],no[:low:high],no[:low:high][,no[:low:high]]] Defaults to first 3 (or 4) bands with automatic computed scaling.

    [--compression {JPEG [--jpeg-quality quality]|LZW|PACKBITS|DEFLATE [--zlevel level]|CCITTRLE|CCITTFAX3|CCITTFAX4|NONE}]

    [--indexed [--pct palette_file]] Defaults to computing a color palette like rgb2pct.py or use the one supplied in palette_file e.g. as VRT.
    [--no-data value]

    [--co "NAME=VALUE"]* Provide GDAL GeoTIFF format options (see http://www.gdal.org/frmt_gtiff.html).

    [--config file] Read configuration from file. Options given on the command line override it. File shall be formated as config file (cf. eoxserver.conf).
    
    [-f --force] Overwrite existing outfiles.

    infile [outfiles_basename] Outfiles are named either "<infile-without-extension>_proc.tif" and "<infile-without-extension>_proc.xml" or "<outfiles_basename>.tif" and "<outfiles_basename>.xml"

ngeo_preprocess.py

Takes browse report given via --browse-report parameter and pre-processes all included browse files into GeoTIFF files converted to RGB using default internal tiling, internal overviews, no compression, and 0 as no-data value, as well as EOxServer simple XML EO metadata files.

The outfiles are ready to be used with the eoxs_register command of an EOxServer instance.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!Caution, you will loose data quality!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The data quality of <outfiles_basename>.tif is most likely different from because of the pre-processing applied. Reasons include re-projection, compression, conversion to RGB, indexing, etc.

ngeo_preprocess.py

    [--no-metadata]

    --browse-report file File holds a single browse XML element or a complete browseReport plus optional index as defined by ngEO.

    [--crs CRS] CRS used for outfile. Default same as used in geo-referencing information.
    [--no-tiling]
    [--no-overview]

    [--orig-bands | --rgba] Don't convert to RGB but leave bands as in infile.
    [--bands no[:low:high],no[:low:high],no[:low:high][,no[:low:high]]] Defaults to first 3 (or 4) bands with automatic computed scaling.

    [--compression {JPEG [--jpeg-quality quality]|LZW|PACKBITS|DEFLATE [--zlevel level]|CCITTRLE|CCITTFAX3|CCITTFAX4|NONE}]

    [--indexed [--pct palette_file]] Defaults to computing a color palette like rgb2pct.py or use the one supplied in palette_file e.g. as VRT.
    [--no-data value]

    [--co "NAME=VALUE"]* Provide GDAL GeoTIFF format options (see http://www.gdal.org/frmt_gtiff.html).

    [--config file] Read configuration from file. Options given on the command line override it. File shall be formated as config file (cf. eoxserver.conf).

    [-f --force] Overwrite existing outfiles.
    
    [--out-dir directory]

    [infile] [outfiles_basename] Infile read from browse report or overridden on command line. Outfiles are named either "<infile-without-extension>_proc.tif" and "<infile-without-extension>_proc.xml" or "<outfiles_basename>.tif" and "<outfiles_basename>.xml"

eoxserver.processing.preprocess

class PreProcessor():
    def __init__(self, config_params):
        config
        outputcoverage
    
    def _XXX():
        ...

    def preprocess(self, infile, outfiles):
        do